Data Ingestion with Python Cookbook offers a practical approach to designing and implementing data ingestion pipelines. It presents real-world examples with the most widely recognized open source tools on the market to answer commonly asked questions and overcome challenges.
You’ll be introduced to designing and working with or without data schemas, as well as creating monitored pipelines with Airflow and data observability principles, all while following industry best practices. The book also addresses challenges associated with reading different data sources and data formats. As you progress through the book, you’ll gain a broader understanding of error logging best practices, troubleshooting techniques, data orchestration, monitoring, and storing logs for further consultation.
By the end of the book, you’ll have a fully automated set that enables you to start ingesting and monitoring your data pipeline effortlessly, facilitating seamless integration with subsequent stages of the ETL process.
publication date
2023-05-31
what you will learn
Implement data observability using monitoring tools
Automate your data ingestion pipeline
Read analytical and partitioned data, whether schema or non-schema based
Debug and prevent data loss through efficient data monitoring and logging
Establish data access policies using a data governance framework
Construct a data orchestration framework to improve data quality
key features
Harness best practices to create a Python and PySpark data ingestion pipeline * Seamlessly automate and orchestrate your data pipelines using Apache Airflow * Build a monitoring framework by integrating the concept of data observability into your pipelines
approach
A step-by-step recipe-based approach helps you avoid common pitfalls while building a robust and resilient ingestion pipeline. This book will also address important challenges that readers face around data observability, data accessibility, and data discovery using popular tools like pyspark and airflow
audience
This book is for data engineers and data enthusiasts seeking a comprehensive understanding of the data ingestion process using popular tools in the open source community. For more advanced learners, this book takes on the theoretical pillars of data governance while providing practical examples of real-world scenarios commonly encountered by data engineers.
short description
The Data Ingestion with Python Cookbook presents a collection of practical recipes to help you get started with the process of data ingestion from various sources or data files. This book offers a range of code recipes and solutions for creating efficient data ingestion pipelines while addressing common issues related to it.
subtitle
A practical guide to ingesting, monitoring, and identifying errors in the data ingestion process
keywords
Data Integration; Data analytics; data processing; data types; ETL solutions; Parquet; Schemas; MongoDB; SQL; Data Discovery; AWS; data analytics books; Data Replication
Product ISBN
9781837632602
