Data Pipelines with Apache Airflow, Second Edition: Orchestration for data and AI

by Julian de Ruiter, Ismael Cabral, Kris Geusebroek

Simplify, streamline, and scale your data operations with data pipelines built on Apache Airflow Data Pipelines with Apache Airflow has empowered thousands of data engineers to build more successful data platforms. This new second edition has been fully revised for Airflow 3 with coverage of all the latest features of Apache Airflow, including the Taskflow API, deferrable operators, and Large Language Model integration. Filled with real-world scenarios and examples, you'll be carefully guided from Airflow novice to expert. In Data Pipelines with Apache Airflow, Second Edition you'll learn how • Master the core concepts of Airflow architecture and workflow design • Schedule data pipelines using the Dataset API and time tables, including complex irregular schedules • Develop custom Airflow components for your specific needs • Implement comprehensive testing strategies for your pipelines • Apply industry best practices for building and maintaining Airflow workflows • Deploy and operate Airflow in production environments • Orchestrate workflows in container-native environments • Build and deploy Machine Learning and Generative AI models using Airflow Using real-world scenarios and examples, Data Pipelines with Apache Airflow, Second Edition teaches you how to simplify and automate data pipelines, reduce operational overhead, and smoothly integrate all the technologies in your stack. Part reference and part tutorial, each technique is illustrated with engaging hands-on examples, from training machine learning models for generative AI to optimizing delivery routes. About the Technology Apache Airflow provides a unified platform for collecting, consolidating, cleaning, and analyzing data. With its easy-to-use UI, powerful scheduling and monitoring features, plug-and-play options, and flexible Python scripting, Airflow makes it easy to implement secure, consistent pipelines for any data or AI task. About the book Data Pipelines with Apache Airflow, Second Edition teaches you how to build, monitor, and maintain effective data workflows. This new edition adds comprehensive coverage of Airflow 3 features, such as event-driven scheduling, dynamic task mapping, DAG versioning, and Airflow’s entirely new UI. The numerous examples address common use cases like data ingestion and transformation and connecting to multiple data sources, along with AI-aware techniques such as building RAG systems. What's inside • Deploying data pipelines as Airflow DAGs • Time and event-based scheduling strategies • Integrating with databases, LLMs, and AI models • Deploying Airflow using Kubernetes About the reader For data engineers, machine learning engineers, DevOps, and sysadmins with intermediate Python skills. About the author Julian de Ruiter, Ismael Cabral, Kris Geusebroek, Daniel van der Ende, and Bas Harenslak are seasoned data engineers and Airflow experts.

512 pages, Paperback

Published January 27, 2026

Reviews

[Alireza Aghamohammadi · Rating 5 out of 5]

This book is most useful for data engineers. First, you learn how to write a workflow in the form of a DAG. A DAG is a graph without cycles that defines the tasks along with their dependencies. Airflow has two very important components: 1. Airflow DAG Processor and 2. Airflow Scheduler. The first one parses the DAG and creates a set of tasks, while the second ensures that these tasks are executed at the times you have specified. Airflow also has a number of workers responsible for running these tasks. The book goes into a lot of detail. In roughly 500 pages, it uses lots of examples to help you understand the entire process I just described. I believe this is the best book you can find to learn Apache Airflow.

[Raja · Rating 5 out of 5]

I started reading this book, here is my progress review

Ch-1

- Explained the advantages of following acyclic graphs-based data processing instead of the cyclic model with clear examples.
- Provided a list of data pipeline and processing tools, this is a good list, which helps in picking the right tool.
- explained how Airflow stands out while building the pipeline using a programmatic approach.
- Exaplined Airflow architecture and the core of Airflow, the DAG engine explained very clearly.
- The way how scheduler works.
- I am actually fascinated by the point that processing the data in chunks but only upto allowed timewindow, so you only process what you can in the given timewindow .

CH-2
- After a good theory chapter, we got into practicals quickly in this chapter.
- Explained how to write a pipeline with simple examples.
- The role of task and operator and how operators are useful for executing other domain code, such bash, python code.
- This is good and allows us to add lot of customisation using Python if functionality is missing or needed.
- I did a small practical case with airflow web interface.

[Juan Venegas · Rating 5 out of 5]

If you work daily with Airflow, this is an amazing book. I used to look online for little tutorials, especially when it comes to operating Airflow in production environments, and with this book I finally got the structured learning I needed.

I have it on my desk all the time to go back and forth through some concepts, and it's so satisfying to have it there as a guide. It even comes with a free ePub copy, which I honestly don't use because I just carry the book with me everywhere 😅

[Giorgia Corrado · Rating 5 out of 5]

If you're looking to understand Apache Airflow rather than just follow tutorials, this book is a very good choice. The explanations are clear, structured, and technically precise without being overwhelming.

The visualizations are particularly helpful, they clarify architecture and workflow concepts that can otherwise feel abstract. The numerous code examples are practical and well-paced, guiding you step-by-step through real-world use cases.

[Alexander · Rating 5 out of 5]

After about eight years without touching Airflow, Data Pipelines with Apache Airflow, Second Edition was the ideal way for me to get back on board. The clear explanations and up‑to‑date examples around Airflow 3, the Taskflow API, and modern data and AI pipelines helped me get back up to speed very quickly.

[Piotr J · Rating 5 out of 5]

This book is the best resource I found on Airflow. I don't know any better way to solidify your knowledge about Data Pipelines. I highly recommend it to any data engineer or any engineer who's working with pipelines.

[Vinicios Wentz · Rating 5 out of 5]

If you want to learn all the core feature of airflow, this book is worth having

[Ana Claudia · Rating 5 out of 5]

A well-structured, up-to-date guide that balances theory and hands-on practice ideal for engineers looking to deepen their understanding of Airflow and apply it confidently in real-world projects.