Community on Apache Airflow

Agentic Workloads on Airflow: Observable, Retryable, and Auditable by Design

Wed, 15 Apr 2026 00:00:00 +0000

A question like “How does AI tool usage vary across Airflow versions?” has a natural SQL shape: one cross-tabulation, one result. A question like “What does a typical Airflow deployment look like for practitioners who are actively using AI in their workflow?” does not. It requires querying executor type, deployment method, cloud provider, and Airflow version independently, each filtered to the same respondent group, then synthesizing the results into a coherent picture. No single query returns the answer. The answer emerges from the relationship between all of them.

Ask Your Survey Anything: Building AI Analysis Pipelines with Airflow 3

Wed, 15 Apr 2026 00:00:00 +0000

The 2025 Airflow Community Survey collected responses from nearly 6,000 practitioners across 168 questions. You can open a spreadsheet and filter, or write SQL by hand. But what if you could just ask a question and have Airflow figure out the query, run it, and bring the result back for your approval?

This post builds two pipelines that do exactly that, using the apache-airflow-providers-common-ai provider for Airflow 3.

The first pipeline is interactive: a human reviews the question before it reaches the LLM and approves the result before the DAG finishes. The second is scheduled: it downloads fresh survey data, validates the schema, runs the query unattended, and emails the result.

Introducing the Common AI Provider: LLM and AI Agent Support for Apache Airflow

Tue, 14 Apr 2026 00:00:00 +0000

At Airflow Summit 2025, we previewed what native AI integration in Apache Airflow could look like. Today we’re shipping it.

apache-airflow-providers-common-ai 0.1.0 adds LLM and agent capabilities directly to Airflow. Not a wrapper around another framework, but a provider package that plugs into the orchestrator you already run. It’s built on Pydantic AI and supports 20+ model providers (OpenAI, Anthropic, Google, Azure, Bedrock, Ollama, and more) through a single install.

pip install 'apache-airflow-providers-common-ai'

Requires Apache Airflow 3.0+.

Introducing the Apache Airflow Registry

Thu, 19 Mar 2026 00:00:00 +0000

Today we’re launching the Apache Airflow Registry — a searchable catalog of every official Airflow provider and its modules, live at airflow.apache.org/registry/.

Need an S3 operator? A Snowflake hook? An OpenAI sensor? The Registry helps you find, compare, and configure the right components for your data pipelines — without digging through docs or PyPI pages.

By the Numbers


98	Official providers
1,602	Modules (operators, hooks, sensors, triggers, transfers, and more)
329M+	Monthly PyPI downloads across all providers
125+	Integrations with cloud platforms, databases, ML tools, and messaging services

Search Everything

Hit Cmd+K from any page and start typing. Results show up instantly, grouped by Providers and Modules, with type badges so you can tell a hook from an operator at a glance.

Airflow Survey 2025

Thu, 22 Jan 2026 00:00:00 +0000

The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics.

View raw data

Download survey responses (CSV)

Airflow Survey 2024

Thu, 27 Feb 2025 00:00:00 +0000

View raw data

Download survey responses (CSV)

Airflow Survey 2023

Thu, 21 Sep 2023 00:00:00 +0000

View Raw Data

Airflow Survey 2022

Fri, 17 Jun 2022 00:00:00 +0000

Airflow User Survey 2022

This year’s survey has come and gone, and with it we’ve got a new batch of data for everyone! We collected 210 responses over two weeks. We continue to see growth in both contributions and downloads over the last two years, and expect that trend will continue through 2022.

The raw response data will be made available here soon, in the meantime, feel free to email john.thomas@astronomer.io for a copy.

Airflow Summit 2022

Mon, 16 May 2022 00:00:00 +0000

The biggest Airflow Event of the Year returns May 23–27! Airflow Summit 2022 will bring together the global community of Apache Airflow practitioners and data leaders.

What’s on the Agenda

During the free conference, you will hear about Apache Airflow best practices, trends in building data pipelines, data governance, Airflow and machine learning, and the future of Airflow. There will also be a series of presentations on non-code contributions driving the open-source project.

Airflow Summit 2021

Sun, 21 Mar 2021 00:00:00 +0000

Airflow Summit 2021 is here!

The summit will be held online, July 8-16, 2021. Join us from all over the world to find out how Airflow is being used by leading companies, what is its roadmap and how you can participate in its development.

Useful information:

The official website: https://airflowsummit.org
Call for proposals is open until 12 April 2021. To submit your talk go to https://sessionize.com/airflow-summit-2021/
In case of any questions reach out to us via info@airflowsummit.org

Airflow Survey 2020

Tue, 09 Mar 2021 00:00:00 +0000

Apache Airflow Survey 2020

World of data processing tools is growing steadily. Apache Airflow seems to be already considered as crucial component of this complex ecosystem. We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.

It’s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this is true, we will learn next year!

Journey with Airflow as an Outreachy Intern

Sun, 30 Aug 2020 00:00:00 +0000

Outreachy is a program which organises three months paid internships with FOSS projects for people who are typically underrepresented in those projects.

Contribution Period

The first thing I had to do was choose a project under an organisation. After going through all the projects I chose “Extending the REST API of Apache Airflow”, because I had a good idea of what REST API(s) are, so I thought it would be easier to get started with the contributions. The next step was to set up Airflow’s dev environment which thanks to Breeze, was a breeze. Since I had never contributed to FOSS before so this part was overwhelming but there were plenty of issues labelled “good first issues” with detailed descriptions and some even had code snippets so luckily that nudged me in the right direction. These things about Airflow and the positive vibes from the community were the reasons why I chose to stick with Airflow as my Outreachy project.

Apache Airflow For Newcomers

Mon, 17 Aug 2020 00:00:00 +0000

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. A workflow is a sequence of tasks that processes a set of data. You can think of workflow as the path that describes how tasks go from being undone to done. Scheduling, on the other hand, is the process of planning, controlling, and optimizing when a particular task should be done.

Authoring Workflow in Apache Airflow.

Airflow makes it easy to author workflows using python scripts. A Directed Acyclic Graph (DAG) represents a workflow in Airflow. It is a collection of tasks in a way that shows each task’s relationships and dependencies. You can have as many DAGs as you want, and Airflow will execute them according to the task’s relationships and dependencies. If task B depends on the successful execution of another task A, it means Airflow will run task A and only run task B after task A. This dependency is very easy to express in Airflow. For example, the above scenario is expressed as

Airflow Survey 2019

Wed, 11 Dec 2019 00:00:00 +0000

Apache Airflow Survey 2019

Apache Airflow is growing faster than ever. Thus, receiving and adjusting to our users’ feedback is a must. We created survey and we got 308 responses. Let’s see who Airflow users are, how they play with it, and what they miss.

Overview of the user

What best describes your current occupation?

	No.	%
Data Engineer	194	62.99%
Developer	34	11.04%
Architect	23	7.47%
Data Scientist	19	6.17%
Data Analyst	13	4.22%
DevOps	13	4.22%
IT Administrator	2	0.65%
Machine Learning Engineer	2	0.65%
Manager	2	0.65%
Operations	2	0.65%
Chief Data Officer	1	0.32%
Engineering Manager	1	0.32%
Intern	1	0.32%
Product owner	1	0.32%
Quant	1	0.32%

In your day to day job, what do you use Airflow for?

New Airflow website

Wed, 11 Dec 2019 00:00:00 +0000

The brand new Airflow website has arrived! Those who have been following the process know that the journey to update the old Airflow website started at the beginning of the year. Thanks to sponsorship from the Cloud Composer team at Google that allowed us to collaborate with Polidea and with their design studio Utilo, and deliver an awesome website.

Documentation of open source projects is key to engaging new contributors in the maintenance, development, and adoption of software. We want the Apache Airflow community to have the best possible experience to contribute and use the project. We also took this opportunity to make the project more accessible, and in doing so, increase its reach.

ApacheCon Europe 2019 — Thoughts and Insights by Airflow Committers

Fri, 22 Nov 2019 00:00:00 +0000

Is it possible to create an organization that delivers tens of projects used by millions, nearly no one is paid for doing their job, and still, it has been fruitfully carrying on for more than 20 years? Apache Software Foundation proves it is possible. For the last two decades, ASF has been crafting a model called the Apache Way—a way of organizing and leading tech open source projects. Due to this approach, which is strongly based on the “community over code” motto, we can enjoy such awesome projects like Apache Spark, Flink, Beam, or Airflow (and many more).