Apache Airflow Blog

Apache Airflow 3.2.0: Data-Aware Workflows at Scale

2026-04-07T00:00:00Z

We’re proud to announce the release of Apache Airflow 3.2.0! Airflow 3.1 puts humans at the center of automated workflows. 3.2 brings that same precision to data: Asset partitioning for granular pipeline orchestration, multi-team deployments for enterprise scale, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/3.2.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.2.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html
🐳 Docker Image: docker pull apache/airflow:3.2.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.2.0

🗂️ Asset Partitioning (AIP-76): Only the Right Work Gets Triggered

Asset partitioning has been one of the most requested additions to data-aware scheduling. If you work with date-partitioned S3 paths, Hive table partitions, BigQuery partitions, or really any partitioned data store, you’ve dealt with this: An upstream task updates one partition, and every downstream Dag fires regardless of which slice actually changed. It’s wasteful, and for large deployments it creates real operational noise.

Asset partitioning in 3.2 makes this granular. Downstream Dags trigger only when the specific partition they care about gets updated. It’s the biggest change to data-aware scheduling since Assets were introduced, and it turns partition-driven orchestration into something Airflow handles natively rather than something you work around.

Key Capabilities

Partition-driven scheduling: Dags trigger on specific partition updates, not every asset change
CronPartitionTimetable: Schedule Dags against partitions using cron expressions. Also available in the Task SDK
Backfill for partitioned Dags: Backfill historical partitions without re-triggering everything downstream (#61464)
Multi-asset partitions: A single Dag can listen for partitions across multiple assets, which matters when your downstream work depends on several sources aligning (#60577)

For more advanced use cases, there are temporal and range partition mappers (#61522, #55247) for mapping time ranges and value ranges to partition keys, a partition key field on Dag run references (#61725) so you can inspect exactly which partition triggered a run, and PartitionedAssetTimetable for full control over how partition events from multiple assets get resolved into a unified trigger.

Example: Three upstream ingestion Dags each write to a separate asset on an hourly cadence. The downstream Dag only triggers when all three have updated the same hourly partition. Since the three assets don’t share a partition key natively, a mapper resolves them into a common key.

from __future__ import annotations

from airflow.sdk import (
    DAG,
    Asset,
    CronPartitionTimetable,
    PartitionedAssetTimetable,
    StartOfHourMapper,
    asset,
    task,
)

team_a_player_stats = Asset(uri="file://incoming/player-stats/team_a.csv", name="team_a_player_stats")
combined_player_stats = Asset(uri="file://curated/player-stats/combined.csv", name="combined_player_stats")


with DAG(
    dag_id="ingest_team_a_player_stats",
    schedule=CronPartitionTimetable("0 * * * *", timezone="UTC"),
    tags=["player-stats", "ingestion"],
):

    @task(outlets=[team_a_player_stats])
    def ingest_team_a_stats():
        """Materialize Team A player statistics for the current hourly partition."""
        pass

    ingest_team_a_stats()


@asset(schedule=CronPartitionTimetable("15 * * * *", timezone="UTC"))
def team_b_player_stats():
    pass


with DAG(
    dag_id="clean_and_combine_player_stats",
    schedule=PartitionedAssetTimetable(
        assets=team_a_player_stats & team_b_player_stats,
        default_partition_mapper=StartOfHourMapper(),
    ),
    catchup=False,
):

    @task(outlets=[combined_player_stats])
    def combine_player_stats(dag_run=None):
        """Merge the aligned hourly partitions into a combined dataset."""
        print(dag_run.partition_key)

    combine_player_stats()

See example_asset_partition.py and the Task SDK API docs for PartitionedAssetTimetable and partition mappers.

🏢 Multi-Team Deployments (AIP-67): Airflow for the Enterprise

⚠️ Experimental: Multi-Team support is experimental in Airflow 3.2 and may change in future releases based on user feedback.

Airflow 3.2 introduces multi-team support, allowing organizations to run multiple isolated teams within a single Airflow deployment. Each team can have its own Dags, connections, variables, pools, and executors— enabling true resource and permission isolation without requiring separate Airflow instances per team.

This is particularly valuable for platform teams that serve multiple data engineering or data science teams from shared infrastructure, while maintaining strong boundaries between teams’ resources and access.

Key Capabilities

Per-team resource isolation: Each team has its own Dags, connections, variables, and pools
Per-team executors: Different teams can use different executors (e.g. Celery, Kubernetes, Local, AWS ECS, etc.) and configure them separately — #57837, #57910
Team-scoped authorization: Keycloak and Simple auth managers support team-scoped access control (#61351, #61861)
Team-scoped secrets: Use AIRFLOW_VAR__{TEAM}___{KEY} environment variable or AIRFLOW_CONN_____ pattern for team-specific secrets (#62588)
CLI management: New CLI commands for managing teams (#55283)
UI team selector: Team selector in connection, variable, and pool create/edit forms (#60237, #60474, #61082)
Full API support: team_name field added to Connection, Variable, and Pool APIs (#59336, #57102, #60952)

Enabling Multi-Team

# In airflow.cfg:
[core]
multi_team = True

# Or via environment variable:
export AIRFLOW__CORE__MULTI_TEAM=True

⏰ Deadline Alerts: Now With Synchronous Callbacks (AIP-86)

⚠️ Experimental: Deadline Alerts are experimental in Airflow 3.2 and may change in future releases based on user feedback.

Building on the Deadline Alerts system introduced in Airflow 3.1, this release adds synchronous callback support. In 3.1, callbacks ran through the triggerer (async only), which limited integration options. Synchronous callbacks execute directly via the executor, with optional targeting of a specific executor via the executor parameter.

What’s New in 3.2

SyncCallback support: Unlike AsyncCallback which runs on the triggerer, SyncCallback executes directly on the worker via the executor, with optional targeting of a specific executor
Multiple Deadline Alerts per Dag: Pass a list to the deadline parameter to configure multiple thresholds on a single Dag
Missed-deadline metadata in Grid API: Dag run API now includes missed-deadline information for programmatic monitoring
Improved UX for custom DeadlineReferences: Cleaner developer experience when defining custom deadline reference points (#57222)

with DAG(
    dag_id="sync_deadline",
    deadline=DeadlineAlert(
        reference=DeadlineReference.FIXED_DATETIME(datetime(1980, 8, 10, 2)),
        interval=timedelta(0),
        callback=SyncCallback(
            SlackWebhookNotifier,
            {"text": "Sync Callback; Alert should trigger immediately!"},
        )
    )
):
    EmptyOperator(task_id='empty_task')

🖥️ UI Enhancements

HITL Approval History: The Human-in-the-Loop approval interface now shows the complete audit trail of approvals and rejections for any task. (#56760, #55952)
XCom Management: You can now add, edit, and delete XCom values directly from the UI. (#58921)
Segmented state bar: Collapsed task groups and mapped tasks now show a segmented state bar for at-a-glance status (#61854)
Unified tooltips: Grid and Graph view tooltips now show dates, duration, and child states (#62119)
Filename in Dag Code tab: File identification now shown in the Code tab (#60759)
Copy button for logs: One-click log copying (#61185)
Date range filter: Filter Dag executions by date range (#60772)
Task upstream/downstream filter: Filter by upstream or downstream tasks in Graph and Grid views (#57237)
Data redaction: Sensitive fields are now redacted in the UI and Public API (#59873)
Custom theme support: globalCss and theme config for white-label/custom deployments (#61161, #58411)
Inherit core UI theme in React plugins: Plugin UIs now automatically match the core Airflow theme (#60256)
Task display names in Gantt: task_display_name shown for better readability (#61438)

🚀 Performance Improvements

Rendered Task Instance Fields Cleanup: ~42x Faster. The cleanup job for rendered task instance fields has been rewritten and is roughly 42 times faster for Dags with many mapped tasks. Retention is now based on the N most recent Dag runs rather than N most recent task executions, which is both more intuitive and dramatically more performant. Config renamed: max_num_rendered_ti_fields_per_task → num_dag_runs_to_retain_rendered_fields (old name still works with deprecation warning). (#60951)

Scheduler Improvements. For large-scale deployments, 3.2 addresses several known bottlenecks:

The scheduler no longer loads all TaskInstances into memory, preventing memory spikes on large deployments (#60956)
Faster task dequeuing loop (#61376)
Queue query now enforces max_active_tasks directly, preventing over-queueing (#54103)

API Server Improvements:

Eliminated SerializedDag loads on task start, reducing memory usage (#60803)
serialized_dag data column now uses JSONB on PostgreSQL (#55979)

🔧 Task SDK Evolution & Developer Experience

Task SDK Decoupling Continues

Airflow 3.2 continues moving components from airflow-core into the Task SDK, progressing toward full client-server separation. This enables Dag authors to independently upgrade the Task SDK without requiring Airflow Core upgrades, reducing coordination overhead between Dag authors and Ops teams.

Modules moved to Task SDK in this release (old import paths still work with deprecation warnings):

Exceptions: AirflowSkipException, TaskDeferred, etc. → airflow.sdk.exceptions (#59780)
Serde: airflow.serialization.serde → airflow.sdk.serde; serializers → airflow.sdk.serde.serializers.* (#58900)
SkipMixin / BranchMixIn: Moved to Task SDK; existing imports work via common-compat (#62749, #62776)
Lineage module: Moved to Task SDK for client-server separation (#60968, #61157)
Listeners module: Moved to shared library (#59883)
XCom API: Decoupled from XComEncoder (#58900)

PythonOperator Async Support

PythonOperator now supports async callables. You can pass an async function as the python_callable and the operator will correctly await it, enabling async I/O patterns without needing a custom operator. (#60268)

@task(show_return_value_in_logs=False)
async def load_xml_files(files):
    import asyncio
    from io import BytesIO
    from more_itertools import chunked
    from os import cpu_count
    from tenacity import retry, stop_after_attempt, wait_fixed

    from airflow.providers.sftp.hooks.sftp import SFTPClientPool

    print("number of files:", len(files))

    async with SFTPClientPool(sftp_conn_id=sftp_conn, pool_size=cpu_count()) as pool:
        @retry(stop=stop_after_attempt(3), wait=wait_fixed(5))
        async def download_file(file):
            async with pool.get_sftp_client() as sftp:
                print("downloading:", file)
                buffer = BytesIO()
                async with sftp.open(file, encoding=xml_encoding) as remote_file:
                    data = await remote_file.read()
                    buffer.write(data.encode(xml_encoding))
                    buffer.seek(0)
                return buffer

        for batch in chunked(files, cpu_count() * 2):
            tasks = [asyncio.create_task(download_file(f)) for f in batch]

            # Wait for this batch to finish before starting the next
            for task in asyncio.as_completed(tasks):
                result = await task
                # Do something with result or accumulate it and return it as an XCom

Updated securiy model

We are working on improving isolation and improving security of Airflow deployments and in order to make our users better informed of what expectations they should have for Airflow security, we updated the security model to reflect changes implemented in Airflow 3.2.0 and explain future improvements that we work on in this area. See more: Airflow Security Model.

🙏 Community Appreciation

This release represents the collaborative effort of hundreds of contributors from around the world. Special thanks to our release manager and all the developers, documentarians, testers, and community members who made Airflow 3.2.0 possible.

Thanks to contributors like you, the Airflow project continues to thrive. Whether you’re filing issues, submitting PRs, improving documentation, or helping others in the community, every contribution matters.

🔗 Get Involved

Try the Release: Upgrade your development environment and explore the new features
Join the Conversation: Connect with us on Slack and the dev mailing list
Contribute: Check out our contribution guide
Provide Feedback: Share your experiences and suggestions on GitHub

Apache Airflow 3.2.0 marks a new chapter in data-aware, partition-driven workflow orchestration. We can’t wait to see what you build with it!

]]>

Introducing the Apache Airflow Registry

2026-03-19T00:00:00Z

Today we’re launching the Apache Airflow Registry — a searchable catalog of every official Airflow provider and its modules, live at airflow.apache.org/registry/.

Need an S3 operator? A Snowflake hook? An OpenAI sensor? The Registry helps you find, compare, and configure the right components for your data pipelines — without digging through docs or PyPI pages.

By the Numbers


98	Official providers
1,602	Modules (operators, hooks, sensors, triggers, transfers, and more)
329M+	Monthly PyPI downloads across all providers
125+	Integrations with cloud platforms, databases, ML tools, and messaging services

Search Everything

Hit Cmd+K from any page and start typing. Results show up instantly, grouped by Providers and Modules, with type badges so you can tell a hook from an operator at a glance.

Provider Pages

Each provider gets a dedicated page with everything in one place: install command with copy-to-clipboard, version selector, extras dropdown, compatibility info, connection types, and the full module listing organized by type.

The Amazon provider, for example, has 372 modules across operators, hooks, sensors, triggers, transfers, and more. Module type tabs let you filter to exactly what you’re looking for, and a category sidebar groups modules by AWS service (S3, Lambda, Glue, Step Functions, etc.).

Connection Builder

Click any connection type badge on a provider page, fill in the fields, and the builder generates the connection in three formats — URI, JSON, and Env Var — ready to copy into your configuration.

No more guessing URI encoding or JSON structure.

Explore by Category

Not sure which provider you need? The Explore page organizes providers into categories: Cloud Platforms, Databases, Data Warehouses, Messaging & Notifications, AI & Machine Learning, Data Processing, and more.

Statistics

The Stats page breaks down the ecosystem: 848 operators, 298 hooks, 164 triggers, 157 sensors, 83 transfers, and more — plus top providers by downloads and module count.

JSON API

Every piece of data in the Registry is available as structured JSON — providers, modules, parameters, connections, versions. An API Explorer lets you browse all endpoints interactively.

This makes the Registry accessible to IDE extensions, AI coding assistants, and automation tools.

Light & Dark Mode

Full theme support with dark mode as the default. One click to switch.

Standing on Shoulders

The Apache Airflow PMC would like to thank Astronomer for building and maintaining the Astronomer Registry for years — it was the go-to place to discover Airflow providers and proved the value of a searchable provider catalog. That work directly shaped this community-owned registry.

The Apache Airflow Registry lives at airflow.apache.org, is built from the same repo as the providers, and updates automatically when new versions are published.

What’s Next

This is the first release of the Registry. Here’s what’s coming:

Third-party provider support — we’re exploring options to list community-built providers alongside the official ones
Richer module pages — dedicated pages per module with full parameter docs and usage examples

Get Involved

Explore the Registry and let us know what you think
Join the conversation on Airflow Slack and the dev mailing list
Contribute — the code lives in registry/ in the main Airflow repo
Report issues or request features on GitHub

]]>

Airflow Survey 2025

2026-01-22T00:00:00Z

The interactive report is hosted by Astronomer. The Apache Airflow community thanks Astronomer for running this survey, for sponsoring it and providing the report in this form, and for their effort in marketing, analysis, and preparing the graphics.

View raw data

Download survey responses (CSV)

]]>

Apache Airflow CTL aka airflowctl 0.1.0

2025-10-15T00:00:00Z

We are thrilled to announce the first major release of airflowctl 0.1.0, the new secure, API-driven command-line interface (CLI) for Apache Airflow — built under AIP-81.

This release marks CLI to join the general posture on communicating through API. Airflow CLI joins the modern era of secure, auditable, and remote-first operations.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow-ctl/0.1.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow-ctl/stable/release_notes.html
🪶 Source Code: https://github.com/apache/airflow/tree/main/airflow-ctl

🎯 What is airflowctl?

airflowctl is a new command-line interface for Apache Airflow that interacts exclusively with the Airflow REST API. It provides a secure, auditable, and consistent way to manage Airflow deployments — without direct access to the metadata database.

🔄 Coexistence with Airflow CLI

The Airflow CLI will continue as intended, primarily for admin tasks such as running Airflow components (airflow api-server, airflow scheduler) or managing the metadata database (airflow db init). airflowctl focuses on operational commands that interact with Airflow resources via the API (airflowctl dagrun trigger, airflowctl connection create, etc.).

We defined the commands falls under two main categories:

Remote Commands: Operations that can be provided via API (e.g., managing DAGs, connections, variables, triggering DAG runs) are now available in airflowctl and will be the recommended approach going forward.
Local/Admin Commands: Operations that manage Airflow components or the metadata database will remain in the Airflow CLI.

Of course, in the current state they will both have the remote commands. We are planning a zero-disruption migration path where Remote Commands will be gradually deprecated from the Airflow CLI as they achieve parity in airflowctl.

🔒 Why airflowctl?

Until now, Airflow CLI connected directly to the metadata database, bypassing RBAC, authentication, and API logs. While convenient, this approach limited security, auditing, and remote management capabilities — especially for enterprise environments.

airflowctl changes that by routing every command through the Airflow REST API, bringing:

Authentication & RBAC enforcement
Centralized logging & audit trail
Secure credential storage via Keyring
Remote command execution with zero DB access
Consistency with Airflow UI and API behaviors

🚀 AIP-81: CLI Reimagined Through the API

AIP-81 (“Enhanced Security in CLI via Integration of API”) defined a clear goal:

“The CLI must be a first-class, secure client of the Airflow REST API — not a privileged database actor.”

airflowctl is the direct realization of that vision.

Core design principles:

All remote commands use the REST API
RBAC and auth handled consistently via API layer
Pluggable auth mechanisms (basic auth, OAuth, token, etc.)
Secure credential persistence through system keyring
Extensible to new API endpoints as Airflow evolves

⚙️ Getting Started

pip install apache-airflow-ctl

Once installed, you can connect your CLI to an Airflow instance:

airflowctl auth login --url http://localhost:8080 --username admin --password admin

The password field is interactive by default. You can enter your password securely without echoing it on the terminal. Use the above command without specifying the password and run it.

airflowctl auth login --url http://localhost:8080 --username admin --password

🧩 Command Highlights

Here’s a quick look at some of the most popular commands, now fully API-backed in airflowctl 0.1.0:

🧩 Assets

⚙️ Config

🔑 Connections

🎯 DAG Runs

Trigger and inspect DAG runs securely through the API:

🪣 Variables

All these commands — and many more — operate via Airflow’s public REST API, ensuring secure, logged, and RBAC-controlled execution.

🔐 Key Security Features

🔑 Keyring Integration

No more plaintext tokens or passwords. airflowctl uses your OS-level keyring (e.g., macOS Keychain, Windows Credential Manager, or Linux Secret Service) to store and retrieve authentication tokens securely.

🧱 Role-Based Access Control

Every command is evaluated by Airflow’s RBAC system through the API — ensuring consistent authorization with the web UI and API clients.

🕵️‍♀️ Auditing and Traceability

All CLI commands generate API logs and can be observed through standard audit mechanisms — closing a long-standing gap between the CLI and Airflow’s security model.

📈 Roadmap Highlights

airflowctl 0.1.0 is just the beginning. The foundation is in place for a fully unified, secure CLI experience.

🧩 Coming Soon

Completeness of API coverage
Live log streaming
Worker management
Remote debugging
Incremental deprecation of legacy CLI commands
Over time, the legacy airflow CLI will be incrementally deprecated as commands achieve API parity.

🧭 Migration

Migration requires mapping commands, updating authentication, and re-testing automation to ensure compatibility with the new API-backed architecture. Because airflowctl mirrors the core CLI syntax, most workflows require minimal changes — primarily adjusting authentication and configuration.

Side by side comparison:

Before	After

🙏 Community & Acknowledgments

This release is the result of extensive collaboration across the Apache Airflow community. Many thanks all who worked on AIP-81, the Airflow REST API, Authentication, and the airflowctl implementation.

Leading Contributors

Special thanks to leading contributors of airflowctl: Amar Prakash Pandey, Amogh Desai, Aritra Basu, Aryan Khurana, ayush3singh, Brent Bovenzi, Brunda10, Bugra Ozturk, Daniel Standish, D. Ferruzzi, Deji Ibrahim, Elad Kalif, Ephraim Anierobi, GPK, Guan Ming(Wesley) Chiu, Hussein Awala, Jake Roach, Jarek Potiuk, Jed Cunningham, Jens Scheffler, Jaejun Lee, Kalyan R, Karthikeyan Singaravelan, Kaxil Naik, Kevin Yang, Kiruban Kamaraj, LI,JHE-CHEN, Pierre Jeambrun, Pratiksha, Sam Wheating, Tzu-ping Chung, Valentyn, Vincent, Wei Lee, Yeonguk, Yunchi Pang, Zhen-Lun (Kevin) Hong

✨ In Summary

airflowctl 0.1.0 makes Airflow’s command line:

Before	After
Direct DB access	API-backed security
No RBAC or audit	Centralized auth & logging
Inconsistent behavior	Unified CLI + API experience
Manual secrets	Keyring-secured credentials

Security first. API always. CLI reimagined. The secure CLI foundation lays the groundwork for Airflow’s next generation. A unified, API-first platform for orchestration and operations.

]]>

Apache Airflow 3.1.0: Human-Centered Workflows

2025-09-25T00:00:00Z

We are thrilled to announce the release of Apache Airflow 3.1.0, an update that puts humans at the center of data workflows. This release introduces powerful new capabilities for human decision-making in automated processes, comprehensive internationalization support, and significant developer experience improvements.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/3.1.0/
📚 Core Airflow Docs: https://airflow.apache.org/docs/apache-airflow/3.1.0/
📚 Task SDK Docs: https://airflow.apache.org/docs/task-sdk/1.1.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/3.1.0/release_notes.html
🪶 Sources: https://airflow.apache.org/docs/apache-airflow/3.1.0/installation/installing-from-sources.html
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.1.0

🤝 Human-in-the-Loop (HITL): When Automation Meets Human Judgment

This powerful capability bridges the gap between automated processes and human expertise, making Airflow invaluable for:

AI/ML Model Validation: Pause inference pipelines for human review of model outputs
Content Moderation: Route content through human reviewers before publication
Approval Workflows: Require manager approval for sensitive operations
Data Quality Gates: Allow data stewards to validate critical datasets

HITL tasks pause in a deferred state while presenting intuitive web forms in the Airflow UI. Users with appropriate roles can review context data, DAG parameters, and XCom values before making informed decisions.

Example Code:

from airflow.sdk import DAG
from airflow.providers.standard.operators.hitl import HITLOperator

with DAG("content_moderation", schedule="@daily") as dag:
    moderate_content = HITLOperator(
        task_id="review_content",
        message="Please review this content for publication",
        data_key="content_to_review"
    )

📊 UI Enhancements & Performance

Calendar and Gantt Views Make Their Comeback

Remember those beloved Calendar and Gantt chart views from Airflow 2.x? They’re back, completely rebuilt for the modern React UI after being omitted from the 3.0 release.

The new Calendar view is genuinely interactive with filtering capabilities that make it easy to drill down into specific time periods and dag states.

The Gantt chart is now integrated directly into the grid view and renders much faster than the old version, giving you that timeline perspective without the performance headaches.

Theme Updates That Actually Matter

We’ve refreshed the color palette using modern design principles, making the UI more consistent, professional and most of all taken a careful look at contrast ratios so the UI should be more accessible.

Other Improvements

We’ve added a lot more filtering options across the pages!

Plus, you can now pin your favorite DAGs to keep them at the top of your list or to filter for them easily. It’s one of those small features that makes a huge difference when dealing with 100s of workflows.

Credited to Volker Janz.

📊 UI Development Milestone: Airflow 3.1.0 features 5x more UI pull requests than the 2.10 release and 50% more than Airflow 3.0, demonstrating the community’s commitment to user experience excellence.

⏰ Deadline Alerts: Proactive Workflow Monitoring

Say goodbye to reactive monitoring. Deadline Alerts provide proactive notifications when DAG runs exceed time thresholds, helping ensure SLA compliance and timely completion of critical workflows.

Configure monitoring by specifying:

Reference point: DAG queued time, logical date, or fixed datetime
Interval: Time threshold (positive or negative)
Callback: Notifications via Airflow Notifiers or custom functions

Example Code:

from datetime import timedelta
from airflow.sdk.definitions.deadline import DeadlineAlert, DeadlineReference, AsyncCallback
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier

with DAG(
    "critical_etl",
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(hours=2),
        callback=AsyncCallback(
            SlackWebhookNotifier,
            kwargs={"text": "🚨 Critical ETL missed deadline!"}
        )
    )
) as dag:
    # Your tasks here

Perfect for monitoring daily ETLs, alerting before critical deadlines, or escalating resource-constrained workflows.

🌍 Going Global with 17 Languages

Airflow now speaks your team’s language. Literally. We have added comprehensive internationalization support for 17 languages, including Arabic, Chinese, French, German, Spanish and more. The interface detects your browser preferences automatically, but you can switch languages on the fly without refreshing the page.

For our Arabic and Hebrew users, we’ve built in proper right-to-left (RTL) support

The best part? We have made it straightforward for the community to contribute additional languages with clear contribution guidelines, so this is just the beginning of Airflow’s global reach.

🎨 Build your Airflow, your way

The new React Plugin System (AIP-68) transforms how you extend Airflow’s interface. We have replaced the old Flask-based approach with a modern toolkit that lets you customize Airflow exactly how your team works.

Want to embed your company’s dashboard right in the Airflow UI? Build React applications or iframes that will render inside Airflow’s (nav bar, dashboard, details page, etc.). Want to link to your existing tools seamlessly? Create custom external links to your resources. Want to extend Airflow’s API server? Register FastAPI sub applications and middlewares that fit your specific processes.

The system includes:

External Views for linking to existing tools (external links or embedded iframes)
React Applications support for rendering external react apps
FastAPI Sub Applications to extend the API server
Root Middlewares for intercepting API requests (even core ones)

We’ve already seen teams integrate everything from Wikipedia searches to data lineage visualizations to yes, someone building a snake game to play while waiting on dag runs!

Credited to Tamara Fingerlin.

🔧 Enhanced Developer and Authoring Experience

Task SDK Evolution

Airflow 3.1 advances the decoupling of the Task SDK from Airflow Core through improved DAG serialization. While complete separation arrives in 3.2.0, the foundation enables:

Independent Upgrades: Reduced coordination need between Dag authors and Airflow Ops teams
Forward Compatibility: Dag authors should now write Dags by importing from the airflow.sdk namespace for future-proofing. (Naturally, the old imports still work but issue a warning.)
Deployment Flexibility: Better support for separated component deployment

Python 3.13 Support

Airflow 3.1.0 adds Python 3.13 support while removing Python 3.9 (end-of-life). The platform now supports Python 3.10, 3.11, 3.12, and 3.13.

Inference Execution

A new streaming API endpoint (/dags/{dag_id}/dagRuns/{dag_run_id}/wait) allows applications to watch DAG runs until completion, enabling responsive integration patterns for real-time workflows.

The below example use httpx to trigger a dag run, and emits the final dag run state after it finishes:

import asyncio
import json

import httpx

dag_id = "my-dag"

async def create_and_wait(client):
    # Create a dag run...
    r = await client.post(f"https://my-airflow.example.com/api/v2/dags/{dag_id}/dagRuns")
    run_id = r.json()["dag_run_id"]
    async with client.stream(
        "GET",
        f"https://my-airflow.example.com/api/v2/dags/{dag_id}/dagRuns/{run_id}/wait",
    ) as r:
        async for line in r.aiter_lines():
            pass  # You can do progress report here instead.
    print("Dag run state:", json.loads(line.strip())["state"])

async def main():
    async with httpx.AsyncClient() as client:
        await create_and_wait(client)

asyncio.run(main())

🙏 Amazing Community

Apache Airflow 3.1.0 represents an extraordinary community effort, showcasing the vibrant ecosystem that drives this project forward with 163 contributors making this release possible across 1,400+ commits.

Leading Contributors

Special thanks to our top 20 contributors who drove this release forward: Amogh Desai, Ash Berlin-Taylor, Brent Bovenzi, Bugra Ozturk, Daniel Standish, Elad Kalif, Ephraim Anierobi, GPK, Guan-Ming (Wesley) Chiu, Jarek Potiuk, Jens Scheffler, Karthikeyan Singaravelan, Kaxil Naik, LI,JHE-CHEN, Pierre Jeambrun, Shahar Epstein, Tzu-ping Chung, Vincent, Wei Lee, and Yeonguk Choo.

View all 143 additional contributors

Apache Airflow® 3 is Generally Available!

2025-04-22T00:00:00Z

We announced our intent to focus on Apache Airflow 3.0® as the next big milestone for the Airflow project at the Airflow Summit in September 2024. We are delighted to announce that Airflow 3.0 is now released!

A Major Release, Four Years in the Making

Airflow 3.0 is the biggest release in Airflow’s history—2.0 was released in 2020, and the last 4 years have seen incremental updates and releases every quarter with version 2.10 released in Q4 2024. With over 30 million monthly downloads (up over 30x since 2020) and 80,000 organizations (up from 25,000 in 2020) now using Airflow, we’ve seen an incredible growth in popularity since 2.0.

Over the last four years, Airflow has grown to power business critical data workflows within organizations of all sizes. We have seen an exponential increase in the use cases for Airflow from its beginnings with ETL, ELT, and Reverse ETL, with over 30% of Airflow users using it for MLOps, and 10% using it for GenAI workflows. Airflow 3 is a response to this use case expansion and is the standard for data application development across the enterprise.

Here are some highlights:

Airflow 3 is significantly easier to use for data practitioners and incorporates their key requests for critical changes to Airflow. Early user reactions to features such as the new React based UI, DAG Versioning, and improved Backfill support have been incredibly positive. I was ecstatic to see the reaction from data engineers when I demonstrated this at a recent Airflow meetup.
The seamless UI transition of navigating between Asset-oriented workflows and Task-oriented workflows is beautiful. Once again, Airflow lets the developer choose how you want to develop and navigate without imposing any restrictions.
Introduction of Event Driven Scheduling enables Airflow to seamlessly integrate with messaging providers and react to events happening and data assets being updated outside of Airflow.
The big architecture change with the introduction of the Task Execution Interface and the Task SDKs, enable a stronger security model, including secure, scalable execution across multi-cloud, hybrid-cloud, and local data center deployments.

This is the result of 300+ developers within the Airflow community working together tirelessly for many months and I could not be more proud to be part of this wonderful team. Here are some more details of the release.

Highly requested UX features

DAG Versioning

DAG Versioning has been the most requested feature within Airflow based on the annual Airflow survey. As implemented in Airflow 3, a DAG will run through to completion based on the version at start, even if a new version has been uploaded while this DAG was being run. All DAG runs in the UI are now associated with the version of the DAG as run including the Task structure, the code, the logs, and more. This is described in two AIPs: Improve DAG history (AIP-65) , and DAG Bundles and Parsing (AIP-66).

Backfills improvements

Another long-standing user request has been better support for backfills. Often discussed in the context of machine learning, backfills also apply to traditional ETL and ELT use cases. In Airflow 3, backfills are run within the scheduler for improved control, scalability, and diagnostics. Backfills can now be started from the UI or API, and monitored within the UI.

This was defined as part of “Scheduler-managed backfills” (AIP-78), and an example screenshot is shown below:

Run anywhere, at any time, in any Language

Run anywhere, in any language

A foundational goal of Airflow 3 is allowing execution in any environment, in any language. A key component of this is the Task Execution Interface (AIP-72), which enables the evolution of Airflow into a client-server architecture, which represents one of the most significant architectural shifts in Airflow’s history. This supports Celery, Kubernetes, and Local Executors, but also enables new capabilities. A component of this change is the API server which represents input for the Task Execution Interface. This foundational feature enables multi-cloud deployments and multi-language support in the form of the Task Execution API. The Airflow 3 release includes the Python TaskSDK which enables backward compatibility for existing DAGs. TaskSDKs for additional languages, starting with Golang will be released over the next few months.

To enable data pipelines to be run on edge devices, outside of the core data centers and clouds, the Edge Executor (AIP-69) is available as a provider package with Airflow 3. This is an incremental feature built on top of the Task Execution Interface. Initial incarnations have been released in experimental mode based on Airflow 2x and this executor has now evolved to leverage the Airflow 3 API Server.

Event-driven scheduling and Data Assets

Airflow 3 represents a foundational jump in enabling Airflow to react to events happening outside of Airflow, including data assets being created or updated by external data systems. This was based on the evolution of Datasets into Data Assets and was broken out into several AIPs as detailed below, which are all part of the release.

The fundamental evolution of Datasets into Data Assets has been done as part of “Introducing Data Assets” (AIP-74). This introduces the concept of Watchers which is leveraged by other capabilities detailed below. A significant enhancement around Data Assets is the New Asset-Centric Syntax (AIP-75) for defining Assets easily with DAGs using the Python decorator syntax, which is part of this release.

External event driven scheduling (AIP-82) is based on the foundational Data Assets work described above, specifically Watchers. The initial scope as defined in the AIP is complete and now incorporates a “Common Message Bus” interface. This release also includes an implementation of the above for AWS SQS as an “out of the box” integration, which demonstrates DAGs being triggered upon the arrival of a message in AWS SQS.

Inference execution and hyperparameter tuning

Many ML and AI Engineers are already using Airflow for ML/AI Ops, especially for model training. However, there were challenges for Inference Execution. Enhancing Airflow for Inference Execution by adding support for non-data-interval-Dags (sorry, that’s a mouthful) is an important change. This work is covered as part of “Remove Execution date unique constraint from DAG run” (AIP-83)

Security and usability improvements

UI Modernization

The Airflow UI has been completely rewritten as part of Airflow 3 and incorporates a significantly improved user experience which seamlessly blends Asset-oriented workflows with Task-oriented workflows. This is a dramatic improvement which enables developers to author DAGs as they choose, without being opinionated about “a right way”.

Check out the screenshots in the docs for more.

Recreating it to be based on React and the FastAPI has been a massive project and was broken out into several AIPs as detailed below.

The foundation for the new UI is the REST API and a set of internal APIs for UI Operations (AIP-84) both of which are now based on FastAPI. These APIs are served as part of the API Server described above as part of the Task Execution framework.

The Airflow 3.0 UI has been significantly improved and includes a streamlined user experience workflow encompassing both the Grid and Graph views. The interaction between DAGs and Assets are also more streamlined. User experience is always a work in progress and we very much appreciate your feedback. This is covered in great detail as part of the Modern Web Application proposal (AIP-38).

As part of this project, Flask AppBuilder has now been moved into a separate provider package and is no longer a part of the Core Airflow package. This enables an easier security and maintenance update process, while retaining backwards compatibility. This is documented as part of the “Remove Flask App Builder as a Core Dependency” proposal (AIP-79).

Security

A key benefit of the Task Execution Interface and the API server is Task Isolation. This has often been requested by Airflow enterprise deployments for a better security posture when an Airflow deployment is shared by multiple teams. Further security and authorization patterns can be developed on top of this foundation as more detailed requirements are uncovered.

Improving the CLI and reducing the maintenance burden by having the CLI use the Airflow APIs, rather than direct access is an important evolution for Airflow. We have now split the core Airflow CLI into two parts, the first for local development and backwards compatibility and the second for remote access using the API. The second will be a new provider package called “airflowctl” which can be optionally installed along with Core Airflow. This is documented in more detail as part of the “Enhanced security in CLI via Integration of API” proposal (AIP-81).

An amazing community

This release could not have happened without the inspiration and technical leadership of key contributors who led the AIPs listed above. We thank them all here: Ash Berlin-Taylor, Brent Bovenzi, Bugra Ozturk, Constance Martineau, Daniel Standish, Jed Cunningham, Jens Scheffler, Kaxil Naik, Pierre Jeambrun, Vincent Beck, and Vikram Koka. We also wanted to thank Jarek Potiuk for the critical development infrastructure and packaging work and to Elad Kalif for shepherding all the key provider changes needed. We would like to recognize Wei Lee and Ankit Chaurasia for their work on the upgrade utilities to enable users to easily upgrade to Airflow 3.

Finally, a huge shoutout to Jed Cunningham and Kaxil Naik for the critical part of release management!

Over three hundred developers around the world have contributed to making this release a reality. We thank them all for their contributions. They are listed here in alphabetical order:

Aakcht
Aaron Chen
Abhishek
Adam Turner
Adan
Aditya Yadav
Adrian Lazar
Adrian Perea
Ajit J Gupta
Albert Okiri
Alex Waygood
Alexander Millin
AlteredOracle
Amar Prakash Pandey
Amir Mor
Amogh Desai
Amol Saini
Anakin Skywalker Pactores
Andor Markus
Andre Miranda
Andres Lowrie
Andrew Arochukwu
Andrew Stein
Andrii Abramov
Andrii Korotkov
Andrii Yerko
Ankit Chaurasia
Anthony Lin
Antony Southworth
Aritra Basu
Arjun Pathak
Arnel Jan Sarmiento
Arnout Engelen
Artem Suslov
Arthur Braveheart
Artour
Artur Skarżyński
Arunav Gupta
Aryan Khurana
Ash Berlin-Taylor
AshKatzEm
AutomationDev85
Avihais12344
Azhar Izzannada E
Baitur Ulukbekov
Balthazar Rouberol
Bartosz Jankiewicz
Bas
Ben Chen
Benoit Perigaud
Biswamitra Biswas
Bjorn Olsen
Bluefox9x5
Bohdan Udovenko
Bonnie Why
Boris Morel
Bowrna
Brent Bovenzi
Bugra Ozturk
Błażej Tecław
Castle Cheng
Chris Luedtke
Christian Yarros
Christos Bisias
Collin McNulty
Computer Network Investigation
Constance Martineau
D. Ferruzzi
DShi
Daniel Gellert
Daniel Imberman
Daniel Standish
Daniel van der Ende
Danish Amjad
Danny Liu
David Blain
Derek
Detlev V.
Dewen Kong
Sriraj Dheeraj Turaga
Diogo Rodrigues
Dmitry Astankov
Dmitry Pustoshilov
Dominic Leung
Dong-yeong0
Doug Guthrie
Dylan Melotik
Elad Kalif
Eldar Kasmamytov
Ephraim Anierobi
Eric
Everton Seiei Arakaki
Farhan
Fedor Kobak
Felix Uellendall
Fred Thomsen
Fully.is(풀리)
GPK
Gagan Bhullar
Geonwoo Kim
GlenboLake
Gopal Dirisala
Gregory Borodin
Guan-Ming (Wesley) Chiu
Guangyang Li
Guillaume Lostis
Hari Selvarajan
HassanAlahmed
Hojin Jun
Howard Yoo
Huanjie Guo
Hung
Hussein Awala
Hyunsoo Kang
Ian Buss
Idris Adebisi
Igor Kholopov
IlaiGigi
Indrale Dnyaneshwar
JISHAN GARGACHARYA
Jaejun
Jake Ferriero
Jake Roach
Jakub Dardzinski
James Chaldecott
James Regan
Jarek Potiuk
Jasmin Patel
Jason
Jed Cunningham
Jeff Harrison
Jens Scheffler
Jianzhun Du
Jimmy McBroom
Joao Amaral
João Pedro M Miguel
Joel Labes
Joey Cumines
Joffrey Bienvenu
John Bampton
John C. Merfeld
Johnny1cyber
José Joaquín Virtudes Castro
Joseph Ang
JoshuaXOng
Josix
Julian Maicher
Kacper Kulczak
Kacper Muda
Kalyan R
Kamil Breguła
Karen Braganza
Karthik Dulam
Karthik Ravi
Karthikeyan Singaravelan
Kaxil Naik
Kevin Allen
Kim
Kris
Kunal Bhattacharya
LIU ZHE YOU
Lennox Stevenson
Linh
Lorin Dawson
Lou ✨
Lucy Hu
Lukas Mikelionis
Luyang Liu
Lyndon Fan
M. Olcay Tercanlı
Maciej Obuchowski
Madison Swain-Bowden
Maksim
Marcelo Trylesinski
Marcos Marx
Maria
Mark Andreev
Mark H
Matt Burke
Matt Dupree
Maxim Martynov
Mayuresh Kedari
Mehul Goyal
Mike
Mike Beckhusen
Mikhail Dengin
MishchenkoYuriy
Muhammad Hanif Mohamad Musa
Myles Hollowed
Narendra-Neerukonda
Natsu
Nikita
Niko Oliveira
Nishant Gupta
Nitesh Kumar Dubey Samsung
Nitochkin
Oleg Ovcharuk
Oleksandr Slynko
Omkar P
Owen Leung
Pandycool
Pankaj Koti
Park Jiwon
Pavan Sharma
Peng-Jui Wang
Peter Debelak
Phani Kumar
Pierre Jeambrun
Po-Yu Hsieh
Prajwal7842
Pratiksha
Purna Chander
Rafa
Rahul Madan
Rahul Vats
Ramit Kataria
Rishabh Srivastava
Rushabh Garambha
Ryan Eakman
Ryan Hatter
Rytis Ulys
SAI GANESH S
Sam Lendle
SamLiaoP
Saumil Patel
SaurabhhB
Sean Gabriel Bayron
Sean Rose
Sebastian Daum
SeonghwanLee
Shahar Epstein
Shahbaz Aamir
Shoaib UR Rehman
Shubham Raj
Simon Sawicki
Siva Kumar Edupuganti
Sneha Prabhu
Sooter Saalu
Srabasti Banerjee
Stefan Keidel
Steven Loria
Steven Shidi Zhou
Stijn De Haes
Success Moses
TakawaAkirayo
Tamara Janina Fingerlin
Tamas Palinkas
Tatiana Al-Chueyr
Topher Anderson
Tzu-ping Chung
Usiel Riedl
Utkarsh Sharma
Valentyn
Venkat VJ
Vikram Koka
Vikram Medabalimi
Vikramaditya Gaonkar
Vincent
Vincent Kling
VladaZakharova
Waldemar Hummer
Wang Ran (汪然)
Wei Lee
Wojciech Szlachta
Wonseok Yang
Yeonguk Choo
Yohei Kishimoto
Youngha, Park
Yuan Li
Zach Liu
Zhen-Lun (Kevin) Hong
althati
ambikagarg
atrbgithub
awdavidson
codecae
dan-js
darkag
davidfgcorreia
dominikhei
ellisms
enisnazif
fritz-astronomer
gaurav7261
geraj1010
got686-yandex
harjeevan maan
harry.shi
hikaruhk
hprassad
ipsatrivedi
jaejun
jj.lee
jonhspyro
kanagaraj
kandharvishnu
leoguzman
lucasmo
luoyuliuyin
mahdi alizadeh
majorosdonat
max
mayankymailusfedu
michaeljs-c
morooshka
ninad-opsverse
olegkachur-e
paolomoriello
perry2of5
pgvishnuram
phi-friday
rahulgoyal2987
raphaelauv
rgriffier
rom sharon
saucoide
sbock-slack
sc-anssi
seyoon-lim
simonprydden
skandala23
sonu4578
suyesh-amatya
svellaiyan
tnk-ysk
uzhastik
vatsrahul1001
vfeldsher
xavipuerto
xitep
yangyulely
yunchi
鐘翊修
김영준

What’s Next

We’d love your feedback. Try out the release, open issues, file PRs, or just join the conversation on the Airflow dev list, Slack, and GitHub. Let’s build the future of data orchestration—together.

]]>

Airflow Survey 2024

2025-02-27T00:00:00Z

View raw data

Download survey responses (CSV)

]]>

Apache Airflow 2.10.0 is here

2024-08-08T00:00:00Z

I’m happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.10.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0

Airflow now collects Telemetry data by default

With the release of Airflow 2.10.0, we’ve introduced the collection of basic telemetry data, as outlined here. This data will play a crucial role in helping Airflow maintainers gain a deeper understanding of how Airflow is utilized across various deployments. The insights derived from this information are invaluable in guiding the prioritization of patches, minor releases, and security fixes. Moreover, this data will inform key decisions regarding the development roadmap, ensuring that Airflow continues to evolve in line with community needs.

For those who prefer not to participate in data collection, deployments can easily opt out by setting the [usage_data_collection] enabled option to False or by using the SCARF_ANALYTICS=false environment variable.

Multiple Executor Configuration (formerly “Hybrid Execution”)

Each executor comes with its unique set of strengths and weaknesses, typically balancing latency, isolation, and compute efficiency. Traditionally, an Airflow environment is limited to a single executor, requiring users to make trade-offs, as no single executor is perfectly suited for all types of tasks.

We are introducing a new feature that allows for the concurrent use of multiple executors within a single Airflow environment. This flexibility enables users to take advantage of the specific strengths of different executors for various tasks, improving overall efficiency and mitigating weaknesses. Users can set a default executor for the entire environment and, if necessary, assign particular executors to individual DAGs or tasks.

To configure multiple executors we can pass comma separated list in airflow configuration. The first executor in the list will be the default executor for the environment.

[core]
executor = 'LocalExecutor,CeleryExecutor'

To make it easier for dag authors, we can also specify aliases for executors that can be specified in the executor configuration

[core]
executor = 'LocalExecutor,KubernetesExecutor,my.custom.module.ExecutorClass:ShortName'

DAG authors can specify executors to use at the task

BashOperator(
    task_id="hello_world",
    executor="ShortName",
    bash_command="echo 'hello world!'",
)

@task(executor="KubernetesExecutor")
def hello_world():
    print("hello world!")

We can also specify executors on the DAG level

def hello_world():
    print("hello world!")

def hello_world_again():
    print("hello world again!")

with DAG(
    dag_id="hello_worlds",
    default_args={"executor": "ShortName"},  # Applies to all tasks in the DAG
) as dag:
    # All tasks will use the executor from default args automatically
    hw = hello_world()
    hw_again = hello_world_again()

Dynamic Dataset scheduling through DatasetAlias

Airflow 2.10 comes with DatasetAlias class which can be passed as a value in the outlets, inlets on a task, and schedule on a DAG. An instance of DatasetAlias is resolved dynamically to a real dataset. Downstream can depend on either the resolved dataset or on an alias itself.

DatasetAlias has one argument name that uniquely identifies the dataset. The task must first declare the alias as an outlet, and use outlet_events or yield Metadata to add events to it.

Emit a dataset event during task execution through outlet_events

from airflow.datasets import DatasetAlias

@task(outlets=[DatasetAlias("my-task-outputs")])
def my_task_with_outlet_events(*, outlet_events):
    outlet_events["my-task-outputs"].add(Dataset("s3://bucket/my-task"))

Emit a dataset event during task execution by yielding Metadata

from airflow.datasets.metadata import Metadata

@task(outlets=[DatasetAlias("my-task-outputs")])
def my_task_with_metadata():
    s3_dataset = Dataset("s3://bucket/my-task}")
    yield Metadata(s3_dataset, alias="my-task-outputs")

There are two options for scheduling based on dataset aliases. Schedule based on DatasetAlias or real datasets.

with DAG(dag_id="dataset-alias-producer"):
    @task(outlets=[DatasetAlias("example-alias")])
    def produce_dataset_events(*, outlet_events):
        outlet_events["example-alias"].add(Dataset("s3://bucket/my-task"))

with DAG(dag_id="dataset-consumer", schedule=Dataset("s3://bucket/my-task")):
    ...

with DAG(dag_id="dataset-alias-consumer", schedule=DatasetAlias("example-alias")):
    ...

Dataset Aliases UI Enhancements

Now users can see Dataset Aliases in legend of each cross-dag dependency graph with a corresponded icon/color.

Dark Mode for Airflow UI

Airflow 2.10 comes with new Dark Mode feature which is designed to enhance user experience by offering an alternative visual theme that is easier on the eyes, especially in low-light conditions. You can toggle the crescent icon on the right side of the navigation bar to switch between light and dark mode.

Task Instance History

In Apache Airflow 2.10.0, when a task instance is retried or cleared, its execution history is maintained. You can view this history by clicking on the task instance in the Grid view, allowing you to access information about each attempt, such as logs, execution durations, and any failures. This feature improves transparency into the task’s execution process, making it easier to troubleshoot and analyze your DAGs.

The history displays the final values of the task instance attributes for each specific run. On the log page, you can also access the logs for each attempt of the task instance. This information is valuable for debugging purposes.

Dataset UI Enhancements

The dataset page has been revamped to include a focused dataset events section with additional details such as extras, consuming DAGs, and producing tasks.

We now have separate dependency graph and dataset list pages in new tabs, enhancing the user experience.

Dataset events are now displayed in both the Details tab of each DAG run and within the DAG graph.

Toggle datasets in Graph

We can now toggle the datasets in the DAG graph

Dataset Conditions in DAG Graph view

We now display the graph view with logical gates. Datasets with actual events are highlighted with a different border, making it easier to see what triggered the selected run.

Dataset event info in DAG Graph

For a DAG run, users can now view the dataset events connected to it directly in the graph view.

On-demand DAG Re-parsing

In 2.10 users can now reparse the DAGs on demand using below button on DAG list and DAG detail pages

Additional new features

Here are just a few interesting new features since there are too many to list in full:

Deferrable operators can now execute directly from the triggerer without needing to go through the worker. This is especially efficient for certain operators, like sensors, and can help teams save both time and money.
Crucial executor logs are now integrated into the task logs. If the executor fails to start a task, the relevant error messages will be available in the task logs, simplifying the debugging process.

Contributors

Thanks to everyone who contributed to this release, including Andrey Anshin, Brent Bovenzi, Daniel Standish, Ephraim Anierobi, Hussein Awala, Jarek Potiuk, Jed Cunningham, Jens Scheffler, Tzu-ping Chung, Vincent, and over 63 others!

I hope you enjoy using Apache Airflow 2.10.0!

]]>

Apache Airflow 2.9.0: Dataset and UI Improvements

2024-04-08T00:00:00Z

I’m happy to announce that Apache Airflow 2.9.0 has been released! This time around we have new features for data-aware scheduling and a bunch of UI-related improvements.

Apache Airflow 2.9.0 contains over 550 commits, which include 38 new features, 70 improvements, 31 bug fixes, and 18 documentation changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.9.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.9.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.9.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.9.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.9.0

Airflow 2.9.0 is also the first release that supports Python 3.12. However, Pendulum 2 does not support Python 3.12, so you’ll need to use Pendulum 3 if you upgrade to Python 3.12.

New data-aware scheduling options

Logical operators and conditional expressions for DAG scheduling

When Datasets were added in Airflow 2.4, DAGs only had scheduling support for logical AND combinations of Datasets. Simply, you could schedule against more than one Dataset, but a DAG run would only be created once all the Datasets were updated after the last run. Now in Airflow 2.9, we support logical OR and even arbitrary combinations of AND and OR.

As an example, you can schedule a DAG whenever dataset_1 or dataset_2 are updated :

with DAG(schedule=(dataset_1 | dataset_2), ...):
    ...

You can have arbitrary combinations:

with DAG(schedule=((dataset_1 | dataset_2) & dataset_3), ...):
    ...

You can read more about this new functionality in the data-aware scheduling docs.

Combining Dataset and Time-Based Schedules

Airflow 2.9 comes with a new timetable, DatasetOrTimeSchedule, that allows you to schedule DAGs based on both dataset events and a timetable. Now you have the best of both worlds.

For example, to run whenever dataset_1 updates and at midnight UTC:

with DAG(
    schedule=DatasetOrTimeSchedule(
        timetable=CronTriggerTimetable("0 0 * * *", timezone="UTC"),
        datasets=[dag1_dataset],
    ),
    ...
):
    ...

Dataset Event REST API endpoints

New REST API endpoints have been introduced for creating, listing, and deleting dataset events. This makes it possible for external systems to notify Airflow about dataset updates and unlocks management of event queues for more sophisticated use cases.

See the Dataset API docs for more details.

Dataset UI Enhancements

The DAG’s graph view has been enhanced to display both the datasets it is scheduled on and those in the task outlets, providing a comprehensive overview of the datasets consumed and produced by the DAG.

The main datasets view now allows you to filter for both DAGs and datasets:

When viewing a Dataset, you can now create a manual dataset event through the UI by clicking the play button shown in the top right here:

Custom names for Dynamic Task Mapping

Gone are the days of clicking into index numbers and hunting for the dynamically mapped task you wanted to see! This has been a requested feature ever since task mapping was added in Airflow 2.3, and we are happy it’s finally here.

You can provide a map_index_template to mapped operators:

BashOperator.partial(
    task_id="hello",
    bash_command="echo Hello $NAME",
    map_index_template="{{ task.env['NAME'] }}",
).expand(
    env=[{"NAME": "John"}, {"NAME": "Bob"}, {"NAME": "Fred"}],
)

That template will be rendered after each task finishes running and will populate the name in the UI:

More details on this, including a taskflow example, is available in the dynamic task mapping docs.

Object Storage as XCom Backend

You can now configure Object Storage to be used as an XCom backend, making it much easier to get XCom results into an object store. Deployment managers can configure the object store of their choice, a size threshold to route some results to the Airflow metadata database and some to the object store, and even a compression method to apply before the data is stored.

The following configuration will store anything above 1MB in S3 and will compress it using gzip:

[core]
xcom_backend = airflow.providers.common.io.xcom.backend.XComObjectStoreBackend

[common.io]
xcom_objectstorage_path = s3://conn_id@mybucket/key
xcom_objectstorage_threshold = 1048576
xcom_objectstorage_compression = gzip

See the docs on the object storage xcom backend for more details.

Display names for DAGs and Tasks

Get your emojis ready! You can now set a display name for dags and tasks, separate from the dag_id and task_id. This allows you to have localized display names in the UI, or just use a bunch of emojis.

Using dag_display_name and task_display_name, you can break away from the ascii handcuffs:

with DAG("not_a_fun_dag_id", dag_display_name="📣 Best DAG ever 🎉", ...):
    BashOperator(task_id="some_task", task_display_name="🥳 Fun task!", ...)

Task log grouping

Airflow now has support for arbitrary grouping of task logs.

By default, pre-execute and post-execute logs are grouped and collapsed, making it easier to see your task logs:

You can also use this feature in your task code to make your logs easier to follow:

@task
def big_hello():
    print("::group::Setup our big Hello")
    greeting = ""
    for c in "Hello Airflow 2.9":
        greeting += c
        print(f"Adding {c} to our greeting. Current greeting: {greeting}")
    print("::endgroup::")
    print(greeting)

That custom group is collapsed by default:

And it can be expanded if you want to dig into the details:

UI Modernization

In addition to all the UI improvements mentioned above, we have a bunch more improvements in Airflow 2.9!

The rest of the DAG level views have been moved into React and the grid view interface, allowing for a more cohesive experience. This includes the calendar, task duration, run duration (which replaces landing times), and the audit log. These weren’t just “moved”, they each were improved upon as well.

Here is the new run duration view, which replaces landing times. Users can toggle between landing times and simple run duration:

And the new task duration view. Users can toggle queued time on/off and see the median value across the displayed runs:

Additional new features

Here are just a few interesting new features since there are too many to list in full:

All create/update/delete actions in the REST API are now recorded in the audit log
New on_skipped_callback
Auto pause DAGs after n consecutive failures
Support for Matomo as an analytics tool
New @task.bash TaskFlow decorator
Support regex in dag_id for the DAG pause and resume CLI commands
airflow tasks test now works with deferrable operators

Contributors

Thanks to everyone who contributed to this release, including Amogh Desai, Andrey Anshin, Brent Bovenzi, Daniel Standish, Ephraim Anierobi, Hussein Awala, Jarek Potiuk, Jed Cunningham, Jens Scheffler, Tzu-ping Chung, Vincent Beck, Wei Lee, and over 120 others!

I’d especially like to thank our release manager, Ephraim, for getting this release out the door.

I hope you enjoy using Apache Airflow 2.9.0!

]]>

Vulnerability in long deprecated OpenID authentication method in Flask AppBuilder

2024-02-26T00:00:00Z

Vulnerability in long deprecated OpenID authentication method in Flask AppBuilder

Recently Islam Rzayev made us aware of a vulnerability in the long deprecated OpenID authentication method in Flask AppBuilder. This vulnerability allowed a malicious user to take over the identity of any Airflow UI user by forging a specially crafted request and implementing their own OpenID service. While this is an old, deprecated and almost not used authentication method, we still took the issue seriously.

This issue ONLY affects users who have AUTH_OID set in their webserver_config.py file as AUTH_TYPE. This is a very old and deprecated authentication method that is unlikely to be used by anyone.

We would like to advise even the small number of our users that still use this authentication method to take an immediate action and either upgrade to Apache Airflow 2.8.2 or switch to another authentication method (or apply a workaround we provide if they cannot do either of the above immediately).

Important to stress, because many of the users might get confused by the name, OpenID is NOT the same as OpenID Connect. Those are completely different protocols and while OpenID Connect (also known as OIDC) is a modern, widely used protocol, OpenID is a legacy protocol that has been deprecated more than 10 years ago and since then has been abandoned by almost everyone in the community, including all services in Flask AppBuilder example services that supported it, so it is highly unlikely someone is still using it.

Due to this highly unlikely configuration the Flask AppBuilder CVE is just “Moderate” not “Critical”. It affects a very small (if any) number of users and it’s not likely to be a target for an attack. However, we still advise our users who still use AUTH_OID to apply remediation.

This vulnerability is fixed in Flask Appbuilder 4.3.11 and Apache Airflow 2.8.2 uses that version of Flask Application Builder. We advise users who still use this authentication method to either switch to another authentication method or upgrade to Apache Airflow 2.8.2. If they cannot do either of these solutions quickly, they should apply the workaround provided below.

Impact

When Flask-AppBuilder is set to AUTH_TYPE set to AUTH_OID, it allows an attacker to forge an HTTP request that could deceive the backend into using any requested OpenID service. This vulnerability could grant an attacker unauthorised privilege access if a custom OpenID service is deployed by the attacker and accessible by the backend.

This vulnerability is only exploitable when the application is using OpenID (not OpenID Connect also known as OIDC). Currently, this protocol is regarded as legacy, with significantly reduced usage.

Possible remediation

Change your authentication method - if you are using AUTH_OID, there are almost no commercial services supporting it, it was deprecated 10 years ago and abandoned by nearly everyone in the community 4 years ago. Your best choice is to choose a different authentication method.
Upgrade to Apache Airflow 2.8.2 (which also upgrades to Flask-AppBuilder 4.3.11 that contains a fix)
If upgrade is not possible, apply the workaround below

Workarounds

If upgrade or changing authentication method is not possible add the following to your webserver_config.py file to fix the issue:

import os

from flask import flash, redirect
from flask_appbuilder.security.forms import LoginForm_oid
from flask_appbuilder.security.views import AuthOIDView
from flask_appbuilder.views import expose

from airflow.www.security import AirflowSecurityManager

basedir = os.path.abspath(os.path.dirname(__file__))

class FixedOIDView(AuthOIDView):
    @expose("/login/", methods=["GET", "POST"])
    def login(self, flag=True):
        form = LoginForm_oid()
        if form.validate_on_submit():
            identity_url = None
            for provider in self.appbuilder.sm.openid_providers:
                if provider.get("url") == form.openid.data:
                    identity_url = form.openid.data
            if identity_url is None:
                flash(self.invalid_login_message, "warning")
                return redirect(self.appbuilder.get_url_for_login)
        return super().login(flag=flag)

class FixedAirflowSecurityManager(AirflowSecurityManager):
    authoidview = FixedOIDView

SECURITY_MANAGER_CLASS = FixedAirflowSecurityManager

Credits

Big thanks to Islam Rzayev for finding out and reporting the issue responsibly and to Daniel Gaspar for very close cooperation on this one and coordinating the disclosure together with the Apache Superset where Flask AppBuilder is also used.

]]>

Apache Airflow 2.8.0 is here

2023-12-15T00:00:00Z

I am thrilled to announce the release of Apache Airflow 2.8.0, featuring a host of significant enhancements and new features that will greatly benefit our community.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.8.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.8.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.8.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.8.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.8.0

Airflow Object Storage (AIP-58)

This feature is experimental and subject to change.

Airflow now offers a generic abstraction layer over various object stores like S3, GCS, and Azure Blob Storage, enabling the use of different storage systems in DAGs without code modification.

In addition, it allows you to use most of the standard Python modules, like shutil, that can work with file-like objects.

Here is an example of how to use the new feature to open a file:

from airflow.io.path import ObjectStoragePath

base = ObjectStoragePath("s3://my-bucket/", conn_id="aws_default")  # conn_id is optional

@task
def read_file(path: ObjectStoragePath) -> str:
    with path.open() as f:
        return f.read()

The above example is just the tip of the iceberg. The new feature allows you to configure an alternative backend for a scheme or protocol.

Here is an example of how to configure a custom backend for the dbfs scheme:

from airflow.io.path import ObjectStoragePath
from airflow.io.store import attach

from fsspec.implementations.dbfs import DBFSFileSystem

attach(protocol="dbfs", fs=DBFSFileSystem(instance="myinstance", token="mytoken"))
base = ObjectStoragePath("dbfs://my-location/")

For more information: Airflow Object Storage

The support for a specific object storage system depends on the installed providers, with out-of-the-box support for the file scheme.

Ship logs from other components to Task logs

This feature seamlessly integrates task-related messages from various Airflow components, including the Scheduler and Executors, into the task logs. This integration allows users to easily track error messages and other relevant information within a single log view.

Presently, suppose a task is terminated by the scheduler before initiation, times out due to prolonged queuing, or transitions into a zombie state. In that case, it is not recorded in the task log. With this enhancement, in such situations, it becomes feasible to dispatch an error message to the task log for convenient visibility on the UI.

This feature can be toggled, for more information see “enable_task_context_logger” in the logging configuration documentation.

Listener hooks for Datasets

Please note that listeners are still experimental and subject to change.

This feature enables users to subscribe to Dataset creation and update events using listener hooks. It’s particularly useful to trigger external processes based on a Dataset being created or updated.

Using Extra Index URLs with PythonVirtualEnvOperator and Caching

This feature allows you to specify extra index URLs to PythonVirtualEnvOperator (+corresponding decorator) to be able to install virtualenvs with (private) additional Python package repositories.

You can also reuse the virtualenvs by caching them in a specified directory and reusing them in subsequent runs. This can be achieved by setting the venv_cache_path to a file system folder on your worker

For more information: PythonVirtualenvOperator

Web UI improvements

There are a number of improvements to the Web UI in this release, including:

Add multiselect to run state in grid view:

The grid view now supports multiselect for run states. This allows you to select multiple states to filter the dag runs shown in the grid view.

Improved visibility of task status in the Graph view

You can now see the status of a task in the graph view through the border color of the task. This makes it easier to see the status of a task at a glance.

Raw HTML code in DAG docs and DAG params descriptions is disabled by default

As part of our continuous quest to make airflow more secure by default, we have disabled raw HTML code in DAG docs and DAG params descriptions by default. We care for your security, and “secure by default” is one of the things we follow strongly.

Other notable UI improvements include:

Simplify DAG trigger UI
Hide logical date and run id in trigger UI form
Move external logs links to top of react logs page

Additional new features and improvements can be found in the Airflow 2.8.0 release notes.

Contributors

Thanks to everyone who contributed to this release, including Amogh Desai, Andrey Anshin, Bolke de Bruin, Daniel Dyląg, Daniel Standish, Ephraim Anierobi, Hussein Awala, Jarek Potiuk, Jed Cunningham, Jens Scheffler, mhenc, Miroslav Šedivý, Pankaj Koti, Tzu-ping Chung, Vincent, and everyone else who committed, all 110 of you! You are what makes Airflow the successful project that it is!

I hope you enjoy using Apache Airflow 2.8.0!

]]>

Airflow Survey 2023

2023-09-21T00:00:00Z

View Raw Data

]]>

Apache Airflow 2.7.0 is here

2023-08-18T00:00:00Z

I’m happy to announce that Apache Airflow 2.7.0 has been released! Some notable features have been added that we are excited for the community to use.

Apache Airflow 2.7.0 contains over 500 commits, which include 40 new features, 49 improvements, 53 bug fixes, and 15 documentation changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.7.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.7.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.7.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.7.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.7.0

Airflow 2.7.0 is a release that focuses on security. The Airflow security team, working together with security researchers, identified a number of areas that required strengthening of security. This resulted in, among others things, an improved description of the Airflow security model, a better explanation of our security policy and the disabling of certain, potentially dangerous, features by default - like, for example, connection testing (#32052).

Airflow 2.7.0 is also the first release that drops support for end-of-life Python 3.7. This allows Airflow users and maintainers to make use of features and improvements in Python 3.8, and unlocks newer versions of our dependencies.

Setup and Teardown (AIP-52)

Airflow now has first class support for the concept of setup and teardown tasks. These tasks have special behavior in that:

Teardown tasks will still run, no matter what state the upstream tasks end up in
Teardown tasks failing won’t, by default, cause the DAG run to fail
Automatically clear setup/teardown tasks when clearing a dependent task

You can read more about setup and teardown in the Introducing Setup and Teardown tasks blog post, or in the setup and teardown docs.

Cluster Activity UI

There is a new top level page in Airflow, the Cluster Activity page. This gives an overview of the cluster, including component health, dag and task state counts, and more!

Graph and gantt views moved into the Grid view UI

The graph and gantt views have been rewritten and moved into the now familiar grid view. This makes it easier to jump between task details, logs, graph, and gantt views without losing your place in a complicated DAG.

Enable deferrable mode for all deferable tasks with 1 config setting

Airflow 2.7.0 comes with a new config option, default_deferrable, which allows admins to enable deferrable mode for all deferrable tasks without requiring any DAG modifications. Simply set it in your config and enjoy async tasks!

OpenLineage built-in integration

OpenLineage provides a spec standardizing operational lineage collection and distribution across the data ecosystem that projects – open source or proprietary – implement.

With 2.7.0, OpenLineage changes from a plugin implementation maintained in the OpenLineage project to a built-in feature of Airflow. As a plugin, OpenLineage depended on Airflow and operators’ internals, making it brittle. Built-in OpenLineage support in Airflow makes publishing operational lineage through the OpenLineage ecosystem easier and more reliable. It has been implemented by moving the openlineage-airflow package from the OpenLineage project to an apache-airflow-providers-openlineage provider in the base Airflow Docker image, where it can be easily enabled by configuration. Also, lineage extraction logic that was included in Extractors in that package has been moved into each corresponding provider package along with unit tests, eliminating the need for Extractors in most cases. For this purpose, a new optional API for Operators (get_openlineage_facets_on_{start(), complete(ti), failure(ti)}, documented here) can be used. Having the extraction logic in each provider ensures the stability of the lineage contract in each operator and makes adding lineage coverage to custom operators easier.

Some executors moved into providers

Some of the executors that were shipped in core Airflow have moved into their respective providers for Airflow 2.7.0. The great benefit of this is to allow faster bug-fix releases as providers are released independently of core. The following providers have been moved and require certain minimum providers versions:

In order to use Celery executors, install the celery provider version 3.3.0+
In order to use the Kubernetes executor, install the kubernetes provider version 7.4.0+
In order to use the Dask executor, install any version of the daskexecutor provider

If you use the official docker images, all of these providers come preinstalled.

Additional new features

Here are just a few interesting new features, since there are too many to list in full:

Pools can now consider tasks in the deferred state as running (#32709)
chain_linear, like chain but allowing sequential tasks (#31927)
Grid view now supports keyboard shortcuts! (#30950)
Mark task groups as success or failed (#30478)
Fail_stop, allowing all remaining and running tasks to be failed on the first failure in a DAG (#29406)

Contributors

Thanks to everyone who contributed to this release, including Akash Sharma, Amogh Desai, Brent Bovenzi, D. Ferruzzi, Daniel Standish, Ephraim Anierobi, Hussein Awala, Jarek Potiuk, Jed Cunningham, Karthikeyan Singaravelan, Maciej Obuchowski, Niko Oliveira, Pankaj Koti, Pankaj Singh, Pierre Jeambrun, Tzu-ping Chung, Utkarsh Sharma, Vincent Beck, and over 74 others!

I’d especially like to thank our release manager, Ephraim, for getting this release out the door.

I hope you enjoy using Apache Airflow 2.7.0!

]]>

Introducing Setup and Teardown tasks

2023-08-18T00:00:00Z

In data pipelines, commonly we need to create infrastructure resources, like a cluster or GPU nodes in an existing cluster, before doing the actual “work” and delete them after the work is done. Airflow 2.7 adds “setup” and “teardown” tasks to better support this type of pipeline. This blog post aims to highlight the key features so you know what’s possible. For full documentation on how to use setup and teardown tasks, see the setup and teardown docs.

Why setup and teardown?

Before we dig into examples, let me state at high level what setup and teardown bring to the table.

More expressive dependencies

Before setup and teardown, upstream and downstream relationships could only mean one thing: “this comes before that”. With setup and teardown, in effect we can say “this requires that”. And what it means in practice is, if you clear your task, and it requires a setup, that setup will be cleared too. And if that setup has a teardown, that will run again as well.

Separating the work from the infra

Sometimes the part of the dag you care about is not, say, the cleanup task. For example, suppose you have a dag that loads some data and then deletes temp files. As long as the data loads, you want your dag to be marked successful. By default, this is how teardown tasks work; that is, they are ignored when determining dag run state.

Simple case

A simple example is one setup / teardown pair, and one normal or “work” task.

Setups and teardowns are indicated by the up and down arrows, respectively. From that we can see that .create_cluster is a setup task and delete_cluster is a teardown. The link between a setup and a teardown is always dotted to highlight the special relationship.

Some things to observe:

If create_cluster fails, neither run_query nor delete_cluster will run.
If create_cluster succeeds and run_query fails, then delete_cluster will still run.
If create_cluster is skipped, run_query and delete_cluster will be skipped
By default, if run_query succeeds, and delete_cluster fails, then the dag run will still be marked successful. (This behavior can be overridden).

Authoring with task groups

When we set something downstream of a task group, any teardowns in the task group are ignored. This reflects the assumption that in general, we probably don’t want to stop dag execution just because a teardown fails. So, let’s wrap the above dag in a task group and see what happens:

And here’s how we linked those groups in the code:

with TaskGroup("do_emr") as do_emr:
    create_cluster_task = create_cluster()
    run_query(create_cluster_task) >> delete_cluster(create_cluster_task)

with TaskGroup("load") as load:
    create_config_task = create_configuration()
    load_data(create_config_task) >> delete_configuration(create_config_task)

do_emr >> load

In this code, each group has a teardown, and we just arrow the first group to the second. As advertised, delete_cluster, a teardown task, is ignored. This has two important consequences: one, even if it fails, the load group will still run; and two, delete_cluster and create_configuration can run in parallel (generally speaking, we’d imagine you don’t want to wait for teardown operations to complete before continuing onto other tasks in the dag). Of course, you can override this behavior by adding an arrow between delete_cluster and create_configuration. Further, the success of this dag will depend only on whether the load_data task completes successfully.

Conclusion

There’s a lot of detail we’re omitting here about exactly how to write dags with setup and teardown tasks, and for that please head over to the setup and teardown docs. But hopefully this post gives you enough of an idea of what is possible with setup and teardown tasks that you can begin to see where they can improve your data pipelines in Airflow.

Curious to know what else is new in Airflow 2.7? Head over to the main Airflow 2.7 blog post to find out!

Acknowledgements

Setup and Teardown was the product of AIP-52. Thanks to everyone who contributed to it, including those that read and voted on the AIP. Special thanks to Ash Berlin-Taylor, Brent Bovenzi, Daniel Standish, Ephraim Anierobi, Jed Cunningham, Rahul Vats, and Vikram Koka.

]]>

what's new in Apache Airflow 2.6.0

2023-04-30T00:00:00Z

I am excited to announce that Apache Airflow 2.6.0 has been released, bringing many minor features and improvements to the community.

Apache Airflow 2.6.0 contains over 500 commits, which include 42 new features, 58 improvements, 38 bug fixes, and 17 documentation changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.6.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.6.0/
🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.6.0/release_notes.html
🐳 Docker Image: “docker pull apache/airflow:2.6.0”
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.6.0

As the changelog is quite large, the following are some notable new features that shipped in this release.

Trigger logs can now be viewed in webserver

Trigger logs have now been added to task logs. They appear right alongside the rest of the logs from your task.

Adding this feature required changes across the entire Airflow logging stack, so be sure to update your providers if you are using remote logging.

Grid view improvements

The grid view has received a number of minor improvements in this release.

Most notably, there is now a graph tab in the grid view. This offers a more integrated graph representation of the DAG, where choosing a task in either the grid or graph will highlight the same task in both views.

You can also filter upstream and downstream from a single task. For example, in the screenshot above, describe_integrity is the selected task. If you choose to filter downstream, this is the result:

Trigger UI based on DAG level params

A user-friendly form is now shown to users triggering runs for DAGs with DAG level params.

See the Params docs for more details.

Consolidation of handling stuck queued tasks

Airflow now has a single configuration, [scheduler] task_queued_timeout, to handle tasks that get stuck in queued for too long. With a simpler implementation than the outgoing code handling these tasks, tasks stuck in queued will no longer slip through the cracks and stay stuck.

For more details, see the Unsticking Airflow: Stuck Queued Tasks are No More in 2.6.0 Medium post.

Cluster Policy hooks can come from plugins

Cluster policy hooks (e.g. dag_policy), can now come from Airflow plugins in addition to Airflow local settings. By allowing multiple hooks to be defined, it makes it easier for more than one team to run hooks in a single Airflow instance.

See the cluster policy docs for more details.

Notification support added

The notifications framework allows you to send messages to external systems when a task instance/DAG run changes state. For example, you can easily post a message to Slack

with DAG(
    “slack_notifier_example”,
    start_date=datetime(2023, 1, 1),
    on_success_callback=[
        send_slack_notification(
            text="The DAG {{ dag.dag_id }} succeeded",
            channel="#general",
            username="Airflow",
        )
    ],
):

As of today, Slack is the only system supported out of the box. However, watch this space as more integrations will be added soon.

You can also create notifiers for your own use, refer to the notifier how-to docs for more details.

Thanks to the contributors

Thanks to everyone who contributed to this release, including Andrey Anshin, Ash Berlin-Taylor, Brent Bovenzi, Daniel Standish, Ephraim Anierobi, Hussein Awala, Jarek Potiuk, Jed Cunningham, Josh Fell, Michael Petro, Niko Oliveira, Pierre Jeambrun, Tzu-ping Chung, Victor Chiapaikeo, and over 120 others!

I’d especially like to thank our release manager, Ephraim, for getting this release out the door.

I hope you enjoy using Apache Airflow 2.6.0!

]]>

Apache Airflow 2.5.0: Tick-Tock

2022-12-02T00:00:00Z

Apache Airflow 2.5 has just been released, barely two and a half months after 2.4!

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.5.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.5.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.5.0/release_notes.html
🐳 Docker Image: docker pull apache/airflow:2.5.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.5.0

This quicker release cadence is a departure from our previous habit of releasing every five-to-seven months and was a deliberate effort to listen to you, our users, and get the changes and improvements into your workflows earlier.

Usability improvements to the Datasets UI

When we released Dataset aware scheduling in September we knew that the tools we gave to manage the Datasets were very much a Minimum Viable Product, and in the last two months the committers and contributors have been hard at work at making the UI much more usable when it comes to Datasets.

But we aren’t done yet - keep an eye out for more improvements coming over the next couple of releases too.

Greatly improved `airflow dags test` command

This airflow subcommand has been rethought and re-optimized to make it much easier to test your DAGs locally - the major changes are:

a. Task logs are visible right there in the console, instead of hidden away inside the task log files b. It is about an order of magnitude quicker to run the tasks than before (i.e. it gets to running the task code so much quicker) c. Everything runs in one process, so you can put a breakpoint in your IDE, and configure it to run airflow dags test then debug code!

Auto tailing task logs in the Grid view

Hopefully the headline says enough. It’s lovely, go check it out.

More improvements to Dynamic-Task mapping

In a similar vein to the improvements to the Dataset (UI), we have continued to iterate on and improve the feature we first added in Airflow 2.3, Dynamic Task Mapping, and 2.5 includes dozens of improvements.

Thanks to the contributors

Andrey Anshin, Ash Berlin-Taylor, blag, Bolke de Bruin, Brent Bovenzi, Chenglong Yan, Daniel Standish, Dov Benyomin Sohacheski, Elad Kalif, Ephraim Anierobi, Jarek Potiuk, Jed Cunningham, Jorrick Sleijster, Michael Petro, Niko, Pierre Jeambrun, Tzu-ping Chung and many more, over 75 of you. Thank you!

And a special thank you to Ephraim who tirelessly worked behind the scenes as release manager!

A much shorter change log than 2.4, but I think you’ll agree, some great changes.

]]>

Apache Airflow 2.4.0: That Data Aware Release

2022-09-19T00:00:00Z

Apache Airflow 2.4.0 contains over 650 “user-facing” commits (excluding commits to providers or chart) and over 870 total. That includes 46 new features, 39 improvements, 52 bug fixes, and several documentation changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.4.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.4.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.4.0/release_notes.html
🐳 Docker Image: docker pull apache/airflow:2.4.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.4.0

Data-aware scheduling (AIP-48)

This one is big. Airflow now has the ability to schedule DAGs based on other tasks updating datasets.

What does this mean, exactly? This is a great new feature that lets DAG authors create smaller, more self-contained DAGs, which chain together into a larger data-based workflow. If you are currently using ExternalTaskSensor or TriggerDagRunOperator you should take a look at datasets – in most cases you can replace them with something that will speed up the scheduling!

But enough talking, lets have a short example. First lets write a simple DAG with a task called my_task that produces a dataset called my-dataset:

from airflow import Dataset


dataset = Dataset(uri='my-dataset')

with DAG(dag_id='producer', ...)
    @task(outlets=[dataset])
    def my_task():
        ...

Datasets are defined by a URI. Now, we can create a second DAG (consumer) that gets scheduled whenever this dataset changes:


from airflow import Dataset


dataset = Dataset(uri='my-dataset')

with DAG(dag_id='dataset-consumer', schedule=[dataset]):
    ...

With these two DAGs, the instant my_task finishes, Airflow will create the DAG run for the dataset-consumer workflow.

We know that what exists right now won’t fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation.

Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in the future - and it’s part of our goal to have smaller releases to get new features in your hands sooner!

For more information on datasets, see the documentation on Data-aware scheduling. That includes details on how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what a dataset is (hint: don’t include “date partitions” in a dataset, it’s higher level than that).

Easier management of conflicting python dependencies using the new ExternalPythonOperator

As much as we wish all python libraries could be used happily together that sadly isn’t the world we live in, and sometimes there are conflicts when trying to install multiple python libraries in an Airflow install – right now we hear this a lot with dbt-core.

To make this easier we have introduced @task.external_python (and the matching ExternalPythonOperator) that lets you run a python function as an Airflow task in a pre-configured virtual env, or even a whole different python version. For example:

@task.external_python(python='/opt/venvs/task_deps/bin/python')
def my_task(data_interval_start, data_interval_env)
    print(f'Looking at data between {data_interval_start} and {data_interval_end}')
    ...

There are a few subtleties as to what you need installed in the virtual env depending on which context variables you access, so be sure to read the how-to on using the ExternalPythonOperator

More improvements to Dynamic Task Mapping (AIP-42)

You asked, we listened. Dynamic task mapping now includes support for:

expand_kwargs: To assign multiple parameters to a non-TaskFlow operator.
zip: To combine multiple things without cross-product.
map: To transform the parameters just before the task is run.

For more information on dynamic task mapping, see the new sections of the doc on Transforming Mapped Data, Combining upstream data (aka “zipping”), and Assigning multiple parameters to a non-TaskFlow operator.

Auto-register DAGs used in a context manager (no more `as dag:` needed)

This one is a small quality of life improvement, and I don’t want to admit how many times I forgot the as dag:, or worse, had as dag: repeated.


with DAG(dag_id="example") as dag:
  ...


@dag
def dag_maker():
    ...


dag2 = dag_maker()

can become


with DAG(dag_id="example"):
    ...


@dag
def my_dag():
    ...


my_dag()

If you want to disable the behaviour for any reason, set auto_register=False on the DAG:

# This dag will not be picked up by Airflow as it's not assigned to a variable
with DAG(dag_id="example", auto_register=False):
    ...

Additional improvements

With over 650 commits the full list of features, fixes and changes is too big to go in to here (check out the release notes for a full list), but some noteworthy or interesting small features include:

Auto-refresh on the home page
Add @task.short_circuit TaskFlow decorator
Add roles delete command to cli
Add support for TaskGroup in ExternalTaskSensor
Add @task.kubernetes taskflow decorator
Add experimental parsing_context to enable optimization of Dynamic DAG handling in workers
Consolidate to one schedule param
Allow showing non-sensitive config values in Admin -> Configuration (rather than all or nothing)
Operator name separate from class (no more _PythonDecoratedOperator when using TaskFlow)

Contributors

Thanks to everyone who contributed to this release, including Andrey Anshin, Ash Berlin-Taylor, Bartłomiej Hirsz, Brent Bovenzi, Chenglong Yan, D. Ferruzzi, Daniel Standish, Drew Hubl, Elad Kalif, Ephraim Anierobi, Jarek Potiuk, Jed Cunningham, Josh Fell, Mark Norman Francis, Niko, Tzu-ping Chung, Vincent, Wojciech Januszek, chethanuk-plutoflume, pierrejeambrun, and everyone else who committed, all 152 of you! You are what makes Airflow the successful project that it is!

]]>

Airflow Survey 2022

2022-06-17T00:00:00Z

Airflow User Survey 2022

This year’s survey has come and gone, and with it we’ve got a new batch of data for everyone! We collected 210 responses over two weeks. We continue to see growth in both contributions and downloads over the last two years, and expect that trend will continue through 2022.

The raw response data will be made available here soon, in the meantime, feel free to email john.thomas@astronomer.io for a copy.

TL;DR

Overview of the user

Like previous years, more than half of the Airflow users are Data Engineers (54%). Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%) are also active Airflow users! There was a slight increase in the representation of Solutions Architect roles compared to results from 2020 and 2019 .
Airflow is used and popular in bigger companies, 64% of Airflow users work for companies with 200+ employees which is an 11 percent increase compared to 2020.
62% of the survey participants have more than 6 Airflow users in their company.
More Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in 2020 and 2019. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow ( 85.7% in 2019 and 92% in 2020 ), only 1% of users are not likely to recommend (3.6% in 2019 and 3.5% in 2020).
Airflow documentation is a critical source of information, with more than 90% (15% increase compared to results from 2020) of survey participants using the documentation. Airflow documentation is also one of the top areas to improve! What’s interesting, also Stack Overflow usage is critical, with about 60% users declaring to use it as a source of information (24% increase compared to results from 2020).

Deployments

85% of the Airflow users have between 1 and 7 active Airflow instances. 62.5% of the Airflow users have between 11 and 250 DAGs in their largest Airflow instance. 75% of the surveyed Airflow users have between 1 and 100 tasks per DAG.
Close to 85% of users use one of the Airflow 2 versions, 9.2% users still use 1.10.15, while the remaining 6.3% are still using older Airflow 1 versions. The good news is that the majority of users on Airflow 1 are planning migration to Airflow 2 quite soon, with resources and capacity being the main blockers.
In comparison to results from 2020, more users were interested in monitoring in general and specifically in using tools such as external monitoring services (40.7%, up from 29.6%) and information from metabase (35.7%, up from 25.1%).
Celery (52.7%) and Kubernetes (39.4%) are the most common executors used.

Usage

81.3% of Airflow users who responded to the survey don’t have any customisation of Airflow.
Xcom (69.8%) is the most popular method to pass inputs and outputs between tasks, however Saving and Retrieving Inputs and Outputs from Storage still plays an important role (49%).
Lineage itself is a quite new topic for Airflow users, most of them don’t use lineage solutions but might be interested if supported by Airflow (47.5%), are not familiar with data lineage (29%) or that data lineage is not their concern (13%).
The Airflow web UI is used heavily for Monitoring Runs (95.9%), Accessing Task Logs (89.8%), Manually triggering DAGs (85.2%), Clearing Tasks (82.7%) and Marking Tasks as successful (60.7%). The top 3 views used are: List of DAGs, Task Logs and DAG Runs, which is very similar to results from 2020 and 2019.

Community and contribution

Most Airflow users (57.1%) are aware they could contribute but do not, and an additional 21.7% contribute very rarely. 14.8% of users were not aware they could contribute. There is much more to be done to engage our community to be more active contributors and raise the current 6.4% of users who actively contribute, especially considering that one important blocker for contribution is lack of knowledge on how to start (37.7%).

The future of Airflow

The top area for improvement is still the Airflow web UI (49.5%), closely followed by more telemetry for logging, monitoring and alerting purposes (48%). However all those efforts should go in line with improved documentation (36.6.%) and resources about using the Airflow, especially when we take into account the need of onboarding new users (36.6%).
DAG Versioning(66.2%) is a winner for new features in Airflow, and it’s not a surprise as this feature may positively impact daily work of Airflow users. It is followed by three other ideas: Dependency management and Data-driven scheduling (42.6%), More dynamic task structure (42.1%) and Multi-Tenancy (37.9%).

Overview of the user

What best describes your current occupation? (single choice)


	No.	%
Data Engineer	114	54%
Solutions Architect	27	13%
Developer	25	12%
DevOps	12	6%
Data Scientist	8	4%
Support Engineer	5	2%
Data Analyst	3	1%
Business Analyst	2	1%
Other	14	7%

According to the survey, more than half of Airflow users are Data Engineers (54%). Roles of the remaining Airflow users might be broken down into Solutions Architects (13%), Developers (12%), DevOps (6%) and Data Scientists (4%). The 2022 results are similar to those from 2019 and 2020 with a slight increase in the representation of Solutions Architect roles.

How often do you interact with Airflow? (single choice)


	No.	%
Every day	154	73%
At least once per week	36	17%
At least once per month	11	5%
Less than once per month	9	4%

Users who took the survey are actively using Airflow as part of their current role. 73% of Airflow users who responded use it on a daily basis, 17% weekly.

How many people work at your company? (single choice)


	No.	%
201-5000	85	41%
5000+	49	23%
51-200	46	22%
11-50	20	10%
1-10	9	4%

Airflow is a framework that is used and popular in bigger companies, 64% of Airflow users who responded (compared to 52.7% in 2020) work for companies bigger than 200 employees (41% in companies size 201-5000 and 23% in companies size 5000+).

How many people at your company use Airflow? (single choice)


	No.	%
6-20	80	38%
1-5	61	29%
51-200	49	24%
200+	18	9%

Airflow is generally used by small to medium-sized teams. 62% of the survey participants have more than 6 Airflow users in their company (38% have between 6 and 200 users, 24% between 51-200 users).


	% 2019	% 2020	% 2022
Very Likely	45.4%	61.6%	65.9%
Likely	40.3%	30.4%	26.9%
Neutral	10.7%	5.4%	6.3%
Unlikely	2.6%	1.5%	0.5%
Very Unlikely	1%	1%	0.5%

According to the survey, more Airflow users (65.9%) are willing to recommend Apache Airflow compared to the survey results in 2020 and 2019. There is a general positive trend in a willingness to recommend Airflow, 93% of surveyed Airflow users are willing to recommend Airflow (92% in 2020 and 85.7% in 2019), only 1% of users are not likely to recommend (3.6% in 2019 and 3.5% in 2020 ).

What is your source of information about Airflow? (multiple choice)


	No.	%
Documentation	189	90.4%
Airflow website (Blog, etc.)	142	67.9%
Stack Overflow	126	60.3%
Github Issues	104	49.8%
Slack	96	45.9%
Airflow Summit Videos	88	42.1%
GitHub Discussions	76	36.4%
Airflow Community Webinars	41	19.6%
Astronomer Registry	51	24.4%
Airflow Mailing List	34	16.3%

Airflow documentation is a critical source of information, with more than 90% of survey participants using the documentation. It is of increasing importance compared to results from 2020 where documentation was at about 75% level. Moreover, more than 60% of users are getting information from the Airflow website (67.9% ) and Stack Overflow (60.3%) which is also a big increase compared to 36% level in 2020. What’s interesting is that Slack usage decreased from 63.05% in 2020 to 45.9% in 2022.

Deployments

How many active DAGs do you have in your largest Airflow instance? (single choice)


	No.	%
51-250	66	31.7%
11-50	64	30.8%
5-10	25	12.0%
251-500	20	9.6%
<5	14	6.7%
1000+	10	4.8%
501-1000	9	4.3%

62.5% of the Airflow users surveyed have between 11 and 250 DAGs in their largest Airflow instance.

How many active Airflow instances do you have? (single choice)


	No.	%
1	52	25.2%
2	46	22.3%
4-7	40	19.4%
3	37	18.0%
20+	19	9.2%
8-10	7	3.4%
11-20	5	2.4%

85% of the Airflow users surveyed have between 1 and 7 active Airflow instances, and nearly 50% have only 1 or 2.

What is the maximum number of tasks that you have used in a single DAG?(single choice)


	No.	%
11-25	51	24.5%
26-50	41	19.7%
51-100	35	16.8%
<10	29	13.9%
101-250	23	11.1%
501-1000	9	4.3%
1000-2500	8	3.8%
251-500	8	3.8%
2500-5000	4	1.9%

75% of the surveyed Airflow users have between 1 and 100 tasks per DAG.

How many schedulers do you have in your largest Airflow instance? (single choice)


	No.	%
1	113	55.1%
2	61	29.8%
3	18	8.8%
4+	13	6.3%

More than half of Airflow users who responded to the survey have 1 scheduler in their largest Airflow instance, however it’s important to notice that the second half of Airflow users decided to have 2 schedulers and more.

What executor type do you use? (multiple choice)


	No.	%
Celery	107	52.7 %
Kubernetes	80	39.4%
Local	49	24.1%
Sequential	21	10.3%
CeleryKubernetes	14	6.9%

Celery (52.7%) and Kubernetes (39.4%) are the most common executors used. CeleryKubernetes (6.9%) executor also started to be noticed and used by Airflow users.

If you use the Celery executor, how many workers do you have in your largest Airflow instance? (single choice)


	No.	%
2-5	64	44.8%
10+	28	19.6%
1	26	18.2%
6-10	25	17.5%

Amongst Celery executor users who responded to the survey, close to half the number (44.8%) have between 2 and 5 workers in their largest Airflow instance. It’s notable that nearly a fifth (19.6%) have more than 10 workers.

Which version of Airflow do you currently use? (single choice)


	No.	%
1.10.14 or older	13	6.3%
1.10.15	19	9.2%
2.0.x	23	11.1%
2.1.x	24	11.6%
2.2.x	79	38.2%
2.3.x	49	23.7%

It’s good to see that close to 85% of users who responded to the survey use one of the Airflow 2 versions, 9.2% users still use 1.10.15, while the remaining 6.3% are still using older Airflow 1.10 versions.

The good news is that the majority of users on Airflow 1 are planning migration to Airflow 2 quite soon, as for now they have capacity constraints to undertake such a significant effort in their opinion. However, it can also be noticed in the survey’s comments that some users are generally skeptical towards migration to Airflow 2, they have negative opinions about the new scheduler or compatibility with the helm chart.

As to plans about migration to the newest version of Airflow 2, users who responded to the survey are committed and waiting especially for the features related to dynamic DAGs. However, some users also reported that they are waiting to solve some dependencies they have or they prefer to wait a little bit more for the community to test the new version before they decide to move on.

What metrics do you use to monitor Airflow? (multiple choice)


	No.	%
External monitoring service	81	40.7%
Information from metadatabase	71	35.7%
Statsd	54	27.1%
I do not use monitoring	47	23.6%
Other	14	7%

In comparison to results from 2020, more users are monitoring airflow in some way. External monitoring services (40.7%) and information from metabase (35.7%) started to play a more important role in Airflow monitoring.

How do you deploy Airflow? (multiple choice)


	No.	%
On virtual machines (for example using AWS EC2)	63	30.6 %
Using a managed service like Astronomer, Google Composer or AWS MWAA	54	26.2 %
On Kubernetes (using Apache Airflow’s helm chart)	46	22.3%
On premises	43	20.9%
On Kubernetes (using custom deployments)	39	18.9%
On Kubernetes (using another helm chart)	21	10.2%
Other	13	6.5%

More than half of Airflow users who responded (51.4%) deploy Airflow on Kubernetes. This is about 20 percent more than in 2020. The remaining top deployment methods are on virtual machines (30.6%) and via managed services (26.2%).

How do you distribute your DAGs from your developer environment to the cloud? (single choice)


	No.	%
Using a synchronizing process (Git sync, GCS fuse, etc)	100	49%
Bake them into the docker image	51	25%
Shared files system	30	14.7%
Other	16	7.9%
I don’t know	7	3.4%

According to the survey responses, the most popular way of distributing DAGs is a synchronizing process, about half of Airflow users (49%) use this process to distribute DAGs from developer environments to the cloud.

Usage

Do you have any customisation of Airflow? (single choice)


	No.	%
No, we use vanilla airflow	165	81.3%
Yes, we have a separate fork	13	6.4%
Yes, we use a 3rd-party fork	12	5.9%
Yes, we’ve backpropagated bug fixes to an older version	13	6.4%

More Airflow users (81.3%) don’t have any customisation of Airflow (compared to 75.9% in 2020). Those Airflow users who have customisations (18.7%) decided to introduce them mainly to separate development and production workflows, to backport bug fixes, due to security fixes or to run a backfill command on Kubernetes pod.

Which Metadata Database do you use? (single choice)


	No.	%I
PostgreSQL 13	86	43.9%
PostgreSQL 12	74	37.8%
MySQL 8	22	11.2%
MySQL 5	9	4.6%
MariaDB	4	2.0%
MsSQL	1	0.5%

According to the survey responses, the most popular metadata databases are PostgreSQL 13 (43.9%) and PostgreSQL 12 (37.8%). This represents a sharp increase from 2020, up from 68.9% to 81.7% total on PostgreSQL, with a corresponding decrease in MySQL, down from 23% to 15%. This is an interesting result taking into account community discussion about not adding support for more database backend or even deciding on single database support.

What’s the primary method by which you integrate with providers and external services in your Airflow DAGs? (single choice)


	No.	%
Using existing dedicated operators / hooks	70	34.5%
Using Bash/Python operators	58	28.6%
Using custom operators / hooks	50	24.6%
Using KubernetesPodOperator	25	12.3%

According to the survey responses, the following ways of using Airflow to connect to external services are the most popular: Using existing dedicated operators / hooks (34.5%), Using Bash/Python operators (28.6%), Using custom operators / hooks (24.6%). Using KubernetesPodOperator (12.3%) is less popular regarding the survey responses. The integration with providers and external services methods ranking is similar to the one from 2020.

What providers do you use in your Airflow DAGs? (multiple choice)


	No.	%
Amazon Web Services	112	55.4%
Google Cloud Platform / Google APIs	79	39.1%
Internal company systems	75	37.1%
Hadoop / Spark / Flink / Other Apache software	57	28.2%
Microsoft Azure	17	8.4%
Other	21	10.5%
I do not use external services in my Airflow DAGs	14	6.9%

It’s not surprising that Amazon Web Services (55.4% vs 59.6% in 2020), on the next three positions Google Cloud Platform (39.1% vs 47.7% in 2020 ), Internal company systems (37.1% vs 55.6% in 2020), and other Apache products (28.2% vs 35.47% in 2020) are leading Airflow providers.

How frequently do you upgrade Airflow environments? (single choice)


	No.	%
every 12 months	46	22.9%
every 6 months	49	24.4%
once a quarter	47	23.4%
Whenever there is a newer version	59	29.4%

Different frequencies of Airflow environments upgrades are almost equally popular amongst Airflow users who responded to the survey.

Do you upgrade providers separately from the core? (single choice)


	No.	%
When I need it	83	42.8%
Never - always use the providers that come with Airflow	68	35.1%
I did not know I can upgrade providers separately	32	16.5%
I upgrade providers when they are released	11	5.7%

According to the survey responses, Airflow users most often upgrade providers when they need it (42.8%) or prefer to stay with providers that come with Airflow (35.1%). It’s surprising that 16.5% of Airflow users who responded to the survey were not aware that they can upgrade their providers separately from the core Airflow.

How do you pass inputs and outputs between tasks? (multiple choice)


	No.	%
Xcom	141	69.8%
Saving and retrieving from Storage	99	49%
TaskFlow	37	18.3%
Other	5	2.5%
We don’t	29	14.4%

According to the survey responses, Xcom (69.8%) is the most popular method to pass inputs and outputs between tasks, however Saving and Retrieving Inputs and Outputs from Storage still plays an important role (49%). It’s interesting that close to 15% of Airflow users who responded to the survey declare to not pass any outputs or inputs between tasks.

Do you use a data lineage backend? (multiple choice)


	No.	%
No, but I will use such feature if fully supported in Airflow	95	47.5%
I’m not familiar with data lineage	58	29%
No, data lineage isn’t a concern for my usage	26	13%
Yes, I send lineage to an Open Source lineage repository	15	7.5%
Yes, I send lineage to an Enterprise lineage repository	7	3.5%
Yes, I send lineage to a custom internal lineage repository	9	4.5%

When asked what lineage backend Airflow users use, the answers indicated that, while lineage itself is a quite new topic, there is interest in the feature as a whole. Most Airflow users responded that they don’t use lineage solutions currently but might be interested in the future if supported by Airflow (47.5%), are not familiar with data lineage (29%) or that data lineage is not their concern (13%).

Which interfaces of Airflow do you use as part of your current role? (multiple choice)


	No.	%
Original Airflow Graphical User Interface	189	94%
CLI	98	48.8%
API	80	39.8%
Custom (own created) Airflow Graphical User Interface	12	6%
GCP Composer	1	0.5%

It’s clear that usage of Airflow web UI is important as 94% of users who responded to the survey declare to use it as a part of their current role. Usage of CLI (48.8%) and API (39.8%) goes in pairs but are not so common compared to Airflow web UI usage.

(If GUI Marked) What do you use the GUI for? (multiple choice)


	No.	%
Monitoring Runs	188	95.9%
Accessing Task Logs	176	89.8%
Manually triggering DAGs	167	85.2%
Clearing Tasks	162	82.7%
Marking Tasks as successful	119	60.7%
Other	6	3%

Airflow web UI is used heavily for monitoring: Monitoring Runs (95.9%) and troubleshooting: Accessing Task Logs (89.8%), Manually triggering DAGs (85.2%), Clearing Tasks (82.7%) and Marking Tasks as successful (60.7%).

(if CLI Marked) What do you use the CLI For? (multiple choice)


	No.	%
Backfilling	63	56.8%
Manually triggering DAGs	52	46.8%
Clearing Tasks	26	23.4%
Monitoring Runs	25	22.5%
Accessing Task Logs	21	18.9%
Marking Tasks as successful	11	9.9%
Other	17	15.3%

Compared to Airflow web UI, Airflow CLI is used mainly for Backfilling (56.8%) and Manually triggering DAGs (46.8%).

In Airflow, which UI views are important for you? (multiple choice)


	No.	%
List of DAGs	178	89.4%
Task Logs	162	81.4%
DAG Runs	160	80.4%
Graph view	147	73.9%
Grid/Tree View	138	69.3%
Run Details	117	58.8%
DAG details	111	55.8%
Task Instances	102	51.3%
Task Duration	91	45.7%
Code	90	45.2%
Task Tries	60	30.2%
Gantt	48	21.4%
Landing Times	27	13.6%
Other	4	2%

UI views importance ranking shows that the majority Airflow users use Web UI mostly for monitoring and/or troubleshooting purposes, where the top 3 views are List of DAGs (89.4%), Task Logs (81.4%) and DAG Runs (80.4%). The results are very similar to those from 2020 and 2019.

Community and contribution

Are you participating in the Airflow community discussions? (single choice)


	No.	%
I see them from time to time	99	48.3%
I regularly follow what’s being discussed but don’t participate	53	25.9%
I didn’t know I could	41	20.0%
I actively participate in the discussions	12	5.9%
	No.	%
I know I can but I do not contribute	116	57.1%
Very rarely when it relates to what I need	44	21.7%
I do not know I could	30	14.8%
I regularly contribute by discussing, reviewing and submitting PR	13	6.4%

Results related to the Airflow contribution are very similar to those about participating in the Airflow community discussions. Most of the Airflow users (57.1%) who responded to the survey are aware but do not contribute or contribute very rarely (21.7%). 14.8% of users were not aware they could contribute. Once again, it’s a clear indicator that there is much more to be done to engage our community to be more active contributors and raise the current 6.4% of users who actively contribute.

If you do not contribute - why?


	No.	%
I have no time to contribute even if would like to	65	38.9%
I don’t know how to start	63	37.7%
I don’t have a need to contribute	19	11.4%
I didn’t know I could	12	7.2%
My employer has policy that makes it difficult to contribute	8	4.8%

According to the survey results, the most important blocker for the Airflow contribution is limited time (38.9%), but surprisingly interesting and important blocker is also lack of knowledge on how to start (37.7%), followed by lack of knowledge that it’s possible to contribute (7.2%).

The future of Airflow

In your opinion, what could be improved in Airflow? (multiple choice)


	No.	%
Web UI	100	49.5%
Logging, monitoring and alerting	97	48.0%
Examples, how-to, onboarding documentation	74	36.6%
Technical documentation	74	36.6%
Scheduler performance	56	27.7%
Reliability	52	25.7%
DAG authoring	48	23.8%
REST API	43	21.3%
Authentication and authorization	41	20.3%
External integration e.g. AWS, GCP, Apache products	41	20.3%
Better support for various deployments (Docker-compose/Nomad/Others)	39	19.3%
Everything works fine for me	19	9.4%
I don’t know	4	2.0%

The results are quite self-explanatory. According to the survey results, the top area for improvement is still the Airflow web UI (49.5%), closely followed by more telemetry for logging, monitoring and alerting purposes (48%). However all those efforts should go in line with improved documentation (36.6.%) and resources about using the Airflow, especially when we take into account the need of onboarding new users (36.6%).

Which features would you like to see in Airflow?


	No.	%
DAG Versioning	129	66.2%
Dependency management and Data-driven scheduling	83	42.6%
More dynamic task structure	82	42.1%
Multi-Tenancy	74	37.9%
Signal-based scheduling	67	34.4%
Better Security (Isolation)	65	33.3%
Submitting new DAGs externally via API	53	27.2%
Composable Operators	46	23.6%
Support for native cloud executors (AWS/GCP/Azure etc.)	44	22.6%
Better support for Machine Learning	38	19.5%
Remote CLI	36	18.5%
Support for hybrid executors	22	11.3%

According to the survey results, DAG Versioning is a winner for new features in Airflow, and it’s not a surprise as this feature may positively impact daily work of Airflow users. It is followed by three other ideas: Dependency management and Data-driven scheduling (42.6%), More dynamic task structure (42.1%) and Multi-Tenancy (37.9%). Another interesting point from that question is that only 11.3% think that support for hybrid executors is needed in Airflow.

Data

If you’re interested in taking a look at the raw data yourself, it’s available here: (Airflow User Survey 2022.csv)[/data/survey-responses/airflow-user-survey-responses-2022.csv.zip]

]]>

Airflow Summit 2022

2022-05-16T00:00:00Z

The biggest Airflow Event of the Year returns May 23–27! Airflow Summit 2022 will bring together the global community of Apache Airflow practitioners and data leaders.

What’s on the Agenda

During the free conference, you will hear about Apache Airflow best practices, trends in building data pipelines, data governance, Airflow and machine learning, and the future of Airflow. There will also be a series of presentations on non-code contributions driving the open-source project.

How to Attend

This year’s edition will include a variety of online sessions across different time zones. Additionally, you can take part in local in-person events organized worldwide for data communities to watch the event and network.

Interested?

🪶 Register for Airflow Summit 2022 today

🤝 Check out the in-person events planned for Airflow Summit 2022.

]]>

Apache Airflow 2.3.0 is here

2022-04-30T00:00:00Z

Apache Airflow 2.3.0 contains over 700 commits since 2.2.0 and includes 50 new features, 99 improvements, 85 bug fixes, and several doc changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.3.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.3.0/
🛠️ Release Notes: https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html
🐳 Docker Image: docker pull apache/airflow:2.3.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.3.0

As the changelog is quite large, the following are some notable new features that shipped in this release.

Dynamic Task Mapping(AIP-42)

There’s now first-class support for dynamic tasks in Airflow. What this means is that you can generate tasks dynamically at runtime. Much like using a for loop to create a list of tasks, here you can create the same tasks without having to know the exact number of tasks ahead of time.

You can have a task generate the list to iterate over, which is not possible with a for loop.

Here is an example:

@task
def make_list():
    # This can also be from an API call, checking a database, -- almost anything you like, as long as the
    # resulting list/dictionary can be stored in the current XCom backend.
    return [1, 2, {"a": "b"}, "str"]


@task
def consumer(arg):
    print(list(arg))


with DAG(dag_id="dynamic-map", start_date=datetime(2022, 4, 2)) as dag:
    consumer.expand(arg=make_list())

More information can be found here: Dynamic Task Mapping

Grid View replaces Tree View

Grid view replaces tree view in Airflow 2.3.0.

Screenshots:

Purge history from metadata database

Airflow 2.3.0 introduces a new airflow db clean command that can be used to purge old data from the metadata database.

You would want to use this command if you want to reduce the size of the metadata database.

More information can be found here: Purge history from metadata database

LocalKubernetesExecutor

There is a new executor named LocalKubernetesExecutor. This executor helps you run some tasks using LocalExecutor and run another set of tasks using the KubernetesExecutor in the same deployment based on the task’s queue.

More information can be found here: LocalKubernetesExecutor

DagProcessorManager as standalone process (AIP-43)

As of 2.3.0, you can run the DagProcessorManager as a standalone process. Because DagProcessorManager runs user code, separating it from the scheduler process and running it as an independent process in a different host is a good idea.

The airflow dag-processor cli command will start a new process that will run the DagProcessorManager in a separate process. Before you can run the DagProcessorManager as a standalone process, you need to set the [scheduler] standalone_dag_processor to True.

More information can be found here: dag-processor CLI command

JSON serialization for connections

You can now create connections using the json serialization format.

airflow connections add 'my_prod_db' \
    --conn-json '{
        "conn_type": "my-conn-type",
        "login": "my-login",
        "password": "my-password",
        "host": "my-host",
        "port": 1234,
        "schema": "my-schema",
        "extra": {
            "param1": "val1",
            "param2": "val2"
        }
    }'

You can also use json serialization format when setting the connection in environment variables.

More information can be found here: JSON serialization for connections

Airflow `db downgrade` and Offline generation of SQL scripts

Airflow 2.3.0 introduced a new command airflow db downgrade that will downgrade the database to your chosen version.

You can also generate the downgrade/upgrade SQL scripts for your database and manually run it against your database or just view the SQL queries that would be run by the downgrade/upgrade command.

More information can be found here: Airflow db downgrade and Offline generation of SQL scripts

Reuse of decorated tasks

You can now reuse decorated tasks across your dag files. A decorated task has an override method that allows you to override its arguments.

Here’s an example:

@task
def add_task(x, y):
    print(f"Task args: x={x}, y={y}")
    return x + y


@dag(start_date=datetime(2022, 1, 1))
def mydag():
    start = add_task.override(task_id="start")(1, 2)
    for i in range(3):
        start >> add_task.override(task_id=f"add_start_{i}")(start, i)

More information can be found here: Reuse of decorated DAGs

Other small features

This isn’t a comprehensive list, but some noteworthy or interesting small features include:

Support different timeout value for dag file parsing
airflow dags reserialize command to reserialize dags
Events Timetable
SmoothOperator - Operator that does literally nothing except logging a YouTube link to Sade’s “Smooth Operator”. Enjoy!

Contributors

Thanks to everyone who contributed to this release: Ash Berlin-Taylor, Brent Bovenzi, Daniel Standish, Elad, Ephraim Anierobi, Jarek Potiuk, Jed Cunningham, Josh Fell, Kamil Breguła, Kanthi, Kaxil Naik, Khalid Mammadov, Malthe Borch, Ping Zhang, Tzu-ping Chung and many others who keep making Airflow better for everyone.

]]>

What's new in Apache Airflow 2.2.0

2021-10-11T00:00:00Z

I’m proud to announce that Apache Airflow 2.2.0 has been released. It contains over 600 commits since 2.1.4 and includes 30 new features, 84 improvements, 85 bug fixes, and many internal and doc changes.

Details:

📦 PyPI: https://pypi.org/project/apache-airflow/2.2.0/
📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.2.0/
🛠️ Changelog: https://airflow.apache.org/docs/apache-airflow/2.2.0/changelog.html
🐳 Docker Image: docker pull apache/airflow:2.2.0
🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.2.0

As the changelog is quite large, the following are some notable new features that shipped in this release.

Custom Timetables (AIP-39)

Airflow has historically used cron expressions and timedeltas to represent when a DAG should run. This worked for a lot of use cases, but not all. For example, running daily on Monday-Friday, but not on weekends wasn’t possible.

To provide more scheduling flexibility, determining when a DAG should run is now done with Timetables. Of course, backwards compatibility has been maintained - cron expressions and timedeltas are still fully supported, however, timetables are pluggable so you can add your own custom timetable to fit your needs! For example, you could write a timetable to schedule a DagRun

execution_date has long been confusing to new Airflowers, so as part of this change a new concept has been added to Airflow to replace it named data_interval, which is the period of data that a task should operate on. The following are now available:

logical_date (aka execution_date)
data_interval_start (same value as execution_date for cron)
data_interval_end (aka next_execution_date)

If you write your own timetables, keep in mind they should be idempotent and fast as they are used in the scheduler to create DagRuns.

More information can be found at: Customizing DAG Scheduling with Timetables

Deferrable Tasks (AIP-40)

Deferrable tasks allows operators or sensors to defer themselves until a light-weight async check passes, at which point they can resume executing. Most importantly, this results in the worker slot, and most notably any resources used by it, to be returned to Airflow. This allows simple things like monitoring a job in an external system or watching for an event to be much cheaper.

To support this feature, a new component has been added to Airflow, the triggerer, which is the daemon process that runs the asyncio event loop.

Airflow 2.2.0 ships with 2 deferrable sensors, DateTimeSensorAsync and TimeDeltaSensorAsync, both of which are drop-in replacements for the existing corresponding sensor.

More information can be found at:

Deferrable Operators & Triggers

Custom `@task` decorators and `@task.docker`

Airflow 2.2.0 allows providers to create custom @task decorators in the TaskFlow interface.

The @task.docker decorator is one such decorator that allows you to run a function in a docker container. Airflow handles getting the code into the container and returning xcom - you just worry about your function. This is particularly useful when you have conflicting dependencies between Airflow itself and tasks you need to run.

More information on creating custom @task decorators can be found at: Creating Custom @task Decorators

More information on the @task.docker decorator can be found at: Using the Taskflow API with Docker or Virtual Environments

Validation of DAG params

You can now apply validation on DAG params by passing a Param object for each param. The Param object supports the full json-schema validation specifications.

Currently, this only functions with manually triggered DAGs, but it does set the stage for future params related functionality.

More information can be found at: Params

Other small features

This isn’t a comprehensive list, but some noteworthy or interesting small features include:

Testing Connections from the UI - test the credentials for your Connection actually work
Duplication Connections from the UI
DAGs “Next run” info is shown in the UI, including when the run will actually start
airflow standalone command runs all of the Airflow components directly without docker - great for local development

Contributors

Thanks to everyone who contributed to this release: Andrew Godwin, Ash Berlin-Taylor, Brent Bovenzi, Elad Kalif, Ephraim Anierobi, James Timmins, Jarek Potiuk, Jed Cunningham, Josh Fell, Kamil Breguła, Kaxil Naik, Malthe Borch, Sam Wheating, Sumit Maheshwari, Tzu-ping Chung and many others

]]>

Airflow Summit 2021

2021-03-21T00:00:00Z

Airflow Summit 2021 is here!

The summit will be held online, July 8-16, 2021. Join us from all over the world to find out how Airflow is being used by leading companies, what is its roadmap and how you can participate in its development.

Useful information:

The official website: https://airflowsummit.org
Call for proposals is open until 12 April 2021. To submit your talk go to https://sessionize.com/airflow-summit-2021/
In case of any questions reach out to us via info@airflowsummit.org

]]>

Airflow Survey 2020

2021-03-09T00:00:00Z

Apache Airflow Survey 2020

World of data processing tools is growing steadily. Apache Airflow seems to be already considered as crucial component of this complex ecosystem. We observe steady growth in number of users as well as in an amount of active contributors. So listening and understanding our community is of high importance.

It’s worth to note that the 2020 survey was still mostly about 1.10.X version of Apache Airflow and possibly many drawbacks were addressed in the 2.0 version that was released in December 2020. But if this is true, we will learn next year!

Overview of the user

What best describes your current occupation? (single choice)

	No.	%
Data Engineer	115	56.65
Developer	28	13.79
DevOps	17	8.37
Solutions Architect	14	6.9
Data Scientist	12	5.91
Other	10	4.93
Data Analyst	4	1.97
Support Engineer	3	1.48

Those results are not a surprise as Airflow is a tool dedicated to data-related tasks. The majority of our users are data engineers, scientists or analysts. The 2020 results are similar to those from 2019 with visible slight increase in ML use cases.

Additionally, 79% of users uses Airflow on daily basis and 16% interacts with it at least once a week.

How many people work in your company? (single choice)

	No.	%
200+	107	52.71
51-200	44	21.67
11-50	37	18.23
1-10	15	7.39

How many people in your company use Airflow? (single choice)

	No.	%
1-5	84	41.38
6-20	75	36.95
21-50	23	11.33
50+	21	10.34

Airflow is a software that is used and trusted by big companies. We can also see that Airflow can work fine for teams of different sizes. However, in some cases users may use multiple Airflow instances.

Are you considering moving to other workflow engines? (single choice)

	No.	%
No, we are happy with Airflow	174	85.71
Yes	29	14.29

Nearly 1 out of 7 users is considering migrating to other workflow engines. Their decision is usually justified by need of easier workflow writing experience (12.32%), better UI/UX and faster scheduler (8.37% both).

While the first point may be addressed by TaskFlow API in Airflow 2.0 the other two are definitely addressed in the new major version. And the early feedback from 2.0 users seems to be confirming it.

The alternative engines considered by users are mainly Prefect and Argo. Some participants also mentioned Luigi, Kubeflow or custom solutions.

Are you or your team actively participating in Airflow development - contributing? (single choice)

	No.	%
I wish we could	99	48.77
No	59	29.06
Yes	45	22.17

This is really heart-warming result. It means that 1 out of 5 users contributes actively to our project! But it would be good to learn if there’s something else than time that is stopping people who wish to contribute from doing it. If there are some other obstacles we definitely would like to learn about them so we can improve. That said - if you know something we can improve please reach out via Slack, dev list or Github discussions.

How likely are you to recommend Apache Airflow? (single choice)

	No.	2020 %	2019 %
Very Likely	125	61.58	45.45%
Likely	62	30.54	40.26%
Neutral	11	5.42	10.71%
Unlikely	3	1.48	2.60%
Very unlikely	2	0.99	0.97%

Here is good news! It seems that people are more willing to recommend Apache Airflow than year before.

What is your source of information about Airflow? (multiple choice)

	No.	%
Documentation	154	75.86
Airflow website	139	68.47
Slack	128	63.05
Github	127	62.56
Stack Overflow	72	35.47
Airflow Summit Videos	44	21.67
The dev mailing list	33	16.26
Awesome Apache Airflow repository	21	10.34
Other	15	7.39

Here we see that Airflow documentation is the crucial source of information. What’s interesting is that more than 60% of users are getting information from Github and Slack channels.

Airflow uses cases

Do you have any customisation of Airflow? (single choice)

	No.	%
No, we use vanilla Airflow	154	75.86
Yes, we have small patches (no fork)	34	16.75
Yes, we have separate fork	15	7.39

When onboarding new members to airflow, what is the biggest problem? (multiple choice)

	No.	%
No guide on best practises on developing DAGs	102	50.25
There is no easy option to launch Airflow	64	31.53
Small number of tutorials on different aspects of using Airflow	57	28.08
Documentation is not clear enough	53	26.11
There is no easy option to deploy DAGs to an Airflow instance	52	25.62
No problems	34	16.75
Small number of blogs regarding Airflow	30	14.78

Which interface(s) of Airflow do you use as part of your current role? (multiple choice)

	No.	%
Original Airflow Graphical User Interface	199	98.03
CLI	88	43.35
API	48	23.65
Custom (own created) Airflow Graphical User Interface	12	5.91
Other	3	1.48

Do you combine multiple DAGs? (multiple choice)

	No.	%
Yes, by triggering another DAG	87	42.86
No, I don’t combine multiple DAGs	79	38.92
Yes, through SubDAG	40	19.7
Other	18	8.87

How do you integrate with external services? (multiple choice)

	No.	%
Using existing dedicated operators / hooks	147	72.41
Using Bash / Python operator	140	68.97
Using own custom operators / hooks	138	67.98
Other	12	5.91

What external services do you use in your Airflow DAGs? (multiple choice)

	No.	%
Amazon Web Services	121	59.61
Internal company systems	113	55.67
Google Cloud Platform / Google APIs	97	47.78
Hadoop / Spark / Flink / Other Apache software	72	35.47
Microsoft Azure	21	10.34
Other	19	9.36
I do not use external services in my Airflow DAGs	5	2.46

Do you use Airflow Plugins? If yes, what do you use them for? (multiple choice)

	No.	%
Adding new operators/sensors and hooks	119	58.62
I don’t use Airflow plugins	69	33.99
Adding AppBuilder views & menu items	27	13.3
Adding new executors	17	8.37
Adding OperatorExtraLinks	13	6.4

| Other

Do you use Airflow’s data lineage feature? (single choice)

	No.	%
No, I will use such feature if fully supported in Airflow	105	51.72
No, data lineage isn’t a concern for my usage.	68	33.5
Yes, I use another data lineage product	24	11.82
Yes, I use custom implementation	5	2.46
Yes, I use Airflow’s experimental data lineage feature	1	0.49

When asked what lineage product users use, the answers were varying from custom tools to known product like Amundsen, Atlas or dbt.

Deployment

How many active DAGs do you have in your largest Airflow instance? (open question)

Number of DAGs	No.	%
< 20	64	32
21-40	33	16
41-60	13	6
61-100	32	16
101-200	31	15
201-300	8	4
301-999	12	6
1000+	10	5

What is the maximum number of tasks that you have used in one DAG? (open question)

Number of DAGs	No.	%
< 10	42	21
11-20	31	15
21-30	15	7
31-40	11	5
41-50	22	11
51-100	39	19
101-200	16	8
201-500	16	8
501+	11	5

Which version of Airflow do you use currently? (single choice)

	No.	%
1.10.14	55	27.09
2.0.0+	45	22.17
1.10.12	27	13.3
1.10.10	26	12.81
1.10.11	14	6.9
1.10.5 or older	10	4.93
1.10.9	8	3.94
1.10.13	7	3.45
1.10.6	4	1.97
1.10.7	4	1.97
1.10.8	3	1.48

This was probably one of the most important questions in the survey. While it’s good to see that more than 60% of users use one of three latest Airflow versions, it’s worrying that the rest are using versions that are old or have known security vulnerabilities.

Additionally, more than 20% of users are already using 2.0.0+ versions which is reasonably good information.

What meta-database do you use? (single choice)

	No.	%
Postgres 12	36	17.73
Postgres 9.6	33	16.26
Postgres 11	31	15.27
MySQL 5.7	27	13.3
MySQL 8.0	20	9.85
Postgres 10	20	9.85
Other	19	9.36
Postgres 13	18	8.87

This means that more about 69% of users decide to use Postgres as their meta-database. MySQL is the choice of nearly 24% users. The other responses included some MySQL versions like MariaDB or cloud hosted database like Cloud SQL (used by Google Composer) or AWS Aurora.

It’s good to know that users rather avoid using SQLite in production deployments!

What executor type do you use? (single choice)

	No.	2020	2019
Celery	100	49.26%	44.81%
Kubernetes	48	23.65%	16.88%
Local	40	19.7%	27.60%
Sequential	10	4.93%	7.14%
Other	5	2.46%	3.57

In comparison to previous year it seems that more users use currently Celery and Kubernetes executors and LocalExecutor usage dropped by nearly 8 points. This may suggest that users’ deployments are growing, and they need more scalable solutions.

Among CeleryExecutor users 78% use Redis as a broker, 19% use RabbitMQ and the rest is using other brokers or is not sure what is used in their deployments.

What metrics do you use to monitor Airflow? (multiple choice)

	No.	%
I do not use monitoring	65	32.02
External monitoring service	60	29.56
Information from metadatabase	51	25.12
Statsd	49	24.14
Other	31	15.27

The other responses included mostly information about tools used by users including DataDog and Prometheus exporter.

How do you deploy Airflow? (single choice)

	No.	%
On virtual machines (for example using AWS EC2)	64	31.53
Using a managed service like Astronomer, Google Composer or AWS MWAA	35	17.24
On Kubernetes (using custom deployments)	29	14.29
On premises	28	13.79
On Kubernetes (using another helm chart)	20	9.85
On Kubernetes (using Apache Airflow’s helm chart)	17	8.37
Other	12	5.91

Nearly 33% of users deploys Airflow using some kind of Kubernetes deployment. This is about 10 percent more than in 2019. There’s slightly increase in usage of Airflow via managed services (14.61% in 2019).

Do you use containerisation for deployment? (single choice)

	No.	%
Yes, using helm chart / kubernetes	58	28.57
No, I don’t use containerisation	57	28.08
Yes, single docker image	49	24.14
Yes, using docker compose	39	19.21

Among users who do not use Kubernetes based deployments 58% of them use containerisation. About 42% of those users use docker-compose for deployments.

How do you distribute your DAGs? (single choice)

	No.	%
Using a synchronizing process (Git sync, GCS fuse, etc)	79	38.92
Bake them into the docker image	56	27.59
Shared files system	34	16.75
Other	20	9.85
I don’t know	14	6.9

The most popular way of distributing DAGs seems to be using a synchronizing process. About 40% of users use this process together with Kubernetes deployments.

Future of Airflow

In your opinion, what could be improved in Airflow? (multiple choice)

	No.	%
Web UI	100	49.26
Examples, how-to, onboarding documentation	90	44.33
Logging, monitoring and alerting	90	44.33
Technical documentation	90	44.33
Scheduler performance	83	40.89
DAG authoring	64	31.53
Authentication and authorization	58	28.57
REST API	51	25.12
Other	44	21.67
Reliability	41	20.2
External integration e.g. AWS, GCP, Apache products	36	17.73
Security	28	13.79
CLI	20	9.85
Everything work fine for me	14	6.9
I don’t know	4	1.97

Which features would most interest you? (multiple choice)

	No.	%
DAG versioning	109	53.69
Builtin statistics	71	34.98
Improved data lineage	65	32.02
Scheduling at the start of the interval	63	31.03
Stateless workers	59	29.06
More option to configure schedules (time units, increments)	57	28.08
Multi-tenant deployment	49	24.14
DAG fetcher (AIP-5)	39	19.21
Generic transfer operator	34	16.75
Other	33	16.26
I have everything I need	11	5.42
Nothing	11	5.42

Will you consider migrating to Airflow 2.0? (single choice)

	No.	%
Yes, as soon as possible	81	39.9
Yes, once it’s mature (for example after 2.1)	72	35.47
I am already using Airflow 2.0+	39	19.21
I don’t know yet	8	3.94
No, I do not plan to migrate	3	1.48

What are the features of Airflow 2.0 you are most excited about? (multiple choice)

	No.	%
General performance improvements	133	65.52
Refreshed WebUI	102	50.25
Scheduler HA	99	48.77
Official docker image	84	41.38
@task decorator	56	27.59
Official helm chart	51	25.12
Providers packages	41	20.2
Configurable XCom backends	33	16.26
CeleryKubernetesExecutor	31	15.27
Other	12	5.91

Summary

From an open-source point of view, it is good to see that many people would love to contribute to Apache Airflow. This means that there are resources that if unleashed may make our community even stronger. From a product perspective, it is important to know that users are usually using the latest versions of our software and are willing to upgrade to new ones.

Finally, there are still some things to improve - documentation, onboarding guides and plug-and-play airflow deployments. However, we hope that with the increase of adoption there will be an increase in people willing to share their experience and tools.

Data

If you think I missed something or you simply want to look for insights on your own, the data is available for you here: (Airflow User Survey 2020.csv)[/data/survey-responses/airflow-user-survey-responses-2020.csv.zip]

]]>

Apache Airflow 2.0 is here!

2020-12-17T00:00:00Z

I am proud to announce that Apache Airflow 2.0.0 has been released.

The full changelog is about 3,000 lines long (already excluding everything backported to 1.10), so for now I’ll simply share some of the major features in 2.0.0 compared to 1.10.14:

A new way of writing dags: the TaskFlow API (AIP-31)

(Known in 2.0.0alphas as Functional DAGs.)

DAGs are now much much nicer to author especially when using PythonOperator. Dependencies are handled more clearly and XCom is nicer to use

Fully specified REST API (AIP-32)

We now have a fully supported, no-longer-experimental API with a comprehensive OpenAPI specification

Massive Scheduler performance improvements

As part of AIP-15 (Scheduler HA+performance) and other work Kamil did, we significantly improved the performance of the Airflow Scheduler. It now starts tasks much, MUCH quicker.

Over at Astronomer.io we’ve benchmarked the scheduler—it’s fast (we had to triple check the numbers as we don’t quite believe them at first!)

Scheduler is now HA compatible (AIP-15)

It’s now possible and supported to run more than a single scheduler instance. This is super useful for both resiliency (in case a scheduler goes down) and scheduling performance.

To fully use this feature you need Postgres 9.6+ or MySQL 8+ (MySQL 5, and MariaDB won’t work with more than one scheduler I’m afraid).

There’s no config or other set up required to run more than one scheduler—just start up a scheduler somewhere else (ensuring it has access to the DAG files) and it will cooperate with your existing schedulers through the database.

For more information, read the Scheduler HA documentation.

Task Groups (AIP-34)

SubDAGs were commonly used for grouping tasks in the UI, but they had many drawbacks in their execution behaviour (primarily that they only executed a single task in parallel!) To improve this experience, we’ve introduced “Task Groups”: a method for organizing tasks which provides the same grouping behaviour as a subdag without any of the execution-time drawbacks.

SubDAGs will still work for now, but we think that any previous use of SubDAGs can now be replaced with task groups. If you find an example where this isn’t the case, please let us know by opening an issue on GitHub

For more information, check out the Task Group documentation.

Refreshed UI

We’ve given the Airflow UI a visual refresh and updated some of the styling.

We have also added an option to auto-refresh task states in Graph View so you no longer need to continuously press the refresh button :).

Check out the screenshots in the docs for more.

Smart Sensors for reduced load from sensors (AIP-17)

If you make heavy use of sensors in your Airflow cluster, you might find that sensor execution takes up a significant proportion of your cluster even with “reschedule” mode. To improve this, we’ve added a new mode called “Smart Sensors”.

This feature is in “early-access”: it’s been well-tested by Airbnb and is “stable”/usable, but we reserve the right to make backwards incompatible changes to it in a future release (if we have to. We’ll try very hard not to!)

Read more about it in the Smart Sensors documentation.

Simplified KubernetesExecutor

For Airflow 2.0, we have re-architected the KubernetesExecutor in a fashion that is simultaneously faster, easier to understand, and more flexible for Airflow users. Users will now be able to access the full Kubernetes API to create a .yaml pod_template_file instead of specifying parameters in their airflow.cfg.

We have also replaced the executor_config dictionary with the pod_override parameter, which takes a Kubernetes V1Pod object for a1:1 setting override. These changes have removed over three thousand lines of code from the KubernetesExecutor, which makes it run faster and creates fewer potential errors.

Airflow core and providers: Splitting Airflow into 60+ packages:

Airflow 2.0 is not a monolithic “one to rule them all” package. We’ve split Airflow into core and 61 (for now) provider packages. Each provider package is for either a particular external service (Google, Amazon, Microsoft, Snowflake), a database (Postgres, MySQL), or a protocol (HTTP/FTP). Now you can create a custom Airflow installation from “building” blocks and choose only what you need, plus add whatever other requirements you might have. Some of the common providers are installed automatically (ftp, http, imap, sqlite) as they are commonly used. Other providers are automatically installed when you choose appropriate extras when installing Airflow.

The provider architecture should make it much easier to get a fully customized, yet consistent runtime with the right set of Python dependencies.

But that’s not all: you can write your own custom providers and add things like custom connection types, customizations of the Connection Forms, and extra links to your operators in a manageable way. You can build your own provider and install it as a Python package and have your customizations visible right in the Airflow UI.

Our very own Jarek Potiuk has written about providers in much more detail on Jarek’s blog.

Docs on the providers concept and writing custom providers
Docs on all providers packages available

Security

As part of Airflow 2.0 effort, there has been a conscious focus on Security and reducing areas of exposure. This is represented across different functional areas in different forms. For example, in the new REST API, all operations now require authorization. Similarly, in the configuration settings, the Fernet key is now required to be specified.

Configuration

Configuration in the form of the airflow.cfg file has been rationalized further in distinct sections, specifically around “core”. Additionally, a significant amount of configuration options have been deprecated or moved to individual component-specific configuration files, such as the pod-template-file for Kubernetes execution-related configuration.

Thanks to all of you

We’ve tried to make as few breaking changes as possible and to provide deprecation path in the code, especially in the case of anything called in the DAG. That said, please read through UPDATING.md to check what might affect you. For example: We have re-organized the layout of operators (they now all live under airflow.providers.*) but the old names should continue to work - you’ll just notice a lot of DeprecationWarnings that need to be fixed up.

Thank you so much to all the contributors who got us to this point, in no particular order: Kaxil Naik, Daniel Imberman, Jarek Potiuk, Tomek Urbaszek, Kamil Breguła, Gerard Casas Saez, Xiaodong DENG, Kevin Yang, James Timmins, Yingbo Wang, Qian Yu, Ryan Hamilton and the 100s of others who keep making Airflow better for everyone.

]]>

Journey with Airflow as an Outreachy Intern

2020-08-30T00:00:00Z

Outreachy is a program which organises three months paid internships with FOSS projects for people who are typically underrepresented in those projects.

Contribution Period

The first thing I had to do was choose a project under an organisation. After going through all the projects I chose “Extending the REST API of Apache Airflow”, because I had a good idea of what REST API(s) are, so I thought it would be easier to get started with the contributions. The next step was to set up Airflow’s dev environment which thanks to Breeze, was a breeze. Since I had never contributed to FOSS before so this part was overwhelming but there were plenty of issues labelled “good first issues” with detailed descriptions and some even had code snippets so luckily that nudged me in the right direction. These things about Airflow and the positive vibes from the community were the reasons why I chose to stick with Airflow as my Outreachy project.

Internship Period

My first PR was followed by many new experiences one of them being that I introduced a bug in it;). But nonetheless it made me familiar with the feedback loop and the feedback on my subsequent PRs was the focal point of the overall learning experience I went through, which boosted my confidence to contribute more and move out of my comfort zone. I wanted to learn more about the things that happen under the Airflow’s hood so I started filtering out recent PRs dealing with different components and I would go through the code changes along with discussion that would help me get a better understanding of the whole workflow. Airflow’s mailing list was also a great source of knowledge.

The API related PRs that I worked on helped me with some of the important concepts like:

Pool CRUD endpoints where pools limit the execution parallelism.
Tasks determine the actual work that has to be carried out.
DAG which represents the structure for a collection of tasks. It keeps track of tasks, their dependencies and the sequence in which they have to run.
Dag Runs that are the instantiation of DAG(s) in time.

Through actively and passively participating in discussions I learnt that even if there is a difference of opinion one could always learn from the different approaches, and this PR with more than 300+ comments is the proof of it. I also started reviewing small PRs which gave me the amazing opportunity to interact with new people. Throughout my internship I learnt a lot about different frameworks and technologies but the biggest takeaway for me was that a code is read more often than it’s written, and I started writing code with that in mind.

Wrapping Up

So with my project of extending Airflow’s REST API as well as the Outreachy internship coming to an end I would like to thank my mentors Jarek Potiuk, Kaxil Naik and Kamil Breguła for the patience and the time they invested in mentoring me and the Airflow community for making me feel so welcomed. I plan to stick around and contribute to give back to the community that has been made my summer, one to remember.

]]>

Apache Airflow 1.10.12

2020-08-25T00:00:00Z

Airflow 1.10.12 contains 113 commits since 1.10.11 and includes 5 new features, 23 improvements, 23 bug fixes, and several doc changes.

Details:

PyPI: https://pypi.org/project/apache-airflow/1.10.12/
Docs: https://airflow.apache.org/docs/1.10.12/
Changelog: http://airflow.apache.org/docs/1.10.12/changelog.html

Airflow 1.10.11 has breaking changes with respect to KubernetesExecutor & KubernetesPodOperator so I recommend users to directly upgrade to Airflow 1.10.12 instead.

Some of the noteworthy new features (user-facing) are:

Allow defining Custom XCom class

Until Airflow 1.10.11, the XCom data was only stored in Airflow Metadatabase. From Airflow 1.10.12, users would be able to define custom XCom classes. This will allow users to transfer larger data between tasks. An example here would be to store XCom in S3 or GCS Bucket if the size of data that needs to be stored is larger than XCom.MAX_XCOM_SIZE (48 KB).

PR: https://github.com/apache/airflow/pull/8560

Get Airflow configs with sensitive data from Secret Backends

Users would be able to get the following Airflow configs from Secrets Backend like Hashicorp Vault:

sql_alchemy_conn in [core] section
fernet_key in [core] section
broker_url in [celery] section
flower_basic_auth in [celery] section
result_backend in [celery] section
password in [atlas] section
smtp_password in [smtp] section
bind_password in [ldap] section
git_password in [kubernetes] section

Further improving Airflow’s Secret Management story, from Airflow 1.10.12, users don’t need to hardcode the sensitive config value in airflow.cfg nor then need to use an Environment variable to set this config.

For example, the metadata database connection string can either be set in airflow.cfg like this:

[core]
sql_alchemy_conn_secret = sql_alchemy_conn

This will retrieve config option from the set Secret Backends.

As you can see you just need to add a _secret suffix at the end of the actual config option and the value needs to be the key which the Secrets backend will look for.

Similarly, _secret config options can also be set using a corresponding environment variable. For example:

export AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn

More details: http://airflow.apache.org/docs/1.10.12/howto/set-config.html

Add AirflowClusterPolicyViolation support to airflow_local_settings.py

Users can use Cluster Policies to apply cluster-wide checks on Airflow tasks. You can raise AirflowClusterPolicyViolation in a policy or task mutation hook to prevent a DAG from being imported or prevent a task from being executed if the task is not compliant with your check.

These checks are intended to help teams using Airflow to protect against common beginner errors that may get past a code reviewer, rather than as technical security controls.

For example, don’t run tasks without airflow owners:

def task_must_have_owners(task):
    if not task.owner or task.owner.lower() == conf.get('operators', 'default_owner'):
        raise AirflowClusterPolicyViolation(
            'Task must have non-None non-default owner. Current value: {}'.format(task.owner))

More details: http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks

Launch Pods via YAML files when using KubernetesExecutor and KubernetesPodOperator

As of 1.10.12, users can launch pods via YAML files instead of passing various configurations.

To allow greater flexibility we have deprecated Airflow’s Pod class and instead now use classes and objects from the official Kubernetes API. The POD class will still work but raise a deprecation warning. This feature involved a pretty extensive rewrite of all of our pod creation code.

Initially, we were going to hold off on these features until Airflow 2.0. However, we soon realized that exposing these features in 1.10.x is crucial in preparing users for the 2.0 release to come.

Details: https://github.com/apache/airflow/pull/6230 (Backport commit)

Updating Guide

If you are updating Apache Airflow from a previous version to 1.10.12, please take a note of the following:

Run airflow upgradedb after pip install -U apache-airflow==1.10.12 as 1.10.12 contains 1 database migration.
As of airflow 1.10.12, using the airflow.contrib.kubernetes.Pod class in the pod_mutation_hook is now deprecated. Instead we recommend that users treat the pod parameter as a kubernetes.client.models.V1Pod object. This means that users now have access to the full Kubernetes API when modifying airflow pods for mutating POD.
Previously, when tasks skipped by SkipMixin (such as BranchPythonOperator, BaseBranchOperator and ShortCircuitOperator) are cleared, they execute. Since 1.10.12, when such skipped tasks are cleared, they will be skipped again by the newly introduced NotPreviouslySkippedDep.

Special Note

Python 2

Python 2 has reached end of its life on Jan 2020. Airflow Master no longer supports Python 2. Airflow 1.10.* would be the last series to support Python 2.

We strongly recommend users to use Python >= 3.6

Use Airflow RBAC UI

Airflow 1.10.12 ships with 2 UIs, the default is non-RBAC Flask-admin based UI and Flask-appbuilder based UI.

The Flask-AppBuilder (FAB) based UI allows Role-based Access Control and has more advanced features compared to the legacy Flask-admin based UI. This UI can be enabled by setting rbac=True in [webserver] section in your airflow.cfg.

Flask-admin based UI is deprecated and new features won’t be ported to it. This UI will still be the default for 1.10.* series but would no longer be available from Airflow 2.0

We have moved to GitHub Issues

The Airflow Project has moved from JIRA to GitHub for tracking issues.

So if you find any bugs in Airflow 1.10.12 please create a GitHub Issue for it.

List of Contributors

According to git shortlog, the following people contributed to the 1.10.12 release. Thank you to all contributors!

Alexander Sutcliffe, Andy, Aneesh Joseph, Ash Berlin-Taylor, Aviral Agrawal, BaoshanGu, Beni Ben zikry, Daniel Imberman, Daniel Standish, Danylo Baibak, Ephraim Anierobi, Felix Uellendall, Greg Neiheisel, Hartorn, Jacob Ferriero, Jannik F, Jarek Potiuk, Jinhui Zhang, Kamil Breguła, Kaxil Naik, Kurganov, Luis Magana, Max Arrich, Pete DeJoy, Sumit Maheshwari, Tomek Urbaszek, Vicken Simonian, Vinnie Guimaraes, William Tran, Xiaodong Deng, YI FU, Zikun Zhu, dewaldabrie, pulsar314, retornam, yuqian90

]]>

Apache Airflow For Newcomers

2020-08-17T00:00:00Z

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. A workflow is a sequence of tasks that processes a set of data. You can think of workflow as the path that describes how tasks go from being undone to done. Scheduling, on the other hand, is the process of planning, controlling, and optimizing when a particular task should be done.

Authoring Workflow in Apache Airflow.

Airflow makes it easy to author workflows using python scripts. A Directed Acyclic Graph (DAG) represents a workflow in Airflow. It is a collection of tasks in a way that shows each task’s relationships and dependencies. You can have as many DAGs as you want, and Airflow will execute them according to the task’s relationships and dependencies. If task B depends on the successful execution of another task A, it means Airflow will run task A and only run task B after task A. This dependency is very easy to express in Airflow. For example, the above scenario is expressed as

task_A >> task_B

Also equivalent to

task_A.set_downstream(task_B)

That helps Airflow to know that it needs to execute task A before task B. Tasks can have far more complex relationships to each other than expressed above and Airflow figures out how and when to execute the tasks following their relationships and dependencies.

Before we discuss the architecture of Airflow that makes scheduling, executing, and monitoring of workflow an easy thing, let us discuss the Breeze environment.

Breeze Environment

The breeze environment is the development environment for Airflow where you can run tests, build images, build documentations and so many other things. There are excellent documentation and video on Breeze environment. Please check them out. You enter the Breeze environment by running the ./breeze script. You can run all the commands mentioned here in the Breeze environment.

Scheduler

The scheduler is the component that monitors DAGs and triggers those tasks whose dependencies have been met. It watches over the DAG folder, checking the tasks in each DAG and triggers them once they are ready. It accomplishes this by spawning a process that runs periodically(every minute or so) reading the metadata database to check the status of each task and decides what needs to be done. The metadata database is where the status of all tasks are recorded. The status can be one of running, success, failed, etc.

A task is said to be ready when its dependencies have been met. The dependencies include all the data necessary for the task to be executed. It should be noted that the scheduler won’t trigger your tasks until the period it covers has ended. If a task’s schedule_interval is @daily, the scheduler triggers the task at the end of the day and not at the beginning. This is to ensure that the necessary data needed for the tasks are ready. It is also possible to trigger tasks manually on the UI.

In the Breeze environment, the scheduler is started by running the command airflow scheduler. It uses the configured production environment. The configuration can be specified in airflow.cfg

Executor

Executors are responsible for running tasks. They work with the scheduler to get information about what resources are needed to run a task as the task is queued.

By default, Airflow uses the SequentialExecutor. However, this executor is limited and it is the only executor that can be used with SQLite.

There are many other executors, the difference is on the resources they have and how they choose to use the resources. The available executors are:

Sequential Executor
Debug Executor
Local Executor
Dask Executor
Celery Executor
Kubernetes Executor
Scaling Out with Mesos (community contributed)

CeleryExecutor is a better executor compared to the SequentialExecutor. The CeleryExecutor uses several workers to execute a job in a distributed way. If a worker node is ever down, the CeleryExecutor assign its task to another worker node. This ensures high availability.

The CeleryExecutor works closely with the scheduler which adds a message to the queue and the Celery broker which delivers the message to a Celery worker to execute. You can find more information about the CeleryExecutor and how to configure it at the documentation

Webserver

The webserver is the web interface (UI) for Airflow. The UI is feature-rich. It makes it easy to monitor and troubleshoot DAGs and Tasks.

There are many actions you can perform on the UI. You can trigger a task, monitor the execution including the duration of the task. The UI makes it possible to view the task’s dependencies in a tree view and graph view. You can view task logs in the UI.

The web UI is started with the command airflow webserver in the breeze environment.

Backend

By default, Airflow uses the SQLite backend for storing the configuration information, DAG states, and much other useful information. This should not be used in production as SQLite can cause a data loss.

You can use PostgreSQL or MySQL as a backend for airflow. It is easy to change to PostgreSQL or MySQL.

The command ./breeze --backend mysql selects MySQL as the backend when starting the breeze environment.

Operators

Operators determine what gets done by a task. Airflow has a lot of builtin Operators. Each operator does a specific task. There’s a BashOperator that executes a bash command, the PythonOperator which calls a python function, AwsBatchOperator which executes a job on AWS Batch and many more.

Sensors

Sensors can be described as special operators that are used to monitor a long-running task. Just like Operators, there are many predefined sensors in Airflow. These include

AthenaSensor: Asks for the state of the Query until it reaches a failure state or success state.
AzureCosmosDocumentSensor: Checks for the existence of a document which matches the given query in CosmosDB
GoogleCloudStorageObjectSensor: Checks for the existence of a file in Google Cloud Storage

A list of most of the available sensors can be found in this module

Contributing to Airflow

Airflow is an open source project, everyone is welcome to contribute. It is easy to get started thanks to the excellent documentation on how to get started.

I joined the community about 12 weeks ago through the Outreachy Program and have completed about 40 PRs.

It has been an amazing experience! Thanks to my mentors Jarek and Kaxil, and the community members especially Kamil and Tomek for all their support. I’m grateful!

Thank you so much, Leah E. Cole, for your wonderful reviews.

]]>

Implementing Stable API for Apache Airflow

2020-07-19T00:00:00Z

My Outreachy internship is coming to its ends which is also the best time to look back and reflect on the progress so far.

The goal of my project is to Extend and Improve the Apache Airflow REST API. In this post, I will be sharing my progress so far.

We started a bit late implementing the REST API because it took time for the OpenAPI 3.0 specification we were to use for the project to be merged. Thanks to Kamil, who paved the way for us to start implementing the REST API endpoints. Below are the endpoints I implemented and the challenges I encountered, including how I overcame them.

Implementing The Read-Only Connection Endpoints

The read-only connection endpoints were the first endpoint I implemented. Looking back, I can see how much I have improved.

I started by implementing the database schema for the Connection table using Marshmallow 2. We had to use Marshmallow 2 because Flask-AppBuilder was still using it and Flask-AppBuilder is deeply integrated to Apache Airflow. This meant I had to unlearn Marshmallow 3 that I had been studying before this realization, but thankfully, Marshmallow 3 isn’t too different, so I was able to start using Marshmallow 2 in no time.

This first PR would have been more difficult than it was unless there had been any reference endpoint to look at. Kamil implemented a draft PR in which I took inspiration from. Thanks to this, It was easy for me to write the unit tests. It was also in this endpoint that I learned using parameterized in unit tests :D.

Implementing The Read-Only DagRuns Endpoints

This endpoint came with its many challenges, especially on filtering with datetimes. This was because the connexion library we were using to build the REST API was not validating date-time format in OpenAPI 3.0 specification, what I eventually found out, was intentional. Connexion dropped strict-rfc3339 because of the later license which is not compatible with Apache 2.0 license.

I implemented a workaround on this, by defining a function called conn_parse_datetime in the API utils module. This was later refactored and thankfully, Kamil implemented a decorator that allowed us to have cleaner code on the views while using this function.

Then we tried using rfc3339-validator whose license is compatible with Apache 2.0 licence but later discarded this because with our custom date parser we were able to use duration and not just date times.

Other Endpoints

I implemented some different other endpoints. One peculiar issue I faced was because of Marshmallow 2 not giving error when extra fields are in the request body. I implemented a validate_unknown method on the schema to handle this. Thankfully, Flask-AppBuilder updated to using Marshmallow 3, we quickly updated Flask-AppBuilder in Apache Airflow and started using Marshmallow 3 too.

Here are some PRs I contributed that are related to the REST API:

Add event log endpoints The event log would help users get information on operations performed at the UI
Add CRUD endpoints for connection This PR performs DELETE, PATCH and POST operations on Connection
Add log endpoint This PR enables users to get Task Instances log entries
Move limit & offset to kwargs in views plus work on a configurable maximum limit This helped us in having a neat code on the views and added configurable maximum limit on query results.
Update FlaskAppBuilder to v3 This enabled Airflow to start using v3 of Flask App Builder and also made it possible for the API to use a modern database serializer/deserializer
Add migration guide from the experimental REST API to the stable REST API This would enable users to start using the stable REST API in less time.

Follow-Ups

There is still lots of works to be done on the REST API including writing helpful documentation. I still follow up on these and hopefully, we will complete the REST API before the internship ends.

I am very grateful to my mentors, Jarek and Kaxil for their patience with me and for surviving my never-ending questions. Kamil and Tomek have been very supportive and I appreciate them for their support and amazing code reviews.

Thanks to Leah E. Cole and Karolina Rosół, for their wonderful reviews. I’m grateful.

Thanks for reading!

]]>

Apache Airflow 1.10.10

2020-04-09T00:00:00Z

Airflow 1.10.10 contains 199 commits since 1.10.9 and includes 11 new features, 43 improvements, 44 bug fixes, and several doc changes.

Details:

PyPI: https://pypi.org/project/apache-airflow/1.10.10/
Docs: https://airflow.apache.org/docs/1.10.10/
Changelog: http://airflow.apache.org/docs/1.10.10/changelog.html

Some of the noteworthy new features (user-facing) are:

Allow user to choose timezone to use in the RBAC UI

By default the Web UI will show times in UTC. It is possible to change the timezone shown by using the menu in the top right (click on the clock to activate it):

Screenshot:

Details: https://airflow.apache.org/docs/1.10.10/timezone.html#web-ui

Note: This feature is only available for the RBAC UI (enabled using rbac=True in [webserver] section in your airflow.cfg).

Add Production Docker image support

There are brand-new production images (alpha quality) available for Airflow 1.10.10. You can pull them from the Apache Airflow Dockerhub repository and start using it.

More information about using production images can be found in https://github.com/apache/airflow/blob/master/IMAGES.rst#using-the-images. Soon it will be updated with information how to use images using official helm chart.

To pull the images you can run one of the following commands:

docker pull apache/airflow:1.10.10-python2.7
docker pull apache/airflow:1.10.10-python3.5
docker pull apache/airflow:1.10.10-python3.6
docker pull apache/airflow:1.10.10-python3.7
docker pull apache/airflow:1.10.10 (uses Python 3.6)

Allow Retrieving Airflow Connections & Variables from various Secrets backend

From Airflow 1.10.10, users would be able to get Airflow Variables from Environment Variables.

Details: https://airflow.apache.org/docs/1.10.10/concepts.html#storing-variables-in-environment-variables

A new concept of Secrets Backend has been introduced to retrieve Airflow Connections and Variables.

From Airflow 1.10.10, users can retrieve Connections & Variables using the same syntax (no DAG code change is required), from a secret backend defined in airflow.cfg. If no backend is defined, Airflow falls-back to Environment Variables and then Metadata DB.

Check https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html#configuration for details on how-to configure Secrets backend.

As of 1.10.10, Airflow supports the following Secret Backends:

Hashicorp Vault
GCP Secrets Manager
AWS Parameters Store

Details: https://airflow.apache.org/docs/1.10.10/howto/use-alternative-secrets-backend.html

Example configuration to use Hashicorp Vault as the backend:

[secrets]
backend = airflow.contrib.secrets.hashicorp_vault.VaultBackend
backend_kwargs = {"url": "http://127.0.0.1:8200", "connections_path": "connections", "variables_path": "variables", "mount_point": "airflow"}

Stateless Webserver using DAG Serialization

The Webserver can now run without access to DAG Files when DAG Serialization is turned on. The 2 limitations we had in 1.10.7-1.10.9 ( https://airflow.apache.org/docs/1.10.7/dag-serialization.html#limitations) have been resolved.

The main advantage of this would be reduction in Webserver startup time for large number of DAGs. Without DAG Serialization all the DAGs are loaded in the DagBag during the Webserver startup.

With DAG Serialization, an empty DagBag is created and Dags are loaded from DB only when needed (i.e. when a particular DAG is clicked on in the home page)

Details: http://airflow.apache.org/docs/1.10.10/dag-serialization.html

Tasks using Dummy Operators are no longer sent to executor

The Dummy operators does not actually do any work and are mostly used for organizing/grouping tasks along with BranchPythonOperator.

Previously, when using Kubernetes Executor, the executor would spin up a whole worker pod to execute a dummy task. With Airflow 1.10.10 tasks using Dummy Operators would be scheduled & evaluated by the Scheduler but not sent to the Executor. This should significantly improve execution time and resource usage.

Allow passing DagRun conf when triggering dags via UI

When triggering a DAG from the CLI or the REST API, it s possible to pass configuration for the DAG run as a JSON blob.

From Airflow 1.10.10, when a user clicks on Trigger Dag button, a new screen confirming the trigger request, and allowing the user to pass a JSON configuration blob would be shown.

Screenshot:

Details: https://github.com/apache/airflow/pull/7312

Updating Guide

If you are updating Apache Airflow from a previous version to 1.10.10, please take a note of the following:

Run airflow upgradedb after pip install -U apache-airflow==1.10.10 as 1.10.10 contains 3 database migrations.
If you have used none_failed trigger rule in your DAG, change it to use the new none_failed_or_skipped trigger rule. As previously implemented, the actual behavior of none_failed trigger rule would skip the current task if all parents of the task had also skipped. This was not in-line with what was documented about that trigger rule. We have changed the implementation to match the documentation, hence if you need the old behavior use none_failed_or_skipped.

More details in https://github.com/apache/airflow/pull/7464.
Setting empty string to an Airflow Variable will now return an empty string, it previously returned None.

Example:
```
>> Variable.set('test_key', '')
>> Variable.get('test_key')
```
The above code returned None previously, now it will return ‘’.
When a task is marked as success by a user in Airflow UI, function defined in on_success_callback will be called.

Special Note / Deprecations

Python 2

Python 2 has reached end of its life on Jan 2020. Airflow Master no longer supports Python 2. Airflow 1.10.* would be the last series to support Python 2.

We strongly recommend users to use Python >= 3.6

Use Airflow RBAC UI

Airflow 1.10.10 ships with 2 UIs, the default is non-RBAC Flask-admin based UI and Flask-appbuilder based UI.

Flask-admin based UI is deprecated and new features won’t be ported to it. This UI will still be the default for 1.10.* series but would no longer be available from Airflow 2.0

Running Airflow on MacOS

Run export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES in your scheduler environmentIf you are running Airflow on MacOS and get the following error in the Scheduler logs:

objc[1873]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[1873]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

This error occurs because of added security to restrict multiprocessing & multithreading in Mac OS High Sierra and above.

We have moved to GitHub Issues

The Airflow Project has moved from JIRA to GitHub for tracking issues.

So if you find any bugs in Airflow 1.10.10 please create a GitHub Issue for it.

List of Contributors

According to git shortlog, the following people contributed to the 1.10.10 release. Thank you to all contributors!

ANiteckiP, Alex Guziel, Alex Lue, Anita Fronczak, Ash Berlin-Taylor, Benji Visser, Bhavika Tekwani, Brad Dettmer, Chris McLennon, Cooper Gillan, Daniel Imberman, Daniel Standish, Felix Uellendall, Jarek Potiuk, Jiajie Zhong, Jithin Sukumar, Kamil Breguła, Kaxil Naik, Kengo Seki, Kris, Kumpan Anton, Lokesh Lal, Louis Guitton, Louis Simoneau, Luyao Yang, Noël Bardelot, Omair Khan, Philipp Großelfinger, Ping Zhang, RasPavel, Ray, Robin Edwards, Ry Walker, Saurabh, Sebastian Brandt, Tomek Kzukowski, Tomek Urbaszek, Van-Duyet Le, Xiaodong Deng, Xinbin Huang, Yu Qian, Zacharya, atrbgithub, cong-zhu, retornam

]]>

Apache Airflow 1.10.8 & 1.10.9

2020-02-23T00:00:00Z

Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new features, 42 improvements, 36 bug fixes, and several doc changes.

We released 1.10.9 on the same day as one of the Flask dependencies (Werkzeug) released 1.0 which broke Airflow 1.10.8.

Details:

PyPI: https://pypi.org/project/apache-airflow/1.10.9/
Docs: https://airflow.apache.org/docs/1.10.9/
Changelog (1.10.8): http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07
Changelog (1.10.9): http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10

Some of the noteworthy new features (user-facing) are:

Add tags to DAGs and use it for filtering in the UI

In order to filter DAGs (e.g. by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button.

For example:

In your Dag file, pass a list of tags you want to add to DAG object:

dag = DAG(
    dag_id='example_dag_tag',
    schedule_interval='0 0 * * *',
    tags=['example']
)

Screenshot:

Note: This feature is only available for the RBAC UI (enabled using rbac=True in [webserver] section in your airflow.cfg).

Special Note / Deprecations

Python 2

Python 2 has reached end of its life on Jan 2020. Airflow Master no longer supports Python 2. Airflow 1.10.* would be the last series to support Python 2.

We strongly recommend users to use Python >= 3.6

Use Airflow RBAC UI

Airflow 1.10.9 ships with 2 UIs, the default is non-RBAC Flask-admin based UI and Flask-appbuilder based UI.

The Flask-AppBuilder (FAB) based UI is allowed Role-based Access Control and has more advanced features compared to the legacy Flask-admin based UI. This UI can be enabled by setting rbac=True in [webserver] section in your airflow.cfg.

Flask-admin based UI is deprecated and new features won’t be ported to it. This UI will still be the default for 1.10.* series but would no longer be available from Airflow 2.0

List of Contributors

According to git shortlog, the following people contributed to the 1.10.8 and 1.10.9 release. Thank you to all contributors!

Anita Fronczak, Ash Berlin-Taylor, BasPH, Bharat Kashyap, Bharath Palaksha, Bhavika Tekwani, Bjorn Olsen, Brian Phillips, Cooper Gillan, Daniel Cohen, Daniel Imberman, Daniel Standish, Gabriel Eckers, Hossein Torabi, Igor Khrol, Jacob, Jarek Potiuk, Jay, Jiajie Zhong, Jithin Sukumar, Kamil Breguła, Kaxil Naik, Kousuke Saruta, Mustafa Gök, Noël Bardelot, Oluwafemi Sule, Pete DeJoy, QP Hou, Qian Yu, Robin Edwards, Ry Walker, Steven van Rossum, Tomek Urbaszek, Xinbin Huang, Yuen-Kuei Hsueh, Yu Qian, Zacharya, ZxMYS, rconroy293, tooptoop4

]]>

Experience in Google Season of Docs 2019 with Apache Airflow

2019-12-20T00:00:00Z

I came across Google Season of Docs (GSoD) almost by accident, thanks to my extensive HackerNews and Twitter addiction. I was familiar with the Google Summer of Code but not with this program. It turns out it was the inaugural phase. I read the details, and the process felt a lot like GSoC except that this was about documentation.

About Me

I have been writing tech articles on medium as well as my blog for the past 1.5 years. Blogging helps me test my understanding of the concepts as untangling the toughest of ideas in simple sentences requires a considerable time investment.

Also, I have been working as a Software Developer for the past three years, which involves writing documentation for my projects as well. I completed my B.Tech from IIT Roorkee. During my stay in college, I applied for GSoC once but didn’t make it through in the final list of selected candidates.

I saw GSoD as an excellent opportunity to improve my technical writing skills using feedback from the open-source community. I contributed some bug fixes and features to Apache Superset and Apache Druid, but this would be my first contribution as a technical writer.

Searching for the organization

About 40+ organizations were participating in the GSoD. However, there were two which came as the right choice for me in the first instant. The first one was Apache Airflow because I had already used Airflow extensively and also contributed some custom operators inside the forked version of my previous company.

The second one was Apache Cassandra, on which I also had worked extensively but hadn’t done any code or doc changes.

Considering the total experience, I decided to go with the Airflow.

Project selection

After selecting the org, the next step was to choose the project. Again, my previous experience played a role here, and I ended up picking the How to create a workflow . The aim of the project was to write documentation which will help users in creating complex as well as custom DAGs. The final deliverables were a bit different, though. More on that later.

After submitting my application, I got involved in my job until one day, I saw a mail from Google confirming my selection as a Technical Writer for the project.

Community Bonding

Getting selected is just a beginning. I got the invite to the Airflow Slack channel where most of the discussions happened. My mentor was Ash-Berlin Taylor from Apache Airflow. I started talking to my mentor to get a general sense of what deliverables were expected. The deliverables were documented in confluence.

A page for how to create a DAG that also includes:
- Revamping the page related to scheduling a DAG
- Adding tips for specific DAG conditions, such as rerunning a failed task
A page for developing custom operators that includes:
- Describing mechanisms that are important when creating an operator, such as template fields, UI color, hooks, connection, etc.
- Describing the responsibility between the operator and the hook
- Considerations for dealing with shared resources (such as connections and hooks)
A page that describes how to define the relationships between tasks. The page should include information about:
- ** >> << **
- set upstream / set downstream
- helpers method ex. chain
A page that describes the communication between tasks that also includes:
- Revamping the page related to macros and XCOM

My mentor set the expectation early on that the deliverables were sort of like guidelines and not strict rules. If I wanted to, I could choose to work on something else related to the project also, which was not under deliverables. After connecting with the mentor, I started engaging with the overall Airflow community. The people in the community were helpful, especially Kamil Bregula. Kamil helped me in getting started with the guidelines to follow while writing the documentation for Airflow.

Doc Development

I picked DAG run as my first deliverable. I chose this topic as some parts of it were already documented but needed some additional text. I split the existing Scheduling & Triggers page into two new pages.

Schedulers
DAG Runs

Most of the details unrelated to schedulers were moved to DAG runs page, and then missing points such as how to re-run a task or DAG were added. Once I was satisfied with my version, I asked my mentor and Kamil to review it. For the first version, I shared the text in the Google Docs file in which the reviewers added comments. However, the document started getting messy, and it became difficult to track the changes. The time had come now to raise a proper Pull Request.

This was the time when I faced my first challenge. The documentation of Apache Airflow is written using RST(reStructuredText) syntax, with which I was entirely unfamiliar. I had mostly worked in Markdown. I spent the next couple of days understanding the syntax. Fortunately, it was quite easy to get acquainted. I raised the Pull Request and waited for the comments. Finally, after a few days when I saw the comments, they were mostly related to two things - grammar and formatting. There were also comments related to what I had missed or misinterpreted.

Using correct grammar

After discussing with Kamil, I decided to follow Google’s Developer Documentation Guidelines. These guidelines contain almost everything you’ll need to consider while writing good documentation, such as always to use active voice. Secondly, I installed the Grammarly app. After writing a doc, I used to put it in Grammarly to check for errors. Then I corrected the errors, made some more changes, and then again pushed it to Grammarly. This was an iterative process until I arrived with a version of the doc, which was grammatically correct but not seemed to have been written by an AI.

Formatting

Formatting involves writing notes and tips, marking the airflow components correctly in the text, and making sure a user who is skimming through the docs doesn’t miss the critical text. This required a bit of trial and error. I studied the current pattern in Airflow docs and made changes, pushed commits, incorporated new review comments, and then so on.

In the end, all the reviewers approved the PR, but it was not merged until two months later. This was because we doubted if some more pages, such as Concepts, should also be split up, resulting in a better-structured document. In the end, we decided to delay it until we discussed it with the broader community.

My second PR was a completely new document. It was related to How to create your custom operator. For this, since now I was familiar with most of the syntax, I directly raised the PR without going via Google Docs. I received a lot of comments again, but this time they were more related to what I had written rather than how I had written it. e.g., Describing in detail how to use template fields and clean up my code examples. The fewer grammatical & formatting error comments showed I had made progress. The PR was accepted within two weeks and gave me a huge confidence boost.

After my second PR, I was in a bit of a deadlock. My last remaining deliverable was related to Macros, but the scope wasn’t clear. I talked to my mentor, and he told me he didn’t mind if I can go off-track to work on something else while the community figured out what changes were needed. We discussed a lot of ideas. In the end, I decided to go with the Best Practices guide inspired by my mentors’ talk on Apache Airflow in a meetup. Having faced challenges while running Airflow in production myself, I was highly motivated to write something like this so that other developers don’t suffer. The first draft was ready within two weeks. I called it Running Airflow in Production. However, after adding a few more pieces to the document, I realized it was better to call it Best Practices guide, which most of the open-source projects contained.

People were enthusiastic about this pull request since a lot of them faced the challenges described in the doc. I had hit the nail on the head. After some deliberation over the next 1-2 weeks, my PR got accepted.

I then returned to my first PR and started making some changes related to the new review comments. After this, I discussed with my mentor about specific elements that were bugging him, such as getting people to understand how the schedule interval works in as few words as possible. After a lot of trial and error, we arrived at a version with which both of us could make peace.

Final Evaluation

On 12th September, I received mail from Google about the successful completion of the project. This meant my mentor liked my work. The Airflow community also appreciated the contributions.

My documents were finally published on Airflow website -

I also started getting invited in the PR reviews of other developers. I am looking forward to more contributions to the project in the coming year.

]]>

Airflow Survey 2019

2019-12-11T00:00:00Z

Apache Airflow Survey 2019

Apache Airflow is growing faster than ever. Thus, receiving and adjusting to our users’ feedback is a must. We created survey and we got 308 responses. Let’s see who Airflow users are, how they play with it, and what they miss.

Overview of the user

What best describes your current occupation?

	No.	%
Data Engineer	194	62.99%
Developer	34	11.04%
Architect	23	7.47%
Data Scientist	19	6.17%
Data Analyst	13	4.22%
DevOps	13	4.22%
IT Administrator	2	0.65%
Machine Learning Engineer	2	0.65%
Manager	2	0.65%
Operations	2	0.65%
Chief Data Officer	1	0.32%
Engineering Manager	1	0.32%
Intern	1	0.32%
Product owner	1	0.32%
Quant	1	0.32%

In your day to day job, what do you use Airflow for?

	No.	%
Data processing (ETL)	298	96.75%
Artificial Intelligence and Machine Learning Pipelines	90	29.22%
Automating DevOps operations	64	20.78%

According to the survey, most of the Airflow users are the “data” people. Moreover, 28.57% uses Airflow to both ETL and ML pipelines meaning that those two fields are somehow connected. Only five respondents use Airflow for DevOps operations only, That means that other 59 people who use Airflow for DevOps stuff use it also for ETL / ML purposes.

How many active DAGs do you have in your largest Airflow instance?

	No.	%
0-20	115	37.34%
21-40	65	21.10%
41-60	44	14.29%
61-100	28	9.09%
101-200	28	9.09%
201-300	7	2.27%
301-999	8	2.60%
1000+	13	4.22%

The majority of users do not exceed 100 active DAGs per Airflow instance. However, as we can see there are users who exceed thousands of DAGs with a maximum number 5000.

What is the maximum number of tasks that you have used in one DAG?

	No.	%
0-10	61	19.81%
11-20	60	19.48%
21-30	31	10.06%
31-40	21	6.82%
41-50	26	8.44%
51-100	36	11.69%
101-200	28	9.09%
201-500	21	6.82%
501+	24	11.54%

The given maximum number of tasks in a single DAG was 10 000 (!). The number of tasks depends on the purposes of a DAG, so it’s rather hard to say if users have “simple” or “complicated” workflows.

When onboarding new members to Airflow, what is the biggest problem?

	No.	%
No guide on best practises on developing DAGs	160	51.95%
Small number of tutorials on different aspects of using Airflow	57	18.51%
Documentation is not clear enough	42	13.64%
Small number of blogs regarding Airflow	6	1.95%
Other	43	13.96%

This is an important result. Using Airflow is all about writing and scheduling DAGs. No guide or any other complete resource on best practices for developing Dags is a big problem. Diving deep in the “other” answers, we can find that:

Airflow’s “magic” (scheduler, executors, schedule times) is hard to understand
DAG testing is not easy to do and to explain
Airflow UI needs some love.

How likely are you to recommend Apache Airflow?

	No.	%
Very Likely	140	45.45%
Likely	124	40.26%
Neutral	33	10.71%
Unlikely	8	2.60%
Very unlikely	3	0.97%

This means that more than 85% of people who use Airflow like it. It seems Airflow does its job nicely. However, we have to remember that this survey is likely biased - it’s more likely that you respond to the survey if you like the tool you use. Should we focus then on those 11 people who did not like Airflow? It’s a good question.

Airflow usage

Which interface(s) of Airflow do you use as part of your current role?

	No.	%
Original Airflow Graphical User Interface	297	96.43%
CLI	126	40.91%
Original Airflow Graphical User Interface, CLI	117	37.99%
API	60	19.48%
Original Airflow Graphical User Interface, CLI, API	32	10.39%
Custom (own created) Airflow Graphical User Interface	25	8.12%

It’s visible that usage of CLI goes in pair with using Airflow web UI. Our survey included some UX related questions to allow us to understand how users use Airflow webserver.

What do you use the Graphical User Interface for?

What do you use CLI for?

In Airflow, which UI view(s) are important for you?

Here we see that the majority uses Web UI mostly for monitoring purposes:

Monitoring DAGs
Accessing logs

An interesting result is that many people seem not to use backfilling as there’s no other way than to do it by CLI.

What executor type do you use?

	No.	%
Celery	138	44.81%
Local	85	27.60%
Kubernetes	52	16.88%
Sequential	22	7.14%
Other	11	3.57

The other option mostly consisted of information that someone uses a few types or is migrating from one executor to another. What can be observed is an increase in usage of Local and Kubernetes executors when compared to results from an earlier survey done by Ash.

Do you use Kubernetes-based deployments for Airflow?

	No.	%
No - we do not plan to use Kubernetes near term	88	28.57%
Yes - setup on our own via Helm Chart or similar	65	21.10%
Not yet - but we use Kubernetes in our organization and we could move	61	19.81%
Yes - via managed service in the cloud (Composer / Astronomer etc.)	45	14.61%
Not yet - but we plan to deploy Kubernetes in our organization soon	42	13.64%
Other	7	2.27%

The most interesting thing is that there’s nearly 30% of users who do not use Kubernetes, and they are not going to move. This means we should keep other deployment options in mind when working on Airflow 2.0. On the other hand, almost 70% of the users already use Kubernetes, or it’s a viable option for them.

Do you combine multiple DAGs?

	No.	%
No, I don’t combine multiple DAGs	127	41.23%
Yes, through SubDAG	73	23.70%
Yes, by triggering another DAG	72	23.38%
Other	36	11.69%

In the other category, 9 people explicitly mentioned using ExternalTaskSensor, and I think it could be treated as running subDAGs by triggering other DAGs.

Do you use Airflow Plugins? If yes, what do you use it for?

	No.	%
Adding new operators/sensors and hooks	187	60.71%
I don’t use Airflow plugins	109	35.39%
Adding AppBuilder views & menu items	31	10.06%
Adding new executor	18	5.84%
Adding OperatorExtraLinks	7	2.27%

The high percentage - 60% for “Adding new operators/sensors and hooks” is quite a surprising result for some of us - especially that you do not actually need to use the plugin mechanism to add any of those. Those are standard python objects, and you can simply drop your hooks/operators/sensors code to PYTHONPATH environment variable and they will work. It seems that this may be a result of a lack of best practices guide.

Plugins are more useful for adding views and menu items - yet only 10%. OperatorExtraLinks are even more useful (though relatively new) feature, so it’s not entirely surprising they are hardly used.

It was also kind of surprising that someone at all uses plugins to use their own executors. We considered removing that option recently - but now we have to rethink our approach.

What metrics do you use to monitor Airflow?

There were a lot of different responses. Some use Prometheus and other services, others do not use any monitoring. One of the interesting responses linked to this solution for airflow_operators_metrics.

External services

What external services do you use in your Airflow DAGs?

	No.	%
Amazon Web Services	160	51.95%
Internal company systems	150	48.7%
Hadoop / Spark / Flink / Other Apache software	119	38.64%
Google Cloud Platform / Google APIs	112	36.36%
Microsoft Azure	28	9.09%
I do not use external services in my Airflow DAGs	18	5.84%

It’s not surprising that Amazon Web Services is leading the way as they are considered the most mature cloud provider. Internal system and other Apache products on the next two positions are quite understandable if we take into account that the majority uses Airflow for ETL processes.

What external services do you use in your Airflow DAGs? (Mixed providers)

	No.	%
Google Cloud Platform / Google APIs, Amazon Web Services	44	14.29%
Amazon Web Services, Microsoft Azure	5	1.62%
Google Cloud Platform / Google APIs, Microsoft Azure	4	1.3%

This result is not surprising because companies usually prefer to stick with one cloud provider.

How do you integrate with external services?

	No.	%
Using Bash / Python operator	220	71.43%
Using existing, dedicated operators / hooks	217	70.45%
Using own, custom operators / hooks	216	70.13%

We had some anecdotal evidence that people use more Python/Bash operators than the dedicated ones - but it looks like all ways of using Airflow to connect to external services are equally popular.

What can be improved

In your opinion, what could be improved in Airflow?

	No.	%
Scheduler performance	189	61.36%
Web UI	180	58.44%
Logging, monitoring and alerting	145	47.08%
Examples, how-to, onboarding documentation	143	46.43%
Technical documentation	137	44.48%
Reliability	112	36.36%
REST API	96	31.17%
Authentication and authorization	89	28.9%
External integration e.g. AWS, GCP, Apache product	49	15.91%
CLI	41	13.31%
I don’t know	5	1.62%

The results are rather quite self-explaining. Improved performance of Airflow, better UI, and more telemetry are desirable. But this should go in pair with improved documentation and resources about using the Airflow, especially when we take into account the problem of onboarding new users.

Another interesting point from that question is that only 16% think that operators should be extended and improved. This suggests that we should focus on improving Airflow core instead of adding more and more integrations.

What would be the most interesting feature for you?

	No.	%
Production-ready Airflow docker image	175	56.82%
Declarative way of writing DAGs / automated DAGs generation	155	50.32%
Horizontal Autoscaling	122	39.61%
Asynchronous Operators	97	31.49%
Stateless web server	81	26.3%
Knative Executor	48	15.58%
I already have all I need	13	4.22%

Production Docker image wins, and it’s not a surprise. We all know that deploying Airflow is not a plug and play process, and that’s why the official image is being worked on by Jarek Potiuk. An unexpected result is that half of the users would like to have a declarative way of creating DAGs. That seems to be something that is “against Airflow” as we always emphasize the possibility of writing workflows in pure python. Stories about DAG generators are not new and confirm that there’s a need for a way to declare DAGs.

]]>

New Airflow website

2019-12-11T00:00:00Z

The brand new Airflow website has arrived! Those who have been following the process know that the journey to update the old Airflow website started at the beginning of the year. Thanks to sponsorship from the Cloud Composer team at Google that allowed us to collaborate with Polidea and with their design studio Utilo, and deliver an awesome website.

Documentation of open source projects is key to engaging new contributors in the maintenance, development, and adoption of software. We want the Apache Airflow community to have the best possible experience to contribute and use the project. We also took this opportunity to make the project more accessible, and in doing so, increase its reach.

In the past three and a half months, we have updated everything: created a more efficient landing page, enhanced information architecture, and improved UX & UI. Most importantly, the website now has capabilities to be translated into many languages. This is our effort to foster a more inclusive community around Apache Airflow, and we look forward to seeing contributions in Spanish, Chinese, Russian, and other languages as well!

We built our website on Docsy, a platform that is easy to use and contribute to. Follow these steps to set up your environment and to create your first pull request. You may also use the new website for your own open source project as a template. All of our code is open and hosted on GitHub.

Share your questions, comments, and suggestions with us, to help us improve the website. We hope that this new design makes finding documentation about Airflow easier, and that its improved accessibility increases adoption and use of Apache Airflow around the world.

Happy browsing!

]]>

ApacheCon Europe 2019 — Thoughts and Insights by Airflow Committers

2019-11-22T00:00:00Z

Is it possible to create an organization that delivers tens of projects used by millions, nearly no one is paid for doing their job, and still, it has been fruitfully carrying on for more than 20 years? Apache Software Foundation proves it is possible. For the last two decades, ASF has been crafting a model called the Apache Way—a way of organizing and leading tech open source projects. Due to this approach, which is strongly based on the “community over code” motto, we can enjoy such awesome projects like Apache Spark, Flink, Beam, or Airflow (and many more).

After this year’s ApacheCon, Polidea’s engineers talked with Committers of Apache projects, such as—Aizhamal Nurmamat kyzy, Felix Uellendall, and Fokko Driesprong—about insights to what makes the ASF such an amazing organization.

You can read the insights after the ApacheCon 2019.

]]>

Documenting using local development environment

2019-11-22T00:00:00Z

Documenting local development environment of Apache Airflow

From Sept to November 2019 I have been participating in a wonderful initiative, Google Season of Docs.

I had a pleasure to contribute to the Apache Airflow open source project as a technical writer. My initial assignment was an extension to the GitHub-based Contribution guide.

From the very first days I have been pretty closely involved into inter-project communications via emails/slack and had regular 1:1s with my mentor, Jarek Potiuk.

I got infected with Jarek’s enthusiasm to ease the on-boarding experience for Airflow contributors. I do share this strategy and did my best to improve the structure, language and DX. As a result, Jarek and I extended the current contributor’s docs and ended up with the Contributing guide navigating the users through the project infrastructure and providing a workflow example based on a real-life use case; the Testing guide with an overview of a complex testing infrastructure for Apache Airflow; and two guides dedicated to the Breeze dev environment and local virtual environment (my initial assignment).

I’m deeply grateful to my mentor and Airflow developers for their feedback, patience and help while I was breaking through new challenges (I’ve never worked on an open source project before), and for their support of all my ideas! I think a key success factor for any contributor is a responsive, supportive and motivated team, and I was lucky to join such a team for 3 months.

Documents I worked on:

]]>

It's a "Breeze" to develop Apache Airflow

2019-11-22T00:00:00Z

The story behind the Airflow Breeze tool

Initially, we started contributing to this fantastic open-source project [Apache Airflow] with a team of three which then grew to five. When we kicked it off a year ago, I realized pretty soon where the biggest bottlenecks and areas for improvement in terms of productivity were. Even with the help of our client, who provided us with a “homegrown” development environment it took us literally days to set it up and learn some basics.

That is how the journey to increased productivity in Apache Airflow began. The result? The Airflow Breeze open-source tool. Jarek Potiuk, an Airflow Committer, will tell you all about it.

You can learn how and why it’s a “Breeze” to Develop Apache Airflow.

]]>