• Home
• Core Concepts

Core Concepts

Here you can find detailed documentation about each one of the core concepts of Apache Airflow® and how to use them, as well as a high-level architectural overview.

Architecture

• Architecture Overview
  • Airflow components
  • Deploying Airflow components
  • Architecture Diagrams
  • Workloads
  • Control Flow
  • User interface

Workloads

• Dags
  • Declaring a DAG
  • Loading dags
  • Running dags
  • DAG Assignment
  • Default Arguments
  • The DAG decorator
  • Control Flow
  • Dynamic dags
  • DAG Visualization
  • DAG & Task Documentation
  • Packaging dags
  • .airflowignore
  • DAG Dependencies
  • DAG pausing, deactivation and deletion
  • DAG Auto-pausing (Experimental)
• DAG Runs
  • DAG Run Status
  • Re-run DAG
  • External Triggers
  • To Keep in Mind
• Tasks
  • Relationships
  • Task Instances
  • Timeouts
  • SLAs
  • Special Exceptions
  • Task Instance Heartbeat Timeout
  • Executor Configuration
• Operators
  • Jinja Templating
  • Reserved params keyword
• Sensors
• TaskFlow
  • Context
  • Logging
  • Passing Arbitrary Objects As Arguments
  • Sensors and the TaskFlow API
  • History
• Executor
  • Executor Types
  • Using Multiple Executors Concurrently
  • Writing Your Own Executor
• Auth manager
  • Available auth managers to use
  • Why pluggable auth managers?
  • Writing your own auth manager
  • Next Steps
• Object Storage
  • Cloud Object Stores are not real file systems
  • Basic Use
  • Configuration
  • Path API
  • Extensions
  • Copying and Moving
  • External Integrations
• Backfill
  • Control over data reprocessing
  • Concurrency control
  • Run ordering
  • Dry run
  • Example:

Communication

• XComs
  • Object Storage XCom Backend
  • Custom XCom Backends
  • Verifying Custom XCom Backend usage in Containers
• Variables
• Params
  • DAG-level Params
  • Task-level Params
  • Referencing Params in a Task
  • JSON Schema Validation
  • Use Params to Provide a Trigger UI Form
  • Disabling Runtime Param Modification

Debugging

• Debugging Airflow dags

Previous Next

Was this entry helpful?