Content
Command Line Interface and Environment Variables Reference¶
Command Line Interface¶
Airflow has a very rich command line interface that allows for many types of operation on a DAG, starting services, and supporting development and testing.
Note
For more information on usage CLI, see Using the Command Line Interface
Providers that implement executors might contribute additional commands to the CLI. Here are the commands contributed by the community providers:
Celery Executor and related CLI commands: Celery Executor Commands
Kubernetes Executor and related CLI commands: Kubernetes Executor Commands
AWS and related CLI commands: Amazon CLI Commands
The
users
androles
CLI commands are described in FAB provider documentation FAB CLI Commands
Usage: airflow [-h] GROUP_OR_COMMAND ...
Positional Arguments¶
- GROUP_OR_COMMAND
Possible choices: cheat-sheet, config, connections, dag-processor, dags, db, info, internal-api, jobs, kerberos, plugins, pools, providers, roles, rotate-fernet-key, scheduler, standalone, sync-perm, tasks, triggerer, users, variables, version, webserver
Sub-commands¶
cheat-sheet¶
Display cheat sheet
airflow cheat-sheet [-h] [-v]
Named Arguments¶
- -v, --verbose
Make logging output more verbose
Default: False
config¶
View configuration
airflow config [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: get-value, list
Sub-commands¶
get-value¶
Print the value of the configuration
airflow config get-value [-h] [-v] section option
- section
The section name
- option
The option name
- -v, --verbose
Make logging output more verbose
Default: False
list¶
List options for the configuration
airflow config list [-h] [--color {auto,off,on}] [-c] [-a] [-p] [-d] [-V] [-e]
[-s] [--section SECTION] [-v]
- --color
Possible choices: auto, off, on
Do emit colored output (default: auto)
Default: “auto”
- -c, --comment-out-everything
Comment out all configuration options. Useful as starting point for new installation
Default: False
- -a, --defaults
Show only defaults - do not include local configuration, sources, includes descriptions, examples, variables. Comment out everything.
Default: False
- -p, --exclude-providers
Exclude provider configuration (they are included by default)
Default: False
- -d, --include-descriptions
Show descriptions for the configuration variables
Default: False
- -V, --include-env-vars
Show environment variable for each option
Default: False
- -e, --include-examples
Show examples for the configuration variables
Default: False
- -s, --include-sources
Show source of the configuration variable
Default: False
- --section
The section name
- -v, --verbose
Make logging output more verbose
Default: False
connections¶
Manage connections
airflow connections [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: add, create-default-connections, delete, export, get, import, list, test
Sub-commands¶
add¶
Add a connection
airflow connections add [-h] [--conn-description CONN_DESCRIPTION]
[--conn-extra CONN_EXTRA] [--conn-host CONN_HOST]
[--conn-json CONN_JSON] [--conn-login CONN_LOGIN]
[--conn-password CONN_PASSWORD]
[--conn-port CONN_PORT] [--conn-schema CONN_SCHEMA]
[--conn-type CONN_TYPE] [--conn-uri CONN_URI]
conn_id
- conn_id
Connection id, required to get/add/delete/test a connection
- --conn-description
Connection description, optional when adding a connection
- --conn-extra
Connection Extra field, optional when adding a connection
- --conn-host
Connection host, optional when adding a connection
- --conn-json
Connection JSON, required to add a connection using JSON representation
- --conn-login
Connection login, optional when adding a connection
- --conn-password
Connection password, optional when adding a connection
- --conn-port
Connection port, optional when adding a connection
- --conn-schema
Connection schema, optional when adding a connection
- --conn-type
Connection type, required to add a connection without conn_uri
- --conn-uri
Connection URI, required to add a connection without conn_type
create-default-connections¶
Creates all the default connections from all the providers
airflow connections create-default-connections [-h] [-v]
- -v, --verbose
Make logging output more verbose
Default: False
delete¶
Delete a connection
airflow connections delete [-h] [--color {auto,off,on}] [-v] conn_id
- conn_id
Connection id, required to get/add/delete/test a connection
- --color
Possible choices: auto, off, on
Do emit colored output (default: auto)
Default: “auto”
- -v, --verbose
Make logging output more verbose
Default: False
export¶
All connections can be exported in STDOUT using the following command: airflow connections export - The file format can be determined by the provided file extension. E.g., The following command will export the connections in JSON format: airflow connections export /tmp/connections.json The –file-format parameter can be used to control the file format. E.g., the default format is JSON in STDOUT mode, which can be overridden using: airflow connections export - –file-format yaml The –file-format parameter can also be used for the files, for example: airflow connections export /tmp/connections –file-format json. When exporting in env file format, you control whether URI format or JSON format is used to serialize the connection by passing uri or json with option –serialization-format.
airflow connections export [-h] [--file-format {json,yaml,env}]
[--format {json,yaml,env}]
[--serialization-format {json,uri}] [-v]
file
- file
Output file path for exporting the connections
- --file-format
Possible choices: json, yaml, env
File format for the export
- --format
Possible choices: json, yaml, env
Deprecated – use –file-format instead. File format to use for the export.
- --serialization-format
Possible choices: json, uri
When exporting as .env format, defines how connections should be serialized. Default is uri.
- -v, --verbose
Make logging output more verbose
Default: False
get¶
Get a connection
airflow connections get [-h] [--color {auto,off,on}]
[-o table, json, yaml, plain] [-v]
conn_id
- conn_id
Connection id, required to get/add/delete/test a connection
- --color
Possible choices: auto, off, on
Do emit colored output (default: auto)
Default: “auto”
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
import¶
Connections can be imported from the output of the export command. The filetype must by json, yaml or env and will be automatically inferred.
airflow connections import [-h] [--overwrite] [-v] file
- file
Import connections from a file
- --overwrite
Overwrite existing entries if a conflict occurs
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
list¶
List connections
airflow connections list [-h] [--conn-id CONN_ID]
[-o table, json, yaml, plain] [-v]
- --conn-id
If passed, only items with the specified connection ID will be displayed
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
dag-processor¶
Start a standalone Dag Processor instance
airflow dag-processor [-h] [-D] [-p] [-l LOG_FILE] [-n NUM_RUNS] [--pid [PID]]
[--stderr STDERR] [--stdout STDOUT] [-S SUBDIR] [-v]
Named Arguments¶
- -D, --daemon
Daemonize instead of running in the foreground
Default: False
- -p, --do-pickle
Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code
Default: False
- -l, --log-file
Location of the log file
- -n, --num-runs
Set the number of runs to execute before exiting
Default: -1
- --pid
PID file location
- --stderr
Redirect stderr to this file
- --stdout
Redirect stdout to this file
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
dags¶
Manage DAGs
airflow dags [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: backfill, delete, details, list, list-import-errors, list-jobs, list-runs, next-execution, pause, report, reserialize, show, show-dependencies, state, test, trigger, unpause
Sub-commands¶
backfill¶
Run subsections of a DAG for a specified date range. If reset_dag_run option is used, backfill will first prompt users whether airflow should clear all the previous dag_run and task_instances within the backfill date range. If rerun_failed_tasks is used, backfill will auto re-run the previous failed task instances within the backfill date range
airflow dags backfill [-h] [-c CONF] [--continue-on-failures]
[--delay-on-limit DELAY_ON_LIMIT] [--disable-retry] [-x]
[-n] [-e END_DATE] [-i] [-I] [-l] [-m] [--pool POOL]
[--rerun-failed-tasks] [--reset-dagruns] [-B]
[-s START_DATE] [-S SUBDIR] [-t TASK_REGEX]
[--treat-dag-as-regex] [--treat-dag-id-as-regex] [-v]
[-y]
dag_id
- dag_id
The id of the dag
- -c, --conf
JSON string that gets pickled into the DagRun’s conf attribute
- --continue-on-failures
if set, the backfill will keep going even if some of the tasks failed
Default: False
- --delay-on-limit
Amount of time in seconds to wait when the limit on maximum active dag runs (max_active_runs) has been reached before trying to execute a dag run again
Default: 1.0
- --disable-retry
if set, the backfill will set tasks as failed without retrying.
Default: False
- -x, --donot-pickle
Do not attempt to pickle the DAG object to send over to the workers, just tell the workers to run their version of the code
Default: False
- -n, --dry-run
Perform a dry run for each task. Only renders Template Fields for each task, nothing else
Default: False
- -e, --end-date
Override end_date YYYY-MM-DD
- -i, --ignore-dependencies
Skip upstream tasks, run only the tasks matching the regexp. Only works in conjunction with task_regex
Default: False
- -I, --ignore-first-depends-on-past
Ignores depends_on_past dependencies for the first set of tasks only (subsequent executions in the backfill DO respect depends_on_past)
Default: False
- -l, --local
Run the task using the LocalExecutor
Default: False
- -m, --mark-success
Mark jobs as succeeded without running them
Default: False
- --pool
Resource pool to use
- --rerun-failed-tasks
if set, the backfill will auto-rerun all the failed tasks for the backfill date range instead of throwing exceptions
Default: False
- --reset-dagruns
if set, the backfill will delete existing backfill-related DAG runs and start anew with fresh, running DAG runs
Default: False
- -B, --run-backwards
if set, the backfill will run tasks from the most recent day first. if there are tasks that depend_on_past this option will throw an exception
Default: False
- -s, --start-date
Override start_date YYYY-MM-DD
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -t, --task-regex
The regex to filter specific task_ids (optional)
- --treat-dag-as-regex
Deprecated – use –treat-dag-id-as-regex instead
Default: False
- --treat-dag-id-as-regex
if set, dag_id will be treated as regex instead of an exact string
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
delete¶
Delete all DB records related to the specified DAG
airflow dags delete [-h] [-v] [-y] dag_id
- dag_id
The id of the dag
- -v, --verbose
Make logging output more verbose
Default: False
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
details¶
Get DAG details given a DAG id
airflow dags details [-h] [-o table, json, yaml, plain] [-v] dag_id
- dag_id
The id of the dag
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
list¶
List all the DAGs
airflow dags list [-h] [--columns COLUMNS] [-o table, json, yaml, plain]
[-S SUBDIR] [-v]
- --columns
List of columns to render. (default: [‘dag_id’, ‘fileloc’, ‘owner’, ‘is_paused’])
Default: (‘dag_id’, ‘fileloc’, ‘owners’, ‘is_paused’)
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
list-import-errors¶
List all the DAGs that have import errors
airflow dags list-import-errors [-h] [-o table, json, yaml, plain] [-S SUBDIR]
[-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
list-jobs¶
List the jobs
airflow dags list-jobs [-h] [-d DAG_ID] [--limit LIMIT]
[-o table, json, yaml, plain]
[--state running, success, restarting, failed] [-v]
- -d, --dag-id
The id of the dag
- --limit
Return a limited number of records
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- --state
Possible choices: running, success, restarting, failed
Only list the jobs corresponding to the state
- -v, --verbose
Make logging output more verbose
Default: False
list-runs¶
List DAG runs given a DAG id. If state option is given, it will only search for all the dagruns with the given state. If no_backfill option is given, it will filter out all backfill dagruns for given dag id. If start_date is given, it will filter out all the dagruns that were executed before this date. If end_date is given, it will filter out all the dagruns that were executed after this date.
airflow dags list-runs [-h] -d DAG_ID [-e END_DATE] [--no-backfill]
[-o table, json, yaml, plain] [-s START_DATE]
[--state queued, running, success, failed] [-v]
- -d, --dag-id
The id of the dag
- -e, --end-date
Override end_date YYYY-MM-DD
- --no-backfill
filter all the backfill dagruns given the dag id
Default: False
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -s, --start-date
Override start_date YYYY-MM-DD
- --state
Possible choices: queued, running, success, failed
Only list the DAG runs corresponding to the state
- -v, --verbose
Make logging output more verbose
Default: False
next-execution¶
Get the next execution datetimes of a DAG. It returns one execution unless the num-executions option is given
airflow dags next-execution [-h] [-n NUM_EXECUTIONS] [-S SUBDIR] [-v] dag_id
- dag_id
The id of the dag
- -n, --num-executions
The number of next execution datetimes to show
Default: 1
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
pause¶
Pause one or more DAGs. This command allows to halt the execution of specified DAGs, disabling further task scheduling. Use –treat-dag-id-as-regex to target multiple DAGs by treating the –dag-id as a regex pattern.
airflow dags pause [-h] [-o table, json, yaml, plain] [-S SUBDIR]
[--treat-dag-id-as-regex] [-v] [-y]
dag_id
- dag_id
The id of the dag
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- --treat-dag-id-as-regex
if set, dag_id will be treated as regex instead of an exact string
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
report¶
Show DagBag loading report
airflow dags report [-h] [-o table, json, yaml, plain] [-S SUBDIR] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
reserialize¶
Drop all serialized dags from the metadata DB. This will cause all DAGs to be reserialized from the DagBag folder. This can be helpful if your serialized DAGs get out of sync with the version of Airflow that you are running.
airflow dags reserialize [-h] [--clear-only] [-S SUBDIR] [-v]
- --clear-only
If passed, serialized DAGs will be cleared but not reserialized.
Default: False
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
show¶
The –imgcat option only works in iTerm.
For more information, see: https://www.iterm2.com/documentation-images.html
The –save option saves the result to the indicated file.
The file format is determined by the file extension. For more information about supported format, see: https://www.graphviz.org/doc/info/output.html
If you want to create a PNG file then you should execute the following command: airflow dags show <DAG_ID> –save output.png
If you want to create a DOT file then you should execute the following command: airflow dags show <DAG_ID> –save output.dot
airflow dags show [-h] [--imgcat] [-s SAVE] [-S SUBDIR] [-v] dag_id
- dag_id
The id of the dag
- --imgcat
Displays graph using the imgcat tool.
Default: False
- -s, --save
Saves the result to the indicated file.
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
show-dependencies¶
The –imgcat option only works in iTerm.
For more information, see: https://www.iterm2.com/documentation-images.html
The –save option saves the result to the indicated file.
The file format is determined by the file extension. For more information about supported format, see: https://www.graphviz.org/doc/info/output.html
If you want to create a PNG file then you should execute the following command: airflow dags show-dependencies –save output.png
If you want to create a DOT file then you should execute the following command: airflow dags show-dependencies –save output.dot
airflow dags show-dependencies [-h] [--imgcat] [-s SAVE] [-S SUBDIR] [-v]
- --imgcat
Displays graph using the imgcat tool.
Default: False
- -s, --save
Saves the result to the indicated file.
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
state¶
Get the status of a dag run
airflow dags state [-h] [-S SUBDIR] [-v] dag_id execution_date
- dag_id
The id of the dag
- execution_date
The execution date of the DAG
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
test¶
Execute one single DagRun for a given DAG and execution date.
The –imgcat-dagrun option only works in iTerm.
For more information, see: https://www.iterm2.com/documentation-images.html
If –save-dagrun is used, then, after completing the backfill, saves the diagram for current DAG Run to the indicated file. The file format is determined by the file extension. For more information about supported format, see: https://www.graphviz.org/doc/info/output.html
If you want to create a PNG file then you should execute the following command: airflow dags test <DAG_ID> <EXECUTION_DATE> –save-dagrun output.png
If you want to create a DOT file then you should execute the following command: airflow dags test <DAG_ID> <EXECUTION_DATE> –save-dagrun output.dot
airflow dags test [-h] [-c CONF] [--imgcat-dagrun]
[--mark-success-pattern MARK_SUCCESS_PATTERN]
[--save-dagrun SAVE_DAGRUN] [--show-dagrun] [-S SUBDIR]
[--use-executor] [-v]
dag_id [execution_date]
- dag_id
The id of the dag
- execution_date
The execution date of the DAG (optional)
- -c, --conf
JSON string that gets pickled into the DagRun’s conf attribute
- --imgcat-dagrun
After completing the dag run, prints a diagram on the screen for the current DAG Run using the imgcat tool.
Default: False
- --mark-success-pattern
Don’t run task_ids matching the regex <MARK_SUCCESS_PATTERN>, mark them as successful instead. Can be used to skip e.g. dependency check sensors or cleanup steps in local testing.
- --save-dagrun
After completing the backfill, saves the diagram for current DAG Run to the indicated file.
- --show-dagrun
After completing the backfill, shows the diagram for current DAG Run.
The diagram is in DOT language
Default: False
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- --use-executor
Use an executor to test the DAG. By default it runs the DAG without an executor. If set, it uses the executor configured in the environment.
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
trigger¶
Trigger a new DAG run. If DAG is paused then dagrun state will remain queued, and the task won’t run.
airflow dags trigger [-h] [-c CONF] [-e EXEC_DATE] [--no-replace-microseconds]
[-o table, json, yaml, plain] [-r RUN_ID] [-S SUBDIR]
[-v]
dag_id
- dag_id
The id of the dag
- -c, --conf
JSON string that gets pickled into the DagRun’s conf attribute
- -e, --exec-date
The execution date of the DAG
- --no-replace-microseconds
whether microseconds should be zeroed
Default: True
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -r, --run-id
Helps to identify this run
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
unpause¶
Resume one or more DAGs. This command allows to restore the execution of specified DAGs, enabling further task scheduling. Use –treat-dag-id-as-regex to target multiple DAGs treating the –dag-id as a regex pattern.
airflow dags unpause [-h] [-o table, json, yaml, plain] [-S SUBDIR]
[--treat-dag-id-as-regex] [-v] [-y]
dag_id
- dag_id
The id of the dag
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- --treat-dag-id-as-regex
if set, dag_id will be treated as regex instead of an exact string
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
db¶
Database operations
airflow db [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: check, check-migrations, clean, downgrade, drop-archived, export-archived, init, migrate, reset, shell, upgrade
Sub-commands¶
check¶
Check if the database can be reached
airflow db check [-h] [--retry RETRY] [--retry-delay RETRY_DELAY] [-v]
- --retry
Retry database check upon failure
Default: 0
- --retry-delay
Wait time between retries in seconds
Default: 1
- -v, --verbose
Make logging output more verbose
Default: False
check-migrations¶
Check if migration have finished (or continually check until timeout)
airflow db check-migrations [-h] [-t MIGRATION_WAIT_TIMEOUT] [-v]
- -t, --migration-wait-timeout
timeout to wait for db to migrate
Default: 60
- -v, --verbose
Make logging output more verbose
Default: False
clean¶
Purge old records in metastore tables
airflow db clean [-h] --clean-before-timestamp CLEAN_BEFORE_TIMESTAMP
[--dry-run] [--skip-archive] [-t TABLES] [-v] [-y]
- --clean-before-timestamp
The date or timestamp before which data should be purged. If no timezone info is supplied then dates are assumed to be in airflow default timezone. Example: ‘2022-01-01 00:00:00+01:00’
- --dry-run
Perform a dry run
Default: False
- --skip-archive
Don’t preserve purged records in an archive table.
Default: False
- -t, --tables
Table names to perform maintenance on (use comma-separated list). Options: [‘callback_request’, ‘celery_taskmeta’, ‘celery_tasksetmeta’, ‘dag’, ‘dag_run’, ‘dataset_event’, ‘import_error’, ‘job’, ‘log’, ‘session’, ‘sla_miss’, ‘task_fail’, ‘task_instance’, ‘task_instance_history’, ‘task_reschedule’, ‘trigger’, ‘xcom’]
- -v, --verbose
Make logging output more verbose
Default: False
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
downgrade¶
Downgrade the schema of the metadata database. You must provide either –to-revision or –to-version. To print but not execute commands, use option –show-sql-only. If using options –from-revision or –from-version, you must also use –show-sql-only, because if actually running migrations, we should only migrate from the current Alembic revision.
airflow db downgrade [-h] [--from-revision FROM_REVISION]
[--from-version FROM_VERSION] [-s] [-r TO_REVISION]
[-n TO_VERSION] [-v] [-y]
- --from-revision
(Optional) If generating sql, may supply a from Alembic revision
- --from-version
(Optional) If generating sql, may supply a from version
- -s, --show-sql-only
Don’t actually run migrations; just print out sql scripts for offline migration. Required if using either –from-revision or –from-version.
Default: False
- -r, --to-revision
The Alembic revision to downgrade to. Note: must provide either –to-revision or –to-version.
- -n, --to-version
(Optional) If provided, only run migrations up to this version.
- -v, --verbose
Make logging output more verbose
Default: False
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
drop-archived¶
Drop archived tables created through the db clean command
airflow db drop-archived [-h] [-t TABLES] [-y]
- -t, --tables
Table names to perform maintenance on (use comma-separated list). Options: [‘callback_request’, ‘celery_taskmeta’, ‘celery_tasksetmeta’, ‘dag’, ‘dag_run’, ‘dataset_event’, ‘import_error’, ‘job’, ‘log’, ‘session’, ‘sla_miss’, ‘task_fail’, ‘task_instance’, ‘task_instance_history’, ‘task_reschedule’, ‘trigger’, ‘xcom’]
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
export-archived¶
Export archived data from the archive tables
airflow db export-archived [-h] [--drop-archives] [--export-format {csv}]
--output-path DIRPATH [-t TABLES] [-y]
- --drop-archives
Drop the archive tables after exporting. Use with caution.
Default: False
- --export-format
Possible choices: csv
The file format to export the cleaned data
Default: “csv”
- --output-path
The path to the output directory to export the cleaned data. This directory must exist.
- -t, --tables
Table names to perform maintenance on (use comma-separated list). Options: [‘callback_request’, ‘celery_taskmeta’, ‘celery_tasksetmeta’, ‘dag’, ‘dag_run’, ‘dataset_event’, ‘import_error’, ‘job’, ‘log’, ‘session’, ‘sla_miss’, ‘task_fail’, ‘task_instance’, ‘task_instance_history’, ‘task_reschedule’, ‘trigger’, ‘xcom’]
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
init¶
Undocumented
airflow db init [-h] [-v]
- -v, --verbose
Make logging output more verbose
Default: False
migrate¶
Migrate the schema of the metadata database. Create the database if it does not exist To print but not execute commands, use option --show-sql-only
. If using options --from-revision
or --from-version
, you must also use --show-sql-only
, because if actually running migrations, we should only migrate from the current Alembic revision.
airflow db migrate [-h] [--from-revision FROM_REVISION]
[--from-version FROM_VERSION] [-s] [-r TO_REVISION]
[-n TO_VERSION] [-m] [-v]
- --from-revision
(Optional) If generating sql, may supply a from Alembic revision
- --from-version
(Optional) If generating sql, may supply a from version
- -s, --show-sql-only
Don’t actually run migrations; just print out sql scripts for offline migration. Required if using either –from-revision or –from-version.
Default: False
- -r, --to-revision
(Optional) If provided, only run migrations up to and including this Alembic revision.
- -n, --to-version
(Optional) The airflow version to upgrade to. Note: must provide either –to-revision or –to-version.
- -m, --use-migration-files
Use migration files to perform migration
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
reset¶
Burn down and rebuild the metadata database
airflow db reset [-h] [-s] [-m] [-v] [-y]
- -s, --skip-init
Only remove tables; do not perform db init.
Default: False
- -m, --use-migration-files
Use migration files to perform migration
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
shell¶
Runs a shell to access the database
airflow db shell [-h] [-v]
- -v, --verbose
Make logging output more verbose
Default: False
upgrade¶
Undocumented
airflow db upgrade [-h] [--from-revision FROM_REVISION]
[--from-version FROM_VERSION] [-s] [-r TO_REVISION]
[-n TO_VERSION] [-m] [-v]
- --from-revision
(Optional) If generating sql, may supply a from Alembic revision
- --from-version
(Optional) If generating sql, may supply a from version
- -s, --show-sql-only
Don’t actually run migrations; just print out sql scripts for offline migration. Required if using either –from-revision or –from-version.
Default: False
- -r, --to-revision
(Optional) If provided, only run migrations up to and including this Alembic revision.
- -n, --to-version
(Optional) The airflow version to upgrade to. Note: must provide either –to-revision or –to-version.
- -m, --use-migration-files
Use migration files to perform migration
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
info¶
Show information about current Airflow and environment
airflow info [-h] [--anonymize] [--file-io] [-o table, json, yaml, plain] [-v]
Named Arguments¶
- --anonymize
Minimize any personal identifiable information. Use it when sharing output with others.
Default: False
- --file-io
Send output to file.io service and returns link.
Default: False
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
internal-api¶
Start a Airflow Internal API instance
airflow internal-api [-h] [-A ACCESS_LOGFILE] [-L ACCESS_LOGFORMAT] [-D] [-d]
[-E ERROR_LOGFILE] [-H HOSTNAME] [-l LOG_FILE]
[--pid [PID]] [-p PORT] [--ssl-cert SSL_CERT]
[--ssl-key SSL_KEY] [--stderr STDERR] [--stdout STDOUT]
[-t WORKER_TIMEOUT] [-k {sync,eventlet,gevent,tornado}]
[-w WORKERS]
Named Arguments¶
- -A, --access-logfile
The logfile to store the access log. Use ‘-‘ to print to stdout
- -L, --access-logformat
The access log format for gunicorn logs
- -D, --daemon
Daemonize instead of running in the foreground
Default: False
- -d, --debug
Use the server that ships with Flask in debug mode
Default: False
- -E, --error-logfile
The logfile to store the error log. Use ‘-‘ to print to stderr
- -H, --hostname
Set the hostname on which to run the web server
Default: “0.0.0.0”
- -l, --log-file
Location of the log file
- --pid
PID file location
- -p, --port
The port on which to run the server
Default: 9080
- --ssl-cert
Path to the SSL certificate for the webserver
Default: “”
- --ssl-key
Path to the key to use with the SSL certificate
Default: “”
- --stderr
Redirect stderr to this file
- --stdout
Redirect stdout to this file
- -t, --worker-timeout
The timeout for waiting on Internal API workers
Default: 120
- -k, --workerclass
Possible choices: sync, eventlet, gevent, tornado
The worker class to use for Gunicorn
Default: “sync”
- -w, --workers
Number of workers to run the Internal API-on
Default: 4
jobs¶
Manage jobs
airflow jobs [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: check
Sub-commands¶
check¶
Checks if job(s) are still alive
airflow jobs check [-h] [--allow-multiple] [--hostname HOSTNAME]
[--job-type {BackfillJob,LocalTaskJob,SchedulerJob,TriggererJob,DagProcessorJob}]
[--limit LIMIT] [--local] [-v]
- --allow-multiple
If passed, this command will be successful even if multiple matching alive jobs are found.
Default: False
- --hostname
The hostname of job(s) that will be checked.
- --job-type
Possible choices: BackfillJob, LocalTaskJob, SchedulerJob, TriggererJob, DagProcessorJob
The type of job(s) that will be checked.
- --limit
The number of recent jobs that will be checked. To disable limit, set 0.
Default: 1
- --local
If passed, this command will only show jobs from the local host (those with a hostname matching what hostname_callable returns).
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
examples: To check if the local scheduler is still working properly, run:
$ airflow jobs check –job-type SchedulerJob –local”
To check if any scheduler is running when you are using high availability, run:
$ airflow jobs check –job-type SchedulerJob –allow-multiple –limit 100
kerberos¶
Start a kerberos ticket renewer
airflow kerberos [-h] [-D] [-k [KEYTAB]] [-l LOG_FILE] [-o] [--pid [PID]]
[--stderr STDERR] [--stdout STDOUT] [-v]
[principal]
Positional Arguments¶
- principal
kerberos principal
Named Arguments¶
- -D, --daemon
Daemonize instead of running in the foreground
Default: False
- -k, --keytab
keytab
Default: “airflow.keytab”
- -l, --log-file
Location of the log file
- -o, --one-time
Run airflow kerberos one time instead of forever
Default: False
- --pid
PID file location
- --stderr
Redirect stderr to this file
- --stdout
Redirect stdout to this file
- -v, --verbose
Make logging output more verbose
Default: False
plugins¶
Dump information about loaded plugins
airflow plugins [-h] [-o table, json, yaml, plain] [-v]
Named Arguments¶
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
pools¶
Manage pools
airflow pools [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: delete, export, get, import, list, set
Sub-commands¶
delete¶
Delete pool
airflow pools delete [-h] [-o table, json, yaml, plain] [-v] NAME
- NAME
Pool name
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
export¶
Export all pools
airflow pools export [-h] [-v] FILEPATH
- FILEPATH
Export all pools to JSON file
- -v, --verbose
Make logging output more verbose
Default: False
get¶
Get pool size
airflow pools get [-h] [-o table, json, yaml, plain] [-v] NAME
- NAME
Pool name
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
import¶
Import pools
airflow pools import [-h] [-v] FILEPATH
- FILEPATH
Import pools from JSON file. Example format:
{ "pool_1": {"slots": 5, "description": "", "include_deferred": true}, "pool_2": {"slots": 10, "description": "test", "include_deferred": false} }
- -v, --verbose
Make logging output more verbose
Default: False
list¶
List pools
airflow pools list [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
set¶
Configure pool
airflow pools set [-h] [--include-deferred] [-o table, json, yaml, plain] [-v]
NAME slots description
- NAME
Pool name
- slots
Pool slots
- description
Pool description
- --include-deferred
Include deferred tasks in calculations for Pool
Default: False
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
providers¶
Display providers
airflow providers [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: auth, auth-managers, behaviours, configs, executors, get, hooks, lazy-loaded, links, list, logging, notifications, secrets, triggers, widgets
Sub-commands¶
auth¶
Get information about API auth backends provided
airflow providers auth [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
auth-managers¶
Get information about auth managers provided
airflow providers auth-managers [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
behaviours¶
Get information about registered connection types with custom behaviours
airflow providers behaviours [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
configs¶
Get information about provider configuration
airflow providers configs [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
executors¶
Get information about executors provided
airflow providers executors [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
get¶
Get detailed information about a provider
airflow providers get [-h] [--color {auto,off,on}] [-f]
[-o table, json, yaml, plain] [-v]
provider_name
- provider_name
Provider name, required to get provider information
- --color
Possible choices: auto, off, on
Do emit colored output (default: auto)
Default: “auto”
- -f, --full
Full information about the provider, including documentation information.
Default: False
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
hooks¶
List registered provider hooks
airflow providers hooks [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
lazy-loaded¶
Checks that provider configuration is lazy loaded
airflow providers lazy-loaded [-h] [-v]
- -v, --verbose
Make logging output more verbose
Default: False
links¶
List extra links registered by the providers
airflow providers links [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
list¶
List installed providers
airflow providers list [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
logging¶
Get information about task logging handlers provided
airflow providers logging [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
notifications¶
Get information about notifications provided
airflow providers notifications [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
secrets¶
Get information about secrets backends provided
airflow providers secrets [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
triggers¶
List registered provider triggers
airflow providers triggers [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
widgets¶
Get information about registered connection form widgets
airflow providers widgets [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
roles¶
Manage roles
airflow roles [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: add-perms, create, del-perms, delete, export, import, list
Sub-commands¶
add-perms¶
Add roles permissions
airflow roles add-perms [-h] -a [ACTION [ACTION ...]] -r
[RESOURCE [RESOURCE ...]] [-v]
[role [role ...]]
- role
The name of a role
- -a, --action
The action of permissions
- -r, --resource
The name of permissions
- -v, --verbose
Make logging output more verbose
Default: False
create¶
Create role
airflow roles create [-h] [-v] [role [role ...]]
- role
The name of a role
- -v, --verbose
Make logging output more verbose
Default: False
del-perms¶
Delete roles permissions
airflow roles del-perms [-h] [-a [ACTION [ACTION ...]]] -r
[RESOURCE [RESOURCE ...]] [-v]
[role [role ...]]
- role
The name of a role
- -a, --action
The action of permissions
- -r, --resource
The name of permissions
- -v, --verbose
Make logging output more verbose
Default: False
delete¶
Delete role
airflow roles delete [-h] [-v] [role [role ...]]
- role
The name of a role
- -v, --verbose
Make logging output more verbose
Default: False
export¶
Export roles (without permissions) from db to JSON file
airflow roles export [-h] [-p] [-v] file
- file
Export all roles to JSON file
- -p, --pretty
Format output JSON file by sorting role names and indenting by 4 spaces
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
import¶
Import roles (without permissions) from JSON file to db
airflow roles import [-h] [-v] file
- file
Import roles from JSON file
- -v, --verbose
Make logging output more verbose
Default: False
list¶
List roles
airflow roles list [-h] [-o table, json, yaml, plain] [-p] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -p, --permission
Show role permissions
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
rotate-fernet-key¶
Rotate all encrypted connection credentials and variables; see https://airflow.apache.org/docs/apache-airflow/stable/howto/secure-connections.html#rotating-encryption-keys
airflow rotate-fernet-key [-h]
scheduler¶
Start a scheduler instance
airflow scheduler [-h] [-D] [-p] [-l LOG_FILE] [-n NUM_RUNS] [--pid [PID]]
[-s] [--stderr STDERR] [--stdout STDOUT] [-S SUBDIR] [-v]
Named Arguments¶
- -D, --daemon
Daemonize instead of running in the foreground
Default: False
- -p, --do-pickle
Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code
Default: False
- -l, --log-file
Location of the log file
- -n, --num-runs
Set the number of runs to execute before exiting
Default: -1
- --pid
PID file location
- -s, --skip-serve-logs
Don’t start the serve logs process along with the workers
Default: False
- --stderr
Redirect stderr to this file
- --stdout
Redirect stdout to this file
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
Signals:
SIGUSR2: Dump a snapshot of task state being tracked by the executor.
- Example:
pkill -f -USR2 “airflow scheduler”
sync-perm¶
Update permissions for existing roles and optionally DAGs
airflow sync-perm [-h] [--include-dags] [-v]
Named Arguments¶
- --include-dags
If passed, DAG specific permissions will also be synced.
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
tasks¶
Manage tasks
airflow tasks [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: clear, failed-deps, list, render, run, state, states-for-dag-run, test
Sub-commands¶
clear¶
Clear a set of task instance, as if they never ran
airflow tasks clear [-h] [-R] [-d] [-e END_DATE] [-X] [-x] [-f] [-r]
[-s START_DATE] [-S SUBDIR] [-t TASK_REGEX] [-u] [-v] [-y]
dag_id
- dag_id
The id of the dag
- -R, --dag-regex
Search dag_id as regex instead of exact string
Default: False
- -d, --downstream
Include downstream tasks
Default: False
- -e, --end-date
Override end_date YYYY-MM-DD
- -X, --exclude-parentdag
Exclude ParentDAGS if the task cleared is a part of a SubDAG
Default: False
- -x, --exclude-subdags
Exclude subdags
Default: False
- -f, --only-failed
Only failed jobs
Default: False
- -r, --only-running
Only running jobs
Default: False
- -s, --start-date
Override start_date YYYY-MM-DD
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -t, --task-regex
The regex to filter specific task_ids (optional)
- -u, --upstream
Include upstream tasks
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
- -y, --yes
Do not prompt to confirm. Use with care!
Default: False
failed-deps¶
Returns the unmet dependencies for a task instance from the perspective of the scheduler. In other words, why a task instance doesn’t get scheduled and then queued by the scheduler, and then run by an executor.
airflow tasks failed-deps [-h] [--map-index MAP_INDEX] [-S SUBDIR] [-v]
dag_id task_id execution_date_or_run_id
- dag_id
The id of the dag
- task_id
The id of the task
- execution_date_or_run_id
The execution_date of the DAG or run_id of the DAGRun
- --map-index
Mapped task index
Default: -1
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
list¶
List the tasks within a DAG
airflow tasks list [-h] [-S SUBDIR] [-t] [-v] dag_id
- dag_id
The id of the dag
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -t, --tree
Deprecated - use dags show instead. Display tasks in a tree. Note that generating the tree can be slow and the output very large for some DAGs.
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
render¶
Render a task instance’s template(s)
airflow tasks render [-h] [--map-index MAP_INDEX] [-S SUBDIR] [-v]
dag_id task_id execution_date_or_run_id
- dag_id
The id of the dag
- task_id
The id of the task
- execution_date_or_run_id
The execution_date of the DAG or run_id of the DAGRun
- --map-index
Mapped task index
Default: -1
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
run¶
Run a single task instance
airflow tasks run [-h] [--cfg-path CFG_PATH] [-d {check,ignore,wait}] [-f]
[-A] [-i] [-I] [-N] [-l] [--map-index MAP_INDEX] [-m]
[-p PICKLE] [--pool POOL] [--read-from-db] [--ship-dag]
[-S SUBDIR] [-v]
dag_id task_id execution_date_or_run_id
- dag_id
The id of the dag
- task_id
The id of the task
- execution_date_or_run_id
The execution_date of the DAG or run_id of the DAGRun
- --cfg-path
Path to config file to use instead of airflow.cfg
- -d, --depends-on-past
Possible choices: check, ignore, wait
Determine how Airflow should deal with past dependencies. The default action is check, Airflow will check if the past dependencies are met for the tasks having depends_on_past=True before run them, if ignore is provided, the past dependencies will be ignored, if wait is provided and depends_on_past=True, Airflow will wait the past dependencies until they are met before running or skipping the task
Default: “check”
- -f, --force
Ignore previous task instance state, rerun regardless if task already succeeded/failed
Default: False
- -A, --ignore-all-dependencies
Ignores all non-critical dependencies, including ignore_ti_state and ignore_task_deps
Default: False
- -i, --ignore-dependencies
Ignore task-specific dependencies, e.g. upstream, depends_on_past, and retry delay dependencies
Default: False
- -I, --ignore-depends-on-past
Deprecated – use –depends-on-past ignore instead. Ignore depends_on_past dependencies (but respect upstream dependencies)
Default: False
- -N, --interactive
Do not capture standard output and error streams (useful for interactive debugging)
Default: False
- -l, --local
Run the task using the LocalExecutor
Default: False
- --map-index
Mapped task index
Default: -1
- -m, --mark-success
Mark jobs as succeeded without running them
Default: False
- -p, --pickle
Serialized pickle object of the entire dag (used internally)
- --pool
Resource pool to use
- --read-from-db
Read dag from DB instead of dag file
Default: False
- --ship-dag
Pickles (serializes) the DAG and ships it to the worker
Default: False
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
state¶
Get the status of a task instance
airflow tasks state [-h] [--map-index MAP_INDEX] [-S SUBDIR] [-v]
dag_id task_id execution_date_or_run_id
- dag_id
The id of the dag
- task_id
The id of the task
- execution_date_or_run_id
The execution_date of the DAG or run_id of the DAGRun
- --map-index
Mapped task index
Default: -1
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -v, --verbose
Make logging output more verbose
Default: False
states-for-dag-run¶
Get the status of all task instances in a dag run
airflow tasks states-for-dag-run [-h] [-o table, json, yaml, plain] [-v]
dag_id execution_date_or_run_id
- dag_id
The id of the dag
- execution_date_or_run_id
The execution_date of the DAG or run_id of the DAGRun
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
test¶
Test a task instance. This will run a task without checking for dependencies or recording its state in the database
airflow tasks test [-h] [-n] [--env-vars ENV_VARS] [--map-index MAP_INDEX]
[-m] [-S SUBDIR] [-t TASK_PARAMS] [-v]
dag_id task_id [execution_date_or_run_id]
- dag_id
The id of the dag
- task_id
The id of the task
- execution_date_or_run_id
The execution_date of the DAG or run_id of the DAGRun (optional)
- -n, --dry-run
Perform a dry run for each task. Only renders Template Fields for each task, nothing else
Default: False
- --env-vars
Set env var in both parsing time and runtime for each of entry supplied in a JSON dict
- --map-index
Mapped task index
Default: -1
- -m, --post-mortem
Open debugger on uncaught exception
Default: False
- -S, --subdir
File location or directory from which to look for the dag. Defaults to ‘[AIRFLOW_HOME]/dags’ where [AIRFLOW_HOME] is the value you set for ‘AIRFLOW_HOME’ config you set in ‘airflow.cfg’
Default: “[AIRFLOW_HOME]/dags”
- -t, --task-params
Sends a JSON params dict to the task
- -v, --verbose
Make logging output more verbose
Default: False
triggerer¶
Start a triggerer instance
airflow triggerer [-h] [--capacity CAPACITY] [-D] [-l LOG_FILE] [--pid [PID]]
[-s] [--stderr STDERR] [--stdout STDOUT] [-v]
Named Arguments¶
- --capacity
The maximum number of triggers that a Triggerer will run at one time.
- -D, --daemon
Daemonize instead of running in the foreground
Default: False
- -l, --log-file
Location of the log file
- --pid
PID file location
- -s, --skip-serve-logs
Don’t start the serve logs process along with the workers
Default: False
- --stderr
Redirect stderr to this file
- --stdout
Redirect stdout to this file
- -v, --verbose
Make logging output more verbose
Default: False
users¶
Manage users
airflow users [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: add-role, create, delete, export, import, list, remove-role, reset-password
Sub-commands¶
add-role¶
Add role to a user
airflow users add-role [-h] [-e EMAIL] -r ROLE [-u USERNAME] [-v]
- -e, --email
Email of the user
- -r, --role
Role of the user. Existing roles include Admin, User, Op, Viewer, and Public
- -u, --username
Username of the user
- -v, --verbose
Make logging output more verbose
Default: False
create¶
Create a user
airflow users create [-h] -e EMAIL -f FIRSTNAME -l LASTNAME [-p PASSWORD] -r
ROLE [--use-random-password] -u USERNAME [-v]
- -e, --email
Email of the user
- -f, --firstname
First name of the user
- -l, --lastname
Last name of the user
- -p, --password
Password of the user, required to create a user without –use-random-password
- -r, --role
Role of the user. Existing roles include Admin, User, Op, Viewer, and Public
- --use-random-password
Do not prompt for password. Use random string instead. Required to create a user without –password
Default: False
- -u, --username
Username of the user
- -v, --verbose
Make logging output more verbose
Default: False
examples: To create an user with “Admin” role and username equals to “admin”, run:
- $ airflow users create
–username admin –firstname FIRST_NAME –lastname LAST_NAME –role Admin –email admin@example.org
delete¶
Delete a user
airflow users delete [-h] [-e EMAIL] [-u USERNAME] [-v]
- -e, --email
Email of the user
- -u, --username
Username of the user
- -v, --verbose
Make logging output more verbose
Default: False
export¶
Export all users
airflow users export [-h] [-v] FILEPATH
- FILEPATH
Export all users to JSON file
- -v, --verbose
Make logging output more verbose
Default: False
import¶
Import users
airflow users import [-h] [-v] FILEPATH
- FILEPATH
Import users from JSON file. Example format:
[ { "email": "foo@bar.org", "firstname": "Jon", "lastname": "Doe", "roles": ["Public"], "username": "jondoe" } ]
- -v, --verbose
Make logging output more verbose
Default: False
list¶
List users
airflow users list [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
remove-role¶
Remove role from a user
airflow users remove-role [-h] [-e EMAIL] -r ROLE [-u USERNAME] [-v]
- -e, --email
Email of the user
- -r, --role
Role of the user. Existing roles include Admin, User, Op, Viewer, and Public
- -u, --username
Username of the user
- -v, --verbose
Make logging output more verbose
Default: False
reset-password¶
Reset a user’s password
airflow users reset-password [-h] [-e EMAIL] [-p PASSWORD]
[--use-random-password] [-u USERNAME] [-v]
- -e, --email
Email of the user
- -p, --password
Password of the user, required to create a user without –use-random-password
- --use-random-password
Do not prompt for password. Use random string instead. Required to create a user without –password
Default: False
- -u, --username
Username of the user
- -v, --verbose
Make logging output more verbose
Default: False
examples: To reset an user with username equals to “admin”, run:
- $ airflow users reset-password
–username admin
variables¶
Manage variables
airflow variables [-h] COMMAND ...
Positional Arguments¶
- COMMAND
Possible choices: delete, export, get, import, list, set
Sub-commands¶
delete¶
Delete variable
airflow variables delete [-h] [-v] key
- key
Variable key
- -v, --verbose
Make logging output more verbose
Default: False
export¶
All variables can be exported in STDOUT using the following command: airflow variables export -
airflow variables export [-h] [-v] file
- file
Export all variables to JSON file
- -v, --verbose
Make logging output more verbose
Default: False
get¶
Get variable
airflow variables get [-h] [-d VAL] [-j] [-v] key
- key
Variable key
- -d, --default
Default value returned if variable does not exist
- -j, --json
Deserialize JSON variable
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
import¶
Import variables
airflow variables import [-h] [-a {overwrite,fail,skip}] [-v] file
- file
Import variables from JSON file
- -a, --action-on-existing-key
Possible choices: overwrite, fail, skip
Action to take if we encounter a variable key that already exists.
Default: “overwrite”
- -v, --verbose
Make logging output more verbose
Default: False
list¶
List variables
airflow variables list [-h] [-o table, json, yaml, plain] [-v]
- -o, --output
Possible choices: table, json, yaml, plain
Output format. Allowed values: json, yaml, plain, table (default: table)
Default: “table”
- -v, --verbose
Make logging output more verbose
Default: False
set¶
Set variable
airflow variables set [-h] [--description DESCRIPTION] [-j] [-v] key VALUE
- key
Variable key
- VALUE
Variable value
- --description
Variable description, optional when setting a variable
- -j, --json
Serialize JSON variable
Default: False
- -v, --verbose
Make logging output more verbose
Default: False
webserver¶
Start a Airflow webserver instance
airflow webserver [-h] [-A ACCESS_LOGFILE] [-L ACCESS_LOGFORMAT] [-D] [-d]
[-E ERROR_LOGFILE] [-H HOSTNAME] [-l LOG_FILE] [--pid [PID]]
[-p PORT] [--ssl-cert SSL_CERT] [--ssl-key SSL_KEY]
[--stderr STDERR] [--stdout STDOUT] [-t WORKER_TIMEOUT]
[-k {sync,eventlet,gevent,tornado}] [-w WORKERS]
Named Arguments¶
- -A, --access-logfile
The logfile to store the webserver access log. Use ‘-‘ to print to stdout
Default: “-“
- -L, --access-logformat
The access log format for gunicorn logs
Default: “”
- -D, --daemon
Daemonize instead of running in the foreground
Default: False
- -d, --debug
Use the server that ships with Flask in debug mode
Default: False
- -E, --error-logfile
The logfile to store the webserver error log. Use ‘-‘ to print to stderr
Default: “-“
- -H, --hostname
Set the hostname on which to run the web server
Default: “0.0.0.0”
- -l, --log-file
Location of the log file
- --pid
PID file location
- -p, --port
The port on which to run the server
Default: 8080
- --ssl-cert
Path to the SSL certificate for the webserver
Default: “”
- --ssl-key
Path to the key to use with the SSL certificate
Default: “”
- --stderr
Redirect stderr to this file
- --stdout
Redirect stdout to this file
- -t, --worker-timeout
The timeout for waiting on webserver workers
Default: 120
- -k, --workerclass
Possible choices: sync, eventlet, gevent, tornado
The worker class to use for Gunicorn
Default: “sync”
- -w, --workers
Number of workers to run the webserver on
Default: 4
Environment Variables¶
- AIRFLOW__{SECTION}__{KEY}¶
Sets options in the Airflow configuration. This takes priority over the value in the
airflow.cfg
file.Replace the
{SECTION}
placeholder with any section and the{KEY}
placeholder with any key in that specified section.For example, if you want to set the
dags_folder
options in[core]
section, then you should set theAIRFLOW__CORE__DAGS_FOLDER
environment variable.For more information, see: Setting Configuration Options.
- AIRFLOW__{SECTION}__{KEY}_CMD¶
For any specific key in a section in Airflow, execute the command the key is pointing to. The result of the command is used as a value of the
AIRFLOW__{SECTION}__{KEY}
environment variable.This is only supported by the following config options:
sql_alchemy_conn
in[database]
sectionfernet_key
in[core]
sectionbroker_url
in[celery]
sectionflower_basic_auth
in[celery]
sectionresult_backend
in[celery]
sectionpassword
in[atlas]
sectionsmtp_password
in[smtp]
sectionsecret_key
in[webserver]
section
- AIRFLOW__{SECTION}__{KEY}_SECRET¶
For any specific key in a section in Airflow, retrieve the secret from the configured secrets backend. The returned value will be used as the value of the
AIRFLOW__{SECTION}__{KEY}
environment variable.See Secrets Backends for more information on available secrets backends.
This form of environment variable configuration is only supported for the same subset of config options as
AIRFLOW__{SECTION}__{KEY}_CMD
- AIRFLOW_CONFIG¶
The path to the Airflow configuration file.
- AIRFLOW_CONN_{CONN_ID}¶
Defines a new connection with the name
{CONN_ID}
using the URI value.For example, if you want to create a connection named
PROXY_POSTGRES_TCP
, you can create a keyAIRFLOW_CONN_PROXY_POSTGRES_TCP
with the connection URI as the value.For more information, see: Storing connections in environment variables.
- AIRFLOW_HOME¶
The root directory for the Airflow content. This is the default parent directory for Airflow assets such as DAGs and logs.
- AIRFLOW_VAR_{KEY}¶
Defines an Airflow variable. Replace the
{KEY}
placeholder with the variable name.For more information, see: Managing Variables.