Configuration Reference¶
This page contains the list of all available Airflow configurations for the
apache-airflow-providers-openlineage
provider that can be set in the airflow.cfg
file or using environment variables.
Note
The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.
Note
For more information see Setting Configuration Options.
Sections:
[openlineage]¶
This section applies settings for OpenLineage integration. More about configuration and it’s precedence can be found at https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html#transport-setup
config_path¶
Specify the path to the YAML configuration file. This ensures backwards compatibility with passing config through the openlineage.yml file.
- Type
string
- Default
''
- Environment Variable
AIRFLOW__OPENLINEAGE__CONFIG_PATH
- Example
full/path/to/openlineage.yml
disable_source_code¶
Disable the inclusion of source code in OpenLineage events by setting this to true. By default, several Operators (e.g. Python, Bash) will include their source code in the events unless disabled.
- Type
boolean
- Default
None
- Environment Variable
AIRFLOW__OPENLINEAGE__DISABLE_SOURCE_CODE
disabled¶
Disable sending events without uninstalling the OpenLineage Provider by setting this to true.
- Type
boolean
- Default
False
- Environment Variable
AIRFLOW__OPENLINEAGE__DISABLED
disabled_for_operators¶
New in version 1.1.0.
Exclude some Operators from emitting OpenLineage events by passing a string of semicolon separated full import paths of Operators to disable.
- Type
string
- Default
''
- Environment Variable
AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS
- Example
airflow.operators.bash.BashOperator;airflow.operators.python.PythonOperator
extractors¶
Register custom OpenLineage Extractors by passing a string of semicolon separated full import paths.
- Type
string
- Default
None
- Environment Variable
AIRFLOW__OPENLINEAGE__EXTRACTORS
- Example
full.path.to.ExtractorClass;full.path.to.AnotherExtractorClass
namespace¶
Set namespace that the lineage data belongs to, so that if you use multiple OpenLineage producers, events coming from them will be logically separated.
- Type
string
- Default
None
- Environment Variable
AIRFLOW__OPENLINEAGE__NAMESPACE
- Example
my_airflow_instance_1
selective_enable¶
New in version 1.7.0.
If this setting is enabled, OpenLineage integration won’t collect and emit metadata, unless you explicitly enable it per DAG or Task using enable_lineage method.
- Type
boolean
- Default
False
- Environment Variable
AIRFLOW__OPENLINEAGE__SELECTIVE_ENABLE
transport¶
Pass OpenLineage Client transport configuration as JSON string. It should contain type of the transport and additional options (different for each transport type). For more details see: https://openlineage.io/docs/client/python/#built-in-transport-types
Currently supported types are:
HTTP
Kafka
Console
File
- Type
string
- Default
''
- Environment Variable
AIRFLOW__OPENLINEAGE__TRANSPORT
- Example
{"type": "http", "url": "http://localhost:5000", "endpoint": "api/v1/lineage"}