Configuration Reference¶

This page contains the list of all available Airflow configurations for the apache-airflow-providers-elasticsearch provider that can be set in the airflow.cfg file or using environment variables.

Note

The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.

Note

For more information see Setting Configuration Options.

Sections:

[elasticsearch]
[elasticsearch_configs]

[elasticsearch]¶

end_of_log_mark¶

Used to mark the end of a log stream for a task

Type: string
Default: end_of_log
Environment Variable: AIRFLOW__ELASTICSEARCH__END_OF_LOG_MARK

frontend¶

Qualified URL for an elasticsearch frontend (like Kibana) with a template argument for log_id Code will construct log_id using the log_id template from the argument above. NOTE: scheme will default to https if one is not provided

Type: string
Default: ''
Environment Variable: AIRFLOW__ELASTICSEARCH__FRONTEND
Example: http://localhost:5601/app/kibana#/discover?_a=(columns:!(message),query:(language:kuery,query:'log_id: "{log_id}"'),sort:!(log.offset,asc))

host¶

Elasticsearch host

Type: string
Default: ''
Environment Variable: AIRFLOW__ELASTICSEARCH__HOST

host_field¶

The field where host name is stored (normally either host or host.name)

Type: string
Default: host
Environment Variable: AIRFLOW__ELASTICSEARCH__HOST_FIELD

index_patterns¶

Comma separated list of index patterns to use when searching for logs (default: _all). The index_patterns_callable takes precedence over this.

Type: string
Default: _all
Environment Variable: AIRFLOW__ELASTICSEARCH__INDEX_PATTERNS
Example: something-*

index_patterns_callable¶

A string representing the full path to the Python callable path which accept TI object and return comma separated list of index patterns. This will takes precedence over index_patterns.

Type: string
Default: ''
Environment Variable: AIRFLOW__ELASTICSEARCH__INDEX_PATTERNS_CALLABLE
Example: module.callable

json_fields¶

Log fields to also attach to the json output, if enabled

Type: string
Default: asctime, filename, lineno, levelname, message
Environment Variable: AIRFLOW__ELASTICSEARCH__JSON_FIELDS

json_format¶

Instead of the default log formatter, write the log lines as JSON

Type: string
Default: False
Environment Variable: AIRFLOW__ELASTICSEARCH__JSON_FORMAT

log_id_template¶

Format of the log_id, which is used to query for a given tasks logs

Type: string
Default: {dag_id}-{task_id}-{run_id}-{map_index}-{try_number}
Environment Variable: AIRFLOW__ELASTICSEARCH__LOG_ID_TEMPLATE

offset_field¶

The field where offset is stored (normally either offset or log.offset)

Type: string
Default: offset
Environment Variable: AIRFLOW__ELASTICSEARCH__OFFSET_FIELD

write_stdout¶

Write the task logs to the stdout of the worker, rather than the default files

Type: string
Default: False
Environment Variable: AIRFLOW__ELASTICSEARCH__WRITE_STDOUT

[elasticsearch_configs]¶

http_compress¶

Type: string
Default: False
Environment Variable: AIRFLOW__ELASTICSEARCH_CONFIGS__HTTP_COMPRESS

verify_certs¶

Type: string
Default: True
Environment Variable: AIRFLOW__ELASTICSEARCH_CONFIGS__VERIFY_CERTS