Configuration Reference

This page contains the list of all available Airflow configurations for the apache-airflow-providers-cncf-kubernetes provider that can be set in the airflow.cfg file or using environment variables.

Note

The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.

Note

For more information see Setting Configuration Options.

[kubernetes_executor]

api_client_retry_configuration

Kwargs to override the default urllib3 Retry used in the kubernetes API client

Type

string

Default

''

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__API_CLIENT_RETRY_CONFIGURATION

Example

{ "total": 3, "backoff_factor": 0.5 }

cluster_context

When running with in_cluster=False change the default cluster_context or config_file options to Kubernetes client. Leave blank these to use default behaviour like kubectl has.

Type

string

Default

None

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__CLUSTER_CONTEXT

config_file

Path to the kubernetes configfile to be used when in_cluster is set to False

Type

string

Default

None

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__CONFIG_FILE

delete_option_kwargs

Optional keyword arguments to pass to the delete_namespaced_pod kubernetes client core_v1_api method when using the Kubernetes Executor. This should be an object and can contain any of the options listed in the v1DeleteOptions class defined here: https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19

Type

string

Default

''

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__DELETE_OPTION_KWARGS

Example

{"grace_period_seconds": 10}

delete_worker_pods

If True, all worker pods will be deleted upon termination

Type

string

Default

True

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS

delete_worker_pods_on_failure

If False (and delete_worker_pods is True), failed worker pods will not be deleted so users can investigate them. This only prevents removal of worker pods where the worker itself failed, not when the task it ran failed.

Type

string

Default

False

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS_ON_FAILURE

enable_tcp_keepalive

Enables TCP keepalive mechanism. This prevents Kubernetes API requests to hang indefinitely when idle connection is time-outed on services like cloud load balancers or firewalls.

Type

boolean

Default

True

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__ENABLE_TCP_KEEPALIVE

in_cluster

Use the service account kubernetes gives to pods to connect to kubernetes cluster. It’s intended for clients that expect to be running inside a pod running on kubernetes. It will raise an exception if called from a process not running in a kubernetes environment.

Type

string

Default

True

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__IN_CLUSTER

kube_client_request_args

Keyword parameters to pass while calling a kubernetes client core_v1_api methods from Kubernetes Executor provided as a single line formatted JSON dictionary string. List of supported params are similar for all core_v1_apis, hence a single config variable for all apis. See: https://raw.githubusercontent.com/kubernetes-client/python/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/api/core_v1_api.py

Type

string

Default

''

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__KUBE_CLIENT_REQUEST_ARGS

logs_task_metadata

Flag to control the information added to kubernetes executor logs for better traceability

Type

boolean

Default

False

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__LOGS_TASK_METADATA

multi_namespace_mode

Allows users to launch pods in multiple namespaces. Will require creating a cluster-role for the scheduler, or use multi_namespace_mode_namespace_list configuration.

Type

boolean

Default

False

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__MULTI_NAMESPACE_MODE

multi_namespace_mode_namespace_list

If multi_namespace_mode is True while scheduler does not have a cluster-role, give the list of namespaces where the scheduler will schedule jobs Scheduler needs to have the necessary permissions in these namespaces.

Type

string

Default

''

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__MULTI_NAMESPACE_MODE_NAMESPACE_LIST

namespace

The Kubernetes namespace where airflow workers should be created. Defaults to default

Type

string

Default

default

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE

pod_template_file

Path to the YAML pod file that forms the basis for KubernetesExecutor workers.

Type

string

Default

''

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__POD_TEMPLATE_FILE

ssl_ca_cert

Path to a CA certificate to be used by the Kubernetes client to verify the server’s SSL certificate.

Type

string

Default

''

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__SSL_CA_CERT

task_publish_max_retries

The Maximum number of retries for queuing the task to the kubernetes scheduler when failing due to Kube API exceeded quota errors before giving up and marking task as failed. -1 for unlimited times.

Type

integer

Default

0

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__TASK_PUBLISH_MAX_RETRIES

tcp_keep_cnt

When the enable_tcp_keepalive option is enabled, if Kubernetes API does not respond to a keepalive probe, TCP retransmits the probe tcp_keep_cnt number of times before a connection is considered to be broken.

Type

integer

Default

6

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_CNT

tcp_keep_idle

When the enable_tcp_keepalive option is enabled, TCP probes a connection that has been idle for tcp_keep_idle seconds.

Type

integer

Default

120

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_IDLE

tcp_keep_intvl

When the enable_tcp_keepalive option is enabled, if Kubernetes API does not respond to a keepalive probe, TCP retransmits the probe after tcp_keep_intvl seconds.

Type

integer

Default

30

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_INTVL

verify_ssl

Set this to false to skip verifying SSL certificate of Kubernetes python client.

Type

boolean

Default

True

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__VERIFY_SSL

worker_container_repository

The repository of the Kubernetes Image for the Worker to Run

Type

string

Default

''

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_REPOSITORY

worker_container_tag

The tag of the Kubernetes Image for the Worker to Run

Type

string

Default

''

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_TAG

worker_pod_pending_fatal_container_state_reasons

New in version 8.1.0.

If the worker pods are in a pending state due to a fatal container state reasons, then fail the task and delete the worker pod if delete_worker_pods is True and delete_worker_pods_on_failure is True.

Type

string

Default

CreateContainerConfigError,ErrImagePull,CreateContainerError,ImageInspectError, InvalidImageName

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__WORKER_POD_PENDING_FATAL_CONTAINER_STATE_REASONS

worker_pods_creation_batch_size

Number of Kubernetes Worker Pod creation calls per scheduler loop. Note that the current default of “1” will only launch a single pod per-heartbeat. It is HIGHLY recommended that users increase this number to match the tolerance of their kubernetes cluster for better performance.

Type

string

Default

1

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE

worker_pods_queued_check_interval

How often in seconds to check for task instances stuck in “queued” status without a pod

Type

integer

Default

60

Environment Variable

AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_QUEUED_CHECK_INTERVAL

[local_kubernetes_executor]

This section only applies if you are using the LocalKubernetesExecutor in [core] section above

kubernetes_queue

Define when to send a task to KubernetesExecutor when using LocalKubernetesExecutor. When the queue of a task is the value of kubernetes_queue (default kubernetes), the task is executed via KubernetesExecutor, otherwise via LocalExecutor

Type

string

Default

kubernetes

Environment Variable

AIRFLOW__LOCAL_KUBERNETES_EXECUTOR__KUBERNETES_QUEUE

Was this entry helpful?