Configuration Reference¶
This page contains the list of all available Airflow configurations for the
apache-airflow-providers-cncf-kubernetes
provider that can be set in the airflow.cfg
file or using environment variables.
Note
The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.
Note
For more information see Setting Configuration Options.
[kubernetes_executor]¶
api_client_retry_configuration¶
Kwargs to override the default urllib3 Retry used in the kubernetes API client
- Type
string
- Default
''
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__API_CLIENT_RETRY_CONFIGURATION
- Example
{ "total": 3, "backoff_factor": 0.5 }
cluster_context¶
When running with in_cluster=False change the default cluster_context or config_file
options to Kubernetes client. Leave blank these to use default behaviour like kubectl
has.
- Type
string
- Default
None
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__CLUSTER_CONTEXT
config_file¶
Path to the kubernetes configfile to be used when in_cluster
is set to False
- Type
string
- Default
None
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__CONFIG_FILE
delete_option_kwargs¶
Optional keyword arguments to pass to the delete_namespaced_pod
kubernetes client
core_v1_api
method when using the Kubernetes Executor.
This should be an object and can contain any of the options listed in the v1DeleteOptions
class defined here:
https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
- Type
string
- Default
''
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__DELETE_OPTION_KWARGS
- Example
{"grace_period_seconds": 10}
delete_worker_pods¶
If True, all worker pods will be deleted upon termination
- Type
string
- Default
True
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS
delete_worker_pods_on_failure¶
If False (and delete_worker_pods is True), failed worker pods will not be deleted so users can investigate them. This only prevents removal of worker pods where the worker itself failed, not when the task it ran failed.
- Type
string
- Default
False
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS_ON_FAILURE
enable_tcp_keepalive¶
Enables TCP keepalive mechanism. This prevents Kubernetes API requests to hang indefinitely when idle connection is time-outed on services like cloud load balancers or firewalls.
- Type
boolean
- Default
True
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__ENABLE_TCP_KEEPALIVE
in_cluster¶
Use the service account kubernetes gives to pods to connect to kubernetes cluster. It’s intended for clients that expect to be running inside a pod running on kubernetes. It will raise an exception if called from a process not running in a kubernetes environment.
- Type
string
- Default
True
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__IN_CLUSTER
kube_client_request_args¶
Keyword parameters to pass while calling a kubernetes client core_v1_api methods from Kubernetes Executor provided as a single line formatted JSON dictionary string. List of supported params are similar for all core_v1_apis, hence a single config variable for all apis. See: https://raw.githubusercontent.com/kubernetes-client/python/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/api/core_v1_api.py
- Type
string
- Default
''
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__KUBE_CLIENT_REQUEST_ARGS
logs_task_metadata¶
Flag to control the information added to kubernetes executor logs for better traceability
- Type
boolean
- Default
False
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__LOGS_TASK_METADATA
multi_namespace_mode¶
Allows users to launch pods in multiple namespaces. Will require creating a cluster-role for the scheduler, or use multi_namespace_mode_namespace_list configuration.
- Type
boolean
- Default
False
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__MULTI_NAMESPACE_MODE
multi_namespace_mode_namespace_list¶
If multi_namespace_mode is True while scheduler does not have a cluster-role, give the list of namespaces where the scheduler will schedule jobs Scheduler needs to have the necessary permissions in these namespaces.
- Type
string
- Default
''
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__MULTI_NAMESPACE_MODE_NAMESPACE_LIST
namespace¶
The Kubernetes namespace where airflow workers should be created. Defaults to default
- Type
string
- Default
default
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE
pod_template_file¶
Path to the YAML pod file that forms the basis for KubernetesExecutor workers.
See also
- Type
string
- Default
''
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__POD_TEMPLATE_FILE
ssl_ca_cert¶
Path to a CA certificate to be used by the Kubernetes client to verify the server’s SSL certificate.
- Type
string
- Default
''
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__SSL_CA_CERT
task_publish_max_retries¶
The Maximum number of retries for queuing the task to the kubernetes scheduler when failing due to Kube API exceeded quota errors before giving up and marking task as failed. -1 for unlimited times.
- Type
integer
- Default
0
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__TASK_PUBLISH_MAX_RETRIES
tcp_keep_cnt¶
When the enable_tcp_keepalive option is enabled, if Kubernetes API does not respond to a keepalive probe, TCP retransmits the probe tcp_keep_cnt number of times before a connection is considered to be broken.
- Type
integer
- Default
6
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_CNT
tcp_keep_idle¶
When the enable_tcp_keepalive option is enabled, TCP probes a connection that has been idle for tcp_keep_idle seconds.
- Type
integer
- Default
120
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_IDLE
tcp_keep_intvl¶
When the enable_tcp_keepalive option is enabled, if Kubernetes API does not respond to a keepalive probe, TCP retransmits the probe after tcp_keep_intvl seconds.
- Type
integer
- Default
30
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_INTVL
verify_ssl¶
Set this to false to skip verifying SSL certificate of Kubernetes python client.
- Type
boolean
- Default
True
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__VERIFY_SSL
worker_container_repository¶
The repository of the Kubernetes Image for the Worker to Run
- Type
string
- Default
''
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_REPOSITORY
worker_container_tag¶
The tag of the Kubernetes Image for the Worker to Run
- Type
string
- Default
''
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_TAG
worker_pod_pending_fatal_container_state_reasons¶
New in version 8.1.0.
If the worker pods are in a pending state due to a fatal container state reasons, then fail the task and delete the worker pod if delete_worker_pods is True and delete_worker_pods_on_failure is True.
- Type
string
- Default
CreateContainerConfigError,ErrImagePull,CreateContainerError,ImageInspectError, InvalidImageName
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__WORKER_POD_PENDING_FATAL_CONTAINER_STATE_REASONS
worker_pods_creation_batch_size¶
Number of Kubernetes Worker Pod creation calls per scheduler loop. Note that the current default of “1” will only launch a single pod per-heartbeat. It is HIGHLY recommended that users increase this number to match the tolerance of their kubernetes cluster for better performance.
- Type
string
- Default
1
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE
worker_pods_queued_check_interval¶
How often in seconds to check for task instances stuck in “queued” status without a pod
- Type
integer
- Default
60
- Environment Variable
AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_QUEUED_CHECK_INTERVAL
[local_kubernetes_executor]¶
This section only applies if you are using the LocalKubernetesExecutor
in
[core]
section above
kubernetes_queue¶
Define when to send a task to KubernetesExecutor
when using LocalKubernetesExecutor
.
When the queue of a task is the value of kubernetes_queue
(default kubernetes
),
the task is executed via KubernetesExecutor
,
otherwise via LocalExecutor
- Type
string
- Default
kubernetes
- Environment Variable
AIRFLOW__LOCAL_KUBERNETES_EXECUTOR__KUBERNETES_QUEUE