CeleryKubernetes Executor

Note

As of Airflow 2.7.0, you need to install both the celery and cncf.kubernetes provider package to use this executor. This can be done by installing apache-airflow-providers-celery>=3.3.0 and apache-airflow-providers-cncf-kubernetes>=7.4.0 or by installing Airflow with the celery and cncf.kubernetes extras: pip install 'apache-airflow[celery,cncf.kubernetes]'.

The CeleryKubernetesExecutor allows users to run simultaneously a CeleryExecutor and a KubernetesExecutor. An executor is chosen to run a task based on the task’s queue.

CeleryKubernetesExecutor inherits the scalability of the CeleryExecutor to handle the high load at the peak time and runtime isolation of the KubernetesExecutor.

The configuration parameters of the Celery Executor can be found in the Celery provider’s Configuration Reference.

When to use CeleryKubernetesExecutor

The CeleryKubernetesExecutor should only be used at certain cases, given that it requires setting up the CeleryExecutor and the KubernetesExecutor.

We recommend considering the CeleryKubernetesExecutor when your use case meets:

  1. The number of tasks needed to be scheduled at the peak exceeds the scale that your Kubernetes cluster can comfortably handle

  2. A relative small portion of your tasks requires runtime isolation.

  3. You have plenty of small tasks that can be executed on Celery workers but you also have resource-hungry tasks that will be better to run in predefined environments.

Was this entry helpful?