airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils

Module Contents

Classes

Singleton

Metaclass that allows to implement singleton pattern.

ResourceVersion

Singleton for tracking resourceVersion from Kubernetes.

KubernetesJobWatcher

Watches for Kubernetes jobs.

AirflowKubernetesScheduler

Airflow Scheduler for Kubernetes.

Functions

get_base_pod_from_template(pod_template_file, kube_config)

Get base pod from template.

Attributes

T

airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.T[source]
class airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.Singleton[source]

Bases: type, Generic[T]

Metaclass that allows to implement singleton pattern.

__call__(*args, **kwargs)[source]

Call self as a function.

class airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.ResourceVersion[source]

Singleton for tracking resourceVersion from Kubernetes.

resource_version: dict[str, str][source]
class airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher(namespace, watcher_queue, resource_version, scheduler_job_id, kube_config)[source]

Bases: multiprocessing.Process, airflow.utils.log.logging_mixin.LoggingMixin

Watches for Kubernetes jobs.

run()[source]

Perform watching.

process_error(event)[source]

Process error response.

process_status(pod_name, namespace, status, annotations, resource_version, event)[source]
class airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.AirflowKubernetesScheduler(kube_config, result_queue, kube_client, scheduler_job_id)[source]

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Airflow Scheduler for Kubernetes.

run_pod_async(pod, **kwargs)[source]

Run POD asynchronously.

run_next(next_job)[source]

Receives the next job to run, builds the pod, and creates it.

delete_pod(pod_name, namespace)[source]

Delete Pod from a namespace; does not raise if it does not exist.

patch_pod_executor_done(*, pod_name, namespace)[source]

Add a “done” annotation to ensure we don’t continually adopt pods.

sync()[source]

Check the status of all currently running kubernetes jobs.

If a job is completed, its status is placed in the result queue to be sent back to the scheduler.

process_watcher_task(task)[source]

Process the task by watcher.

terminate()[source]

Terminates the watcher.

airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.get_base_pod_from_template(pod_template_file, kube_config)[source]

Get base pod from template.

Reads either the pod_template_file set in the executor_config or the base pod_template_file set in the airflow.cfg to craft a “base pod” that will be used by the KubernetesExecutor

Parameters

  • pod_template_file (str | None) – absolute path to a pod_template_file.yaml or None

  • kube_config (Any) – The KubeConfig class generated by airflow that contains all kube metadata

Returns

a V1Pod that can be used as the base pod for k8s tasks

Return type

kubernetes.client.models.V1Pod

Was this entry helpful?