airflow.providers.cncf.kubernetes.utils.pod_manager

Launches PODs.

Module Contents

Classes

PodPhase

Possible pod phases.

PodOperatorHookProtocol

Protocol to define methods relied upon by KubernetesPodOperator.

PodLoggingStatus

Return the status of the pod and last log time when exiting from fetch_container_logs.

PodManager

Create, monitor, and otherwise interact with Kubernetes pods for use with the KubernetesPodOperator.

OnFinishAction

Action to take when the pod finishes.

Functions

should_retry_start_pod(exception)

Check if an Exception indicates a transient error and warrants retrying.

get_container_status(pod, container_name)

Retrieve container status.

container_is_running(pod, container_name)

Examine V1Pod pod to determine whether container_name is running.

container_is_completed(pod, container_name)

Examine V1Pod pod to determine whether container_name is completed.

container_is_succeeded(pod, container_name)

Examine V1Pod pod to determine whether container_name is completed and succeeded.

container_is_terminated(pod, container_name)

Examine V1Pod pod to determine whether container_name is terminated.

get_container_termination_message(pod, container_name)

check_exception_is_kubernetes_api_unauthorized(exc)

exception airflow.providers.cncf.kubernetes.utils.pod_manager.PodLaunchFailedException[source]

Bases: airflow.exceptions.AirflowException

When pod launching fails in KubernetesPodOperator.

airflow.providers.cncf.kubernetes.utils.pod_manager.should_retry_start_pod(exception)[source]

Check if an Exception indicates a transient error and warrants retrying.

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodPhase[source]

Possible pod phases.

See https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase.

PENDING = 'Pending'[source]
RUNNING = 'Running'[source]
FAILED = 'Failed'[source]
SUCCEEDED = 'Succeeded'[source]
terminal_states[source]
class airflow.providers.cncf.kubernetes.utils.pod_manager.PodOperatorHookProtocol[source]

Bases: Protocol

Protocol to define methods relied upon by KubernetesPodOperator.

Subclasses of KubernetesPodOperator, such as GKEStartPodOperator, may use hooks that don’t extend KubernetesHook. We use this protocol to document the methods used by KPO and ensure that these methods exist on such other hooks.

property core_v1_client: kubernetes.client.CoreV1Api[source]

Get authenticated client object.

property is_in_cluster: bool[source]

Expose whether the hook is configured with load_incluster_config or not.

get_pod(name, namespace)[source]

Read pod object from kubernetes API.

get_namespace()[source]

Return the namespace that defined in the connection.

get_xcom_sidecar_container_image()[source]

Return the xcom sidecar image that defined in the connection.

get_xcom_sidecar_container_resources()[source]

Return the xcom sidecar resources that defined in the connection.

airflow.providers.cncf.kubernetes.utils.pod_manager.get_container_status(pod, container_name)[source]

Retrieve container status.

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_running(pod, container_name)[source]

Examine V1Pod pod to determine whether container_name is running.

If that container is present and running, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_completed(pod, container_name)[source]

Examine V1Pod pod to determine whether container_name is completed.

If that container is present and completed, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_succeeded(pod, container_name)[source]

Examine V1Pod pod to determine whether container_name is completed and succeeded.

If that container is present and completed and succeeded, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_terminated(pod, container_name)[source]

Examine V1Pod pod to determine whether container_name is terminated.

If that container is present and terminated, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.get_container_termination_message(pod, container_name)[source]
airflow.providers.cncf.kubernetes.utils.pod_manager.check_exception_is_kubernetes_api_unauthorized(exc)[source]
exception airflow.providers.cncf.kubernetes.utils.pod_manager.PodLaunchTimeoutException[source]

Bases: airflow.exceptions.AirflowException

When pod does not leave the Pending phase within specified timeout.

exception airflow.providers.cncf.kubernetes.utils.pod_manager.PodNotFoundException[source]

Bases: airflow.exceptions.AirflowException

Expected pod does not exist in kube-api.

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodLoggingStatus[source]

Return the status of the pod and last log time when exiting from fetch_container_logs.

running: bool[source]
last_log_time: pendulum.DateTime | None[source]
class airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager(kube_client, callbacks=None)[source]

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Create, monitor, and otherwise interact with Kubernetes pods for use with the KubernetesPodOperator.

run_pod_async(pod, **kwargs)[source]

Run POD asynchronously.

delete_pod(pod)[source]

Delete POD.

create_pod(pod)[source]

Launch the pod asynchronously.

await_pod_start(pod, startup_timeout=120, startup_check_interval=1)[source]

Wait for the pod to reach phase other than Pending.

Parameters
  • pod (kubernetes.client.models.v1_pod.V1Pod) –

  • startup_timeout (int) – Timeout (in seconds) for startup of the pod (if pod is pending for too long, fails task)

  • startup_check_interval (int) – Interval (in seconds) between checks

Returns

Return type

None

await_container_completion(pod, container_name)[source]

Wait for the given container in the given pod to be completed.

Parameters
  • pod (kubernetes.client.models.v1_pod.V1Pod) – pod spec that will be monitored

  • container_name (str) – name of the container within the pod to monitor

await_pod_completion(pod, istio_enabled=False, container_name='base')[source]

Monitor a pod and return the final state.

Parameters
  • istio_enabled (bool) – whether istio is enabled in the namespace

  • pod (kubernetes.client.models.v1_pod.V1Pod) – pod spec that will be monitored

  • container_name (str) – name of the container within the pod

Returns

tuple[State, str | None]

Return type

kubernetes.client.models.v1_pod.V1Pod

parse_log_line(line)[source]

Parse K8s log line and returns the final state.

Parameters

line (str) – k8s log line

Returns

timestamp and log message

Return type

tuple[pendulum.DateTime | None, str]

container_is_running(pod, container_name)[source]

Read pod and checks if container is running.

container_is_terminated(pod, container_name)[source]

Read pod and checks if container is terminated.

read_pod_logs(pod, container_name, tail_lines=None, timestamps=False, since_seconds=None, follow=True, post_termination_timeout=120, **kwargs)[source]

Read log from the POD.

get_container_names(pod)[source]

Return container names from the POD except for the airflow-xcom-sidecar container.

read_pod_events(pod)[source]

Read events from the POD.

read_pod(pod)[source]

Read POD information.

await_xcom_sidecar_container_start(pod, timeout=900, log_interval=30)[source]

Check if the sidecar container has reached the ‘Running’ state before performing do_xcom_push.

extract_xcom(pod)[source]

Retrieve XCom value and kill xcom sidecar container.

extract_xcom_json(pod)[source]

Retrieve XCom value and also check if xcom json is valid.

extract_xcom_kill(pod)[source]

Kill xcom sidecar container.

class airflow.providers.cncf.kubernetes.utils.pod_manager.OnFinishAction[source]

Bases: str, enum.Enum

Action to take when the pod finishes.

KEEP_POD = 'keep_pod'[source]
DELETE_POD = 'delete_pod'[source]
DELETE_SUCCEEDED_POD = 'delete_succeeded_pod'[source]

Was this entry helpful?