• Home
• airflow.providers.cncf.kubernetes
• airflow.providers.cncf.kubernetes.utils
• airflow.providers.cncf.kubernetes.utils.pod_manager

airflow.providers.cncf.kubernetes.utils.pod_manager

Launches PODs.

Module Contents

Classes

PodPhase	Possible pod phases.
PodOperatorHookProtocol	Protocol to define methods relied upon by KubernetesPodOperator.
PodLoggingStatus	Used for returning the status of the pod and last log time when exiting from fetch_container_logs.
PodManager	Create, monitor, and otherwise interact with Kubernetes pods for use with the KubernetesPodOperator.
OnFinishAction	Action to take when the pod finishes.

Functions

should_retry_start_pod(exception)	Check if an Exception indicates a transient error and warrants retrying.
get_container_status(pod, container_name)	Retrieves container status.
container_is_running(pod, container_name)	Examines V1Pod pod to determine whether container_name is running.
container_is_completed(pod, container_name)	Examines V1Pod pod to determine whether container_name is completed.
container_is_succeeded(pod, container_name)	Examines V1Pod pod to determine whether container_name is completed and succeeded.
container_is_terminated(pod, container_name)	Examines V1Pod pod to determine whether container_name is terminated.
get_container_termination_message(pod, container_name)

exception airflow.providers.cncf.kubernetes.utils.pod_manager.PodLaunchFailedException

Bases: airflow.exceptions.AirflowException

When pod launching fails in KubernetesPodOperator.

airflow.providers.cncf.kubernetes.utils.pod_manager.should_retry_start_pod(exception)

Check if an Exception indicates a transient error and warrants retrying.

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodPhase

Possible pod phases.

See https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase.

PENDING = 'Pending'

RUNNING = 'Running'

FAILED = 'Failed'

SUCCEEDED = 'Succeeded'

terminal_states

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodOperatorHookProtocol

Bases: Protocol

Protocol to define methods relied upon by KubernetesPodOperator.

Subclasses of KubernetesPodOperator, such as GKEStartPodOperator, may use hooks that don’t extend KubernetesHook. We use this protocol to document the methods used by KPO and ensure that these methods exist on such other hooks.

property core_v1_client: kubernetes.client.CoreV1Api

Get authenticated client object.

property is_in_cluster: bool

Expose whether the hook is configured with load_incluster_config or not.

get_pod(name, namespace)

Read pod object from kubernetes API.

get_namespace()

Returns the namespace that defined in the connection.

get_xcom_sidecar_container_image()

Returns the xcom sidecar image that defined in the connection.

get_xcom_sidecar_container_resources()

Returns the xcom sidecar resources that defined in the connection.

airflow.providers.cncf.kubernetes.utils.pod_manager.get_container_status(pod, container_name)

Retrieves container status.

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_running(pod, container_name)

Examines V1Pod pod to determine whether container_name is running.

If that container is present and running, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_completed(pod, container_name)

Examines V1Pod pod to determine whether container_name is completed.

If that container is present and completed, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_succeeded(pod, container_name)

Examines V1Pod pod to determine whether container_name is completed and succeeded.

If that container is present and completed and succeeded, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_terminated(pod, container_name)

Examines V1Pod pod to determine whether container_name is terminated.

If that container is present and terminated, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.get_container_termination_message(pod, container_name)

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodLoggingStatus

Used for returning the status of the pod and last log time when exiting from fetch_container_logs.

running: bool

last_log_time: pendulum.DateTime | None

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager(kube_client, progress_callback=None)

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Create, monitor, and otherwise interact with Kubernetes pods for use with the KubernetesPodOperator.

run_pod_async(pod, **kwargs)

Runs POD asynchronously.

delete_pod(pod)

Deletes POD.

create_pod(pod)

Launches the pod asynchronously.

await_pod_start(pod, startup_timeout=120)

Waits for the pod to reach phase other than Pending.

Parameters

• pod (kubernetes.client.models.v1_pod.V1Pod) –

• startup_timeout (int) – Timeout (in seconds) for startup of the pod (if pod is pending for too long, fails task)

Returns

Return type

None

follow_container_logs(pod, container_name)

fetch_container_logs(pod, container_name, *, follow=False, since_time=None, post_termination_timeout=120)

Follows the logs of container and streams to airflow logging.

Returns when container exits.

Between when the pod starts and logs being available, there might be a delay due to CSR not approved and signed yet. In such situation, ApiException is thrown. This is why we are retrying on this specific exception.

fetch_requested_container_logs(pod, container_logs, follow_logs=False)

Follow the logs of containers in the specified pod and publish it to airflow logging.

Returns when all the containers exit.

await_container_completion(pod, container_name)

Waits for the given container in the given pod to be completed.

Parameters

• pod (kubernetes.client.models.v1_pod.V1Pod) – pod spec that will be monitored

• container_name (str) – name of the container within the pod to monitor

await_pod_completion(pod, istio_enabled=False, container_name='base')

Monitors a pod and returns the final state.

Parameters

• istio_enabled (bool) – whether istio is enabled in the namespace

• pod (kubernetes.client.models.v1_pod.V1Pod) – pod spec that will be monitored

• container_name (str) – name of the container within the pod

Returns

tuple[State, str | None]

Return type

kubernetes.client.models.v1_pod.V1Pod

parse_log_line(line)

Parse K8s log line and returns the final state.

Parameters

line (str) – k8s log line

Returns

timestamp and log message

Return type

tuple[pendulum.DateTime | None, str]

container_is_running(pod, container_name)

Reads pod and checks if container is running.

container_is_terminated(pod, container_name)

Reads pod and checks if container is terminated.

read_pod_logs(pod, container_name, tail_lines=None, timestamps=False, since_seconds=None, follow=True, post_termination_timeout=120)

Reads log from the POD.

get_container_names(pod)

Return container names from the POD except for the airflow-xcom-sidecar container.

read_pod_events(pod)

Reads events from the POD.

read_pod(pod)

Read POD information.

await_xcom_sidecar_container_start(pod)

extract_xcom(pod)

Retrieves XCom value and kills xcom sidecar container.

extract_xcom_json(pod)

Retrieves XCom value and also checks if xcom json is valid.

extract_xcom_kill(pod)

Kills xcom sidecar container.

class airflow.providers.cncf.kubernetes.utils.pod_manager.OnFinishAction

Bases: str, enum.Enum

Action to take when the pod finishes.

KEEP_POD = 'keep_pod'

DELETE_POD = 'delete_pod'

DELETE_SUCCEEDED_POD = 'delete_succeeded_pod'

Previous Next

Was this entry helpful?