• Home
• airflow.providers.cncf.kubernetes
• airflow.providers.cncf.kubernetes.utils
• airflow.providers.cncf.kubernetes.utils.pod_manager

airflow.providers.cncf.kubernetes.utils.pod_manager

Launches PODs

Module Contents

Classes

PodPhase	Possible pod phases
PodLogsConsumer	PodLogsConsumer is responsible for pulling pod logs from a stream with checking a container status before
PodLoggingStatus	Used for returning the status of the pod and last log time when exiting from fetch_container_logs
PodManager	Helper class for creating, monitoring, and otherwise interacting with Kubernetes pods

Functions

should_retry_start_pod(exception)	Check if an Exception indicates a transient error and warrants retrying
get_container_status(pod, container_name)	Retrieves container status
container_is_running(pod, container_name)	Examines V1Pod pod to determine whether container_name is running.
container_is_terminated(pod, container_name)	Examines V1Pod pod to determine whether container_name is terminated.
get_container_termination_message(pod, container_name)

exception airflow.providers.cncf.kubernetes.utils.pod_manager.PodLaunchFailedException

Bases: airflow.exceptions.AirflowException

When pod launching fails in KubernetesPodOperator.

airflow.providers.cncf.kubernetes.utils.pod_manager.should_retry_start_pod(exception)

Check if an Exception indicates a transient error and warrants retrying

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodPhase

Possible pod phases See https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase.

PENDING = 'Pending'

RUNNING = 'Running'

FAILED = 'Failed'

SUCCEEDED = 'Succeeded'

terminal_states

airflow.providers.cncf.kubernetes.utils.pod_manager.get_container_status(pod, container_name)

Retrieves container status

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_running(pod, container_name)

Examines V1Pod pod to determine whether container_name is running. If that container is present and running, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.container_is_terminated(pod, container_name)

Examines V1Pod pod to determine whether container_name is terminated. If that container is present and terminated, returns True. Returns False otherwise.

airflow.providers.cncf.kubernetes.utils.pod_manager.get_container_termination_message(pod, container_name)

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodLogsConsumer(response, pod, pod_manager, container_name, post_termination_timeout=120, read_pod_cache_timeout=120)

PodLogsConsumer is responsible for pulling pod logs from a stream with checking a container status before reading data. This class is a workaround for the issue https://github.com/apache/airflow/issues/23497

Parameters

• response (urllib3.response.HTTPResponse) – HTTP response with logs

• pod (kubernetes.client.models.v1_pod.V1Pod) – Pod instance from Kubernetes client

• pod_manager (PodManager) – Pod manager instance

• container_name (str) – Name of the container that we’re reading logs from

• post_termination_timeout (int) – (Optional) The period of time in seconds representing for how long time logs are available after the container termination.

• read_pod_cache_timeout (int) – (Optional) The container’s status cache lifetime. The container status is cached to reduce API calls.

__iter__()

The generator yields log items divided by the ‘n’ symbol.

logs_available()

read_pod()

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodLoggingStatus

Used for returning the status of the pod and last log time when exiting from fetch_container_logs

running: bool

last_log_time: DateTime | None

class airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager(kube_client=None, in_cluster=True, cluster_context=None)

Bases: airflow.utils.log.logging_mixin.LoggingMixin

Helper class for creating, monitoring, and otherwise interacting with Kubernetes pods for use with the KubernetesPodOperator

run_pod_async(pod, **kwargs)

Runs POD asynchronously

delete_pod(pod)

Deletes POD

create_pod(pod)

Launches the pod asynchronously.

await_pod_start(pod, startup_timeout=120)

Waits for the pod to reach phase other than Pending

Parameters

• pod (kubernetes.client.models.v1_pod.V1Pod) –

• startup_timeout (int) – Timeout (in seconds) for startup of the pod (if pod is pending for too long, fails task)

Returns

Return type

None

follow_container_logs(pod, container_name)

fetch_container_logs(pod, container_name, *, follow=False, since_time=None, post_termination_timeout=120)

Follows the logs of container and streams to airflow logging. Returns when container exits.

await_container_completion(pod, container_name)

Waits for the given container in the given pod to be completed

Parameters

• pod (kubernetes.client.models.v1_pod.V1Pod) – pod spec that will be monitored

• container_name (str) – name of the container within the pod to monitor

await_pod_completion(pod)

Monitors a pod and returns the final state

Parameters

pod (kubernetes.client.models.v1_pod.V1Pod) – pod spec that will be monitored

Returns

tuple[State, str | None]

Return type

kubernetes.client.models.v1_pod.V1Pod

parse_log_line(line)

Parse K8s log line and returns the final state

Parameters

line (str) – k8s log line

Returns

timestamp and log message

Return type

tuple[DateTime | None, str]

container_is_running(pod, container_name)

Reads pod and checks if container is running

container_is_terminated(pod, container_name)

Reads pod and checks if container is terminated

read_pod_logs(pod, container_name, tail_lines=None, timestamps=False, since_seconds=None, follow=True, post_termination_timeout=120)

Reads log from the POD

read_pod_events(pod)

Reads events from the POD

read_pod(pod)

Read POD information

await_xcom_sidecar_container_start(pod)

extract_xcom(pod)

Retrieves XCom value and kills xcom sidecar container

Previous Next

Was this entry helpful?