• Home
• airflow.providers.cncf.kubernetes
• airflow.providers.cncf.kubernetes.pod_generator

airflow.providers.cncf.kubernetes.pod_generator

Pod generator.

This module provides an interface between the previous Pod API and outputs a kubernetes.client.models.V1Pod. The advantage being that the full Kubernetes API is supported and no serialization need be written.

Module Contents

Classes

PodGenerator

Contains Kubernetes Airflow Worker configuration logic.

Functions

make_safe_label_value(string)	Normalize a provided label to be of valid length and characters.
datetime_to_label_safe_datestring(datetime_obj)	Transform a datetime string to use as a label.
label_safe_datestring_to_datetime(string)	Transform a label back to a datetime object.
merge_objects(base_obj, client_obj)	Merge objects.
extend_object_field(base_obj, client_obj, field_name)	Add field values to existing objects.

Attributes

log
MAX_LABEL_LEN

airflow.providers.cncf.kubernetes.pod_generator.log

airflow.providers.cncf.kubernetes.pod_generator.MAX_LABEL_LEN = 63

exception airflow.providers.cncf.kubernetes.pod_generator.PodMutationHookException

Bases: airflow.exceptions.AirflowException

Raised when exception happens during Pod Mutation Hook execution.

exception airflow.providers.cncf.kubernetes.pod_generator.PodReconciliationError

Bases: airflow.exceptions.AirflowException

Raised when an error is encountered while trying to merge pod configs.

airflow.providers.cncf.kubernetes.pod_generator.make_safe_label_value(string)

Normalize a provided label to be of valid length and characters.

Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.

If the label value is greater than 63 chars once made safe, or differs in any way from the original value sent to this function, then we need to truncate to 53 chars, and append it with a unique hash.

airflow.providers.cncf.kubernetes.pod_generator.datetime_to_label_safe_datestring(datetime_obj)

Transform a datetime string to use as a label.

Kubernetes doesn’t like “:” in labels, since ISO datetime format uses “:” but not “_” let’s replace “:” with “_”

Parameters

datetime_obj (datetime.datetime) – datetime.datetime object

Returns

ISO-like string representing the datetime

Return type

str

airflow.providers.cncf.kubernetes.pod_generator.label_safe_datestring_to_datetime(string)

Transform a label back to a datetime object.

Kubernetes doesn’t permit “:” in labels. ISO datetime format uses “:” but not “_”, let’s replace “:” with “_”

Parameters

string (str) – str

Returns

datetime.datetime object

Return type

datetime.datetime

class airflow.providers.cncf.kubernetes.pod_generator.PodGenerator(pod=None, pod_template_file=None, extract_xcom=True)

Contains Kubernetes Airflow Worker configuration logic.

Represents a kubernetes pod and manages execution of a single pod. Any configuration that is container specific gets applied to the first container in the list of containers.

Parameters

• pod (kubernetes.client.models.V1Pod | None) – The fully specified pod. Mutually exclusive with pod_template_file

• pod_template_file (str | None) – Path to YAML file. Mutually exclusive with pod

• extract_xcom (bool) – Whether to bring up a container for xcom

gen_pod()

Generate pod.

static add_xcom_sidecar(pod)

Add sidecar.

static from_obj(obj)

Convert to pod from obj.

static from_legacy_obj(obj)

Convert to pod from obj.

static reconcile_pods(base_pod, client_pod)

Merge Kubernetes Pod objects.

Parameters

• base_pod (kubernetes.client.models.V1Pod) – has the base attributes which are overwritten if they exist in the client pod and remain if they do not exist in the client_pod

• client_pod (kubernetes.client.models.V1Pod | None) – the pod that the client wants to create.

Returns

the merged pods

Return type

kubernetes.client.models.V1Pod

This can’t be done recursively as certain fields are overwritten and some are concatenated.

static reconcile_metadata(base_meta, client_meta)

Merge Kubernetes Metadata objects.

Parameters

• base_meta – has the base attributes which are overwritten if they exist in the client_meta and remain if they do not exist in the client_meta

• client_meta – the spec that the client wants to create.

Returns

the merged specs

static reconcile_specs(base_spec, client_spec)

Merge Kubernetes PodSpec objects.

Parameters

• base_spec (kubernetes.client.models.V1PodSpec | None) – has the base attributes which are overwritten if they exist in the client_spec and remain if they do not exist in the client_spec

• client_spec (kubernetes.client.models.V1PodSpec | None) – the spec that the client wants to create.

Returns

the merged specs

Return type

kubernetes.client.models.V1PodSpec | None

static reconcile_containers(base_containers, client_containers)

Merge Kubernetes Container objects.

Parameters

• base_containers (list[kubernetes.client.models.V1Container]) – has the base attributes which are overwritten if they exist in the client_containers and remain if they do not exist in the client_containers

• client_containers (list[kubernetes.client.models.V1Container]) – the containers that the client wants to create.

Returns

the merged containers

Return type

list[kubernetes.client.models.V1Container]

The runs recursively over the list of containers.

classmethod construct_pod(dag_id, task_id, pod_id, try_number, kube_image, date, args, pod_override_object, base_worker_pod, namespace, scheduler_job_id, run_id=None, map_index=-1, *, with_mutation_hook=False)

Create a Pod.

Construct a pod by gathering and consolidating the configuration from 3 places:

• airflow.cfg

• executor_config

• dynamic arguments

static serialize_pod(pod)

Convert a k8s.V1Pod into a json serializable dictionary.

Parameters

pod (kubernetes.client.models.V1Pod) – k8s.V1Pod object

Returns

Serialized version of the pod returned as dict

Return type

dict

static deserialize_model_file(path)

Generate a Pod from a file.

Parameters

path (str) – Path to the file

Returns

a kubernetes.client.models.V1Pod

Return type

kubernetes.client.models.V1Pod

static deserialize_model_dict(pod_dict)

Deserializes a Python dictionary to k8s.V1Pod.

Unfortunately we need access to the private method _ApiClient__deserialize_model from the kubernetes client. This issue is tracked here; https://github.com/kubernetes-client/python/issues/977.

Parameters

pod_dict (dict | None) – Serialized dict of k8s.V1Pod object

Returns

De-serialized k8s.V1Pod

Return type

kubernetes.client.models.V1Pod

static make_unique_pod_id(pod_id)

Generate a unique Pod name.

Kubernetes pod names must consist of one or more lowercase rfc1035/rfc1123 labels separated by ‘.’ with a maximum length of 253 characters.

Name must pass the following regex for validation ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$

For more details, see: https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/design/identifiers.md

Parameters

pod_id (str) – requested pod name

Returns

str valid Pod name of appropriate length

Return type

str | None

airflow.providers.cncf.kubernetes.pod_generator.merge_objects(base_obj, client_obj)

Merge objects.

Parameters

• base_obj – has the base attributes which are overwritten if they exist in the client_obj and remain if they do not exist in the client_obj

• client_obj – the object that the client wants to create.

Returns

the merged objects

airflow.providers.cncf.kubernetes.pod_generator.extend_object_field(base_obj, client_obj, field_name)

Add field values to existing objects.

Parameters

• base_obj – an object which has a property field_name that is a list

• client_obj – an object which has a property field_name that is a list. A copy of this object is returned with field_name modified

• field_name – the name of the list field

Returns

the client_obj with the property field_name being the two properties appended

Previous Next

Was this entry helpful?