airflow.providers.microsoft.azure.operators.container_instances

Module Contents

Classes

AzureContainerInstancesOperator

Start a container on Azure Container Instances.

Attributes

Volume

DEFAULT_ENVIRONMENT_VARIABLES

DEFAULT_SECURED_VARIABLES

DEFAULT_VOLUMES

DEFAULT_MEMORY_IN_GB

DEFAULT_CPU

airflow.providers.microsoft.azure.operators.container_instances.Volume[source]
airflow.providers.microsoft.azure.operators.container_instances.DEFAULT_ENVIRONMENT_VARIABLES: dict[str, str][source]
airflow.providers.microsoft.azure.operators.container_instances.DEFAULT_SECURED_VARIABLES: Sequence[str] = [][source]
airflow.providers.microsoft.azure.operators.container_instances.DEFAULT_VOLUMES: Sequence[Volume] = [][source]
airflow.providers.microsoft.azure.operators.container_instances.DEFAULT_MEMORY_IN_GB = 2.0[source]
airflow.providers.microsoft.azure.operators.container_instances.DEFAULT_CPU = 1.0[source]
class airflow.providers.microsoft.azure.operators.container_instances.AzureContainerInstancesOperator(*, ci_conn_id, resource_group, name, image, region, registry_conn_id=None, environment_variables=None, secured_variables=None, volumes=None, memory_in_gb=None, cpu=None, gpu=None, command=None, remove_on_error=True, fail_if_exists=True, tags=None, os_type='Linux', restart_policy='Never', ip_address=None, ports=None, subnet_ids=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Start a container on Azure Container Instances.

Parameters

  • ci_conn_id (str) – connection id of a service principal which will be used to start the container instance

  • registry_conn_id (str | None) – connection id of a user which can login to a private docker registry. For Azure use Azure connection id

  • resource_group (str) – name of the resource group wherein this container instance should be started

  • name (str) – name of this container instance. Please note this name has to be unique in order to run containers in parallel.

  • image (str) – the docker image to be used

  • region (str) – the region wherein this container instance should be started

  • environment_variables (dict | None) – key,value pairs containing environment variables which will be passed to the running container

  • secured_variables (str | None) – names of environmental variables that should not be exposed outside the container (typically passwords).

  • volumes (list | None) – list of Volume tuples to be mounted to the container. Currently only Azure Fileshares are supported.

  • memory_in_gb (Any | None) – the amount of memory to allocate to this container

  • cpu (Any | None) – the number of cpus to allocate to this container

  • gpu (Any | None) – GPU Resource for the container.

  • command (list[str] | None) – the command to run inside the container

  • container_timeout – max time allowed for the execution of the container instance.

  • tags (dict[str, str] | None) – azure tags as dict of str:str

  • os_type (str) – The operating system type required by the containers in the container group. Possible values include: ‘Windows’, ‘Linux’

  • restart_policy (str) – Restart policy for all containers within the container group. Possible values include: ‘Always’, ‘OnFailure’, ‘Never’

  • ip_address (azure.mgmt.containerinstance.models.IpAddress | None) – The IP address type of the container group.

  • subnet_ids (list[azure.mgmt.containerinstance.models.ContainerGroupSubnetId] | None) – The subnet resource IDs for a container group

Example:

AzureContainerInstancesOperator(
    ci_conn_id="azure_service_principal",
    registry_conn_id="azure_registry_user",
    resource_group="my-resource-group",
    name="my-container-name-{{ ds }}",
    image="myprivateregistry.azurecr.io/my_container:latest",
    region="westeurope",
    environment_variables={
        "MODEL_PATH": "my_value",
        "POSTGRES_LOGIN": "{{ macros.connection('postgres_default').login }}",
        "POSTGRES_PASSWORD": "{{ macros.connection('postgres_default').password }}",
        "JOB_GUID": "{{ ti.xcom_pull(task_ids='task1', key='guid') }}",
    },
    secured_variables=["POSTGRES_PASSWORD"],
    volumes=[
        ("azure_container_instance_conn_id", "my_storage_container", "my_fileshare", "/input-data", True),
    ],
    memory_in_gb=14.0,
    cpu=4.0,
    gpu=GpuResource(count=1, sku="K80"),
    command=["/bin/echo", "world"],
    task_id="start_container",
)
template_fields: Sequence[str] = ('name', 'image', 'command', 'environment_variables', 'volumes')[source]
template_fields_renderers[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

on_kill()[source]

Override this method to clean up subprocesses when a task instance gets killed.

Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up, or it will leave ghost processes behind.

Was this entry helpful?