airflow.providers.microsoft.azure.hooks.data_factory

Spelling exceptions.

Module Contents

Classes

AzureDataFactoryPipelineRunStatus

Azure Data Factory pipeline operation statuses.

AzureDataFactoryHook

A hook to interact with Azure Data Factory.

AzureDataFactoryAsyncHook

An Async Hook that connects to Azure DataFactory to perform pipeline operations.

Functions

provide_targeted_factory(func)

Provide the targeted factory to the decorated function in case it isn't specified.

get_field(extras, field_name[, strict])

Get field from extra, first checking short name, then for backcompat we check for prefixed name.

provide_targeted_factory_async(func)

Provide the targeted factory to the async decorated function in case it isn't specified.

Attributes

Credentials

AsyncCredentials

T

airflow.providers.microsoft.azure.hooks.data_factory.Credentials[source]
airflow.providers.microsoft.azure.hooks.data_factory.AsyncCredentials[source]
airflow.providers.microsoft.azure.hooks.data_factory.T[source]
airflow.providers.microsoft.azure.hooks.data_factory.provide_targeted_factory(func)[source]

Provide the targeted factory to the decorated function in case it isn’t specified.

If resource_group_name or factory_name is not provided it defaults to the value specified in the connection extras.

class airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryPipelineRunStatus[source]

Azure Data Factory pipeline operation statuses.

QUEUED = 'Queued'[source]
IN_PROGRESS = 'InProgress'[source]
SUCCEEDED = 'Succeeded'[source]
FAILED = 'Failed'[source]
CANCELING = 'Canceling'[source]
CANCELLED = 'Cancelled'[source]
TERMINAL_STATUSES[source]
INTERMEDIATE_STATES[source]
FAILURE_STATES[source]
exception airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryPipelineRunException[source]

Bases: airflow.exceptions.AirflowException

An exception that indicates a pipeline run failed to complete.

airflow.providers.microsoft.azure.hooks.data_factory.get_field(extras, field_name, strict=False)[source]

Get field from extra, first checking short name, then for backcompat we check for prefixed name.

class airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryHook(azure_data_factory_conn_id=default_conn_name)[source]

Bases: airflow.hooks.base.BaseHook

A hook to interact with Azure Data Factory.

Parameters

azure_data_factory_conn_id (str) – The Azure Data Factory connection id.

conn_type: str = 'azure_data_factory'[source]
conn_name_attr: str = 'azure_data_factory_conn_id'[source]
default_conn_name: str = 'azure_data_factory_default'[source]
hook_name: str = 'Azure Data Factory'[source]
classmethod get_connection_form_widgets()[source]

Return connection widgets to add to connection form.

classmethod get_ui_field_behaviour()[source]

Return custom field behaviour.

get_conn()[source]

Return connection for the hook.

refresh_conn()[source]
get_factory(resource_group_name, factory_name, **config)[source]

Get the factory.

Parameters

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Returns

The factory.

Return type

azure.mgmt.datafactory.models.Factory | None

update_factory(factory, resource_group_name, factory_name, if_match=None, **config)[source]

Update the factory.

Parameters

  • factory (azure.mgmt.datafactory.models.Factory) – The factory resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • if_match (str | None) – ETag of the factory entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update. Default value is None.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the factory does not exist.

Returns

The factory.

Return type

azure.mgmt.datafactory.models.Factory

create_factory(factory, resource_group_name, factory_name, **config)[source]

Create the factory.

Parameters

  • factory (azure.mgmt.datafactory.models.Factory) – The factory resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the factory already exists.

Returns

The factory.

Return type

azure.mgmt.datafactory.models.Factory

delete_factory(resource_group_name, factory_name, **config)[source]

Delete the factory.

Parameters

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

get_linked_service(linked_service_name, resource_group_name, factory_name, if_none_match=None, **config)[source]

Get the linked service.

Parameters

  • linked_service_name (str) – The linked service name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • if_none_match (str | None) – ETag of the linked service entity. Should only be specified for get. If the ETag matches the existing entity tag, or if * was provided, then no content will be returned. Default value is None.

  • config (Any) – Extra parameters for the ADF client.

Returns

The linked service.

Return type

azure.mgmt.datafactory.models.LinkedServiceResource | None

update_linked_service(linked_service_name, linked_service, resource_group_name, factory_name, **config)[source]

Update the linked service.

Parameters

  • linked_service_name (str) – The linked service name.

  • linked_service (azure.mgmt.datafactory.models.LinkedServiceResource) – The linked service resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the linked service does not exist.

Returns

The linked service.

Return type

azure.mgmt.datafactory.models.LinkedServiceResource

create_linked_service(linked_service_name, linked_service, resource_group_name, factory_name, **config)[source]

Create the linked service.

Parameters

  • linked_service_name (str) – The linked service name.

  • linked_service (azure.mgmt.datafactory.models.LinkedServiceResource) – The linked service resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the linked service already exists.

Returns

The linked service.

Return type

azure.mgmt.datafactory.models.LinkedServiceResource

delete_linked_service(linked_service_name, resource_group_name, factory_name, **config)[source]

Delete the linked service.

Parameters

  • linked_service_name (str) – The linked service name.

  • resource_group_name (str) – The linked service name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

get_dataset(dataset_name, resource_group_name, factory_name, **config)[source]

Get the dataset.

Parameters

  • dataset_name (str) – The dataset name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Returns

The dataset.

Return type

azure.mgmt.datafactory.models.DatasetResource | None

update_dataset(dataset_name, dataset, resource_group_name, factory_name, **config)[source]

Update the dataset.

Parameters

  • dataset_name (str) – The dataset name.

  • dataset (azure.mgmt.datafactory.models.DatasetResource) – The dataset resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the dataset does not exist.

Returns

The dataset.

Return type

azure.mgmt.datafactory.models.DatasetResource

create_dataset(dataset_name, dataset, resource_group_name, factory_name, **config)[source]

Create the dataset.

Parameters

  • dataset_name (str) – The dataset name.

  • dataset (azure.mgmt.datafactory.models.DatasetResource) – The dataset resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the dataset already exists.

Returns

The dataset.

Return type

azure.mgmt.datafactory.models.DatasetResource

delete_dataset(dataset_name, resource_group_name, factory_name, **config)[source]

Delete the dataset.

Parameters

  • dataset_name (str) – The dataset name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

get_dataflow(dataflow_name, resource_group_name, factory_name, if_none_match=None, **config)[source]

Get the dataflow.

Parameters

  • dataflow_name (str) – The dataflow name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • if_none_match (str | None) – ETag of the data flow entity. Should only be specified for get. If the ETag matches the existing entity tag, or if * was provided, then no content will be returned. Default value is None.

  • config (Any) – Extra parameters for the ADF client.

Returns

The DataFlowResource.

Return type

azure.mgmt.datafactory.models.DataFlowResource

update_dataflow(dataflow_name, dataflow, resource_group_name, factory_name, if_match=None, **config)[source]

Update the dataflow.

Parameters

  • dataflow_name (str) – The dataflow name.

  • dataflow (azure.mgmt.datafactory.models.DataFlowResource | IO) – The dataflow resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • if_match (str | None) – ETag of the data flow entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update. Default value is None.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the dataset does not exist.

Returns

DataFlowResource.

Return type

azure.mgmt.datafactory.models.DataFlowResource

create_dataflow(dataflow_name, dataflow, resource_group_name, factory_name, if_match=None, **config)[source]

Create the dataflow.

Parameters

  • dataflow_name (str) – The dataflow name.

  • dataflow (azure.mgmt.datafactory.models.DataFlowResource) – The dataflow resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • if_match (str | None) – ETag of the factory entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update. Default value is None.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the dataset already exists.

Returns

The dataset.

Return type

azure.mgmt.datafactory.models.DataFlowResource

delete_dataflow(dataflow_name, resource_group_name, factory_name, **config)[source]

Delete the dataflow.

Parameters

  • dataflow_name (str) – The dataflow name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

get_pipeline(pipeline_name, resource_group_name, factory_name, **config)[source]

Get the pipeline.

Parameters

  • pipeline_name (str) – The pipeline name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Returns

The pipeline.

Return type

azure.mgmt.datafactory.models.PipelineResource | None

update_pipeline(pipeline_name, pipeline, resource_group_name, factory_name, **config)[source]

Update the pipeline.

Parameters

  • pipeline_name (str) – The pipeline name.

  • pipeline (azure.mgmt.datafactory.models.PipelineResource) – The pipeline resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the pipeline does not exist.

Returns

The pipeline.

Return type

azure.mgmt.datafactory.models.PipelineResource

create_pipeline(pipeline_name, pipeline, resource_group_name, factory_name, **config)[source]

Create the pipeline.

Parameters

  • pipeline_name (str) – The pipeline name.

  • pipeline (azure.mgmt.datafactory.models.PipelineResource) – The pipeline resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the pipeline already exists.

Returns

The pipeline.

Return type

azure.mgmt.datafactory.models.PipelineResource

delete_pipeline(pipeline_name, resource_group_name, factory_name, **config)[source]

Delete the pipeline.

Parameters

  • pipeline_name (str) – The pipeline name.

  • resource_group_name (str) – The pipeline name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

run_pipeline(pipeline_name, resource_group_name, factory_name, **config)[source]

Run a pipeline.

Parameters

  • pipeline_name (str) – The pipeline name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Returns

The pipeline run.

Return type

azure.mgmt.datafactory.models.CreateRunResponse

get_pipeline_run(run_id, resource_group_name, factory_name, **config)[source]

Get the pipeline run.

Parameters

  • run_id (str) – The pipeline run identifier.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Returns

The pipeline run.

Return type

azure.mgmt.datafactory.models.PipelineRun

get_pipeline_run_status(run_id, resource_group_name, factory_name)[source]

Get a pipeline run’s current status.

Parameters

  • run_id (str) – The pipeline run identifier.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

Returns

The status of the pipeline run.

Return type

str

wait_for_pipeline_run_status(run_id, expected_statuses, resource_group_name, factory_name, check_interval=60, timeout=60 * 60 * 24 * 7)[source]

Wait for a pipeline run to match an expected status.

Parameters

  • run_id (str) – The pipeline run identifier.

  • expected_statuses (str | set[str]) – The desired status(es) to check against a pipeline run’s current status.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • check_interval (int) – Time in seconds to check on a pipeline run’s status.

  • timeout (int) – Time in seconds to wait for a pipeline to reach a terminal status or the expected status.

Returns

Boolean indicating if the pipeline run has reached the expected_status.

Return type

bool

cancel_pipeline_run(run_id, resource_group_name, factory_name, **config)[source]

Cancel the pipeline run.

Parameters

  • run_id (str) – The pipeline run identifier.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

get_trigger(trigger_name, resource_group_name, factory_name, **config)[source]

Get the trigger.

Parameters

  • trigger_name (str) – The trigger name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Returns

The trigger.

Return type

azure.mgmt.datafactory.models.TriggerResource | None

update_trigger(trigger_name, trigger, resource_group_name, factory_name, if_match=None, **config)[source]

Update the trigger.

Parameters

  • trigger_name (str) – The trigger name.

  • trigger (azure.mgmt.datafactory.models.TriggerResource) – The trigger resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • if_match (str | None) – ETag of the trigger entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update. Default value is None.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the trigger does not exist.

Returns

The trigger.

Return type

azure.mgmt.datafactory.models.TriggerResource

create_trigger(trigger_name, trigger, resource_group_name, factory_name, **config)[source]

Create the trigger.

Parameters

  • trigger_name (str) – The trigger name.

  • trigger (azure.mgmt.datafactory.models.TriggerResource) – The trigger resource definition.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Raises

AirflowException – If the trigger already exists.

Returns

The trigger.

Return type

azure.mgmt.datafactory.models.TriggerResource

delete_trigger(trigger_name, resource_group_name, factory_name, **config)[source]

Delete the trigger.

Parameters

  • trigger_name (str) – The trigger name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

start_trigger(trigger_name, resource_group_name, factory_name, **config)[source]

Start the trigger.

Parameters

  • trigger_name (str) – The trigger name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Returns

An Azure operation poller.

Return type

azure.core.polling.LROPoller

stop_trigger(trigger_name, resource_group_name, factory_name, **config)[source]

Stop the trigger.

Parameters

  • trigger_name (str) – The trigger name.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Returns

An Azure operation poller.

Return type

azure.core.polling.LROPoller

rerun_trigger(trigger_name, run_id, resource_group_name, factory_name, **config)[source]

Rerun the trigger.

Parameters

  • trigger_name (str) – The trigger name.

  • run_id (str) – The trigger run identifier.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

cancel_trigger(trigger_name, run_id, resource_group_name, factory_name, **config)[source]

Cancel the trigger.

Parameters

  • trigger_name (str) – The trigger name.

  • run_id (str) – The trigger run identifier.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

test_connection()[source]

Test a configured Azure Data Factory connection.

airflow.providers.microsoft.azure.hooks.data_factory.provide_targeted_factory_async(func)[source]

Provide the targeted factory to the async decorated function in case it isn’t specified.

If resource_group_name or factory_name is not provided it defaults to the value specified in the connection extras.

class airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryAsyncHook(azure_data_factory_conn_id=default_conn_name)[source]

Bases: AzureDataFactoryHook

An Async Hook that connects to Azure DataFactory to perform pipeline operations.

Parameters

azure_data_factory_conn_id (str) – The Azure Data Factory connection id.

default_conn_name: str = 'azure_data_factory_default'[source]
async get_async_conn()[source]

Get async connection and connect to azure data factory.

async refresh_conn()[source]
async get_pipeline_run(run_id, resource_group_name, factory_name, **config)[source]

Connect to Azure Data Factory asynchronously to get the pipeline run details by run id.

Parameters

  • run_id (str) – The pipeline run identifier.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

async get_adf_pipeline_run_status(run_id, resource_group_name, factory_name)[source]

Connect to Azure Data Factory asynchronously and get the pipeline status by run_id.

Parameters

  • run_id (str) – The pipeline run identifier.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

async cancel_pipeline_run(run_id, resource_group_name, factory_name, **config)[source]

Cancel the pipeline run.

Parameters

  • run_id (str) – The pipeline run identifier.

  • resource_group_name (str) – The resource group name.

  • factory_name (str) – The factory name.

  • config (Any) – Extra parameters for the ADF client.

Was this entry helpful?