airflow.providers.microsoft.azure.hooks.data_factory

Module Contents

Classes

PipelineRunInfo

Type class for the pipeline run info dictionary.

AzureDataFactoryPipelineRunStatus

Azure Data Factory pipeline operation statuses.

AzureDataFactoryHook

A hook to interact with Azure Data Factory.

Functions

provide_targeted_factory(func)

Provide the targeted factory to the decorated function in case it isn't specified.

Attributes

Credentials

airflow.providers.microsoft.azure.hooks.data_factory.Credentials[source]
airflow.providers.microsoft.azure.hooks.data_factory.provide_targeted_factory(func)[source]

Provide the targeted factory to the decorated function in case it isn't specified.

If resource_group_name or factory_name is not provided it defaults to the value specified in the connection extras.

class airflow.providers.microsoft.azure.hooks.data_factory.PipelineRunInfo[source]

Bases: airflow.typing_compat.TypedDict

Type class for the pipeline run info dictionary.

run_id :str[source]
factory_name :Optional[str][source]
resource_group_name :Optional[str][source]
class airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryPipelineRunStatus[source]

Azure Data Factory pipeline operation statuses.

QUEUED = Queued[source]
IN_PROGRESS = InProgress[source]
SUCCEEDED = Succeeded[source]
FAILED = Failed[source]
CANCELING = Canceling[source]
CANCELLED = Cancelled[source]
TERMINAL_STATUSES[source]
exception airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryPipelineRunException[source]

Bases: airflow.exceptions.AirflowException

An exception that indicates a pipeline run failed to complete.

class airflow.providers.microsoft.azure.hooks.data_factory.AzureDataFactoryHook(azure_data_factory_conn_id=default_conn_name)[source]

Bases: airflow.hooks.base.BaseHook

A hook to interact with Azure Data Factory.

Parameters

azure_data_factory_conn_id (str) -- The Azure Data Factory connection id.

conn_type :str = azure_data_factory[source]
conn_name_attr :str = azure_data_factory_conn_id[source]
default_conn_name :str = azure_data_factory_default[source]
hook_name :str = Azure Data Factory[source]
static get_connection_form_widgets()[source]

Returns connection widgets to add to connection form

static get_ui_field_behaviour()[source]

Returns custom field behaviour

get_conn(self)[source]

Returns connection for the hook.

get_factory(self, resource_group_name=None, factory_name=None, **config)[source]

Get the factory.

Parameters

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Returns

The factory.

Return type

azure.mgmt.datafactory.models.Factory

update_factory(self, factory, resource_group_name=None, factory_name=None, **config)[source]

Update the factory.

Parameters

  • factory (azure.mgmt.datafactory.models.Factory) -- The factory resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the factory does not exist.

Returns

The factory.

Return type

azure.mgmt.datafactory.models.Factory

create_factory(self, factory, resource_group_name=None, factory_name=None, **config)[source]

Create the factory.

Parameters

  • factory (azure.mgmt.datafactory.models.Factory) -- The factory resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the factory already exists.

Returns

The factory.

Return type

azure.mgmt.datafactory.models.Factory

delete_factory(self, resource_group_name=None, factory_name=None, **config)[source]

Delete the factory.

Parameters

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

get_linked_service(self, linked_service_name, resource_group_name=None, factory_name=None, **config)[source]

Get the linked service.

Parameters

  • linked_service_name (str) -- The linked service name.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Returns

The linked service.

Return type

azure.mgmt.datafactory.models.LinkedServiceResource

update_linked_service(self, linked_service_name, linked_service, resource_group_name=None, factory_name=None, **config)[source]

Update the linked service.

Parameters

  • linked_service_name (str) -- The linked service name.

  • linked_service (azure.mgmt.datafactory.models.LinkedServiceResource) -- The linked service resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the linked service does not exist.

Returns

The linked service.

Return type

azure.mgmt.datafactory.models.LinkedServiceResource

create_linked_service(self, linked_service_name, linked_service, resource_group_name=None, factory_name=None, **config)[source]

Create the linked service.

Parameters

  • linked_service_name (str) -- The linked service name.

  • linked_service (azure.mgmt.datafactory.models.LinkedServiceResource) -- The linked service resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the linked service already exists.

Returns

The linked service.

Return type

azure.mgmt.datafactory.models.LinkedServiceResource

delete_linked_service(self, linked_service_name, resource_group_name=None, factory_name=None, **config)[source]

Delete the linked service.

Parameters

  • linked_service_name (str) -- The linked service name.

  • resource_group_name (Optional[str]) -- The linked service name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

get_dataset(self, dataset_name, resource_group_name=None, factory_name=None, **config)[source]

Get the dataset.

Parameters

  • dataset_name (str) -- The dataset name.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Returns

The dataset.

Return type

azure.mgmt.datafactory.models.DatasetResource

update_dataset(self, dataset_name, dataset, resource_group_name=None, factory_name=None, **config)[source]

Update the dataset.

Parameters

  • dataset_name (str) -- The dataset name.

  • dataset (azure.mgmt.datafactory.models.DatasetResource) -- The dataset resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the dataset does not exist.

Returns

The dataset.

Return type

azure.mgmt.datafactory.models.DatasetResource

create_dataset(self, dataset_name, dataset, resource_group_name=None, factory_name=None, **config)[source]

Create the dataset.

Parameters

  • dataset_name (str) -- The dataset name.

  • dataset (azure.mgmt.datafactory.models.DatasetResource) -- The dataset resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the dataset already exists.

Returns

The dataset.

Return type

azure.mgmt.datafactory.models.DatasetResource

delete_dataset(self, dataset_name, resource_group_name=None, factory_name=None, **config)[source]

Delete the dataset.

Parameters

  • dataset_name (str) -- The dataset name.

  • resource_group_name (Optional[str]) -- The dataset name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

get_pipeline(self, pipeline_name, resource_group_name=None, factory_name=None, **config)[source]

Get the pipeline.

Parameters

  • pipeline_name (str) -- The pipeline name.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Returns

The pipeline.

Return type

azure.mgmt.datafactory.models.PipelineResource

update_pipeline(self, pipeline_name, pipeline, resource_group_name=None, factory_name=None, **config)[source]

Update the pipeline.

Parameters

  • pipeline_name (str) -- The pipeline name.

  • pipeline (azure.mgmt.datafactory.models.PipelineResource) -- The pipeline resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the pipeline does not exist.

Returns

The pipeline.

Return type

azure.mgmt.datafactory.models.PipelineResource

create_pipeline(self, pipeline_name, pipeline, resource_group_name=None, factory_name=None, **config)[source]

Create the pipeline.

Parameters

  • pipeline_name (str) -- The pipeline name.

  • pipeline (azure.mgmt.datafactory.models.PipelineResource) -- The pipeline resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the pipeline already exists.

Returns

The pipeline.

Return type

azure.mgmt.datafactory.models.PipelineResource

delete_pipeline(self, pipeline_name, resource_group_name=None, factory_name=None, **config)[source]

Delete the pipeline.

Parameters

  • pipeline_name (str) -- The pipeline name.

  • resource_group_name (Optional[str]) -- The pipeline name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

run_pipeline(self, pipeline_name, resource_group_name=None, factory_name=None, **config)[source]

Run a pipeline.

Parameters

  • pipeline_name (str) -- The pipeline name.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Returns

The pipeline run.

Return type

azure.mgmt.datafactory.models.CreateRunResponse

get_pipeline_run(self, run_id, resource_group_name=None, factory_name=None, **config)[source]

Get the pipeline run.

Parameters

  • run_id (str) -- The pipeline run identifier.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Returns

The pipeline run.

Return type

azure.mgmt.datafactory.models.PipelineRun

get_pipeline_run_status(self, run_id, resource_group_name=None, factory_name=None)[source]

Get a pipeline run's current status.

Parameters

  • run_id (str) -- The pipeline run identifier.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

Returns

The status of the pipeline run.

Return type

str

wait_for_pipeline_run_status(self, run_id, expected_statuses, resource_group_name=None, factory_name=None, check_interval=60, timeout=60 * 60 * 24 * 7)[source]

Waits for a pipeline run to match an expected status.

Parameters

  • run_id (str) -- The pipeline run identifier.

  • expected_statuses (Union[str, Set[str]]) -- The desired status(es) to check against a pipeline run's current status.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • check_interval (int) -- Time in seconds to check on a pipeline run's status.

  • timeout (int) -- Time in seconds to wait for a pipeline to reach a terminal status or the expected status.

Returns

Boolean indicating if the pipeline run has reached the expected_status.

Return type

bool

cancel_pipeline_run(self, run_id, resource_group_name=None, factory_name=None, **config)[source]

Cancel the pipeline run.

Parameters

  • run_id (str) -- The pipeline run identifier.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

get_trigger(self, trigger_name, resource_group_name=None, factory_name=None, **config)[source]

Get the trigger.

Parameters

  • trigger_name (str) -- The trigger name.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Returns

The trigger.

Return type

azure.mgmt.datafactory.models.TriggerResource

update_trigger(self, trigger_name, trigger, resource_group_name=None, factory_name=None, **config)[source]

Update the trigger.

Parameters

  • trigger_name (str) -- The trigger name.

  • trigger (azure.mgmt.datafactory.models.TriggerResource) -- The trigger resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the trigger does not exist.

Returns

The trigger.

Return type

azure.mgmt.datafactory.models.TriggerResource

create_trigger(self, trigger_name, trigger, resource_group_name=None, factory_name=None, **config)[source]

Create the trigger.

Parameters

  • trigger_name (str) -- The trigger name.

  • trigger (azure.mgmt.datafactory.models.TriggerResource) -- The trigger resource definition.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Raises

AirflowException -- If the trigger already exists.

Returns

The trigger.

Return type

azure.mgmt.datafactory.models.TriggerResource

delete_trigger(self, trigger_name, resource_group_name=None, factory_name=None, **config)[source]

Delete the trigger.

Parameters

  • trigger_name (str) -- The trigger name.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

start_trigger(self, trigger_name, resource_group_name=None, factory_name=None, **config)[source]

Start the trigger.

Parameters

  • trigger_name (str) -- The trigger name.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Returns

An Azure operation poller.

Return type

azure.core.polling.LROPoller

stop_trigger(self, trigger_name, resource_group_name=None, factory_name=None, **config)[source]

Stop the trigger.

Parameters

  • trigger_name (str) -- The trigger name.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

Returns

An Azure operation poller.

Return type

azure.core.polling.LROPoller

rerun_trigger(self, trigger_name, run_id, resource_group_name=None, factory_name=None, **config)[source]

Rerun the trigger.

Parameters

  • trigger_name (str) -- The trigger name.

  • run_id (str) -- The trigger run identifier.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

cancel_trigger(self, trigger_name, run_id, resource_group_name=None, factory_name=None, **config)[source]

Cancel the trigger.

Parameters

  • trigger_name (str) -- The trigger name.

  • run_id (str) -- The trigger run identifier.

  • resource_group_name (Optional[str]) -- The resource group name.

  • factory_name (Optional[str]) -- The factory name.

  • config (Any) -- Extra parameters for the ADF client.

test_connection(self)[source]

Test a configured Azure Data Factory connection.

Was this entry helpful?