Azure Data Factory Operators

Azure Data Factory is Azure's cloud ETL service for scale-out serverless data integration and data transformation. It offers a code-free UI for intuitive authoring and single-pane-of-glass monitoring and management.

AzureDataFactoryRunPipelineOperator

Use the AzureDataFactoryRunPipelineOperator to execute a pipeline within a data factory. By default, the operator will periodically check on the status of the executed pipeline to terminate with a "Succeeded" status. This functionality can be disabled for an asynchronous wait -- typically with the AzureDataFactoryPipelineRunSensor -- by setting wait_for_termination to False.

Below is an example of using this operator to execute an Azure Data Factory pipeline.

airflow/providers/microsoft/azure/example_dags/example_adf_run_pipeline.py[source]

    run_pipeline1: BaseOperator = AzureDataFactoryRunPipelineOperator(
        task_id="run_pipeline1",
        pipeline_name="pipeline1",
        parameters={"myParam": "value"},
    )

Here is a different example of using this operator to execute a pipeline but coupled with the AzureDataFactoryPipelineRunSensor to perform an asynchronous wait.

airflow/providers/microsoft/azure/example_dags/example_adf_run_pipeline.py[source]

    run_pipeline2: BaseOperator = AzureDataFactoryRunPipelineOperator(
        task_id="run_pipeline2",
        pipeline_name="pipeline2",
        wait_for_termination=False,
    )

    pipeline_run_sensor: BaseOperator = AzureDataFactoryPipelineRunStatusSensor(
        task_id="pipeline_run_sensor",
        run_id=run_pipeline2.output["run_id"],
    )

Reference

For further information, please refer to the Microsoft documentation:

Was this entry helpful?