airflow.providers.microsoft.azure.hooks.synapse
¶
Module Contents¶
Classes¶
Azure Synapse Spark Job operation statuses. |
|
A hook to interact with Azure Synapse. |
|
Azure Synapse pipeline operation statuses. |
|
A base hook class to create session and connection to Azure Synapse using connection id. |
|
A hook to interact with Azure Synapse Pipeline. |
Attributes¶
- class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapseSparkBatchRunStatus[source]¶
Azure Synapse Spark Job operation statuses.
- class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapseHook(azure_synapse_conn_id=default_conn_name, spark_pool='')[source]¶
Bases:
airflow.hooks.base.BaseHook
A hook to interact with Azure Synapse.
- Parameters
azure_synapse_conn_id (str) – The Azure Synapse connection id.
spark_pool (str) – The Apache Spark pool used to submit the job
- classmethod get_connection_form_widgets()[source]¶
Return connection widgets to add to connection form.
- run_spark_job(payload)[source]¶
Run a job in an Apache Spark pool.
- Parameters
payload (azure.synapse.spark.models.SparkBatchJobOptions) – Livy compatible payload which represents the spark job that a user wants to submit.
- wait_for_job_run_status(job_id, expected_statuses, check_interval=60, timeout=60 * 60 * 24 * 7)[source]¶
Wait for a job run to match an expected status.
- Parameters
job_id (int | None) – The job run identifier.
expected_statuses (str | set[str]) – The desired status(es) to check against a job run’s current status.
check_interval (int) – Time in seconds to check on a job run’s status.
timeout (int) – Time in seconds to wait for a job to reach a terminal status or the expected status.
- class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapsePipelineRunStatus[source]¶
Azure Synapse pipeline operation statuses.
- exception airflow.providers.microsoft.azure.hooks.synapse.AzureSynapsePipelineRunException[source]¶
Bases:
airflow.exceptions.AirflowException
An exception that indicates a pipeline run failed to complete.
- class airflow.providers.microsoft.azure.hooks.synapse.BaseAzureSynapseHook(azure_synapse_conn_id=default_conn_name, **kwargs)[source]¶
Bases:
airflow.hooks.base.BaseHook
A base hook class to create session and connection to Azure Synapse using connection id.
- Parameters
azure_synapse_conn_id (str) – The Azure Synapse connection id.
- class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapsePipelineHook(azure_synapse_workspace_dev_endpoint, azure_synapse_conn_id=default_conn_name, **kwargs)[source]¶
Bases:
BaseAzureSynapseHook
A hook to interact with Azure Synapse Pipeline.
- Parameters
azure_synapse_conn_id (str) – The Azure Synapse connection id.
azure_synapse_workspace_dev_endpoint (str) – The Azure Synapse Workspace development endpoint.
- run_pipeline(pipeline_name, **config)[source]¶
Run a Synapse pipeline.
- Parameters
pipeline_name (str) – The pipeline name.
config (Any) – Extra parameters for the Synapse Artifact Client.
- Returns
The pipeline run Id.
- Return type
azure.synapse.artifacts.models.CreateRunResponse
- get_pipeline_run(run_id)[source]¶
Get the pipeline run.
- Parameters
run_id (str) – The pipeline run identifier.
- Returns
The pipeline run.
- Return type
azure.synapse.artifacts.models.PipelineRun
- wait_for_pipeline_run_status(run_id, expected_statuses, check_interval=60, timeout=60 * 60 * 24 * 7)[source]¶
Wait for a pipeline run to match an expected status.
- Parameters
run_id (str) – The pipeline run identifier.
expected_statuses (str | set[str]) – The desired status(es) to check against a pipeline run’s current status.
check_interval (int) – Time in seconds to check on a pipeline run’s status.
timeout (int) – Time in seconds to wait for a pipeline to reach a terminal status or the expected status.
- Returns
Boolean indicating if the pipeline run has reached the
expected_status
.- Return type