airflow.providers.amazon.aws.triggers.emr
¶
Module Contents¶
Classes¶
Poll for the status of EMR steps until they reach terminal state. |
|
Asynchronously poll the boto3 API and wait for the JobFlow to finish executing. |
|
Asynchronously poll the boto3 API and wait for the JobFlow to finish terminating. |
|
Poll for the status of EMR container until reaches terminal state. |
|
Poll for the status of EMR container until reaches terminal state. |
|
Poll an Emr Serverless application and wait for it to be created. |
|
Poll an Emr Serverless application and wait for it to be started. |
|
Poll an Emr Serverless application and wait for it to be stopped. |
|
Poll an Emr Serverless job run and wait for it to be completed. |
|
Poll an Emr Serverless application and wait for it to be deleted. |
|
Trigger for canceling a list of jobs in an EMR Serverless application. |
- class airflow.providers.amazon.aws.triggers.emr.EmrAddStepsTrigger(job_flow_id, step_ids, waiter_delay, waiter_max_attempts, aws_conn_id='aws_default')[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Poll for the status of EMR steps until they reach terminal state.
- class airflow.providers.amazon.aws.triggers.emr.EmrCreateJobFlowTrigger(job_flow_id, aws_conn_id=None, waiter_delay=30, waiter_max_attempts=60)[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Asynchronously poll the boto3 API and wait for the JobFlow to finish executing.
- Parameters
- class airflow.providers.amazon.aws.triggers.emr.EmrTerminateJobFlowTrigger(job_flow_id, aws_conn_id=None, waiter_delay=30, waiter_max_attempts=60)[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Asynchronously poll the boto3 API and wait for the JobFlow to finish terminating.
- Parameters
- class airflow.providers.amazon.aws.triggers.emr.EmrContainerTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', waiter_delay=30, waiter_max_attempts=sys.maxsize)[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Poll for the status of EMR container until reaches terminal state.
- Parameters
virtual_cluster_id (str) – Reference Emr cluster id
job_id (str) – job_id to check the state
aws_conn_id (str | None) – Reference to AWS connection id
waiter_delay (int) – polling period in seconds to check for the status
waiter_max_attempts (int) – The maximum number of attempts to be made. Defaults to an infinite wait.
- class airflow.providers.amazon.aws.triggers.emr.EmrStepSensorTrigger(job_flow_id, step_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Poll for the status of EMR container until reaches terminal state.
- Parameters
job_flow_id (str) – job_flow_id which contains the step check the state of
step_id (str) – step to check the state of
waiter_delay (int) – polling period in seconds to check for the status
waiter_max_attempts (int) – The maximum number of attempts to be made
aws_conn_id (str | None) – Reference to AWS connection id
- class airflow.providers.amazon.aws.triggers.emr.EmrServerlessCreateApplicationTrigger(application_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Poll an Emr Serverless application and wait for it to be created.
- Parameters
- Waiter_delay
polling period in seconds to check for the status
- class airflow.providers.amazon.aws.triggers.emr.EmrServerlessStartApplicationTrigger(application_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Poll an Emr Serverless application and wait for it to be started.
- Parameters
- Waiter_delay
polling period in seconds to check for the status
- class airflow.providers.amazon.aws.triggers.emr.EmrServerlessStopApplicationTrigger(application_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Poll an Emr Serverless application and wait for it to be stopped.
- Parameters
- Waiter_delay
polling period in seconds to check for the status
- class airflow.providers.amazon.aws.triggers.emr.EmrServerlessStartJobTrigger(application_id, job_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Poll an Emr Serverless job run and wait for it to be completed.
- Parameters
- Waiter_delay
polling period in seconds to check for the status
- class airflow.providers.amazon.aws.triggers.emr.EmrServerlessDeleteApplicationTrigger(application_id, waiter_delay=30, waiter_max_attempts=60, aws_conn_id='aws_default')[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Poll an Emr Serverless application and wait for it to be deleted.
- Parameters
- Waiter_delay
polling period in seconds to check for the status
- class airflow.providers.amazon.aws.triggers.emr.EmrServerlessCancelJobsTrigger(application_id, aws_conn_id, waiter_delay, waiter_max_attempts)[source]¶
Bases:
airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger
Trigger for canceling a list of jobs in an EMR Serverless application.
- Parameters
- property hook_instance: airflow.providers.amazon.aws.hooks.base_aws.AwsGenericHook[source]¶
This property is added for backward compatibility.