airflow.providers.amazon.aws.sensors.glue
¶
Module Contents¶
Classes¶
Waits for an AWS Glue Job to reach any of the status below. |
|
Waits for an AWS Glue data quality ruleset evaluation run to reach any of the status below. |
|
Waits for an AWS Glue data quality recommendation run to reach any of the status below. |
- class airflow.providers.amazon.aws.sensors.glue.GlueJobSensor(*, job_name, run_id, verbose=False, aws_conn_id='aws_default', **kwargs)[source]¶
Bases:
airflow.sensors.base.BaseSensorOperator
Waits for an AWS Glue Job to reach any of the status below.
‘FAILED’, ‘STOPPED’, ‘SUCCEEDED’
See also
For more information on how to use this sensor, take a look at the guide: Wait on an AWS Glue job state
- Parameters
- class airflow.providers.amazon.aws.sensors.glue.GlueDataQualityRuleSetEvaluationRunSensor(*, evaluation_run_id, show_results=True, verify_result_status=True, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), poke_interval=120, max_retries=60, aws_conn_id='aws_default', **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.sensors.base_aws.AwsBaseSensor
[airflow.providers.amazon.aws.hooks.glue.GlueDataQualityHook
]Waits for an AWS Glue data quality ruleset evaluation run to reach any of the status below.
‘FAILED’, ‘STOPPED’, ‘STOPPING’, ‘TIMEOUT’, ‘SUCCEEDED’
See also
For more information on how to use this sensor, take a look at the guide: Wait on an AWS Glue Data Quality Evaluation Run
- Parameters
evaluation_run_id (str) – The AWS Glue data quality ruleset evaluation run identifier.
verify_result_status (bool) – Validate all the ruleset rules evaluation run results, If any of the rule status is Fail or Error then an exception is thrown. (default: True)
show_results (bool) – Displays all the ruleset rules evaluation run results. (default: True)
deferrable (bool) – If True, the sensor will operate in deferrable mode. This mode requires aiobotocore module to be installed. (default: False, but can be overridden in config file by setting default_deferrable to True)
poke_interval (int) – Polling period in seconds to check for the status of the job. (default: 120)
max_retries (int) – Number of times before returning the current state. (default: 60)
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is
None
or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html
- class airflow.providers.amazon.aws.sensors.glue.GlueDataQualityRuleRecommendationRunSensor(*, recommendation_run_id, show_results=True, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), poke_interval=120, max_retries=60, aws_conn_id='aws_default', **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.sensors.base_aws.AwsBaseSensor
[airflow.providers.amazon.aws.hooks.glue.GlueDataQualityHook
]Waits for an AWS Glue data quality recommendation run to reach any of the status below.
‘FAILED’, ‘STOPPED’, ‘STOPPING’, ‘TIMEOUT’, ‘SUCCEEDED’
See also
For more information on how to use this sensor, take a look at the guide: Wait on an AWS Glue Data Quality Recommendation Run
- Parameters
recommendation_run_id (str) – The AWS Glue data quality rule recommendation run identifier.
show_results (bool) – Displays the recommended ruleset (a set of rules), when recommendation run completes. (default: True)
deferrable (bool) – If True, the sensor will operate in deferrable mode. This mode requires aiobotocore module to be installed. (default: False, but can be overridden in config file by setting default_deferrable to True)
poke_interval (int) – Polling period in seconds to check for the status of the job. (default: 120)
max_retries (int) – Number of times before returning the current state. (default: 60)
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is
None
or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html