airflow.providers.amazon.aws.sensors.comprehend
¶
Module Contents¶
Classes¶
General sensor behavior for Amazon Comprehend. |
|
Poll the state of the pii entities detection job until it reaches a completed state; fails if the job fails. |
|
Poll the state of the document classifier until it reaches a completed state; fails if the job fails. |
- class airflow.providers.amazon.aws.sensors.comprehend.ComprehendBaseSensor(deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.sensors.base_aws.AwsBaseSensor
[airflow.providers.amazon.aws.hooks.comprehend.ComprehendHook
]General sensor behavior for Amazon Comprehend.
- Subclasses must implement following methods:
get_state()
- Subclasses must set the following fields:
INTERMEDIATE_STATES
FAILURE_STATES
SUCCESS_STATES
FAILURE_MESSAGE
- Parameters
deferrable (bool) – If True, the sensor will operate in deferrable mode. This mode requires aiobotocore module to be installed. (default: False, but can be overridden in config file by setting default_deferrable to True)
- class airflow.providers.amazon.aws.sensors.comprehend.ComprehendStartPiiEntitiesDetectionJobCompletedSensor(*, job_id, max_retries=75, poke_interval=120, **kwargs)[source]¶
Bases:
ComprehendBaseSensor
Poll the state of the pii entities detection job until it reaches a completed state; fails if the job fails.
See also
For more information on how to use this sensor, take a look at the guide: Wait for an Amazon Comprehend Start PII Entities Detection Job
- Parameters
job_id (str) – The id of the Comprehend pii entities detection job.
deferrable – If True, the sensor will operate in deferrable mode. This mode requires aiobotocore module to be installed. (default: False, but can be overridden in config file by setting default_deferrable to True)
poke_interval (int) – Polling period in seconds to check for the status of the job. (default: 120)
max_retries (int) – Number of times before returning the current state. (default: 75)
aws_conn_id – The Airflow connection used for AWS credentials. If this is
None
or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html
- class airflow.providers.amazon.aws.sensors.comprehend.ComprehendCreateDocumentClassifierCompletedSensor(*, document_classifier_arn, fail_on_warnings=False, max_retries=75, poke_interval=120, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), aws_conn_id='aws_default', **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.sensors.base_aws.AwsBaseSensor
[airflow.providers.amazon.aws.hooks.comprehend.ComprehendHook
]Poll the state of the document classifier until it reaches a completed state; fails if the job fails.
See also
For more information on how to use this sensor, take a look at the guide: Wait for an Amazon Comprehend Document Classifier
- Parameters
document_classifier_arn (str) – The arn of the Comprehend document classifier.
fail_on_warnings (bool) – If set to True, the document classifier training job will throw an error when the status is TRAINED_WITH_WARNING. (default False)
deferrable (bool) – If True, the sensor will operate in deferrable mode. This mode requires aiobotocore module to be installed. (default: False, but can be overridden in config file by setting default_deferrable to True)
poke_interval (int) – Polling period in seconds to check for the status of the job. (default: 120)
max_retries (int) – Number of times before returning the current state. (default: 75)
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is
None
or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html
- FAILURE_STATES: tuple[str, Ellipsis] = ('DELETING', 'STOP_REQUESTED', 'STOPPED', 'IN_ERROR')[source]¶