airflow.providers.amazon.aws.sensors.glue_catalog_partition¶

Classes¶

GlueCatalogPartitionSensor

Waits for a partition to show up in AWS Glue Catalog.

Module Contents¶

class airflow.providers.amazon.aws.sensors.glue_catalog_partition.GlueCatalogPartitionSensor(*, table_name, expression="ds='{{ ds }}'", database_name='default', poke_interval=60 * 3, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.sensors.base_aws.AwsBaseSensor[airflow.providers.amazon.aws.hooks.glue_catalog.GlueCatalogHook]

Waits for a partition to show up in AWS Glue Catalog.

See also

For more information on how to use this sensor, take a look at the guide: Wait on an AWS Glue Catalog Partition

Parameters:

table_name (str) – The name of the table to wait for, supports the dot notation (my_database.my_table)
expression (str) – The partition clause to wait for. This is passed as is to the AWS Glue Catalog API’s get_partitions function, and supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01". See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html #aws-glue-api-catalog-partitions-GetPartitions
database_name (str) – The name of the catalog database where the partitions reside.
poke_interval (int) – Time in seconds that the job should wait in between each tries
deferrable (bool) – If true, then the sensor will wait asynchronously for the partition to show up in the AWS Glue Catalog. (default: False, but can be overridden in config file by setting default_deferrable to True)
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

ui_color = '#C5CAE9'[source]¶

table_name[source]¶

expression = "ds='{{ ds }}'"[source]¶

database_name = 'default'[source]¶

poke_interval = 180[source]¶

deferrable = True[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

poke(context)[source]¶

Check for existence of the partition in the AWS Glue Catalog table.

execute_complete(context, event=None)[source]¶