airflow.providers.amazon.aws.sensors.glue_catalog_partition

Module Contents

Classes

GlueCatalogPartitionSensor

Waits for a partition to show up in AWS Glue Catalog.

class airflow.providers.amazon.aws.sensors.glue_catalog_partition.GlueCatalogPartitionSensor(*, table_name, expression="ds='{{ ds }}'", aws_conn_id='aws_default', region_name=None, database_name='default', poke_interval=60 * 3, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Waits for a partition to show up in AWS Glue Catalog.

Parameters
  • table_name (str) – The name of the table to wait for, supports the dot notation (my_database.my_table)

  • expression (str) – The partition clause to wait for. This is passed as is to the AWS Glue Catalog API’s get_partitions function, and supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01". See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html #aws-glue-api-catalog-partitions-GetPartitions

  • aws_conn_id (str | None) – ID of the Airflow connection where credentials and extra configuration are stored

  • region_name (str | None) – Optional aws region name (example: us-east-1). Uses region from connection if not specified.

  • database_name (str) – The name of the catalog database where the partitions reside.

  • poke_interval (int) – Time in seconds that the job should wait in between each tries

  • deferrable (bool) – If true, then the sensor will wait asynchronously for the partition to show up in the AWS Glue Catalog. (default: False, but can be overridden in config file by setting default_deferrable to True)

template_fields: Sequence[str] = ('database_name', 'table_name', 'expression')[source]
ui_color = '#C5CAE9'[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

poke(context)[source]

Check for existence of the partition in the AWS Glue Catalog table.

execute_complete(context, event=None)[source]
get_hook()[source]

Get the GlueCatalogHook.

hook()[source]

Was this entry helpful?