airflow.contrib.sensors.gcs_sensor

Module Contents

class airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectSensor(bucket, object, google_cloud_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)

Bases:airflow.sensors.base_sensor_operator.BaseSensorOperator

Checks for the existence of a file in Google Cloud Storage. Create a new GoogleCloudStorageObjectSensor.

param bucket

The Google cloud storage bucket where the object is.

type bucket

str

param object

The name of the object to check in the Google cloud storage bucket.

type object

str

param google_cloud_storage_conn_id

The connection ID to use when connecting to Google cloud storage.

type google_cloud_storage_conn_id

str

param delegate_to

The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

type delegate_to

str

template_fields = ['bucket', 'object']

ui_color = #f0eee4

poke(self, context)

airflow.contrib.sensors.gcs_sensor.ts_function(context)

Default callback for the GoogleCloudStorageObjectUpdatedSensor. The default behaviour is check for the object being updated after execution_date + schedule_interval.

class airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectUpdatedSensor(bucket, object, ts_func=ts_function, google_cloud_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)

Bases:airflow.sensors.base_sensor_operator.BaseSensorOperator

Checks if an object is updated in Google Cloud Storage. Create a new GoogleCloudStorageObjectUpdatedSensor.

param bucket

The Google cloud storage bucket where the object is.

type bucket

str

param object

The name of the object to download in the Google cloud storage bucket.

type object

str

param ts_func

Callback for defining the update condition. The default callback returns execution_date + schedule_interval. The callback takes the context as parameter.

type ts_func

function

param google_cloud_storage_conn_id

The connection ID to use when connecting to Google cloud storage.

type google_cloud_storage_conn_id

str

param delegate_to

The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

type delegate_to

str

template_fields = ['bucket', 'object']

template_ext = ['.sql']

ui_color = #f0eee4

poke(self, context)

class airflow.contrib.sensors.gcs_sensor.GoogleCloudStoragePrefixSensor(bucket, prefix, google_cloud_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)

Bases:airflow.sensors.base_sensor_operator.BaseSensorOperator

Checks for the existence of a files at prefix in Google Cloud Storage bucket. Create a new GoogleCloudStorageObjectSensor.

param bucket

The Google cloud storage bucket where the object is.

type bucket

str

param prefix

The name of the prefix to check in the Google cloud storage bucket.

type prefix

str

param google_cloud_storage_conn_id

The connection ID to use when connecting to Google cloud storage.

type google_cloud_storage_conn_id

str

param delegate_to

The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

type delegate_to

str

template_fields = ['bucket', 'prefix']

ui_color = #f0eee4

poke(self, context)