airflow.providers.google.cloud.triggers.pubsub

This module contains Google Cloud Pubsub triggers.

Module Contents

Classes

PubsubPullTrigger

Initialize the Pubsub Pull Trigger with needed parameters.
class airflow.providers.google.cloud.triggers.pubsub.PubsubPullTrigger(project_id, subscription, max_messages, ack_messages, gcp_conn_id, messages_callback=None, poke_interval=10.0, impersonation_chain=None)[source]

Bases: airflow.triggers.base.BaseTrigger

Initialize the Pubsub Pull Trigger with needed parameters.

Parameters

  • project_id (str) – the Google Cloud project ID for the subscription (templated)

  • subscription (str) – the Pub/Sub subscription name. Do not include the full subscription path.

  • max_messages (int) – The maximum number of messages to retrieve per PubSub pull request

  • ack_messages (bool) – If True, each message will be acknowledged immediately rather than by any downstream tasks

  • gcp_conn_id (str) – Reference to google cloud connection id

  • messages_callback (Callable[[list[google.cloud.pubsub_v1.types.ReceivedMessage], airflow.utils.context.Context], Any] | None) – (Optional) Callback to process received messages. Its return value will be saved to XCom. If you are pulling large messages, you probably want to provide a custom callback. If not provided, the default implementation will convert ReceivedMessage objects into JSON-serializable dicts using google.protobuf.json_format.MessageToDict function.

  • poke_interval (float) – polling period in seconds to check for the status

  • impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

serialize()[source]

Serialize PubsubPullTrigger arguments and classpath.

async run()[source]

Run the trigger in an asynchronous context.

The trigger should yield an Event whenever it wants to fire off an event, and return None if it is finished. Single-event triggers should thus yield and then immediately return.

If it yields, it is likely that it will be resumed very quickly, but it may not be (e.g. if the workload is being moved to another triggerer process, or a multi-event trigger was being used for a single-event task defer).

In either case, Trigger classes should assume they will be persisted, and then rely on cleanup() being called when they are no longer needed.

async message_acknowledgement(pulled_messages)[source]

Was this entry helpful?