airflow.providers.google.cloud.transfers.bigquery_to_sql

Base operator for BigQuery to SQL operators.

Module Contents

Classes

BigQueryToSqlBaseOperator

Fetch data from a BigQuery table (alternatively fetch selected columns) and insert it into an SQL table.
class airflow.providers.google.cloud.transfers.bigquery_to_sql.BigQueryToSqlBaseOperator(*, dataset_table, target_table_name, selected_fields=None, gcp_conn_id='google_cloud_default', database=None, replace=False, batch_size=1000, location=None, impersonation_chain=None, dataset_id=None, table_id=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Fetch data from a BigQuery table (alternatively fetch selected columns) and insert it into an SQL table.

This is a BaseOperator; an abstract class. Refer to children classes which are related to specific SQL databases (MySQL, MsSQL, Postgres…).

Note

If you pass fields to selected_fields which are in different order than the order of columns already in BQ table, the data will still be in the order of BQ table. For example if the BQ table has 3 columns as [A,B,C] and you pass ‘B,A’ in the selected_fields the data would still be of the form 'A,B' and passed through this form to the SQL database.

Parameters

  • dataset_table (str) – A dotted <dataset>.<table>: the big query table of origin

  • target_table_name (str | None) – target SQL table

  • selected_fields (list[str] | str | None) – List of fields to return (comma-separated). If unspecified, all fields are returned.

  • gcp_conn_id (str) – reference to a specific Google Cloud hook.

  • database (str | None) – name of database which overwrite defined one in connection

  • replace (bool) – Whether to replace instead of insert

  • batch_size (int) – The number of rows to take in each batch

  • location (str | None) – The location used for the operation.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('target_table_name', 'impersonation_chain', 'dataset_id', 'table_id')[source]
abstract get_sql_hook()[source]

Return a concrete SQL Hook (a PostgresHook for instance).

Persist the connection to the SQL provider.

execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Was this entry helpful?