Google Cloud BigQuery Transfer Operator to BigQuery¶

Google Cloud BigQuery is Google Cloud’s serverless data warehouse offering. This operator can be used to copy data from one BigQuery table to another.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Select or create a Cloud Platform project using the Cloud Console.

Enable billing for your project, as described in the Google Cloud documentation.

Enable the API, as described in the Cloud Console documentation.
Install API libraries via pip.
pip install 'apache-airflow[google]'
Detailed information is available for Installation.
Setup a Google Cloud Connection.

Operator¶

Copying data from one BigQuery table to another is performed with the BigQueryToBigQueryOperator operator.

Use Jinja templating with source_project_dataset_tables, destination_project_dataset_table, labels, impersonation_chain to define values dynamically.

You may include multiple source tables, as well as define a write_disposition and a create_disposition. For more information, please refer to the links above.

Copying BigQuery tables¶

The following Operator copies data from one or more BigQuery tables to another.

tests/system/google/cloud/bigquery/example_bigquery_to_bigquery.py[source]

copy_selected_data = BigQueryToBigQueryOperator(
    task_id="copy_selected_data",
    source_project_dataset_tables=f"{DATASET_NAME}.{ORIGIN}",
    destination_project_dataset_table=f"{DATASET_NAME}.{TARGET}",
)

Reference¶

For further information, look at: