Google Cloud BigQuery Transfer Operator to Google Cloud Storage¶

Google Cloud BigQuery is Google Cloud’s serverless data warehouse offering. Google Cloud Storage (GCS) is a managed service for storing unstructured data. This operator can be used to export data from BigQuery tables into files in a Cloud Storage bucket.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Select or create a Cloud Platform project using the Cloud Console.

Enable billing for your project, as described in the Google Cloud documentation.

Enable the API, as described in the Cloud Console documentation.
Install API libraries via pip.
pip install 'apache-airflow[google]'
Detailed information is available for Installation.
Setup a Google Cloud Connection.

Operator¶

File transfer from GCS to BigQuery is performed with the BigQueryToGCSOperator operator.

Use Jinja templating with source_project_dataset_table, destination_cloud_storage_uris, export_format, labels, impersonation_chain, job_id to define values dynamically.

You may define multiple destination URIs, as well as other settings such as compression and export_format. For more information, please refer to the links above.

Importing files¶

The following Operator imports one or more files from GCS into a BigQuery table.

tests/system/providers/google/cloud/bigquery/example_bigquery_to_gcs.py[source]

bigquery_to_gcs = BigQueryToGCSOperator(
    task_id="bigquery_to_gcs",
    source_project_dataset_table=f"{DATASET_NAME}.{TABLE}",
    destination_cloud_storage_uris=[f"gs://{BUCKET_NAME}/{BUCKET_FILE}"],
)

Reference¶

For further information, look at: