Azure Blob Storage to Google Cloud Storage (GCS) Transfer Operator¶

The Google Cloud Storage (GCS) is used to store large data from various applications. This is also the same with Azure Blob Storage. This page shows how to transfer data from Azure Blob Storage to GCS.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Select or create a Cloud Platform project using the Cloud Console.

Enable billing for your project, as described in the Google Cloud documentation.

Enable the API, as described in the Cloud Console documentation.
Install API libraries via pip.
pip install 'apache-airflow[google]'
Detailed information is available for Installation.
Setup a Google Cloud Connection.

Transfer Data from Azure Blob Storage to Google Cloud Storage¶

Operator transfers data from Azure Blob Storage to specified bucket in Google Cloud Storage

Use the AzureBlobStorageToGCSOperator to transfer data from Azure Blob Storage to Google Cloud Storage.

Example usage:

tests/system/google/cloud/azure/example_azure_blob_to_gcs.py[source]

    transfer_files_to_gcs = AzureBlobStorageToGCSOperator(
        task_id="transfer_files_to_gcs",
        # azure args
        container_name=AZURE_CONTAINER_NAME,
        blob_name=BLOB_NAME,
        # GCP args
        bucket_name=GCP_BUCKET_NAME,
        object_name=GCP_OBJECT_NAME,
        filename=GCP_BUCKET_FILE_PATH,
        gzip=False,
        impersonation_chain=None,
    )

Reference¶

For further information, look at: