Google Cloud Storage Transfer Operator to Samba¶

Google has a service Google Cloud Storage. This service is used to store large data from various applications. Samba is the standard Windows interoperability suite of programs for Linux and Unix. Samba has provided secure, stable and fast file and print services for clients using the SMB/CIFS protocol

Operator¶

Transfer files between Google Storage and Samba is performed with the GCSToSambaOperator operator.

Use Jinja templating with source_bucket, source_object, destination_path, impersonation_chain to define values dynamically.

Copying a single file¶

The following Operator copies a single file.

tests/system/samba/example_gcs_to_samba.py[source]

copy_file_from_gcs_to_samba = GCSToSambaOperator(
    task_id="file-copy-gcs-to-samba",
    samba_conn_id=SMB_CONN,
    source_bucket=BUCKET_NAME,
    source_object=GCS_SRC_FILE,
    destination_path=DESTINATION_PATH_1,
)

Moving a single file¶

To move the file use the move_object parameter. Once the file is copied to SMB, the original file from the Google Storage is deleted. The destination_path parameter defines the full path of the file on the Samba server.

tests/system/samba/example_gcs_to_samba.py[source]

move_file_from_gcs_to_samba = GCSToSambaOperator(
    task_id="file-move-gcs-to-samba",
    samba_conn_id=SMB_CONN,
    source_bucket=BUCKET_NAME,
    source_object=GCS_SRC_FILE_IN_DIR,
    destination_path=DESTINATION_PATH_1,
    move_object=True,
)

Copying a directory¶

Use the wildcard in source_path parameter to copy a directory.

tests/system/samba/example_gcs_to_samba.py[source]

copy_dir_from_gcs_to_samba = GCSToSambaOperator(
    task_id="dir-copy-gcs-to-samba",
    samba_conn_id=SMB_CONN,
    source_bucket=BUCKET_NAME,
    source_object=GCS_SRC_DIR,
    destination_path=DESTINATION_PATH_2,
)

Moving specific files¶

Use the wildcard in source_path parameter to move the specific files. The destination_path defines the path that is prefixed to all copied files.

tests/system/samba/example_gcs_to_samba.py[source]

move_dir_from_gcs_to_samba = GCSToSambaOperator(
    task_id="dir-move-gcs-to-samba",
    samba_conn_id=SMB_CONN,
    source_bucket=BUCKET_NAME,
    source_object=GCS_SRC_DIR,
    destination_path=DESTINATION_PATH_3,
    keep_directory_structure=False,
)