IO Operators

These operators perform various operations on a filesystem or object storage.

Transfer a file

Use the FileTransferOperator to copy a file from one location to another. Parameters of the operator are:

  • src - source path as a str or ObjectStoragePath

  • dst - destination path as a str or ObjectStoragePath

  • src_conn_id - source connection id (default: None)

  • dst_conn_id - destination connection id (default: None)

  • overwrite - overwrite destination (default: False)

If the src and the dst are both on the same object storage, copy will be performed in the object storage. Otherwise the data will be streamed from the source to the destination.

The example below shows how to instantiate the FileTransferOperator task.

tests/system/common/io/example_file_transfer_local_to_s3.py[source]

transfer = FileTransferOperator(src=temp_file_path, dst=AWS_BUCKET, task_id="transfer")

Was this entry helpful?