Upload data from Local Filesystem to Azure Data Lake¶

The Azure Data Lake (ADL) make it easy to store data of any size, shape, and speed. This page shows how to upload data from local filesystem to ADL.

Prerequisite Tasks¶

To use these operators, you must do a few things:

Create necessary resources using AZURE PORTAL or AZURE CLI.
Install API libraries via pip.
pip install 'apache-airflow[azure]'
Detailed information is available Installation of Airflow®
Setup Connection.

LocalFilesystemToADLSOperator¶

LocalFilesystemToADLSOperator allows you to upload data from local filesystem to ADL.

Below is an example of using this operator to upload a file to ADL.

tests/system/microsoft/azure/example_local_to_adls.py[source]

    upload_file = LocalFilesystemToADLSOperator(
        task_id="upload_task",
        local_path=LOCAL_FILE_PATH,
        remote_path=REMOTE_FILE_PATH,
    )

Reference¶

For further information, look at:

Azure Data lake Storage Documentation