Azure DataLake Storage Operators¶
Prerequisite Tasks¶
To use these operators, you must do a few things:
Create necessary resources using AZURE PORTAL or AZURE CLI.
Install API libraries via pip.
pip install 'apache-airflow[azure]'Detailed information is available Installation of Airflow®
ADLSCreateObjectOperator¶
ADLSCreateObjectOperator
allows you to
upload data to Azure DataLake Storage
Below is an example of using this operator to upload data to ADL.
upload_data = ADLSCreateObjectOperator(
task_id="upload_data",
file_system_name="Fabric",
file_name=REMOTE_FILE_PATH,
data="Hello world",
replace=True,
)
ADLSDeleteOperator¶
Use the
ADLSDeleteOperator
to remove
file(s) from Azure DataLake Storage
Below is an example of using this operator to delete a file from ADL.
remove_file = ADLSDeleteOperator(task_id="delete_task", path=REMOTE_FILE_PATH, recursive=True)
ADLSListOperator¶
Use the
ADLSListOperator
to list all
file(s) from Azure DataLake Storage
Below is an example of using this operator to list files from ADL.
adls_files = ADLSListOperator(
task_id="adls_files",
path="folder/output/*.parquet",
azure_data_lake_conn_id="azure_data_lake_default",
)