airflow.providers.microsoft.azure.operators.adls

Module Contents

Classes

ADLSCreateObjectOperator

Creates a new object from passed data to Azure Data Lake on specified file.

ADLSDeleteOperator

Delete files in the specified path.

ADLSListOperator

List all files from the specified path.

Attributes

DEFAULT_AZURE_DATA_LAKE_CONN_ID

airflow.providers.microsoft.azure.operators.adls.DEFAULT_AZURE_DATA_LAKE_CONN_ID = 'azure_data_lake_default'[source]
class airflow.providers.microsoft.azure.operators.adls.ADLSCreateObjectOperator(*, file_system_name, file_name, data, length=None, replace=False, azure_data_lake_conn_id=DEFAULT_AZURE_DATA_LAKE_CONN_ID, **kwargs)[source]

Bases: airflow.models.BaseOperator

Creates a new object from passed data to Azure Data Lake on specified file.

See also

For more information on how to use this operator, take a look at the guide: ADLSCreateObjectOperator

Parameters

  • file_system_name (str) – Name of the file system or instance of FileSystemProperties.

  • file_name (str) – Name of the file which needs to be created in the file system.

  • data (bytes | str | Iterable[AnyStr] | IO[AnyStr]) – The data that will be uploaded.

  • length (int | None) – Size of the data in bytes (optional).

  • replace (bool) – Whether to forcibly overwrite existing files/directories. If False and remote path is a directory, will quit regardless if any files would be overwritten or not. If True, only matching filenames are actually overwritten.

  • azure_data_lake_conn_id (str) – Reference to the Azure Data Lake connection.

template_fields: Sequence[str] = ('file_system_name', 'file_name', 'data')[source]
ui_color = '#e4f0e8'[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.microsoft.azure.operators.adls.ADLSDeleteOperator(*, path, recursive=False, ignore_not_found=True, azure_data_lake_conn_id=DEFAULT_AZURE_DATA_LAKE_CONN_ID, **kwargs)[source]

Bases: airflow.models.BaseOperator

Delete files in the specified path.

See also

For more information on how to use this operator, take a look at the guide: ADLSDeleteOperator

Parameters

  • path (str) – A directory or file to remove

  • recursive (bool) – Whether to loop into directories in the location and remove the files

  • ignore_not_found (bool) – Whether to raise error if file to delete is not found

  • azure_data_lake_conn_id (str) – Reference to the Azure Data Lake connection.

template_fields: Sequence[str] = ('path',)[source]
ui_color = '#901dd2'[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.microsoft.azure.operators.adls.ADLSListOperator(*, path, azure_data_lake_conn_id=DEFAULT_AZURE_DATA_LAKE_CONN_ID, **kwargs)[source]

Bases: airflow.models.BaseOperator

List all files from the specified path.

This operator returns a python list with the names of files which can be used by

xcom in the downstream tasks.

See also

For more information on how to use this operator, take a look at the guide: ADLSListOperator

Parameters

  • path (str) – The Azure Data Lake path to find the objects. Supports glob strings (templated)

  • azure_data_lake_conn_id (str) – Reference to the Azure Data Lake connection.

template_fields: Sequence[str] = ('path',)[source]
ui_color = '#901dd2'[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Was this entry helpful?