airflow.providers.amazon.aws.hooks.datasync
¶
Interact with AWS DataSync, using the AWS boto3
library.
Module Contents¶
Classes¶
Interact with AWS DataSync. |
- class airflow.providers.amazon.aws.hooks.datasync.DataSyncHook(wait_interval_seconds=30, *args, **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook
Interact with AWS DataSync.
Provide thick wrapper around
boto3.client("datasync")
.Additional arguments (such as
aws_conn_id
) may be specified and are passed down to the underlying AwsBaseHook.- Parameters
wait_interval_seconds (int) – Time to wait between two consecutive calls to check TaskExecution status. Defaults to 30 seconds.
- Raises
ValueError – If wait_interval_seconds is not between 0 and 15*60 seconds.
- TASK_EXECUTION_INTERMEDIATE_STATES = ('INITIALIZING', 'QUEUED', 'LAUNCHING', 'PREPARING', 'TRANSFERRING', 'VERIFYING')[source]¶
- create_location(location_uri, **create_location_kwargs)[source]¶
Create a new location.
See also
- Parameters
location_uri (str) – Location URI used to determine the location type (S3, SMB, NFS, EFS).
create_location_kwargs – Passed to
DataSync.Client.create_location_*
methods.
- Returns
LocationArn of the created Location.
- Raises
AirflowException – If location type (prefix from
location_uri
) is invalid.- Return type
- get_location_arns(location_uri, case_sensitive=False, ignore_trailing_slash=True)[source]¶
Return all LocationArns which match a LocationUri.
- Parameters
- Returns
List of LocationArns.
- Raises
AirflowBadRequest – if
location_uri
is empty- Return type
- create_task(source_location_arn, destination_location_arn, **create_task_kwargs)[source]¶
Create a Task between the specified source and destination LocationArns.
See also
- update_task(task_arn, **update_task_kwargs)[source]¶
Update a Task.
See also
- Parameters
task_arn (str) – The TaskArn to update.
update_task_kwargs – Passed to
boto.update_task()
, See AWS boto3 datasync documentation.
- delete_task(task_arn)[source]¶
Delete a Task.
See also
- Parameters
task_arn (str) – The TaskArn to delete.
- get_task_arns_for_location_arns(source_location_arns, destination_location_arns)[source]¶
Return list of TaskArns which use both a specified source and destination LocationArns.
- start_task_execution(task_arn, **kwargs)[source]¶
Start a TaskExecution for the specified task_arn.
Each task can have at most one TaskExecution. Additional keyword arguments send to
start_task_execution
boto3 method.
- cancel_task_execution(task_execution_arn)[source]¶
Cancel a TaskExecution for the specified
task_execution_arn
.- Parameters
task_execution_arn (str) – TaskExecutionArn.
- Raises
AirflowBadRequest – If
task_execution_arn
is empty.
- describe_task_execution(task_execution_arn)[source]¶
Get description for the specified
task_execution_arn
.
- get_current_task_execution_arn(task_arn)[source]¶
Get current TaskExecutionArn (if one exists) for the specified
task_arn
.