HTTP to Amazon S3¶
Use the HttpToS3Operator
transfer content from a http endpoint to an Amazon Simple Storage Service (S3) file.
Prerequisite Tasks¶
To use these operators, you must do a few things:
Create necessary resources using AWS Console or AWS CLI.
Install API libraries via pip.
pip install 'apache-airflow[amazon]'Detailed information is available Installation of Airflow®
Operators¶
HTTP to Amazon S3 transfer operator¶
This operator copies data from a HTTP endpoint to an Amazon S3 file.
To get more information about this operator visit:
HttpToS3Operator
Example usage:
tests/system/amazon/aws/example_http_to_s3.py
http_to_s3_task = HttpToS3Operator(
task_id="http_to_s3_task",
http_conn_id=conn_id_name,
endpoint="/test_file",
s3_bucket=s3_bucket,
s3_key=s3_key,
replace=True,
)