AWS: Amazon Web Services

Airflow has support for Amazon Web Services.

All hooks are based on airflow.providers.amazon.aws.hooks.base_aws.

Services

These integrations allow you to perform various operations within the Amazon Web Services.

Amazon CloudWatch Logs

Hooks

airflow.providers.amazon.aws.hooks.logs.

Provider

apache-airflow-providers-amazon

Product documentation

Amazon CloudWatch Logs

Amazon Elastic Container Registry (ECR)

Hooks

airflow.providers.amazon.aws.hooks.ecr.

Provider

apache-airflow-providers-amazon

Product documentation

Amazon Elastic Container Registry (ECR)

Amazon Kinesis Data Firehose

Hooks

airflow.providers.amazon.aws.hooks.kinesis.

Provider

apache-airflow-providers-amazon

Product documentation

Amazon Kinesis Data Firehose

Amazon Systems Manager (SSM)

Hooks

airflow.providers.amazon.aws.hooks.ssm.

Provider

apache-airflow-providers-amazon

Product documentation

Amazon Systems Manager (SSM)

AWS Security Token Service (STS)

Hooks

airflow.providers.amazon.aws.hooks.sts.

Provider

apache-airflow-providers-amazon

Product documentation

AWS Security Token Service (STS)

Transfers

These integrations allow you to copy data from/to Amazon Web Services.

Amazon DynamoDB to Amazon Simple Storage Service (S3)

Source product documentation

Amazon DynamoDB

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

Amazon DynamoDB to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.dynamodb_to_s3

Provider

apache-airflow-providers-amazon

Google Cloud Storage (GCS) to Amazon Simple Storage Service (S3)

Source product documentation

Google Cloud Storage (GCS)

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

Google Cloud Storage to Amazon S3 transfer operator

Python API

airflow.providers.amazon.aws.transfers.gcs_to_s3

Provider

apache-airflow-providers-amazon

Amazon Glacier to Google Cloud Storage (GCS)

Source product documentation

Amazon Glacier

Target product documentation

Google Cloud Storage (GCS)

Operator guide

Amazon S3 Glacier to GCS

Python API

airflow.providers.amazon.aws.transfers.glacier_to_gcs

Provider

apache-airflow-providers-amazon

Google to Amazon Simple Storage Service (S3)

Source product documentation

Google

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

Google API to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.google_api_to_s3

Provider

apache-airflow-providers-amazon

Apache Hive to Amazon DynamoDB

Source product documentation

Apache Hive

Target product documentation

Amazon DynamoDB

Operator guide

Apache Hive to Amazon DynamoDB

Python API

airflow.providers.amazon.aws.transfers.hive_to_dynamodb

Provider

apache-airflow-providers-amazon

Hypertext Transfer Protocol (HTTP) to Amazon Simple Storage Service (S3)

Source product documentation

Hypertext Transfer Protocol (HTTP)

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

HTTP to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.http_to_s3

Provider

apache-airflow-providers-amazon

Internet Message Access Protocol (IMAP) to Amazon Simple Storage Service (S3)

Source product documentation

Internet Message Access Protocol (IMAP)

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

Imap Attachment to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.imap_attachment_to_s3

Provider

apache-airflow-providers-amazon

MongoDB to Amazon Simple Storage Service (S3)

Source product documentation

MongoDB

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

MongoDB to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.mongo_to_s3

Provider

apache-airflow-providers-amazon

Amazon Redshift to Amazon Simple Storage Service (S3)

Source product documentation

Amazon Redshift

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

Amazon Redshift to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.redshift_to_s3

Provider

apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Amazon Redshift

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

Amazon Redshift

Operator guide

Amazon S3 to Amazon Redshift

Python API

airflow.providers.amazon.aws.transfers.s3_to_redshift

Provider

apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to SSH File Transfer Protocol (SFTP)

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

SSH File Transfer Protocol (SFTP)

Operator guide

Amazon S3 to SFTP

Python API

airflow.providers.amazon.aws.transfers.s3_to_sftp

Provider

apache-airflow-providers-amazon

SSH File Transfer Protocol (SFTP) to Amazon Simple Storage Service (S3)

Source product documentation

SSH File Transfer Protocol (SFTP)

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

SFTP to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.sftp_to_s3

Provider

apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to File Transfer Protocol (FTP)

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

File Transfer Protocol (FTP)

Operator guide

Amazon S3 to FTP

Python API

airflow.providers.amazon.aws.transfers.s3_to_ftp

Provider

apache-airflow-providers-amazon

Exasol to Amazon Simple Storage Service (S3)

Source product documentation

Exasol

Target product documentation

Amazon Simple Storage Service (S3)

Python API

airflow.providers.amazon.aws.transfers.exasol_to_s3

Provider

apache-airflow-providers-amazon

File Transfer Protocol (FTP) to Amazon Simple Storage Service (S3)

Source product documentation

File Transfer Protocol (FTP)

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

FTP to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.ftp_to_s3

Provider

apache-airflow-providers-amazon

Salesforce to Amazon Simple Storage Service (S3)

Source product documentation

Salesforce

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

Salesforce to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.salesforce_to_s3

Provider

apache-airflow-providers-amazon

Local to Amazon Simple Storage Service (S3)

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

Local Filesystem to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.local_to_s3

Provider

apache-airflow-providers-amazon

Common SQL to Amazon Simple Storage Service (S3)

Source product documentation

Common SQL

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

SQL to Amazon S3

Python API

airflow.providers.amazon.aws.transfers.sql_to_s3

Provider

apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Common SQL

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

Common SQL

Operator guide

Amazon S3 to SQL

Python API

airflow.providers.amazon.aws.transfers.s3_to_sql

Provider

apache-airflow-providers-amazon

Amazon Web Services to Amazon Web Services

Source product documentation

Amazon Web Services

Target product documentation

Amazon Web Services

Python API

airflow.providers.amazon.aws.transfers.base

Provider

apache-airflow-providers-amazon

Microsoft Azure Blob Storage to Amazon Simple Storage Service (S3)

Source product documentation

Microsoft Azure Blob Storage

Target product documentation

Amazon Simple Storage Service (S3)

Operator guide

Azure Blob Storage to Amazon S3 transfer operator

Python API

airflow.providers.amazon.aws.transfers.azure_blob_to_s3

Provider

apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Amazon DynamoDB

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

Amazon DynamoDB

Operator guide

Amazon S3 to DynamoDB

Python API

airflow.providers.amazon.aws.transfers.s3_to_dynamodb

Provider

apache-airflow-providers-amazon

Amazon Simple Storage Service (S3) to Apache Hive

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

Apache Hive

Python API

airflow.providers.apache.hive.transfers.s3_to_hive

Provider

apache-airflow-providers-apache-hive

Amazon Simple Storage Service (S3) to Google Cloud Storage (GCS)

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

Google Cloud Storage (GCS)

Operator guide

Transfer Data from Amazon S3 to Google Cloud Storage

Python API

airflow.providers.google.cloud.transfers.s3_to_gcs

Provider

apache-airflow-providers-google

Amazon Simple Storage Service (S3) to Microsoft Azure Blob Storage

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

Microsoft Azure Blob Storage

Operator guide

Amazon S3 to Azure Blob Storage Transfer Operator

Python API

airflow.providers.microsoft.azure.transfers.s3_to_wasb

Provider

apache-airflow-providers-microsoft-azure

Amazon Simple Storage Service (S3) to MySQL

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

MySQL

Python API

airflow.providers.mysql.transfers.s3_to_mysql

Provider

apache-airflow-providers-mysql

Amazon Simple Storage Service (S3) to Snowflake

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

Snowflake

Operator guide

CopyFromExternalStageToSnowflakeOperator

Python API

airflow.providers.snowflake.transfers.copy_into_snowflake

Provider

apache-airflow-providers-snowflake

Amazon Simple Storage Service (S3) to Teradata

Source product documentation

Amazon Simple Storage Service (S3)

Target product documentation

Teradata

Operator guide

S3ToTeradataOperator

Python API

airflow.providers.teradata.transfers.s3_to_teradata

Provider

apache-airflow-providers-teradata

Was this entry helpful?