airflow.contrib.operators.sftp_operator

Module Contents

class airflow.contrib.operators.sftp_operator.SFTPOperation

Bases:object

PUT = put

GET = get

class airflow.contrib.operators.sftp_operator.SFTPOperator(ssh_hook=None, ssh_conn_id=None, remote_host=None, local_filepath=None, remote_filepath=None, operation=SFTPOperation.PUT, confirm=True, create_intermediate_dirs=False, *args, **kwargs)

Bases:airflow.models.BaseOperator

SFTPOperator for transferring files from remote host to local or vice a versa. This operator uses ssh_hook to open sftp transport channel that serve as basis for file transfer.

Parameters

• ssh_hook (airflow.contrib.hooks.ssh_hook.SSHHook) – predefined ssh_hook to use for remote execution. Either ssh_hook or ssh_conn_id needs to be provided.

• ssh_conn_id (str) – connection id from airflow Connections. ssh_conn_id will be ignored if ssh_hook is provided.

• remote_host (str) – remote host to connect (templated) Nullable. If provided, it will replace the remote_host which was defined in ssh_hook or predefined in the connection of ssh_conn_id.

• local_filepath (str) – local file path to get or put. (templated)

• remote_filepath (str) – remote file path to get or put. (templated)

• operation (str) – specify operation ‘get’ or ‘put’, defaults to put

• confirm (bool) – specify if the SFTP operation should be confirmed, defaults to True

• create_intermediate_dirs (bool) –

  create missing intermediate directories when copying from remote to local and vice-versa. Default is False.

  Example: The following task would copy file.txt to the remote host at /tmp/tmp1/tmp2/ while creating tmp,``tmp1`` and tmp2 if they don’t exist. If the parameter is not passed it would error as the directory does not exist.

  put_file = SFTPOperator(
      task_id="test_sftp",
      ssh_conn_id="ssh_default",
      local_filepath="/tmp/file.txt",
      remote_filepath="/tmp/tmp1/tmp2/file.txt",
      operation="put",
      create_intermediate_dirs=True,
      dag=dag
  )
  

template_fields = ['local_filepath', 'remote_filepath', 'remote_host']

execute(self, context)

airflow.contrib.operators.sftp_operator._make_intermediate_dirs(sftp_client, remote_directory)

Create all the intermediate directories in a remote host

Parameters

• sftp_client – A Paramiko SFTP client.

• remote_directory – Absolute Path of the directory containing the file

Returns