airflow.providers.ftp.hooks.ftp

Module Contents

Classes

FTPHook

Interact with FTP.

FTPSHook

Interact with FTPS.

class airflow.providers.ftp.hooks.ftp.FTPHook(ftp_conn_id=default_conn_name)[source]

Bases: airflow.hooks.base.BaseHook

Interact with FTP.

Errors that may occur throughout but should be handled downstream. You can specify mode for data transfers in the extra field of your connection as {"passive": "true"}.

Parameters

ftp_conn_id (str) – The ftp connection id reference.

conn_name_attr = 'ftp_conn_id'[source]
default_conn_name = 'ftp_default'[source]
conn_type = 'ftp'[source]
hook_name = 'FTP'[source]
__enter__()[source]
__exit__(exc_type, exc_val, exc_tb)[source]
get_conn()[source]

Return an FTP connection object.

close_conn()[source]

Close the connection; an error will occur if the connection was never opened.

describe_directory(path)[source]

Return a dictionary of {filename: {attributes}} for all files on a remote system which supports MLSD.

Parameters

path (str) – full path to the remote directory

list_directory(path)[source]

Return a list of files on the remote system.

Parameters

path (str) – full path to the remote directory to list

create_directory(path)[source]

Create a directory on the remote system.

Parameters

path (str) – full path to the remote directory to create

delete_directory(path)[source]

Delete a directory on the remote system.

Parameters

path (str) – full path to the remote directory to delete

retrieve_file(remote_full_path, local_full_path_or_buffer, callback=None, block_size=8192)[source]

Transfer the remote file to a local location.

If local_full_path_or_buffer is a string path, the file will be put at that location; if it is a file-like buffer, the file will be written to the buffer but not closed.

Parameters
  • remote_full_path (str) – full path to the remote file

  • local_full_path_or_buffer (Any) – full path to the local file or a file-like buffer

  • callback (Callable | None) – callback which is called each time a block of data is read. if you do not use a callback, these blocks will be written to the file or buffer passed in. if you do pass in a callback, note that writing to a file or buffer will need to be handled inside the callback. [default: output_handle.write()]

  • block_size (int) – file is transferred in chunks of default size 8192 or as set by user

hook = FTPHook(ftp_conn_id="my_conn")

remote_path = "/path/to/remote/file"
local_path = "/path/to/local/file"


# with a custom callback (in this case displaying progress on each read)
def print_progress(percent_progress):
    self.log.info("Percent Downloaded: %s%%" % percent_progress)


total_downloaded = 0
total_file_size = hook.get_size(remote_path)
output_handle = open(local_path, "wb")


def write_to_file_with_progress(data):
    total_downloaded += len(data)
    output_handle.write(data)
    percent_progress = (total_downloaded / total_file_size) * 100
    print_progress(percent_progress)


hook.retrieve_file(remote_path, None, callback=write_to_file_with_progress)

# without a custom callback data is written to the local_path
hook.retrieve_file(remote_path, local_path)
store_file(remote_full_path, local_full_path_or_buffer, block_size=8192)[source]

Transfers a local file to the remote location.

If local_full_path_or_buffer is a string path, the file will be read from that location; if it is a file-like buffer, the file will be read from the buffer but not closed.

Parameters
  • remote_full_path (str) – full path to the remote file

  • local_full_path_or_buffer (Any) – full path to the local file or a file-like buffer

  • block_size (int) – file is transferred in chunks of default size 8192 or as set by user

delete_file(path)[source]

Remove a file on the FTP Server.

Parameters

path (str) – full path to the remote file

rename(from_name, to_name)[source]

Rename a file.

Parameters
  • from_name (str) – rename file from name

  • to_name (str) – rename file to name

get_mod_time(path)[source]

Return a datetime object representing the last time the file was modified.

Parameters

path (str) – remote file path

get_size(path)[source]

Return the size of a file (in bytes).

Parameters

path (str) – remote file path

test_connection()[source]

Test the FTP connection by calling path with directory.

class airflow.providers.ftp.hooks.ftp.FTPSHook(ftp_conn_id=default_conn_name)[source]

Bases: FTPHook

Interact with FTPS.

get_conn()[source]

Return an FTPS connection object.

Was this entry helpful?