airflow.providers.qubole.hooks.qubole

Qubole hook.

Module Contents

Classes

QuboleHook

Hook for Qubole communication.

Functions

flatten_list(list_of_lists)

Flatten the list.

filter_options(options)

Remove options from the list.

get_options_list(command_class)

Get options list.

build_command_args()

Build Command argument from command and options.

Attributes

log

COMMAND_CLASSES

POSITIONAL_ARGS

airflow.providers.qubole.hooks.qubole.log[source]
airflow.providers.qubole.hooks.qubole.COMMAND_CLASSES[source]
airflow.providers.qubole.hooks.qubole.POSITIONAL_ARGS[source]
airflow.providers.qubole.hooks.qubole.flatten_list(list_of_lists)[source]

Flatten the list.

airflow.providers.qubole.hooks.qubole.filter_options(options)[source]

Remove options from the list.

airflow.providers.qubole.hooks.qubole.get_options_list(command_class)[source]

Get options list.

airflow.providers.qubole.hooks.qubole.build_command_args()[source]

Build Command argument from command and options.

class airflow.providers.qubole.hooks.qubole.QuboleHook(*args, **kwargs)[source]

Bases: airflow.hooks.base.BaseHook

Hook for Qubole communication.

conn_name_attr: str = 'qubole_conn_id'[source]
default_conn_name = 'qubole_default'[source]
conn_type = 'qubole'[source]
hook_name = 'Qubole'[source]
static get_ui_field_behaviour()[source]

Returns custom field behaviour.

static handle_failure_retry(context)[source]

Handle retries in case of failures.

execute(context)[source]

Execute call.

kill(ti)[source]

Kill (cancel) a Qubole command.

Parameters

ti – Task Instance of the dag, used to determine the Quboles command id

Returns

response from Qubole

get_results(ti=None, fp=None, inline=True, delim=None, fetch=True, include_headers=False)[source]

Get results (or just s3 locations) of a command from Qubole and save into a file.

Parameters
  • ti – Task Instance of the dag, used to determine the Quboles command id

  • fp – Optional file pointer, will create one and return if None passed

  • inline (bool) – True to download actual results, False to get s3 locations only

  • delim – Replaces the CTL-A chars with the given delim, defaults to ‘,’

  • fetch (bool) – when inline is True, get results directly from s3 (if large)

Returns

file location containing actual results or s3 locations of results

Return type

str

get_log(ti)[source]

Get Logs of a command from Qubole.

Parameters

ti – Task Instance of the dag, used to determine the Quboles command id

Returns

command log as text

Return type

None

get_jobs_id(ti)[source]

Get jobs associated with a Qubole commands.

Parameters

ti – Task Instance of the dag, used to determine the Quboles command id

Returns

Job information associated with command

Return type

None

create_cmd_args(context)[source]

Creates command arguments.

Was this entry helpful?