airflow.providers.qubole.hooks.qubole
¶
Qubole hook.
Module Contents¶
Classes¶
Hook for Qubole communication. |
Functions¶
|
Flatten the list. |
|
Remove options from the list. |
|
Get options list. |
Build Command argument from command and options. |
Attributes¶
- airflow.providers.qubole.hooks.qubole.filter_options(options)[source]¶
Remove options from the list.
- airflow.providers.qubole.hooks.qubole.build_command_args()[source]¶
Build Command argument from command and options.
- class airflow.providers.qubole.hooks.qubole.QuboleHook(*args, **kwargs)[source]¶
Bases:
airflow.hooks.base.BaseHook
Hook for Qubole communication.
- kill(ti)[source]¶
Kill (cancel) a Qubole command.
- Parameters
ti – Task Instance of the dag, used to determine the Quboles command id
- Returns
response from Qubole
- get_results(ti=None, fp=None, inline=True, delim=None, fetch=True, include_headers=False)[source]¶
Get results (or just s3 locations) of a command from Qubole and save into a file.
- Parameters
ti – Task Instance of the dag, used to determine the Quboles command id
fp – Optional file pointer, will create one and return if None passed
inline (bool) – True to download actual results, False to get s3 locations only
delim – Replaces the CTL-A chars with the given delim, defaults to ‘,’
fetch (bool) – when inline is True, get results directly from s3 (if large)
- Returns
file location containing actual results or s3 locations of results
- Return type
- get_log(ti)[source]¶
Get Logs of a command from Qubole.
- Parameters
ti – Task Instance of the dag, used to determine the Quboles command id
- Returns
command log as text
- Return type
None