airflow.contrib.operators.bigquery_get_data

Module Contents

class airflow.contrib.operators.bigquery_get_data.BigQueryGetDataOperator(dataset_id, table_id, max_results='100', selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None, *args, **kwargs)[source]

Bases:airflow.models.BaseOperator

Fetches the data from a BigQuery table (alternatively fetch data for selected columns) and returns data in a python list. The number of elements in the returned list will be equal to the number of rows fetched. Each element in the list will again be a list where element would represent the columns values for that row.

Example Result: [['Tony', '10'], ['Mike', '20'], ['Steve', '15']]

Note

If you pass fields to selected_fields which are in different order than the order of columns already in BQ table, the data will still be in the order of BQ table. For example if the BQ table has 3 columns as [A,B,C] and you pass ‘B,A’ in the selected_fields the data would still be of the form 'A,B'.

Example:

get_data = BigQueryGetDataOperator(
    task_id='get_data_from_bq',
    dataset_id='test_dataset',
    table_id='Transaction_partitions',
    max_results='100',
    selected_fields='DATE',
    bigquery_conn_id='airflow-service-account'
)
Parameters

  • dataset_id (str) – The dataset ID of the requested table. (templated)

  • table_id (str) – The table ID of the requested table. (templated)

  • max_results (str) – The maximum number of records (rows) to be fetched from the table. (templated)

  • selected_fields (str) – List of fields to return (comma-separated). If unspecified, all fields are returned.

  • bigquery_conn_id (str) – reference to a specific BigQuery hook.

  • delegate_to (str) – The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

template_fields = ['dataset_id', 'table_id', 'max_results'][source]
ui_color = #e4f0e8[source]
execute(self, context)[source]