airflow.providers.http.operators.http¶
Module Contents¶
Classes¶
Calls an endpoint on an HTTP system to execute an action. |
|
Calls an endpoint on an HTTP system to execute an action. |
- class airflow.providers.http.operators.http.HttpOperator(*, endpoint=None, method='POST', data=None, headers=None, pagination_function=None, response_check=None, response_filter=None, extra_options=None, http_conn_id='http_default', log_response=False, auth_type=None, tcp_keep_alive=True, tcp_keep_alive_idle=120, tcp_keep_alive_count=20, tcp_keep_alive_interval=30, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorCalls an endpoint on an HTTP system to execute an action.
See also
For more information on how to use this operator, take a look at the guide: HttpOperator
- Parameters
http_conn_id (str) – The http connection to run the operator against
endpoint (str | None) – The relative part of the full url. (templated)
method (str) – The HTTP method to use, default = “POST”
data (dict[str, Any] | str | None) – The data to pass. POST-data in POST/PUT and params in the URL for a GET request. (templated)
headers (dict[str, str] | None) – The HTTP headers to be added to the GET request
pagination_function (Callable[Ellipsis, Any] | None) – A callable that generates the parameters used to call the API again, based on the previous response. Typically used when the API is paginated and returns for e.g a cursor, a ‘next page id’, or a ‘next page URL’. When provided, the Operator will call the API repeatedly until this callable returns None. The result of the Operator will become by default a list of Response.text objects (instead of a single response object). Same with the other injected functions (like response_check, response_filter, …) which will also receive a list of Response objects. This function receives a Response object form previous call, and should return a nested dictionary with the following optional keys: endpoint, data, headers and extra_options. Those keys will be merged and/or override the parameters provided into the HttpOperator declaration. Parameters are merged when they are both a dictionary (e.g.: HttpOperator.headers will be merged with the `headers dict provided by this function). When merging, dict items returned by this function will override initial ones (e.g: if both HttpOperator.headers and headers have a ‘cookie’ item, the one provided by headers is kept). Parameters are simply overridden when any of them are string (e.g.: HttpOperator.endpoint is overridden by endpoint).
response_check (Callable[Ellipsis, bool] | None) – A check against the ‘requests’ response object. The callable takes the response object as the first positional argument and optionally any number of keyword arguments available in the context dictionary. It should return True for ‘pass’ and False otherwise. If a pagination_function is provided, this function will receive a list of response objects instead of a single response object.
response_filter (Callable[Ellipsis, Any] | None) – A function allowing you to manipulate the response text. e.g response_filter=lambda response: json.loads(response.text). The callable takes the response object as the first positional argument and optionally any number of keyword arguments available in the context dictionary. If a pagination_function is provided, this function will receive a list of response object instead of a single response object.
extra_options (dict[str, Any] | None) – Extra options for the ‘requests’ library, see the ‘requests’ documentation (options to modify timeout, ssl, etc.)
log_response (bool) – Log the response (default: False)
auth_type (type[requests.auth.AuthBase] | None) – The auth type for the service
tcp_keep_alive (bool) – Enable TCP Keep Alive for the connection.
tcp_keep_alive_idle (int) – The TCP Keep Alive Idle parameter (corresponds to
socket.TCP_KEEPIDLE).tcp_keep_alive_count (int) – The TCP Keep Alive count parameter (corresponds to
socket.TCP_KEEPCNT)tcp_keep_alive_interval (int) – The TCP Keep Alive interval parameter (corresponds to
socket.TCP_KEEPINTVL)deferrable (bool) – Run operator in the deferrable mode
- property hook: airflow.providers.http.hooks.http.HttpHook[source]¶
Get Http Hook based on connection type.
- execute(context)[source]¶
Derive when creating an operator.
Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.
- class airflow.providers.http.operators.http.SimpleHttpOperator(**kwargs)[source]¶
Bases:
HttpOperatorCalls an endpoint on an HTTP system to execute an action.
See also
For more information on how to use this operator, take a look at the guide: HttpOperator
- Parameters
http_conn_id – The http connection to run the operator against
endpoint – The relative part of the full url. (templated)
method – The HTTP method to use, default = “POST”
data – The data to pass. POST-data in POST/PUT and params in the URL for a GET request. (templated)
headers – The HTTP headers to be added to the GET request
pagination_function – A callable that generates the parameters used to call the API again, based on the previous response. Typically used when the API is paginated and returns for e.g a cursor, a ‘next page id’, or a ‘next page URL’. When provided, the Operator will call the API repeatedly until this callable returns None. The result of the Operator will become by default a list of Response.text objects (instead of a single response object). Same with the other injected functions (like response_check, response_filter, …) which will also receive a list of Response objects. This function receives a Response object form previous call, and should return a nested dictionary with the following optional keys: endpoint, data, headers and extra_options. Those keys will be merged and/or override the parameters provided into the HttpOperator declaration. Parameters are merged when they are both a dictionary (e.g.: HttpOperator.headers will be merged with the `headers dict provided by this function). When merging, dict items returned by this function will override initial ones (e.g: if both HttpOperator.headers and headers have a ‘cookie’ item, the one provided by headers is kept). Parameters are simply overridden when any of them are string (e.g.: HttpOperator.endpoint is overridden by endpoint).
response_check – A check against the ‘requests’ response object. The callable takes the response object as the first positional argument and optionally any number of keyword arguments available in the context dictionary. It should return True for ‘pass’ and False otherwise. If a pagination_function is provided, this function will receive a list of response objects instead of a single response object.
response_filter – A function allowing you to manipulate the response text. e.g response_filter=lambda response: json.loads(response.text). The callable takes the response object as the first positional argument and optionally any number of keyword arguments available in the context dictionary. If a pagination_function is provided, this function will receive a list of response object instead of a single response object.
extra_options – Extra options for the ‘requests’ library, see the ‘requests’ documentation (options to modify timeout, ssl, etc.)
log_response – Log the response (default: False)
auth_type – The auth type for the service
tcp_keep_alive – Enable TCP Keep Alive for the connection.
tcp_keep_alive_idle – The TCP Keep Alive Idle parameter (corresponds to
socket.TCP_KEEPIDLE).tcp_keep_alive_count – The TCP Keep Alive count parameter (corresponds to
socket.TCP_KEEPCNT)tcp_keep_alive_interval – The TCP Keep Alive interval parameter (corresponds to
socket.TCP_KEEPINTVL)deferrable – Run operator in the deferrable mode