apache-airflow-providers-common-sql

Changelog¶

1.20.0¶

Features¶

Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook (#41916)

1.19.0¶

Features¶

Feature: Added fast_executemany parameter to insert_rows of DbApiHook (#43357)
Make conn id parameters templated in GenericTransfer and also allow passing hook parameters like in BaseSQLOperator (#42891)
Allow capture of replication lag in SqlSensor (#43107)

Misc¶

fix mypy failure (#43206)

1.18.0¶

Features¶

feat(providers/common/sql): add warning to connection setter (#42736)

Bug Fixes¶

FIX: Only pass connection to sqlalchemy engine in JdbcHook (#42705)

1.17.1¶

Bug Fixes¶

fix(providers/common/sql): add dummy connection setter for backward compatibility (#42490)
Changed type hinting for handler function (#42275)

1.17.0¶

Features¶

Note

Connection in DB Hook is now cached to avoid multiple lookups when properties from extras have to be resolved.

Generalize caching of connection in DbApiHook to improve performance (#40751)

Misc¶

feat: log client db messages for provider postgres (#40171)
remove deprecated soft_fail from providers (#41710)

1.16.0¶

Note

This release of provider is only available for Airflow 2.8+ as explained in the Apache Airflow providers support policy.

Bug Fixes¶

fix: rm deprecated import (#41461)

Misc¶

Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)

1.15.0¶

Features¶

Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669)

1.14.2¶

Bug Fixes¶

FIX: DbApiHook.insert_rows unnecessarily restarting connections (#40615)

Misc¶

Enable enforcing pydocstyle rule D213 in ruff. (#40448)

1.14.1¶

Misc¶

implement per-provider tests with lowest-direct dependency resolution (#39946)
Update pandas minimum requirement for Python 3.12 (#40272)
standardizes template fields for 'BaseSQLOperator' and adds 'database' as a templated field (#39826)

1.14.0¶

Features¶

Add 'parameters' as template field for SqlSensor (#39588)

Bug Fixes¶

DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is formatted correctly (#39690)

Misc¶

refactor: The executemany parameter of insert_rows should not be deprecated as for some hooks we don't want to enable a system-wide supports_executemany parameter, that way we can also keep using it in dedicated situations (#39630)
Faster 'airflow_version' imports (#39552)
Simplify 'airflow_version' imports (#39497)
Add typing for SqlSensor (#39773)

1.13.0¶

Note

This release of provider is only available for Airflow 2.7+ as explained in the Apache Airflow providers support policy.

Features¶

Add 'sqlalchemy_url' property to 'DbApiHook' class (#38871)
Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715)

Bug Fixes¶

Fix 'DbApiHook.insert_rows' when 'rows' is a generator (#38972)
Fix 'update-common-sql-api-stubs' pre-commit check (#38915)

Misc¶

Bump minimum Airflow version in providers to Airflow 2.7.0 (#39240)
openlineage, snowflake: do not run external queries for Snowflake (#39113)

1.12.0¶

Features¶

Add hook_params to template_fields for BaseSQLOperator-related Operators (#38724)
Make 'placeholder' of DbApiHook configurable in UI (#38528)

Misc¶

Undeprecating 'DBApiHookForTests._make_common_data_structure' (#38573)

1.11.1¶

Bug Fixes¶

Make 'executemany' keyword arguments only in 'DbApiHook.insert_rows' (#37840)
Limit 'pandas' to '<2.2' (#37748)

1.11.0¶

Features¶

Enhancement: Performance enhancement for insert_rows method DbApiHook with fast executemany + SAP Hana support (#37246)

Bug Fixes¶

Fix SQLThresholdCheckOperator error on falsey vals (#37150)

Misc¶

feat: Switch all class, functions, methods deprecations to decorators (#36876)
Add more-itertools as dependency of common-sql (#37359)

1.10.1¶

Misc¶

Set min pandas dependency to 1.2.5 for all providers and airflow (#36698)

1.10.0¶

Make "placeholder" of ODBC configurable in UI (#36000)

Bug Fixes¶

Return common data structure in DBApi derived classes
SQLCheckOperator fails if returns dict with any False values (#36273)

1.9.0¶

Note

This release of provider is only available for Airflow 2.6+ as explained in the Apache Airflow providers support policy.

Misc¶

Bump minimum Airflow version in providers to Airflow 2.6.0 (#36017)
Add Architecture Decision Record for common.sql introduction (#36015)

1.8.1¶

Misc¶

Add '_make_serializable' method which other SQL operators can overrides when result from cursor is not JSON-serializable (#32319)
Remove backcompat inheritance for DbApiHook (#35754)

1.8.0¶

Note

This release of provider is only available for Airflow 2.5+ as explained in the Apache Airflow providers support policy.

Bug Fixes¶

fix(providers/sql): respect soft_fail argument when exception is raised (#34199)

Misc¶

Bump min airflow version of providers (#34728)
Use 'airflow.exceptions.AirflowException' in providers (#34511)

1.7.2¶

Bug Fixes¶

Fix BigQueryValueCheckOperator deferrable mode optimisation (#34018)

Misc¶

Refactor regex in providers (#33898)

1.7.1¶

Misc¶

Refactor: Better percentage formatting (#33595)
Refactor: Simplify code in smaller providers (#33234)

1.7.0¶

Features¶

Add a new parameter to SQL operators to specify conn id field (#30784)

1.6.2¶

Misc¶

Make SQLExecute Query signature consistent with other SQL operators (#32974)
Get rid of Python2 numeric relics (#33050)

1.6.1¶

Bug Fixes¶

Fix local OpenLineage import in 'SQLExecuteQueryOperator'. (#32400)

Misc¶

Add default port to Openlineage authority method. (#32828)
Add more accurate typing for DbApiHook.run method (#31846)

1.6.0¶

Features¶

openlineage, common.sql: provide OL SQL parser as internal OpenLineage provider API (#31398)

Misc¶

Pass SQLAlchemy engine to construct information schema query. (#32371)

1.5.2¶

Misc¶

Remove Python 3.7 support (#30963)

1.5.1¶

Misc¶

Bring back min-airflow-version for preinstalled providers (#31469)

1.5.0 (YANKED)¶

Warning

This release has been yanked with a reason: This version might cause unconstrained installation of old airflow version lead to Runtime Error.

Note

This release of provider is only available for Airflow 2.4+ as explained in the Apache Airflow providers support policy.

Features¶

Add conditional output processing in SQL operators (#31136)

Misc¶

Remove noisy log from SQL table check (#31037)

1.4.0¶

Features¶

Add option to show output of 'SQLExecuteQueryOperator' in the log (#29954)

Misc¶

Fix Python API docs formatting for Common SQL provider (#29863)

1.3.4¶

Bug Fixes¶

Do not process output when do_xcom_push=False (#29599)

1.3.3¶

Bug Fixes¶

Handle non-compliant behaviour of Exasol cursor (#28744)

1.3.2¶

Bug Fixes¶

fIx isort problems introduced by recent isort release (#28434)
Fix template rendering for Common SQL operators (#28202)
Defer to hook setting for split_statements in SQLExecuteQueryOperator (#28635)

Misc¶

Clarify docstrings for updated DbApiHook (#27966)

1.3.1¶

This release fixes a few errors that were introduced in common.sql operator while refactoring common parts:

_process_output method in SQLExecuteQueryOperator has now consistent semantics and typing, it can also modify the returned (and stored in XCom) values in the operators that derive from the SQLExecuteQueryOperator).
descriptions of all returned results are stored as descriptions property in the DBApiHook
last description of the cursor whether to return single query results values are now exposed in DBApiHook via last_description property.

Lack of consistency in the operator caused 1.3.0 to be yanked - the 1.3.0 should not be used - if you have 1.3.0 installed, upgrade to 1.3.1.

Bug Fixes¶

Restore removed (but used) methods in common.sql (#27843)
Fix errors in Databricks SQL operator introduced when refactoring (#27854)
Bump common.sql provider to 1.3.1 (#27888)
Fixing the behaviours of SQL Hooks and Operators finally (#27912)

1.3.0 (YANKED)¶

Warning

This release has been yanked with a reason: Breaks Google 8.4.0 provider for SQLExecute

Note

This release of provider is only available for Airflow 2.3+ as explained in the Apache Airflow providers support policy.

Misc¶

Move min airflow version to 2.3.0 for all providers (#27196)

Features¶

Add SQLExecuteQueryOperator (#25717)
Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first (#26944)
DbApiHook consistent insert_rows logging (#26758)

Bug Fixes¶

Common sql bugfixes and improvements (#26761)
Use unused SQLCheckOperator.parameters in SQLCheckOperator.execute. (#27599)

1.2.0¶

Features¶

Make placeholder style configurable (#25939)
Better error message for pre-common-sql providers (#26051)

Bug Fixes¶

Fix (and test) SQLTableCheckOperator on postgresql (#25821)
Don't use Pandas for SQLTableCheckOperator (#25822)
Discard semicolon stripping in SQL hook (#25855)

1.1.0¶

Features¶

Improve taskflow type hints with ParamSpec (#25173)
Move all "old" SQL operators to common.sql providers (#25350)
Deprecate hql parameters and synchronize DBApiHook method APIs (#25299)
Unify DbApiHook.run() method with the methods which override it (#23971)
Common SQLCheckOperators Various Functionality Update (#25164)

Bug Fixes¶

Allow Legacy SqlSensor to use the common.sql providers (#25293)
Fix fetch_all_handler & db-api tests for it (#25430)
Align Common SQL provider logo location (#25538)
Fix SQL split string to include ';-less' statements (#25713)

1.0.0¶

Initial version of the provider. Adds SQLColumnCheckOperator and SQLTableCheckOperator. Moves DBApiHook, SQLSensor and ConnectorProtocol to the provider.