Python3.12 #36755
Python3.12 #36755
Conversation
potiuk
requested review from
kaxil,
pierrejeambrun,
ashb and
jedcunningham
as code owners
|
cc: @dirrao @Taragolis -> seems like apache-beam having numpy as dependency is the next problem to solve after pendulum is solved |
|
Looks like we need NumPy 1.26+ - from that long discussion here: numpy/numpy#23808 and So likely the next best thing to do is to exclude apache-beam provider for python 3.12 This is kinda expected, Beam is always dragging us behind |
|
Pushed a fixup marking it for exclusion - let's see. |
|
Seems like 1.26.0 is the first release for numpy which officially support 3.12: |
|
Oh... I've post my comment without refresh page, and you've already found the same things |
|
Seems like Google provider also not compatible with 3.12 yet 46.9 ERROR: Could not find a version that satisfies the requirement google-ads>=22.1.0; extra == "google" (from apache-airflow[google]) (from versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.3.0, 1.3.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.4.1, 3.0.0, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.3.0, 4.0.0, 4.1.0, 4.1.1, 5.0.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.1.0, 6.0.0, 7.0.0, 8.0.0, 8.1.0, 8.2.0, 9.0.0, 10.0.0, 11.0.0, 11.0.1, 11.0.2, 12.0.0, 13.0.0, 14.0.0, 14.0.1, 14.1.0, 15.0.0, 15.1.0, 15.1.1, 16.0.0, 17.0.0, 18.0.0, 18.1.0, 18.2.0, 19.0.0, 20.0.0, 21.0.0, 21.1.0, 21.2.0, 21.3.0, 22.0.0)
246.9 ERROR: No matching distribution found for google-ads>=22.1.0; extra == "google"Latest Issue for add support of Python 3.12 already exists googleads/google-ads-python#813 |
|
All right ... let me exclude google provider too then. At this stage I have a feeling that excluding few - even huge and important - providers and having an open -issue to bring the 3.12 support in would be a good thing. And I know for a fact that google team wants to split the google provider and splitting of ads was the first thing to try anyway, so that might accelerate things a bit. |
|
Pushed. |
|
BTW. I really like how nicely and transpartently the new |
|
I think one day we need to finally resolve Consider splitting Google Provider because google provider is really huge providers (28k+ lines which tracked by our test) and contains quite a few different components:
So if we found the way how it could be done it might prevent the situation that one of this component become a showstopper for others
|
This is precisely the plan I am discussing with Google team :) |
|
So we have |
|
Seems ❯ pip install duckdb==0.9.3.dev2258
Collecting duckdb==0.9.3.dev2258
Obtaining dependency information for duckdb==0.9.3.dev2258 from https://files.pythonhosted.org/packages/3c/43/094637a1939e8ba6ae53a788bd46adfd0b71fe0a6e182c8e6179b2966e09/duckdb-0.9.3.dev2258-cp312-cp312-macosx_11_0_arm64.whl.metadata
Downloading duckdb-0.9.3.dev2258-cp312-cp312-macosx_11_0_arm64.whl.metadata (768 bytes)
Downloading duckdb-0.9.3.dev2258-cp312-cp312-macosx_11_0_arm64.whl (13.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.7/13.7 MB 3.6 MB/s eta 0:00:00
Installing collected packages: duckdb
Successfully installed duckdb-0.9.3.dev2258 |
|
Pushed a change for it :) |
|
One more step and a new error, this time it is related to the LevelDB, which is also part of google provider, but I guess it has separate extra I think this is the same issue: wbolster/plyvel#158 |
|
Running |
|
Build Prod Images still picking the google ads provider. |
|
Yes. Because google provider is installed and epxected to be installed when PROD image is built. So if we do not build it locally for Pythin 3.12, it will install the one from In the meatime - we could take a close look at the failing tests for Python 3.13 https://github.com/apache/airflow/actions/runs/7509965786/job/20448000561?pr=36755) I think they mostly fail because |
potiuk
force-pushed
the
python3.12
branch
2 times, most recently
from
42c1339 to
2dee02f
Compare
|
cc: @dirrao @Taragolis -> Ads released with 3.12 support https://pypi.org/project/google-ads/ - removed the limit from Google provider, let's see. |
|
Duckdb removed devel version we had pinned for 3.12 -> replaced it with >= for the new devel |
This reverts commit 76dee0b.
Finally after a number of dependency upgrades we seem to be able to upgrade to Python 3.12 (pending universal_pathlib 0.2.0 conversion) Several providers are excluded from being installed and wait for Python 3.12, but it should not block Airlfow's general 3.12 support. Co-authored-by: dirrao <[email protected]> (cherry picked from commit 76dee0b)
Finally after a number of dependency upgrades we seem to be able to upgrade to Python 3.12 (pending universal_pathlib 0.2.0 conversion) Several providers are excluded from being installed and wait for Python 3.12, but it should not block Airlfow's general 3.12 support. Co-authored-by: dirrao <[email protected]> (cherry picked from commit 76dee0b)
Finally after a number of dependency upgrades we seem to be able to upgrade to Python 3.12 (pending universal_pathlib 0.2.0 conversion) Several providers are excluded from being installed and wait for Python 3.12, but it should not block Airlfow's general 3.12 support. Co-authored-by: dirrao <[email protected]>
This reverts commit 76dee0b.
Finally after a number of dependency upgrades we seem to be able to upgrade to Python 3.12 (pending universal_pathlib 0.2.0 conversion) Several providers are excluded from being installed and wait for Python 3.12, but it should not block Airlfow's general 3.12 support. Co-authored-by: dirrao <[email protected]> (cherry picked from commit 76dee0b)
ephraimbuddy
added
the
changelog:skip
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in apache/airflow#36755 GitOrigin-RevId: 13012744ada457883e57848f6fc45454d9c25a4c
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in apache/airflow#36755 GitOrigin-RevId: 13012744ada457883e57848f6fc45454d9c25a4c
The Universal Pathlib provides Pathlib-like interface for FSSPEC In 0.1. *It was not very well defined for extension, so the way how we use it for 0.1.* so we used a lot of private methods and attributes that were not defined in the interface an they are broken with version 0.2.0 which is much better suited for extension and supports Python 3.12. We should limit it, unti we migrate to 0.2.0 See: fsspec/universal_pathlib#173 (comment) This is prerequistite to make Airflow compatible with Python 3.12 Tracked in apache/airflow#36755 GitOrigin-RevId: 13012744ada457883e57848f6fc45454d9c25a4c
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.