-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Task]: Update the minor version of protobuf library in the upper bound prior to Beam release. #25590
Comments
@damccorm can you add the milestone 2.48.0 to catch this for the next release? |
The next release is 2.47.0 (the current one is 2.46.0) |
cc: @tvalentyn |
This is still up to date, moving to 2.48 |
This is still up to date (4.23.3<4.24.0), moving to 2.50 |
2.50 release manager here. Please complete work and get it into the main branch in that time, or move this issue to the 2.51 Milestone: https://github.com/apache/beam/milestone/15 |
Already on latest version 4.23.4. Moving to 2.51.0 Milestone |
We're up to date as of now looking at https://pypi.org/project/protobuf/ and https://github.com/apache/beam/blob/master/sdks/python/setup.py#L300 This might change if protobuf gets a fix in for #28246 - but I'll at least move this blocker forward and we can address that in the separate issue |
It looks like we're up to date with the upper bound at <4.26.0 and the latest release is 4.25.1. |
There's one week until the 2.54.0 cut and this issue is tagged for that release, if possible/necessary please complete the necessary work before then, or move this to the 2.55.0 Release Milestone. |
4.25.3 is the latest 4.x release as of 2.55.0 cut day. and it is note: protobuf currently encode its version in minor version number for all language impls; and the major version is reserved for language specific breaking change |
+1. I'm working with @tvalentyn on workaround for that |
We still need to wait for google-api-core to release new version. The latest version of google-api-core has a dependency on protobuf<5 which would result in dependency conflict if we update the protobuf version for beam.
|
Is there any outstanding work on the protobuf major version bump? |
https://pypi.org/pypi/google-api-core/2.19.0/json
The new version requires protobuf < 5.0.0.dev0 |
#30556 more work needed in the future. |
https://pypi.org/pypi/google-api-core/2.19.1/json Looks like the google-api-core package has been updated requiring < 6.0.0.dev0 for protobuf now. Do we want to prioritize this upgrade for 2.58.0, targeting the same upper bound? |
Followed up, google-api-core 2.19.1 supports protobuf 5 but other GCP dependencies (specifically google-cloud-aiplatform) do not. The workaround for the breaking timestamp change introduced in 5.26.0 explored in #30556 can go in before we formally support protobuf 5.x, but will need a little bit more effort to test as a result. I'll be coming back to work on this after the 2.58 release |
Do we need to do anything further for the 2.59.0 release for this, or should I push it to 2.60.0 ? |
It is fine to push to 2.60 |
protobuf 4.25.x still receives update, the latest one being 4.25.5 on Sept 18, 2024. Moving to 2.61.0 |
We now support arbitrary 5.x versions (as of #32679) so we can close this |
What needs to happen?
If a Beam dependency has a flexible upper bound, users will download the most recent compatible version of a dependency at sdk installation time. Overtime, the version used at job submission may become newer than the version installed in a released Beam container. Given that forwards-compatiblity of a protobuf library is not guaranteed, the pipeline may fail.
To mitigate, protobuf library should be specified in install_requires with tight upper bound limiting to last recently released minor version. But if we depend on an old version of a library, it will cause inconveniences users,. Therefore, we should periodically update the upper bound we set, at least once per release cycle.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
The text was updated successfully, but these errors were encountered: