Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various release script fixes #27548

Merged
merged 2 commits into from
Jul 19, 2023
Merged

Various release script fixes #27548

merged 2 commits into from
Jul 19, 2023

Conversation

Abacn
Copy link
Contributor

@Abacn Abacn commented Jul 18, 2023

Fix various minor issue encountered in 2.49.0 release process

Fixes #27521
Fixes #27315
Fixes #27320

================Setting Up Environment Variables===========
Which release version are you working on: 
2.49.0
Which release candidate will be the source of final docker images? (ex: 1)
2
================Pull RC Containers from DockerHub===========
We are using rc2 to push docker images for 2.49.0.
2.49.0rc2: Pulling from apache/beam_python3.8_sdk
...
================Confirming Release and RC version===========
Publishing the following images:
apache/beam_flink1.12_job_server
apache/beam_flink1.13_job_server
apache/beam_flink1.14_job_server
apache/beam_flink1.15_job_server
apache/beam_flink1.16_job_server
apache/beam_go_sdk
apache/beam_java11_sdk
apache/beam_java17_sdk
apache/beam_java8_sdk
apache/beam_java_expansion_service
apache/beam_python3.10_sdk
apache/beam_python3.11_sdk
apache/beam_python3.8_sdk
apache/beam_python3.9_sdk
apache/beam_spark3_job_server
apache/beam_transform_service_controller
apache/beam_typescript_sdk
Do you want to proceed? [y|N]
y
docker push apache/beam_flink1.12_job_server:2.49.0
docker push apache/beam_flink1.12_job_server:latest
docker push apache/beam_flink1.13_job_server:2.49.0
docker push apache/beam_flink1.13_job_server:latest
docker push apache/beam_flink1.14_job_server:2.49.0
docker push apache/beam_flink1.14_job_server:latest
docker push apache/beam_flink1.15_job_server:2.49.0
docker push apache/beam_flink1.15_job_server:latest
docker push apache/beam_flink1.16_job_server:2.49.0
docker push apache/beam_flink1.16_job_server:latest
docker push apache/beam_go_sdk:2.49.0
docker push apache/beam_go_sdk:latest
docker push apache/beam_java11_sdk:2.49.0
docker push apache/beam_java11_sdk:latest
docker push apache/beam_java17_sdk:2.49.0
docker push apache/beam_java17_sdk:latest
docker push apache/beam_java8_sdk:2.49.0
docker push apache/beam_java8_sdk:latest
docker push apache/beam_java_expansion_service:2.49.0
docker push apache/beam_java_expansion_service:latest
docker push apache/beam_python3.10_sdk:2.49.0
docker push apache/beam_python3.10_sdk:latest
docker push apache/beam_python3.11_sdk:2.49.0
docker push apache/beam_python3.11_sdk:latest
docker push apache/beam_python3.8_sdk:2.49.0
docker push apache/beam_python3.8_sdk:latest
docker push apache/beam_python3.9_sdk:2.49.0
docker push apache/beam_python3.9_sdk:latest
docker push apache/beam_spark3_job_server:2.49.0
docker push apache/beam_spark3_job_server:latest
docker push apache/beam_transform_service_controller:2.49.0
docker push apache/beam_transform_service_controller:latest
docker push apache/beam_typescript_sdk:2.49.0
docker push apache/beam_typescript_sdk:latest
  • Change empty file to .json to make RAT_PreCommit happy

Otherwise PreCommit_RAT job will fail. json is explicitly excluded from license check:

"**/*.json",

  • Set base branch to the release branch for start_snapshot_build PR

If cut the release using github action, the auto-generated pull request used for start_snapshot_build default base branch is master. This would then build next version snapshot. The base branch should be release-2.xx.0, as in the release verification PR generated here:

hub pull-request -b apache:${RELEASE_BRANCH} -h ${GITHUB_USERNAME}:${WORKING_BRANCH} -F- <<<"[DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

  • Bump pubsub signal wait for pipeline start to 8 min in integration tests

The flakinesses reported in #27315 are due to worker startup > 5 min. It usually takes 4-6 min depending on the staging bandwidths, 8 min should suffice.

  • Change message head from Warning to Level for PubsubInjector as it follows by an exit(1)

Even when this exit(1) hit, the program exits quitely but the validation script still runs and validation fails with no data written. This should be an Error.

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

* Remove Release_Gradle_Build job from release verification

* Fix publish docker image script to work with remotely built images

* Change empty file to .json to make RAT_PreCommit happy

* Set base branch to the release branch for build snapshot PR

* Bump pubsub signal wait for pipeline start to 8 min in integration tests

* Change message head from Warning to Level for PubsubInjector
  as it follows by an exit(1)
@Abacn
Copy link
Contributor Author

Abacn commented Jul 18, 2023

pylint appears non-deterministic? failed on this PR, added a fix.

@codecov
Copy link

codecov bot commented Jul 18, 2023

Codecov Report

Merging #27548 (759fbf7) into master (c8f68f9) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #27548   +/-   ##
=======================================
  Coverage   71.14%   71.14%           
=======================================
  Files         861      861           
  Lines      104547   104561   +14     
=======================================
+ Hits        74385    74395   +10     
- Misses      28613    28617    +4     
  Partials     1549     1549           
Flag Coverage Δ
python 80.34% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 11 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@github-actions
Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @riteshghorse for label python.
R: @bvolpato for label java.
R: @damccorm for label build.
R: @manavgarg for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Copy link
Contributor

@damccorm damccorm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thank you!

Copy link
Contributor

@bvolpato bvolpato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me too! Thanks Yi!

@Abacn
Copy link
Contributor Author

Abacn commented Jul 19, 2023

SpannerIT flake not related, merging for now

@Abacn Abacn merged commit b2e00ef into apache:master Jul 19, 2023
@Abacn Abacn deleted the releasefixes branch July 19, 2023 14:04
aleksandr-dudko pushed a commit to aleksandr-dudko/beam that referenced this pull request Jul 19, 2023
* Various release script fixes

* Remove Release_Gradle_Build job from release verification

* Fix publish docker image script to work with remotely built images

* Change empty file to .json to make RAT_PreCommit happy

* Set base branch to the release branch for build snapshot PR

* Bump pubsub signal wait for pipeline start to 8 min in integration tests

* Change message head from Warning to Level for PubsubInjector
  as it follows by an exit(1)

* Fix Pylint
cushon pushed a commit to cushon/beam that referenced this pull request May 24, 2024
* Various release script fixes

* Remove Release_Gradle_Build job from release verification

* Fix publish docker image script to work with remotely built images

* Change empty file to .json to make RAT_PreCommit happy

* Set base branch to the release branch for build snapshot PR

* Bump pubsub signal wait for pipeline start to 8 min in integration tests

* Change message head from Warning to Level for PubsubInjector
  as it follows by an exit(1)

* Fix Pylint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants