Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various release script fixes #27548

Merged
merged 2 commits into from
Jul 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/workflows/cut_release_branch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ jobs:
fi
done

cat /tmp/result | sort | uniq | grep -i -E 'precommit|postcommit|validates|vr|example|test|gradle build' | grep -v -i -E 'load|perf|website' >> release/src/main/scripts/jenkins_jobs.txt
cat /tmp/result | sort | uniq | grep -i -E 'precommit|postcommit|validates|vr|example|test' | grep -v -i -E 'load|perf|website' >> release/src/main/scripts/jenkins_jobs.txt
env:
JENKINS_USERNAME: ${{ github.event.inputs.JENKINS_USERNAME }}
JENKINS_TOKEN: ${{ github.event.inputs.JENKINS_TOKEN }}
Expand Down Expand Up @@ -175,6 +175,7 @@ jobs:
REMOTE_NAME: remote_repo
REMOTE_URL: ${{ github.server_url }}/${{ github.repository }}
BRANCH_NAME: snapshot_build-${{ github.event.inputs.RELEASE_VERSION }}
RELEASE_BRANCH: release-${{ github.event.inputs.RELEASE_VERSION }}
steps:
- name: Install Hub
run: |
Expand All @@ -192,15 +193,15 @@ jobs:
run: |
git remote add ${REMOTE_NAME} ${REMOTE_URL}
git checkout -b ${BRANCH_NAME}
touch empty_file.txt
touch empty_file.json
git add -A
git commit -m "Add empty file in order to create PR"
git push -f ${REMOTE_NAME}
- name: Create Pull Request
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
hub pull-request -F- <<<"[DO NOT MERGE]Start snapshot build for release process
hub pull-request -b apache:${RELEASE_BRANCH} -F- <<<"[DO NOT MERGE]Start snapshot build for release process


Run Gradle Publish"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/run_rc_validation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@ jobs:
- name: Create Pull Request
run: |
git checkout -b ${{env.WORKING_BRANCH}} ${{ env.RC_TAG }} --quiet
touch empty_file.txt
git add empty_file.txt
touch empty_file.json
git add empty_file.json
git commit -m "Add empty file in order to create PR" --quiet
git push origin ${{env.WORKING_BRANCH}} --quiet
GITHUB_PR_URL=$(gh pr create -B ${{env.RELEASE_BRANCH}} -H ${{env.WORKING_BRANCH}} -t"[DO NOT MERGE] Run Python RC Validation Tests" -b "PR to run Python ReleaseCandidate Jenkins Job.")
Expand Down
1 change: 0 additions & 1 deletion .test-infra/jenkins/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,6 @@ Beam Jenkins overview page: [link](https://ci-beam.apache.org/)
| beam_Publish_Beam_SDK_Snapshots | [cron](https://ci-beam.apache.org/job/beam_Publish_Beam_SDK_Snapshots/)| N/A | [![Build Status](https://ci-beam.apache.org/job/beam_Publish_Beam_SDK_Snapshots/badge/icon)](https://ci-beam.apache.org/job/beam_Publish_Beam_SDK_Snapshots/) |
| beam_Publish_Docker_Snapshots | [cron](https://ci-beam.apache.org/job/beam_Publish_Docker_Snapshots/)| N/A | [![Build Status](https://ci-beam.apache.org/job/beam_Publish_Docker_Snapshots/badge/icon)](https://ci-beam.apache.org/job/beam_Publish_Docker_Snapshots/) |
| beam_PostRelease_Python_Candidate | [cron](https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/)| `Run Python ReleaseCandidate` | [![Build Status](https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/badge/icon)](https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/) |
| beam_Release_Gradle_Build | [cron](https://ci-beam.apache.org/job/beam_Release_Gradle_Build/) | `Run Release Gradle Build` | [![Build Status](https://ci-beam.apache.org/job/beam_Release_Gradle_Build/badge/icon)](https://ci-beam.apache.org/job/beam_Release_Gradle_Build/)

### Notes:

Expand Down
49 changes: 0 additions & 49 deletions .test-infra/jenkins/job_Release_Gradle_Build.groovy

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ public static Pubsub getClient(final HttpTransport httpTransport, final JsonFact
}
if (credential.getClientAuthentication() != null) {
System.out.println(
"\n***Warning! You are not using service account credentials to "
"\n***Error! You are not using service account credentials to "
+ "authenticate.\nYou need to use service account credentials for this example,"
+ "\nsince user-level credentials do not have enough pubsub quota,\nand so you will run "
+ "out of PubSub quota very quickly.\nSee "
Expand Down
1 change: 0 additions & 1 deletion release/src/main/scripts/jenkins_jobs.txt
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,6 @@ Run Python_Transforms PreCommit,beam_PreCommit_Python_Transforms_Phrase
Run Python_Xlang_Gcp_Dataflow PostCommit,beam_PostCommit_Python_Xlang_Gcp_Dataflow_PR
Run Python_Xlang_Gcp_Direct PostCommit,beam_PostCommit_Python_Xlang_Gcp_Direct_PR
Run RAT PreCommit,beam_PreCommit_RAT_Phrase
Run Release Gradle Build,beam_Release_Gradle_Build
Run SQL PostCommit,beam_PostCommit_SQL_PR
Run SQL PreCommit,beam_PreCommit_SQL_Phrase
Run SQL_Java11 PreCommit,beam_PreCommit_SQL_Java11_Phrase
Expand Down
25 changes: 17 additions & 8 deletions release/src/main/scripts/publish_docker_images.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,18 +37,27 @@ echo "Which release candidate will be the source of final docker images? (ex: 1)
read RC_NUM
RC_VERSION="rc${RC_NUM}"

echo "================Confirming Release and RC version==========="
echo "================Pull RC Containers from DockerHub==========="
IMAGES=$(docker search ${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX} --format "{{.Name}}" --limit 100)
KNOWN_IMAGES=()
echo "We are using ${RC_VERSION} to push docker images for ${RELEASE}."
while read IMAGE; do
# Try pull verified RC from dockerhub.
if docker pull "${IMAGE}:${RELEASE}${RC_VERSION}" 2>/dev/null ; then
KNOWN_IMAGES+=( $IMAGE )
fi
done < <(echo "${IMAGES}")

echo "================Confirming Release and RC version==========="
echo "Publishing the following images:"
IMAGES=$(docker images --filter "reference=apache/beam_*:${RELEASE}${RC_VERSION}" --format "{{.Repository}}")
echo "${IMAGES}"
# Sort by name for easy examination
IFS=$'\n' KNOWN_IMAGES=($(sort <<<"${KNOWN_IMAGES[*]}"))
unset IFS
printf "%s\n" ${KNOWN_IMAGES[@]}
echo "Do you want to proceed? [y|N]"
read confirmation
if [[ $confirmation = "y" ]]; then
echo "${IMAGES}" | while read IMAGE; do
# Pull verified RC from dockerhub.
docker pull "${IMAGE}:${RELEASE}${RC_VERSION}"

for IMAGE in "${KNOWN_IMAGES[@]}"; do
# Tag with ${RELEASE} and push to dockerhub.
docker tag "${IMAGE}:${RELEASE}${RC_VERSION}" "${IMAGE}:${RELEASE}"
docker push "${IMAGE}:${RELEASE}"
Expand All @@ -58,4 +67,4 @@ if [[ $confirmation = "y" ]]; then
docker push "${IMAGE}:latest"
done

fi
fi
4 changes: 2 additions & 2 deletions release/src/main/scripts/run_rc_validation.sh
Original file line number Diff line number Diff line change
Expand Up @@ -271,8 +271,8 @@ echo "This task will create a PR against apache/beam, trigger a jenkins job to r
echo "1. Python quickstart validations(batch & streaming)"
echo "2. Python MobileGame validations(UserScore, HourlyTeamScore)"
if [[ "$python_quickstart_mobile_game" = true && ! -z `which hub` ]]; then
touch empty_file.txt
git add empty_file.txt
touch empty_file.json
git add empty_file.json
git commit -m "Add empty file in order to create PR" --quiet
git push -f ${GITHUB_USERNAME} --quiet
# Create a test PR
Expand Down
4 changes: 2 additions & 2 deletions release/src/main/scripts/verify_release_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -125,8 +125,8 @@ hub version


echo ""
echo "==================== 3 Run Gradle Release Build & PostCommit Tests on Jenkins ==================="
echo "[Current Task] Run Gradle release build and all PostCommit Tests against Release Branch on Jenkins."
echo "==================== 3 Run PostCommit Tests on Jenkins ==================="
echo "[Current Task] Run all PostCommit Tests against Release Branch on Jenkins."
echo "This task will create a PR against apache/beam."
echo "After PR created, you need to comment phrases listed in description in the created PR:"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ public void testFhirIORead() throws Exception {
"waitForAnyMessage", signal.signalSuccessWhen(resources.getCoder(), anyResources -> true));
// wait for any resource

Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
pipeline.apply(signal.signalStart());
PipelineResult job = pipeline.run();
start.get();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ public void testReadPublicData() throws Exception {
messages.apply(
"waitForAnyMessage", signal.signalSuccessWhen(messages.getCoder(), anyMessages -> true));

Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
pipeline.apply(signal.signalStart());
PipelineResult job = pipeline.run();
start.get();
Expand Down Expand Up @@ -79,7 +79,7 @@ public void testReadPubsubMessageId() throws Exception {
"isMessageIdNonNull",
signal.signalSuccessWhen(messages.getCoder(), new NonEmptyMessageIdCheck()));

Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
pipeline.apply(signal.signalStart());
PipelineResult job = pipeline.run();
start.get();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ public void testPubsubLiteWriteReadWithSchemaTransform() throws Exception {
return Objects.requireNonNull(row.getInt64("numberInInt")).intValue();
}));
ids.apply("PubsubSignalTest", signal.signalSuccessWhen(BigEndianIntegerCoder.of(), testIds()));
Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
pipeline.apply("start signal", signal.signalStart());
PipelineResult job = pipeline.run();
start.get();
Expand Down Expand Up @@ -363,7 +363,7 @@ public void testReadWrite() throws Exception {
PCollection<SequencedMessage> messages = readMessages(subscription, pipeline);
PCollection<Integer> ids = messages.apply(MapElements.via(extractIds()));
ids.apply("PubsubSignalTest", signal.signalSuccessWhen(BigEndianIntegerCoder.of(), testIds()));
Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(5));
Supplier<Void> start = signal.waitForStart(Duration.standardMinutes(8));
pipeline.apply(signal.signalStart());
PipelineResult job = pipeline.run();
start.get();
Expand Down
1 change: 0 additions & 1 deletion sdks/python/apache_beam/runners/worker/sdk_worker_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
import logging
import unittest
from collections import namedtuple
from typing import Any

import grpc
import hamcrest as hc
Expand Down
5 changes: 1 addition & 4 deletions website/www/site/content/en/contribute/release-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ There are 2 ways to perform this verification, either running automation script(
```
cd beam/release/src/main/scripts && ./verify_release_build.sh
```
1. Trigger `beam_Release_Gradle_Build` and all Jenkins PostCommit jobs from the PR created by the previous step.
1. Trigger all Jenkins PostCommit jobs from the PR created by the previous step.
You can run [mass_comment.py](https://github.com/apache/beam/blob/master/release/src/main/scripts/mass_comment.py) to do that.
Or manually add one trigger phrase per PR comment.
See [jenkins_jobs.txt](https://github.com/apache/beam/blob/master/release/src/main/scripts/jenkins_jobs.txt)
Expand All @@ -328,9 +328,6 @@ There are 2 ways to perform this verification, either running automation script(
1. Installs ```hub``` with your agreement and setup local git repo;
1. Create a test PR against release branch;

The [`beam_Release_Gradle_Build`](https://ci-beam.apache.org/job/beam_Release_Gradle_Build/) Jenkins job runs `./gradlew build -PisRelease`.
This only verifies that everything builds with unit tests passing.

#### Verify the build succeeds

* Tasks you need to do manually to __verify the build succeed__:
Expand Down