Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a 30 second sleep in mass_comment.py to avoid overwhelming Jenkins #25714

Merged
merged 2 commits into from
Mar 3, 2023

Conversation

damccorm
Copy link
Contributor

@damccorm damccorm commented Mar 3, 2023

I found that running everything at once led to dropped jobs and conflicts between jobs (e.g. Gradle issues, Jenkins worker failures). This helped dramatically (I only had one or two jobs with Jenkins issues when I did this, as opposed to 20+). It also avoids harming the Jenkins experience for others who are planning on doing normal CI.

I also removed Java Portability examples on Dataflow with Java 11 since it doesn't seem to be a real postcommit (see #25600 (comment))

This will cause the script to run for around 1.25 hours, but shouldn't make a huge difference in the time to completion of the jobs themselves since a full Jenkins queue tends to be the blocker here anyways.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI.

@github-actions github-actions bot added the build label Mar 3, 2023
@damccorm
Copy link
Contributor Author

damccorm commented Mar 3, 2023

R: @johnjcasey @kennknowles @Abacn

@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2023

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

Copy link
Member

@kennknowles kennknowles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the obvious question is... can this script be a jenkins job?

@kennknowles
Copy link
Member

So the obvious question is... can this script be a jenkins job?

Or GitHub Action

@codecov
Copy link

codecov bot commented Mar 3, 2023

Codecov Report

Merging #25714 (66a5287) into master (60ac6ef) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #25714      +/-   ##
==========================================
- Coverage   72.82%   72.82%   -0.01%     
==========================================
  Files         775      775              
  Lines      102928   102928              
==========================================
- Hits        74957    74953       -4     
- Misses      26517    26521       +4     
  Partials     1454     1454              
Flag Coverage Δ
python 81.96% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
sdks/python/apache_beam/utils/interactive_utils.py 95.12% <0.00%> (-2.44%) ⬇️
sdks/python/apache_beam/runners/direct/executor.py 96.46% <0.00%> (-0.55%) ⬇️
sdks/python/apache_beam/transforms/combiners.py 93.05% <0.00%> (-0.39%) ⬇️
...ks/python/apache_beam/runners/worker/sdk_worker.py 89.46% <0.00%> (+0.15%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@damccorm
Copy link
Contributor Author

damccorm commented Mar 3, 2023

It actually is part of

sh ./comment_pr_trigger_phrases.sh $GITHUB_PR_URL
- that's currently on broken self-hosted runners though.

This is a good reminder that I forgot to update that job too, just did

@damccorm
Copy link
Contributor Author

damccorm commented Mar 3, 2023

Unfortunately, our half done GH release automation is probably slowly drifting away from our actual release scripts

@damccorm
Copy link
Contributor Author

damccorm commented Mar 3, 2023

Also, arguably we should be solving this problem by hitting Jenkins directly and gathering results at the end instead of using more expensive/indirect GH PR comments.

All of that is more work than bumping the timeout though, thus the current fix

Copy link
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! It would greatly reduce the need of babysitting jenkins

@damccorm damccorm merged commit 39cab94 into master Mar 3, 2023
@damccorm damccorm deleted the users/damccorm/mass-comment-sleep branch March 3, 2023 15:46
ruslan-ikhsan pushed a commit to akvelon/beam that referenced this pull request Mar 10, 2023
apache#25714)

* Add a 30 second sleep in mass_comment.py to avoid overwhelming Jenkins

* Update comment_pr_trigger_phrases.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants