Fix Python PreCommit dependency compatibility test not run or redundant setup #25115
Conversation
…nt setup * Move pyarrow and pandas matrix unit test to preCommitPyCoverage * Fix Dataframe compatibility test not run * Update readme
Codecov Report
@@ Coverage Diff @@
## master #25115 +/- ##
==========================================
- Coverage 73.14% 73.13% -0.01%
==========================================
Files 735 735
Lines 98161 98161
==========================================
- Hits 71796 71789 -7
- Misses 25002 25009 +7
Partials 1363 1363
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Python_Dataframes precommit now has Task like Python_Examples precommit now costs 14 min compared to ~25 min before: https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Commit/275/ other jobs also costs ~10 min shorter Python_Coverage now runs pyarrow and pytorch compatibility matrix tests. Ready for review now. R: @damccorm |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
|
|
||
| // Create a test task for each major version of pyarrow | ||
| toxTask "testPy38pyarrow-0", "py38-pyarrow-0", "${posargs}" | ||
| test.dependsOn "testPy38pyarrow-0" | ||
| preCommitPy38.dependsOn "testPy38pyarrow-0" | ||
| preCommitPyCoverage.dependsOn "testPy38pyarrow-0" |
There was a problem hiding this comment.
This is the right move for these tests, we should probably actually do the same thing for the dataframes tests so that they get counted towards codecoverage.
I'm approving this PR, feel free to merge as is or make that change. If you merge as is, I'll put up a follow up PR to move the dataframes tests into the coverage job.
There was a problem hiding this comment.
thanks, moving dataframe compatibility tests into preCommitPyCoverage
|
Python_Coverage now becomes slowest task and can costs > 1h when jenkins nodes are full. Not ideal. The dependency tasks can actually run in parallel |
|
https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/14639/ actually succeeded. merge for now. |
Followup of #24866
It is found that dataframe compatibility matrix tests are actually not exercised after splitting due to a typo.
Also, sharded precommits set up full version of pyarrow and pytorch compatibility test tasks, but most of them do not have test run. Observed that most of these tests costs >20 min run, where about 10 minutes are contributed by the tox setup for these compability test tasks. This is inefficient. Moved these tasks to be exercised by beam_PreCommit_Python_Coverage which deals with precommit tests exercised by single py version (currently is Py38).
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.