Skip to content

Commit

Permalink
Shard python precommit (#24204)
Browse files Browse the repository at this point in the history
* WIP: Shard python precommit

* Naming

* Formatting

* Allow empty test suites

* Fix no posargs case

* Fix paths

* Split out integration tests

* Do common tasks in IT suite

* Fix name base

* More splitting

* Fix quotes

* try ignore-glob

* Add groovy files per subfolder

* name typo

* Filter dataframes tests

* Logging

* Switch to correct directory syntax

* Merge master + fix paths

* Consolidate back to reasonable set

* Add jobs to README

* Try full quotes

* Add in missing tests

* Move arg lines down
  • Loading branch information
damccorm authored Dec 29, 2022
1 parent c8bf5d5 commit 598324a
Show file tree
Hide file tree
Showing 13 changed files with 265 additions and 48 deletions.
5 changes: 5 additions & 0 deletions .test-infra/jenkins/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,11 @@ Beam Jenkins overview page: [link](https://ci-beam.apache.org/)
| beam_PreCommit_Portable_Python | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Phrase/) | `Run Portable_Python PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron) |
| beam_PreCommit_PythonLint | [commit](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Phrase/) | `Run PythonLint PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron) |
| beam_PreCommit_Python | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Phrase/) | `Run Python PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron) |
| beam_PreCommit_Python_Integration | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Integration_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Integration_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Integration_Phrase/) | `Run Python_Integration PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Integration_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Integration_Cron) |
| beam_PreCommit_Python_Dataframe | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Dataframe_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Dataframe_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Dataframe_Phrase/) | `Run Python_TODO PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Dataframe_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Dataframe_Cron) |
| beam_PreCommit_Python_Examples | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Phrase/) | `Run Python_TODO PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Examples_Cron) |
| beam_PreCommit_Python_Runners | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Phrase/) | `Run Python_TODO PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Runners_Cron) |
| beam_PreCommit_Python_Transforms | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Phrase/) | `Run Python_TODO PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Transforms_Cron) |
| beam_PreCommit_PythonDocker | [commit](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Phrase/) | `Run PythonDocker PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/) |
| beam_PreCommit_PythonDocs| [commit](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Phrase/) | `Run PythonDocs PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/) |
| beam_PreCommit_Python_PVR_Flink | [commit](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Commit/), [cron](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/), [phrase](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Phrase/) | `Run Python_PVR_Flink PreCommit` | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron) |
Expand Down
3 changes: 3 additions & 0 deletions .test-infra/jenkins/job_PreCommit_Python.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Python',
gradleTask: ':pythonPreCommit',
gradleSwitches: [
'-Pposargs=\"apache_beam/*.py apache_beam/coders apache_beam/internal apache_beam/io apache_beam/metrics apache_beam/ml apache_beam/options apache_beam/portability apache_beam/testing apache_beam/tools apache_beam/typehints apache_beam/utils\"' // All other tests are covered by different jobs.
],
timeoutMins: 180,
triggerPathPatterns: [
'^model/.*$',
Expand Down
37 changes: 37 additions & 0 deletions .test-infra/jenkins/job_PreCommit_PythonIT.groovy
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import PrecommitJobBuilder

PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Python_Integration',
gradleTask: ':pythonPreCommitIT',
timeoutMins: 180,
triggerPathPatterns: [
'^model/.*$',
'^sdks/python/.*$',
'^release/.*$',
]
)
builder.build {
// Publish all test results to Jenkins.
publishers {
archiveJunit('**/pytest*.xml')
}
}
40 changes: 40 additions & 0 deletions .test-infra/jenkins/job_PreCommit_Python_Dataframe.groovy
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import PrecommitJobBuilder

PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Python_Dataframe',
gradleTask: ':pythonPreCommit',
gradleSwitches: [
'-Pposargs=apache_beam/dataframe/'
],
timeoutMins: 180,
triggerPathPatterns: [
'^model/.*$',
'^sdks/python/.*$',
'^release/.*$',
]
)
builder.build {
// Publish all test results to Jenkins.
publishers {
archiveJunit('**/pytest*.xml')
}
}
40 changes: 40 additions & 0 deletions .test-infra/jenkins/job_PreCommit_Python_Examples.groovy
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import PrecommitJobBuilder

PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Python_Examples',
gradleTask: ':pythonPreCommit',
gradleSwitches: [
'-Pposargs=apache_beam/examples/'
],
timeoutMins: 180,
triggerPathPatterns: [
'^model/.*$',
'^sdks/python/.*$',
'^release/.*$',
]
)
builder.build {
// Publish all test results to Jenkins.
publishers {
archiveJunit('**/pytest*.xml')
}
}
40 changes: 40 additions & 0 deletions .test-infra/jenkins/job_PreCommit_Python_Runners.groovy
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import PrecommitJobBuilder

PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Python_Runners',
gradleTask: ':pythonPreCommit',
gradleSwitches: [
'-Pposargs=apache_beam/runners/'
],
timeoutMins: 180,
triggerPathPatterns: [
'^model/.*$',
'^sdks/python/.*$',
'^release/.*$',
]
)
builder.build {
// Publish all test results to Jenkins.
publishers {
archiveJunit('**/pytest*.xml')
}
}
40 changes: 40 additions & 0 deletions .test-infra/jenkins/job_PreCommit_Python_Transforms.groovy
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import PrecommitJobBuilder

PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Python_Transforms',
gradleTask: ':pythonPreCommit',
gradleSwitches: [
'-Pposargs=apache_beam/transforms/'
],
timeoutMins: 180,
triggerPathPatterns: [
'^model/.*$',
'^sdks/python/.*$',
'^release/.*$',
]
)
builder.build {
// Publish all test results to Jenkins.
publishers {
archiveJunit('**/pytest*.xml')
}
}
4 changes: 4 additions & 0 deletions build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,10 @@ tasks.register("pythonPreCommit") {
dependsOn(":sdks:python:test-suites:tox:py38:preCommitPy38")
dependsOn(":sdks:python:test-suites:tox:py39:preCommitPy39")
dependsOn(":sdks:python:test-suites:tox:py310:preCommitPy310")
}

tasks.register("pythonPreCommitIT") {
dependsOn(":sdks:python:test-suites:tox:pycommon:preCommitPyCommon")
dependsOn(":sdks:python:test-suites:dataflow:preCommitIT")
dependsOn(":sdks:python:test-suites:dataflow:preCommitIT_V2")
}
Expand Down
7 changes: 1 addition & 6 deletions sdks/python/scripts/run_pytest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,7 @@ pytest -o junit_suite_name=${envname}_no_xdist \
--junitxml=pytest_${envname}_no_xdist.xml -m 'no_xdist' ${pytest_args} --pyargs ${posargs}
status2=$?

# Exit with error if no tests were run in either suite (status code 5).
if [[ $status1 == 5 && $status2 == 5 ]]; then
exit $status1
fi

# Exit with error if one of the statuses has an error that's not 5.
# Exit with error if one of the statuses has an error that's not 5 (no tests run).
if [[ $status1 != 0 && $status1 != 5 ]]; then
exit $status1
fi
Expand Down
12 changes: 7 additions & 5 deletions sdks/python/test-suites/tox/common.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,21 @@

def pythonVersionSuffix = project.ext.pythonVersion.replace('.', '')

toxTask "testPython${pythonVersionSuffix}", "py${pythonVersionSuffix}"
def posargs = project.findProperty("posargs") ?: ""

toxTask "testPython${pythonVersionSuffix}", "py${pythonVersionSuffix}", "${posargs}"
test.dependsOn "testPython${pythonVersionSuffix}"

toxTask "testPy${pythonVersionSuffix}Cloud", "py${pythonVersionSuffix}-cloud"
toxTask "testPy${pythonVersionSuffix}Cloud", "py${pythonVersionSuffix}-cloud", "${posargs}"
test.dependsOn "testPy${pythonVersionSuffix}Cloud"

toxTask "testPy${pythonVersionSuffix}Dask", "py${pythonVersionSuffix}-dask"
toxTask "testPy${pythonVersionSuffix}Dask", "py${pythonVersionSuffix}-dask", "${posargs}"
test.dependsOn "testPy${pythonVersionSuffix}Dask"

toxTask "testPy${pythonVersionSuffix}Cython", "py${pythonVersionSuffix}-cython"
toxTask "testPy${pythonVersionSuffix}Cython", "py${pythonVersionSuffix}-cython", "${posargs}"
test.dependsOn "testPy${pythonVersionSuffix}Cython"

toxTask "testPy38CloudCoverage", "py38-cloudcoverage"
toxTask "testPy38CloudCoverage", "py38-cloudcoverage", "${posargs}"
test.dependsOn "testPy38CloudCoverage"

project.tasks.register("preCommitPy${pythonVersionSuffix}") {
Expand Down
6 changes: 4 additions & 2 deletions sdks/python/test-suites/tox/py37/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,15 @@ applyPythonNature()
// Required to setup a Python 3 virtualenv and task names.
pythonVersion = '3.7'

def posargs = project.findProperty("posargs") ?: ""

task lint {}
check.dependsOn lint

toxTask "lintPy37", "py37-lint"
toxTask "lintPy37", "py37-lint", "${posargs}"
lint.dependsOn lintPy37

toxTask "mypyPy37", "py37-mypy"
toxTask "mypyPy37", "py37-mypy", "${posargs}"
lint.dependsOn mypyPy37

apply from: "../common.gradle"
Expand Down
Loading

0 comments on commit 598324a

Please sign in to comment.