Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix gradle command not found error for Python ValidatesContainer Dataflow ARM #27810

Merged
merged 9 commits into from
Aug 2, 2023
Merged

Fix gradle command not found error for Python ValidatesContainer Dataflow ARM #27810

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6156efb
fix gradle command not found error.
celeste-zeng Aug 2, 2023
8af760a
Update run_validatescontainer.sh, exit if invalid CPU architecture gi…
celeste-zeng Aug 2, 2023
956d56d
Formatting changes.
celeste-zeng Aug 2, 2023
497a738
Remove white spaces.
celeste-zeng Aug 2, 2023
74f99c8
Fix trigger command in README.
celeste-zeng Aug 2, 2023
2179627
update github job.
celeste-zeng Aug 2, 2023
37deceb
Merge branch 'apache:master' into gh-py-fix
celeste-zeng Aug 2, 2023
4d50777
Allow workflow to be manually trigger.
celeste-zeng Aug 2, 2023
025efa3
Fix whitespace.
celeste-zeng Aug 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,4 +137,4 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
| [ PreCommit Website ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml) | N/A |`Run Website PreCommit`| [![.github/workflows/beam_PreCommit_Website.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml) |
| [ PreCommit Website Stage GCS ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) | N/A |`Run Website_Stage_GCS PreCommit`| [![PreCommit Website Stage GCS](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) |
| [ PreCommit Whitespace ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml) | N/A |`Run Whitespace PreCommit`| [![.github/workflows/beam_PreCommit_Whitespace.yml](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml) |
| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.8','3.9','3.10','3.11'] | `Run Python Validates Container Dataflow ARM (matrix_element)`| [![.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) |
| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.8','3.9','3.10','3.11'] | `Run Python ValidatesContainer Dataflow ARM (matrix_element)`| [![.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) |
18 changes: 10 additions & 8 deletions .github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,15 +49,16 @@ concurrency:

jobs:
beam_Python_ValidatesContainer_Dataflow_ARM:
name: beam_Python_ValidatesContainer_Dataflow_ARM
name: beam_Python_ValidatesContainer_Dataflow_ARM (Run Python ValidatesContainer Dataflow ARM 3.x)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we remove 3.x from the name and trigger commands?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Followed what was done here:

beam/.github/workflows/beam_PreCommit_Python_Integration.yml

Line 54 in afa4371

name: beam_PreCommit_Python_Integration (Run Python_Integration PreCommit 3.x)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like we do need to specify matrix element. then, it's intentional. nvm.

strategy:
fail-fast: false
matrix:
python_version: ['3.8','3.9','3.10','3.11']
if: |
github.event_name == 'push' ||
startsWith(github.event.comment.body, 'Run Python ValidatesContainer Dataflow ARM') ||
github.event_name == 'schedule'
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
runs-on: [self-hosted, ubuntu-20.04, main]
env:
MULTIARCH_TAG: $(date +'%Y%m%d-%H%M%S%N')
Expand All @@ -77,7 +78,7 @@ jobs:
pull_request_url: ${{ github.event.issue.pull_request.url }}
github_repository: ${{ github.repository }}
github_token: ${{ secrets.GITHUB_TOKEN }}
github_job: "${{ github.job }} (${{ matrix.python_version }})"
github_job: "beam_Python_ValidatesContainer_Dataflow_ARM (Run Python ValidatesContainer Dataflow ARM 3.x) (${{ matrix.python_version }})"
github_current_run_id: ${{ github.run_id }}
- name: Install Python
uses: actions/setup-python@v4
Expand Down Expand Up @@ -107,17 +108,18 @@ jobs:
echo "py_ver_clean=$PY_VER_CLEAN" >> $GITHUB_OUTPUT
- name: run Python validatesContainerARM script
id: run_script
run: |
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:python:test-suites:dataflow:py${{steps.set_py_ver_clean.outputs.py_ver_clean}}:validatesContainerARM
arguments: |
-PpythonVersion=${{ matrix.python_version }} \
-Pcontainer-architecture-list=arm64,amd64 \
-Pdocker-repository=us.gcr.io/apache-beam-testing/github-actions \
-Pdocker-tag=${{ env.MULTIARCH_TAG }} \
-Ppush-containers
env:
MULTIARCH_TAG: ${{ env.MULTIARCH_TAG }}
USER: github-actions
-Ppush-containers \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the last line-joining intentional?

Copy link
Contributor Author

@celeste-zeng celeste-zeng Aug 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just followed what was done here:

beam/.github/workflows/beam_PreCommit_Python_Integration.yml

Line 116 in afa4371

-PpythonVersion=${{ matrix.python_version }} \
But it shouldn't matter either way?

env:
MULTIARCH_TAG: ${{ env.MULTIARCH_TAG }}
USER: github-actions
- name: Archive code coverage results
uses: actions/upload-artifact@v3
with:
Expand Down
7 changes: 5 additions & 2 deletions sdks/python/container/run_validatescontainer.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ if [[ $# < 2 ]]; then
printf "Usage: \n$> ./sdks/python/container/run_validatescontainer.sh <python_version> <sdk_location> <cpu_architecture>"
printf "\n\tpython_version: [required] Python version used for container build and run tests."
printf " Sample value: 3.9"
printf "\n\tcpu_architecture: [optional] CPU architecture used for container build and run tests."
printf "\n\tcpu_architecture: [optional] CPU architecture used for container build and run tests, default as x86."
printf " Sample value: ARM or x86"
exit 1
fi
Expand Down Expand Up @@ -86,13 +86,16 @@ if [[ "$ARCH" == "x86" ]]; then

# Push the container
gcloud docker -- push $CONTAINER:$TAG
else
elif [[ "$ARCH" == "ARM" ]]; then
# Note: ARM test suites only run on github actions, where multi-arch Python SDK containers are already pushed during build.
# Reset the test suite tag to run ARM pipelines.
TEST_SUITE_TAG="it_dataflow_arm"

# Reset the multi-arch Python SDK container image tag.
TAG=$MULTIARCH_TAG
else
printf "Please give a valid CPU architecture, either x86 or ARM."
exit 1
fi

function cleanup_container {
Expand Down