Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up Python Tests workflow #33396

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Clean up Python Tests workflow #33396

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
354c6c2
Remove static GCP credentials from workflow
damondouglas Dec 10, 2024
2970849
Remove workflow_dispatch blocking input
damondouglas Dec 10, 2024
9cbe9a1
Remove conditional from Python SDK source step
damondouglas Dec 10, 2024
cfbe773
Fix parenthesis error
damondouglas Dec 10, 2024
48afcc7
Remove redundant Dataflow test
damondouglas Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 2 additions & 54 deletions .github/workflows/python_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,6 @@ on:
tags: 'v*'
paths: ['sdks/python/**', 'model/**']
workflow_dispatch:
inputs:
runDataflow:
description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
default: false

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
Expand All @@ -57,16 +53,15 @@ jobs:
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
GCP_REGION: ${{ secrets.GCP_REGION }}
GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
GCP_PYTHON_WHEELS_BUCKET: "not-needed-here"

build_python_sdk_source:
name: 'Build python source distribution'
if: |
needs.check_gcp_variables.outputs.gcp-variables-set == 'true' && (
(github.event_name == 'push' || github.event_name == 'schedule') ||
(github.event_name == 'workflow_dispatch' && github.event.inputs.runDataflow == 'true')
((github.event_name == 'push' || github.event_name == 'schedule') ||
github.event_name == 'workflow_dispatch')
)
needs:
- check_gcp_variables
Expand Down Expand Up @@ -153,50 +148,3 @@ jobs:
working-directory: ./sdks/python
shell: bash
run: python -m apache_beam.examples.wordcount --input MANIFEST.in --output counts

python_wordcount_dataflow:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, we are removing this since this a duplicate test that is already run elsewhere ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chamikaramj Thank you for reviewing. Yes. That's correct.

name: 'Python Wordcount Dataflow'
# TODO(https://github.com/apache/beam/issues/31848) run on Dataflow after fixes credential on macOS/win GHA runner
if: (github.event_name == 'workflow_dispatch' && github.event.inputs.runDataflow == 'true')
needs:
- build_python_sdk_source
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [[self-hosted, ubuntu-20.04, main], macos-latest, windows-latest]
python: ["3.9", "3.10", "3.11", "3.12"]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
python-version: ${{ matrix.python }}
go-version: default
- name: Download source from artifacts
uses: actions/[email protected]
with:
name: python_sdk_source
path: apache-beam-source
- name: Authenticate on GCP
id: auth
uses: google-github-actions/auth@v1
with:
credentials_json: ${{ secrets.GCP_SA_KEY }}
project_id: ${{ secrets.GCP_PROJECT_ID }}
- name: Install requirements
working-directory: ./sdks/python
run: pip install setuptools --upgrade && pip install -e ".[gcp]"
- name: Run WordCount
working-directory: ./sdks/python
shell: bash
run: |
python -m apache_beam.examples.wordcount \
--input gs://dataflow-samples/shakespeare/kinglear.txt \
--output gs://${{ secrets.GCP_TESTING_BUCKET }}/python_wordcount_dataflow/counts \
--runner DataflowRunner \
--project ${{ secrets.GCP_PROJECT_ID }} \
--region ${{ secrets.GCP_REGION }} \
--temp_location gs://${{ secrets.GCP_TESTING_BUCKET }}/tmp/python_wordcount_dataflow/ \
--sdk_location ../../apache-beam-source/apache-beam-source.tar.gz
Loading