The PostCommit Java IO Performance Tests job is flaky

[github-actions]

The PostCommit Java IO Performance Tests is failing over 50% of the time
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_IO_Performance_Tests.yml?query=is%3Afailure+branch%3Amaster to see the logs.

[kennknowles]

Also perma-red

[github-actions]

Reopening since the workflow is still flaky

[ahmedabu98]

@Abacn another test that is using the wrong secret

[damondouglas]

Situation

The Gradle task :it:google-cloud-platform:GCSPerformanceTest repeatedly fails with error:

com.google.auth.oauth2.GoogleAuthException at FileBasedIOLT.java

org.apache.beam.it.gcp.storage.FileBasedIOLT > testTextIOWriteThenRead FAILED	
    java.lang.IllegalArgumentException at FileBasedIOLT.java:191	
        Caused by: java.lang.RuntimeException at FileBasedIOLT.java:191	
            Caused by: java.io.IOException at FileBasedIOLT.java:191	
                Caused by: com.google.auth.oauth2.GoogleAuthException at FileBasedIOLT.java:191	
                    Caused by: com.google.api.client.http.HttpResponseException at FileBasedIOLT.java:191	
org.apache.beam.it.gcp.storage.FileBasedIOLT > classMethod FAILED	
    java.lang.RuntimeException at FileBasedIOLT.java:128	
        Caused by: com.google.cloud.storage.StorageException at FileBasedIOLT.java:128	
            Caused by: com.google.auth.oauth2.GoogleAuthException at FileBasedIOLT.java:128	
                Caused by: com.google.api.client.http.HttpResponseException at FileBasedIOLT.java:128

Background

History of failures go back as far as 9 months, most recently related to an error in GCP credentials. Notably, beam_PostCommit_Java_IO_Performance_Tests.yml has an authentication step using static credentials. Searching through the repository we see this google-github-actions/auth@v1 pattern used in only a few places while there are numerous tests and workflows requiring GCP authentication.

Assessment

The test may be improperly configured using incorrect credentials. More importantly, it uses an outdated pattern via static credentials to authenticate to Google cloud.

Recomendation

Remove the use of google-github-actions/auth@v1.

[damondouglas]

#33338 merged, running manually: https://github.com/apache/beam/actions/runs/12262712166

[damondouglas]

#33338 fixed the GCS Performance test which enabled the BigQueryStorageApiStreamingPerformanceTest to run but we see this breaking. https://github.com/apache/beam/actions/runs/12262712166/job/34225821488

org.apache.beam.it.gcp.bigquery.BigQueryStreamingLT > testExactlyOnceStreaming FAILED
    dev.failsafe.FailsafeException at BigQueryStreamingLT.java:374
        Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException at BigQueryStreamingLT.java:374

[damondouglas]

I'm keeping this ticket assigned to me. Catching up on other tasks in my queue and will research what went wrong with testExactlyOnceStreaming. I feel like we are close to fixing this flakyness and thus worth it to invest the time when available.

[damondouglas]

I see that the BigQuery performance test is failing with this message:

Access Denied: Project <external project>: User does not have bigquery.datasets.create permission in project <external project>.

I'm not including what that external project is in this comment but the point is is that it is not apache-beam-testing for which the IAM roles are bound by the GitHub actions runner node service account.

The solution is either:

• Add the GitHub action runner node service account as an IAM member of the BigQuery dataset/project
• Reconfigure the test to use BigQuery in apache-beam-testing

[damondouglas]

@ahmedabu98 found that BigQueryStreamingLT has a hard coded project name. We just need to configure the test to acquire the GCP project using a Gradle property.