The PostCommit Java SingleStoreIO IT job is flaky

[github-actions]

The PostCommit Java SingleStoreIO IT is failing over 50% of the time
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_SingleStoreIO_IT.yml?query=is%3Afailure+branch%3Amaster to see the logs.

[volatilemolotov]

.take-issue

[github-actions]

Reopening since the workflow is still flaky

[github-actions]

Reopening since the workflow is still flaky

[volatilemolotov]

@Abacn History shows that at some point the key for SA expired. It seems rotated but now it shows some permission issues.

Has anyone rotated the key and touched permissions?

[Abacn]

This job recently has multiple auth issues.

FIrstly it was " ('invalid_grant: Invalid JWT Signature.', 'error': 'invalid_grant', 'error_description': 'Invalid JWT Signature.')" seen in various jobs using secret.GCP_SA_KEY to authenticate. This is intermittent

Then I did #31881 removed the use of secret.GCP_SA_KEY for affected jobs, and use k8s clusters' default credential, then most jobs back green, and there is still Kafka and Singlestore tests failing with

User "117793688972024512976" cannot create resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "beam-performancetests-kafka-io-9954999796": No policy matched.

[Abacn]

fixed by adding corresponding permission to new SA