[Flake]: Spark batch flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle

[kennknowles]

What happened?

This test has been historically flaky, and flaked here:

• https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/10934/ (https://scans.gradle.com/s/kqx3lxltwtorq#failure)

Issue Priority

Priority: 1

Issue Component

Component: runner-spark

[kennknowles]

Forked from #19814

[kennknowles]

Unfortunately not so easy to reproduce. This passed:

for i in `seq 1 100` ; do (
  echo "RUN $i"
  ./gradlew :runners:spark:3:cleanTest --quiet
  ./gradlew :runners:spark:3:validatesRunnerBatch --quiet --tests ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement
) 2>/dev/null
done

Either it depends on resource constraints on Jenkins due to concurrency or it flakes less often.

[kennknowles]

Leaving open but opportunistically moving to a bug I can repro. This one is in the unfortunate state where I know it is real so I can't close it but I cannot repro.

[manuzhang]

shall we downgrade priority for such flaky tests?

[damccorm]

I think this is fixed. If not, it should get auto-flagged by our tooling anyways, so this should be safe to close