Flink streaming flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful #19814

damccorm · 2022-06-04T13:38:30Z

Temporary fail from Jenkins:


java.lang.AssertionError: Function should have been torn down after exception
Expected: is <TEARDOWN>

    but: was <START_BUNDLE>
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.apache.beam.sdk.transforms.ParDoLifecycleTest.lambda$validate$0(ParDoLifecycleTest.java:266)
	at
java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707)
	at org.apache.beam.sdk.transforms.ParDoLifecycleTest.validate(ParDoLifecycleTest.java:264)
	at
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful(ParDoLifecycleTest.java:253)
	at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319)
	at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:349)
	at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:314)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:312)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:396)
	at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at
sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at
java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at
com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
	at
sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at
java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:175)
	at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:157)
	at
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at
java.lang.Thread.run(Thread.java:748)

Standard Error

Aug 26, 2019 2:47:16 PM org.apache.beam.runners.direct.ParDoEvaluatorFactory
createParDoEvaluator
SEVERE: Exception encountered while cleaning up in ParDo evaluator construction
org.apache.beam.sdk.util.UserCodeException:
java.lang.AssertionError: Expected to have a processing method throw an exception
	at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:34)
	at
org.apache.beam.sdk.transforms.ParDoLifecycleTest$ExceptionThrowingFn$DoFnInvoker.invokeTeardown(Unknown
Source)
	at org.apache.beam.runners.direct.DoFnLifecycleManager$TeardownRemovedFnListener.onRemoval(DoFnLifecycleManager.java:113)
	at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1809)
	at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3462)
	at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3438)
	at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3072)
	at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache.remove(LocalCache.java:4236)
	at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4899)
	at
org.apache.beam.runners.direct.DoFnLifecycleManager.remove(DoFnLifecycleManager.java:66)
	at org.apache.beam.runners.direct.ParDoEvaluatorFactory.createParDoEvaluator(ParDoEvaluatorFactory.java:168)
	at
org.apache.beam.runners.direct.ParDoEvaluatorFactory.createEvaluator(ParDoEvaluatorFactory.java:121)
	at
org.apache.beam.runners.direct.ParDoEvaluatorFactory.forApplication(ParDoEvaluatorFactory.java:79)
	at
org.apache.beam.runners.direct.TransformEvaluatorRegistry.forApplication(TransformEvaluatorRegistry.java:169)
	at
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:117)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused
by: java.lang.AssertionError: Expected to have a processing method throw an exception
	at org.junit.Assert.fail(Assert.java:89)
	at
org.apache.beam.sdk.transforms.ParDoLifecycleTest$ExceptionThrowingFn.after(ParDoLifecycleTest.java:398)

Imported from Jira BEAM-8101. Original Jira may contain additional context.
Reported by: janl.

The text was updated successfully, but these errors were encountered:

TheNeuralBit · 2022-06-29T20:13:23Z

@adude3141 it looks like you had the most recent updates on BEAM-8101. Do you have any thoughts on a path forward?

kennknowles · 2022-10-18T18:38:24Z

Did not flake in 100 runs on DirectRunner:

for i in `seq 1 100` ; do (
  echo "RUN $i"
  ./gradlew :runners:direct-java:cleanTest --quiet
  ./gradlew :runners:direct-java:needsRunnerTests --quiet --tests ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful
) 2>/dev/null
done

kennknowles · 2022-10-18T19:02:52Z

Did not flake in 100 runs on Flink batch:

for i in `seq 1 100` ; do (
  echo "RUN $i"
  ./gradlew :runners:flink:1.15:cleanTest --quiet
  ./gradlew :runners:flink:1.15:validatesRunnerBatch --quiet --tests ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful
) 2>/dev/null
done

kennknowles · 2022-10-18T19:03:00Z

We do see recent flakes:

kennknowles · 2022-10-18T19:14:27Z

Notably the flakes only occur in the stateful variant.

kennknowles · 2022-10-18T19:16:55Z

We do see recent flakes in Spark batch:

https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/10934/testReport/org.apache.beam.sdk.transforms/ParDoLifecycleTest/testTeardownCalledAfterExceptionInProcessElement_4/

These may be the same root cause in the StatefulDoFnRunner but may be something else entirely having to do with exception handling in the runner. I will fork the tickets.

kennknowles · 2022-10-18T19:18:09Z

Actually the Spark flake is not in a stateful variant.

kennknowles · 2022-10-18T19:24:03Z

Despite recent flakes in Jenkins, this succeeded:

for i in `seq 1 100` ; do (
  echo "RUN $i"
  ./gradlew :runners:flink:1.15:cleanTest --quiet
  ./gradlew :runners:flink:1.15:validatesRunnerStreaming --quiet --tests ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful
) 2>/dev/null
done

From the logs, it seems clear that the state of the chain moves to failed, and presumably it is taken down in a race condition with calling teardown.

[CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at ParDo(ExceptionThrowing)/ParMultiDo(ExceptionThrowing)) -> FlatMap (FlatMap at ParDo(ExceptionThrowing)/ParMultiDo(ExceptionThrowing).output) (1/2)#0] WARN org.apache.flink.runtime.taskmanager.Task - CHAIN DataSource (at Create.Values/Read(CreateSource) (org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat)) -> FlatMap (FlatMap at ParDo(ExceptionThrowing)/ParMultiDo(ExceptionThrowing)) -> FlatMap (FlatMap at ParDo(ExceptionThrowing)/ParMultiDo(ExceptionThrowing).output) (1/2)#0 (5f615db581224acfcac1a918f0fa2b1c) switched from RUNNING to FAILED with failure cause: org.apache.beam.sdk.util.UserCodeException: java.lang.Exception: Hasn't yet thrown
	at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
	at org.apache.beam.sdk.transforms.ParDoLifecycleTest$ExceptionThrowingFn$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:211)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:188)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.Exception: Hasn't yet thrown
	at org.apache.beam.sdk.transforms.ParDoLifecycleTest$ExceptionThrowingFn.throwIfNecessary(ParDoLifecycleTest.java:453)
	at org.apache.beam.sdk.transforms.ParDoLifecycleTest$ExceptionThrowingFn.perElement(ParDoLifecycleTest.java:433)

kennknowles · 2022-10-18T21:55:45Z

Leaving open but opportunistically moving to a bug I can repro. This one is in the unfortunate state where I know it is real so I can't close it but I cannot repro.

damccorm · 2024-10-31T14:10:47Z

I think this is fixed. If not, it should get auto-flagged by our tooling anyways, so this should be safe to close

damccorm added bug flake P1 runner-direct labels Jun 4, 2022

damccorm added direct runners flink spark and removed runner-direct labels Jun 16, 2022

damccorm added failing test java labels Oct 5, 2022

kennknowles self-assigned this Oct 17, 2022

kennknowles mentioned this issue Oct 18, 2022

Java Validates DirectRunner: testTeardownCalledAfterExceptionInFinishBundleStateful flaky #19812

Closed

kennknowles changed the title ~~Flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful for Direct, Spark, Flink~~ Flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful for Spark, Flink Oct 18, 2022

kennknowles mentioned this issue Oct 18, 2022

[Flake]: Spark batch flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle #23709

Closed

kennknowles removed their assignment Oct 18, 2022

kennknowles mentioned this issue Oct 18, 2022

ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful is flaky in Java ValidatesRunner Flink suite. #21266

Closed

kennknowles added only on Jenkins and removed spark labels Oct 19, 2022

Abacn mentioned this issue Jan 23, 2024

[flink] Fix watermark generation for empty UnboundedSource #30066

Merged

3 tasks

Abacn mentioned this issue Mar 8, 2024

[DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch #30569

Closed

3 tasks

damccorm closed this as completed Oct 31, 2024

github-actions bot added this to the 2.61.0 Release milestone Oct 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flink streaming flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful #19814

Flink streaming flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful #19814

damccorm commented Jun 4, 2022

TheNeuralBit commented Jun 29, 2022

kennknowles commented Oct 18, 2022 •

edited

Loading

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022 •

edited

Loading

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022

damccorm commented Oct 31, 2024

Flink streaming flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful #19814

Flink streaming flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful #19814

Comments

damccorm commented Jun 4, 2022

TheNeuralBit commented Jun 29, 2022

kennknowles commented Oct 18, 2022 • edited Loading

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022 • edited Loading

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022

kennknowles commented Oct 18, 2022

damccorm commented Oct 31, 2024

kennknowles commented Oct 18, 2022 •

edited

Loading

kennknowles commented Oct 18, 2022 •

edited

Loading