[Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process

[gouthammylu]

What happened?

We are creating a data pipeline to read the CSV files and write records to Bigquery and Spanner tables. In our pipeline we are connecting to 3 spanner databases, based on the file name we push data to one of the spanner database tables. The dataflow job was working fine till version 2.48.0. when we upgrade to 2.49.0 or 2.50.0 Dataflow job is in running state for a long time but no data hasbeen processed. We have seen below exception in the Diagnostics section.

Logs:

Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process
at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)
at [email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at app//com.google.common.util.concurrent.OverflowAvoidingLockSupport.parkNanos(OverflowAvoidingLockSupport.java:36)
at app//com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:452)
at app//com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:121)
at app//com.google.cloud.spanner.SessionPool$WaiterFuture.pollUninterruptiblyWithTimeout(SessionPool.java:1591)
at app//com.google.cloud.spanner.SessionPool$WaiterFuture.get(SessionPool.java:1568)
at app//com.google.cloud.spanner.SessionPool$WaiterFuture.get(SessionPool.java:1545)
at app//com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:66)
at app//com.google.cloud.spanner.SessionPool$PooledSessionFuture.get(SessionPool.java:1302)
at app//com.google.cloud.spanner.SessionPool$PooledSessionFuture.lambda$readOnlyTransaction$4(SessionPool.java:1195)
at app//com.google.cloud.spanner.SessionPool$PooledSessionFuture$$Lambda$372/0x0000000800882440.apply(Unknown Source)
at app//com.google.cloud.spanner.SessionPool$AutoClosingReadContext.getReadContextDelegate(SessionPool.java:234)
at app//com.google.cloud.spanner.SessionPool$AutoClosingReadContext$6.load(SessionPool.java:472)
at app//com.google.cloud.spanner.SessionPool$CachedResultSetSupplier.get(SessionPool.java:153)
at app//com.google.cloud.spanner.SessionPool$CachedResultSetSupplier.get(SessionPool.java:141)
at app//com.google.cloud.spanner.ForwardingResultSet.next(ForwardingResultSet.java:54)
at app//com.google.cloud.spanner.SessionPool$AutoClosingReadContext$1.internalNext(SessionPool.java:273)
at app//com.google.cloud.spanner.SessionPool$AutoClosingReadContext$1.next(SessionPool.java:253)
at app//org.apache.beam.sdk.io.gcp.spanner.ReadSpannerSchema.processElement(ReadSpannerSchema.java:95)
at app//org.apache.beam.sdk.io.gcp.spanner.ReadSpannerSchema$DoFnInvoker.invokeProcessElement(Unknown Source)

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Our SpannerIO load tests shows spanner write is functioning without issue, though metrics shows Spanner write could be slower than bigquery write and optimization is possible.

Please open a support ticket to GCP Dataflow; Job logs are important for this kind of investigation.

[gouthammylu]

How to open a support ticket to GCP Dataflow ?

[gouthammylu]

Why reading information schema is taking lot of time? even after two hours we were getting the same exception.

[liferoad]

Please check https://cloud.google.com/dataflow/docs/support/getting-support and feel free to mention this GitHub issue. Thanks.