BigQueryWriteClient in DatasetServiceImpl is not closed, which causes "ManagedChannel allocation site" exceptions

[damccorm]

BigQueryWriteClient in DatasetServiceImpl (added at https://github.com/apache/beam/pull/14309)) is not closed.  This causes the error logs  in gRPC orphan channel clean up. See "bigquery_grpc.log" in attachments which is extracted from GCP Dataflow. I don't think this issue affect pipeline runs except the error logs, but could you take a look at that?

A similar issue is reported for CloudBigtableIO at googleapis/java-bigtable-hbase#2658

 

Imported from Jira BEAM-12356. Original Jira may contain additional context.
Reported by: baeminbo.

[databius]

May I know if there is an update for this? Thanks

[TheNeuralBit]

@databius (and anyone else who encounters this): Could you share stacktraces where this happened and the Beam verision you were using? That will help us pinpoint the client that is not being cleaned up.

FYI @johnjcasey - this comes up a lot. Some of the more recent stacktraces posted in BEAM-12356 may not have been tracked down.

[Tera]

Hi all

I think that I got this error on apache beam 2.41, java 11

this is my stack trace, it happens during reading phase

oct 05, 2022 4:33:02 PM io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
GRAVE: *~*~*~ Channel ManagedChannelImpl{logId=13, target=bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
	at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
	at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
	at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
	at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630)
	at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:388)
	at com.google.api.gax.grpc.ChannelPool.<init>(ChannelPool.java:105)
	at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:83)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:236)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:230)
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:201)
	at com.google.cloud.bigquery.storage.v1.stub.GrpcBigQueryWriteStub.create(GrpcBigQueryWriteStub.java:132)
	at com.google.cloud.bigquery.storage.v1.stub.BigQueryWriteStubSettings.createStub(BigQueryWriteStubSettings.java:144)
	at com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.<init>(BigQueryWriteClient.java:130)
	at com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.create(BigQueryWriteClient.java:111)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.newBigQueryWriteClient(BigQueryServicesImpl.java:1482)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$800(BigQueryServicesImpl.java:151)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:595)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.<init>(BigQueryServicesImpl.java:537)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.getDatasetService(BigQueryServicesImpl.java:190)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.getTargetTable(BigQueryStorageTableSource.java:188)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.getEstimatedSizeBytes(BigQueryStorageTableSource.java:133)
	at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:215)
	at org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:88)
	at org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:80)
	at org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:161)
	at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
	at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:323)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)

hope to have reported all valuable info

[TheNeuralBit]

Thanks @Tera

#23873 should address this instance of the issue.