Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BigtableSession is never closed by Reader causing "ManagedChannel allocation site" exceptions #2658

Open
pdex opened this issue Oct 16, 2020 · 1 comment · Fixed by #2782
Open

BigtableSession is never closed by Reader causing "ManagedChannel allocation site" exceptions #2658

pdex opened this issue Oct 16, 2020 · 1 comment · Fixed by #2782
Assignees
@dmitry-fa
Labels
api: bigtable Issues related to the googleapis/java-bigtable-hbase API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@pdex
Copy link

pdex commented Oct 16, 2020

This isn't a major issue, but it is causing exceptions to be logged that may confuse someone who isn't familiar with grpc-java/netty.

Code example

The BigtableSession created here is never closed.

Stack trace

java.lang.RuntimeException: ManagedChannel allocation site
	at com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
	at com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
	at com.google.bigtable.repackaged.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
	at com.google.bigtable.repackaged.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:517)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession.createNettyChannel(BigtableSession.java:630)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.BigtableSession.<init>(BigtableSession.java:367)
	at com.google.cloud.bigtable.beam.CloudBigtableIO$Reader.initializeScanner(CloudBigtableIO.java:583)
	at com.google.cloud.bigtable.beam.CloudBigtableIO$Reader.start(CloudBigtableIO.java:573)
	at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.start(WorkerCustomSources.java:607)
	at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:361)
	at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:194)
	at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
	at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
	at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:417)
	at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:386)
	at org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:311)
	at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
	at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
	at org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
@product-auto-label product-auto-label bot added the api: bigtable Issues related to the googleapis/java-bigtable-hbase API. label Oct 16, 2020
@yoshi-automation yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels Oct 17, 2020
@kolea2 kolea2 added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. and removed 🚨 This issue needs some love. triage me I really want to be triaged. labels Oct 21, 2020
@dmitry-fa dmitry-fa self-assigned this Dec 2, 2020
@dmitry-fa dmitry-fa mentioned this issue Dec 28, 2020
Merged
@baeminbo baeminbo mentioned this issue Mar 31, 2021
Closed
@yoshi-automation yoshi-automation added the 🚨 This issue needs some love. label Apr 14, 2021
@pdex
Copy link
Author

pdex commented Apr 14, 2021 •
edited
Loading

Just to bring everyone up to speed in one place #2782 was reverted by #2873
citing this comment in #2871

#2913 is incorrect there is not a leak, the underlying grpc-netty library is informing the user that they haven't properly closed a channel. This isn't a big issue for Dataflow jobs.

The ideal solution will close the session when the worker shuts down. But again, if that doesn't happen you can ignore this log message in your Dataflow job.

@crwilcox crwilcox added the priority: p3 Desirable enhancement or fix. May not be included in next release. label Jun 17, 2021
@kolea2 kolea2 removed the priority: p2 Moderately-important priority. Fix may not be included in next release. label Jun 30, 2021
@yoshi-automation yoshi-automation removed the 🚨 This issue needs some love. label Jun 30, 2021
@yoshi-automation yoshi-automation added 🚨 This issue needs some love. and removed 🚨 This issue needs some love. labels Dec 27, 2021
@damccorm damccorm mentioned this issue Jun 4, 2022
Open
@yoshi-automation yoshi-automation added 🚨 This issue needs some love. and removed 🚨 This issue needs some love. labels Jun 25, 2022
baeminbo added a commit to baeminbo/java-bigtable-hbase that referenced this issue Dec 12, 2022
@baeminbo
Close Connection to prevent gRPC leak error messages
e14f81d 
* Reader.close(): fixes googleapis#2658 where original fix googleapis#2782 was rolled back by googleapis#2871 and googleapis#2873. As we don’t use “session” here anymore, I believe we should close “connection” here.
* AbstractCloudBigtableTableDoFn.tearDown(): If an exception happened in ProcessElement, FinishBundle is not executed. We should clean up resources in TearDown. You can see the similar code with BigQueryIO apache/beam#14949.
@baeminbo baeminbo mentioned this issue Dec 12, 2022
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmitry-fa dmitry-fa

Labels
api: bigtable Issues related to the googleapis/java-bigtable-hbase API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
6 participants
@pdex @crwilcox @igorbernstein2 @yoshi-automation @kolea2 @dmitry-fa