BigQuery Storage Write API: Callback doesnt return and blocks indefinitely during appendrows

[NarenSi]

      ApiFuture<AppendRowsResponse> future = streamWriter.append(data, offset);
      ApiFutures.addCallback(future, new DataWriter.AppendCompleteCallback(this), MoreExecutors.directExecutor());

When I try to execute this piece of code in a standalone tool, it works fine. But when I try to integrate it with another project the thread gets parked,blocks indefinitely and the program just hangs. Eventually i get the exception Caused by: java.lang.RuntimeException: io.grpc.StatusRuntimeException: CANCELLED: Timeout waiting for DoneCallback. . Could someone please help.

This is the grpc stack trace.
Caused by: io.grpc.StatusRuntimeException: CANCELLED: Timeout waiting for DoneCallback.
at com.google.cloud.bigquery.storage.v1.StreamWriter.waitForDoneCallback(StreamWriter.java:540)
at com.google.cloud.bigquery.storage.v1.StreamWriter.appendLoop(StreamWriter.java:493)
... 3 more

Version used :

google-cloud-bigquerystorage - 2.14.2
google-cloud-bigquery - 2.13.3

[yirutang]

Which thread gets parked? It is the Future callback, or done callback? Could you try a sync mode which is directly call future.get to see if that can succeed?

[NarenSi]

Which thread gets parked? It is the Future callback, or done callback? Could you try a sync mode which is directly call future.get to see if that can succeed?

Thank you for your reply. The done callback thread is the one that gets parked . I will try to directly call future.get().Actually I am facing a similar issue when I am trying to create a WriteStream. The thread just gets parked and the program hangs.
WriteStream writeStream = bigQueryWriteClient.createWriteStream(createWriteStreamRequest);
The weird behaviour is that these commands work fine in the main thread but they start to hang when i try to execute from another thread.

This is a sample thread dump :-

"admin.bq_tar-0" #274 prio=5 os_prio=31 tid=0x00007fb910445800 nid=0x2cd03 waiting on condition [0x000000030d554000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c336c6c8> (a com.google.api.gax.retrying.CallbackChainRetryingFuture) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:557) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:240) at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1379) at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:53) at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112) at com.google.cloud.bigquery.storage.v1.BigQueryWriteClient.createWriteStream(BigQueryWriteClient.java:239) at com.striim.bigquery.writeAPI.writeAPI.writeDatatoBQ(writeAPI.java:87) - locked <0x00000006c336c778> (a com.writeAPI.writeAPI)

[yirutang]

Maybe try set your server's default executor or something? Most likely no executor is executing the threads:
https://stackoverflow.com/questions/24241335/waiting-at-sun-misc-unsafe-parknative-method

[NarenSi]

Maybe try set your server's default executor or something? Most likely no executor is executing the threads: https://stackoverflow.com/questions/24241335/waiting-at-sun-misc-unsafe-parknative-method

Apologies for the late reply , but yes I tried to set a default executor and the issue still seems to persist.

[NarenSi]

The thread actually keeps waiting for the a response from the object 'CallbackChainRetryingFuture' and the program hangs from then on as it does not receive any response.