Google batch timeout - add google.batch.storage.httpRequestTimeout configuration option

[Shahbaz-mahmood123]

New feature

Currently, when running pipelines in Google batch larger (>1gb) intermediary files not get transferred over to the outdir.

The current default timeout is 20 Seconds and the fix seems to be to increase the timeout: googleapis/google-cloud-java#2243 (comment)

I do not see a configuration in Nextflow to extend the current timeout: google.batch.storage.httpRequestTimeout.

Spoke with Jordi and using Fusion will not help in this case either since its Nextflow doing the copying to the publishDir.

java.io.IOException: All 0 retries failed. Waited a total of 0 ms between attempts
  at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.asIoException(CloudStorageFileSystemProvider.java:1052)
  at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.copy(CloudStorageFileSystemProvider.java:691)
  at nextflow.file.FileHelper.copyPath(FileHelper.groovy:959)
  at nextflow.processor.PublishDir.processFileImpl(PublishDir.groovy:466)
  at nextflow.processor.PublishDir.processFile(PublishDir.groovy:377)
  at nextflow.processor.PublishDir.safeProcessFile(PublishDir.groovy:359)
  at jdk.internal.reflect.GeneratedMethodAccessor524.invoke(Unknown Source)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
  at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
  at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1258)
  at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
  at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1036)
  at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1019)
  at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
  at nextflow.processor.PublishDir$_apply1_closure1.doCall(PublishDir.groovy:332)
  at nextflow.processor.PublishDir$_apply1_closure1.call(PublishDir.groovy)
  at groovy.lang.Closure.run(Closure.java:493)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.google.cloud.storage.StorageException: All 0 retries failed. Waited a total of 0 ms between attempts
  at com.google.cloud.storage.contrib.nio.CloudStorageRetryHandler.handleRetryForStorageException(CloudStorageRetryHandler.java:135)
  at com.google.cloud.storage.contrib.nio.CloudStorageRetryHandler.handleStorageException(CloudStorageRetryHandler.java:115)
  at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.copy(CloudStorageFileSystemProvider.java:688)
  ... 22 common frames omitted
Caused by: com.google.cloud.storage.StorageException: Read timed out
  at com.google.cloud.storage.StorageException.translate(StorageException.java:118)
  at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:287)
  at com.google.cloud.storage.spi.v1.HttpStorageRpc.rewrite(HttpStorageRpc.java:1123)
  at com.google.cloud.storage.spi.v1.HttpStorageRpc.openRewrite(HttpStorageRpc.java:1053)
  at com.google.cloud.storage.StorageImpl.lambda$copy$22(StorageImpl.java:515)
  at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
  at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
  at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
  at com.google.cloud.storage.Retrying.run(Retrying.java:54)
  at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1406)
  at com.google.cloud.storage.StorageImpl.copy(StorageImpl.java:513)
  at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.copy(CloudStorageFileSystemProvider.java:682)
  ... 22 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
  at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
  at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
  at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
  at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
  at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
  at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
  at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
  at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
  at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
  at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
  at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
  at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
  at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
  at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:824)
  at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759)
  at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1688)
  at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
  at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529)
  at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308)
  at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)
  at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152)
  at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
  at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
  at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:525)
  at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:466)
  at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:576)
  at com.google.cloud.storage.spi.v1.HttpStorageRpc.rewrite(HttpStorageRpc.java:1113)
  ... 31 common frames omitted

[bentsherman]

Okay, it looks like the factory method that we use for GCS paths can also accept a StorageOptions: https://cloud.google.com/java/docs/reference/google-cloud-nio/0.124.8/com.google.cloud.storage.contrib.nio.CloudStorageFileSystem?hl=en#com_google_cloud_storage_contrib_nio_CloudStorageFileSystem_forBucket_java_lang_String_com_google_cloud_storage_contrib_nio_CloudStorageConfiguration_

I will draft a PR shortly.