Parameterize GoogleCloudStorage provider in GcsUtil to unblock gcs-co… #33368
Conversation
| GoogleCloudStorage get( | ||
| GoogleCloudStorageOptions options, | ||
| Storage storage, | ||
| Credentials credentials, | ||
| HttpRequestInitializer httpRequestInitializer); |
There was a problem hiding this comment.
These 4 params should cover both the 2.x constructor and the 3.x Builder
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
cc @Abacn - wdyt of this workaround for gcs-connector 3.x? |
|
Hi, thanks for the investigation. Is the builder constructor also supported on 2.x ? If so we can just change to use it in all case and no need extra options exposed to user |
Unfortunately it isn't :/ There is no way to construct a |
thanks @Abacn ! should I update CHANGES.md for the |
|
java precommit tineout for several rerun. It's passing on HEAD, could you please taking a look if it is related to this change? |
Rationale:
I would like to use gcs-connector 3.x, which supports the new Parquet VectorIO feature. However, gcs-connector 3.x also drops Java 8 and targets Java 11, which blocks us from upgrading it directly in Beam, since Beam is still targeting 8 (see #31678).
Additionally, as a Beam user, I can't just upgrade gcs-connector on my end, due to breaking changes in how
GoogleCloudStorageImplis instantiated: in 2.x it has public constructors, but in 3.x it drops the public constructors and enforces a Builder pattern.Therefore, when running on gcs-connector 3.x, my pipeline throws a NoSuchMethodError from
org.apache.beam.sdk.extensions.gcp.util.GcsUtilwhen it tries to invoke the 2.x constructor: https://github.com/apache/beam/blob/v2.61.0/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java#L727This PR adds a pipeline option for a GoogleCloudStorage Provider, so that users who want to use gcs-connector 3.x can be unblocked from doing so. It defaults to invoking the gcs-connector 2.x public constructor, but 3.x users can override it to use the Builder.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.