Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: Access GCS bucket with requester pays #30747

Open
1 of 16 tasks
LeoCBS opened this issue Mar 26, 2024 · 1 comment · May be fixed by #33236
Open
1 of 16 tasks

[Feature Request]: Access GCS bucket with requester pays #30747

LeoCBS opened this issue Mar 26, 2024 · 1 comment · May be fixed by #33236

Comments

@LeoCBS
Copy link

LeoCBS commented Mar 26, 2024

What would you like to happen?

Hello, is there a way to pass the requester pay project using textio.Read("gs://my-bucket")?

Apache beam is returning this error when i try accessing a bucket with this property active:

4: SDF.SplitAndSizeRestrictions[textio.readFn] UID:4 Out:[2] 3: PCollection[ne5_splitnsized] Out:[2] 2: DataSink[S[ee5_splitnsize_i0@localhost:43511]] Coder:W;cwv_ne5_splitnsized<KV;ce5kv_ele_rest_size_lp<KV;ce5kv_ele_rest_lp<LP;c3_lp<R[fileio.Readab leFile]>,KV;c7<offsetrange.Restriction[offsetrange.Restriction;c4];c5,bool;c6>>,double;ce5size>>!GWC caused by: DoFn[UID:10, PID:e3, Name: [github.com/apache/beam/sdks/v2/go/pkg/beam/io/fileio.matchFn](http://github.com/apache/beam/sdks/v2/go/pkg/beam/io/fileio.matchFn)] failed: googleapi: Error 400: Bucket is a requester pays bucket but no user project provided., required source=/go/pkg/mod/[github.com/apache/beam/sdks/v2@v2](http://github.com/apache/beam/sdks/v2@v2). 54.0/go/pkg/beam/core/runtime/harness/harness.go:492 time=2024-03-25T11:58:07.084Z worker.ID=job-001[go-job-1-1711367886775655910]_go worker.endpoint =localhost:43511

Thanks

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@LeoCBS
Copy link
Author

LeoCBS commented Nov 26, 2024

.take-issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant