[Bug]: PubsubIO/PubsubLiteIO is not using the gcp credential set in the PipelineOption

[ns-ydai]

What happened?

My dataflow job is running in Project A, and it needs to use service account which belongs to Project B to read the Pubsub message from Project B. So in the DataflowPipelineOption, I use options.setGcpCredential to set it to use that service account. But when the PubsubIO.readString() tries to read the message, it is still using the GCP credential when launching the dataflow job, not the one I set in the option.

I tried to dive into the code and found when creating PubsubJsonClient, the option doesn't have the gcpCredential field from the pipeline option. Can anyone help on this please? Thanks.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[liferoad]

@dpcollins-google any clue?

[dpcollins-google]

PubsubIO is replaced by dataflow internally and may not respect this. PubsubLiteIO likely just doesn't support this option.

[sriram23kmm]

@dpcollins-google was there any other options to read from other projects pubsub topic using service credentials..?

[dpcollins-google]

Not a question for me, its a question for dataflow people since PubsubIO is replaced internally. @aaltay may know

[scwhittle]

It does appear that PubsubJsonClient uses the credential here

However as Daniel notes the PubsubIO implementation is overridden for dataflow.
You can use --experiment=enable_custom_pubsub_source to disable the read PubsubIO replacement done with dataflow. However you will not benefit from the features: https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#integration-features

However couldn't you instead grant Service account A viewer/subscriber permissions to read from the topic/sub owned by account B?

[scwhittle]

Please reopen above if granting permission to the service account used by Dataflow is not sufficient to fix the issue.