Fix stuck nexmark streaming #29465
Fix stuck nexmark streaming #29465
Conversation
* Cherry pick two previous commits on migrating gcs client The two commits are merged into one: * Reapply "Replace StorageV1 client with GCS client (apache#28079)" (apache#28721) * added project parameter to apiclient * Initialze storage client with project from pipeline option. --------- Co-authored-by: Bjorn Pedersen <[email protected]>
|
Assigning reviewers. If you would like to opt out of this review, comment R: @robertwb added as fallback since no labels match configuration Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| @@ -126,6 +126,9 @@ public InputStatus pollNext(ReaderOutput<WindowedValue<ValueWithRecordId<T>>> ou | |||
| if (reader != null) { | |||
| emitRecord(reader, output); | |||
| return InputStatus.MORE_AVAILABLE; | |||
| } else if (noMoreSplits()) { | |||
There was a problem hiding this comment.
Just to confirm, does noMoreSplits()=true means there won't be split forever (not just currently)
From https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/connector/source/SourceReader.html#notifyNoMoreSplits-- iiuc it is this case
There was a problem hiding this comment.
That's my understanding yes. And it makes sense given the implementation of isAvailable.
|
Run Flink Runner Nexmark Tests |
|
https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Flink_PR/86/ passed. Thank you! |
The two commits are merged into one:
Reapply "Replace StorageV1 client with GCS client (Replace StorageV1 client with GCS client (take 4) #28079)" (Revert "Replace StorageV1 client with GCS client (#28079)" #28721)
added project parameter to apiclient
Initialze storage client with project from pipeline option.
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.