fix(sink): handle Kinesis PutRecords partial success and throttle #17983
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9 tasks
tabVersion
approved these changes
Aug 11, 2024
xiangjinwu
approved these changes
Aug 12, 2024
Comment on lines
+281
to
+282
| // PutRecords doesn’t guarantee the ordering of records. If you need to read records in the same | ||
| // order they are written to the stream, use PutRecord instead of PutRecords, and write to the same shard. |
There was a problem hiding this comment.
Shall we acknowledge this limitation in doc as there is currently no user option to opt for ordering rather than batching?
| } | ||
| if total_payload_size + *size < MAX_TOTAL_RECORD_PAYLOAD_SIZE { | ||
| total_payload_size += *size; | ||
| records.push(record.clone()); |
There was a problem hiding this comment.
The need of clone here feels like a API design flaw in the AWS auto-generated SDK. I don't think we can do anything but just point it out.
Impact:
Every single message written out is cloned at least once.
Root cause:
AWS SDK takes the ownership away but does not give it back for failure retry.
xiangjinwu
reviewed
Aug 13, 2024
src/connector/src/sink/kinesis.rs
Outdated
Comment on lines
344
to
350
| #[expect(rw::format_error)] | ||
| return Err(SinkError::Kinesis(anyhow!( | ||
| "failed to send records. sent {} out of {}, last err: {:?}", | ||
| start_idx, | ||
| total_count, | ||
| e.as_report() | ||
| ))); |
There was a problem hiding this comment.
When wrapping an error and returning to caller (rather than handling/consuming the error as warn! below), use context:
return Err(
SinkError::Kinesis(
anyhow!(e).context(format!("failed to send records. sent {start_idx} out of {total_count}, last err:"))
)
);
xiangjinwu
added
the
need-cherry-pick-release-1.10
wenym1
added a commit
that referenced
this pull request
Aug 14, 2024
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Checklist
./risedev check(or alias,./risedev c)Documentation
Release note
Changes on the doc of kinesis sink:
In kinesis sink, we use PutRecords to send multiple records in a batch to achieve higher throughput. Due to the limitation of kinesis, records might be out of order when using such API. Nevertheless, in the current kinesis sink implementation, we ensure at-least-once delivery semantic, and eventual consistency.