Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the stream arn parameter when the next token is present #482

Merged
merged 2 commits into from
Feb 17, 2023

Conversation

zengyu714
Copy link
Contributor

@zengyu714 zengyu714 commented Feb 17, 2023

Issue #, if available:

Description of changes:

  • Remove the StreamArn parameter when ListShards fetches the next page using NextToken
  • Testing*
    Tested locally with the code snippet
void ShardMap::list_shards(const Aws::String& next_token) {
  Aws::Kinesis::Model::ListShardsRequest req;
  req.SetMaxResults(1);

  if (!next_token.empty()) {
    LOG(info) << "[DEBUG] Updating shard map for stream \"" << stream_ << "\", the max result has been set to 1";
    req.SetNextToken(next_token);
  } else {
...
  }

and verified that the next page of shards can be successfully loaded:

[kpl-daemon-0003] INFO com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2023-02-17 10:28:06.754721] [0x00007c48][0x0000700003987000] [info] [shard_map.cc:105] [DEBUG] Updating shard map for stream "kpltest", the max result has been set to 1
[pool-1-thread-1] INFO com.amazonaws.services.kinesis.producer.sample.SampleProducer - Put 5999 of 200000 so far (3.00 %), 2000 have completed (1.00 %)
[pool-1-thread-1] INFO com.amazonaws.services.kinesis.producer.sample.SampleProducer - Oldest future as of now in millis is 2000
[pool-1-thread-1] INFO com.amazonaws.services.kinesis.producer.sample.SampleProducer - Put 7998 of 200000 so far (4.00 %), 4500 have completed (2.25 %)
[pool-1-thread-1] INFO com.amazonaws.services.kinesis.producer.sample.SampleProducer - Oldest future as of now in millis is 1749
[kpl-daemon-0003] INFO com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2023-02-17 10:28:08.309224] [0x00007c48][0x0000700003b93000] [info] [shard_map.cc:105] [DEBUG] Updating shard map for stream "kpltest", the max result has been set to 1
[pool-1-thread-1] INFO com.amazonaws.services.kinesis.producer.sample.SampleProducer - Put 9998 of 200000 so far (5.00 %), 7000 have completed (3.50 %)
[pool-1-thread-1] INFO com.amazonaws.services.kinesis.producer.sample.SampleProducer - Oldest future as of now in millis is 1499

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Copy link
Contributor

@aakkem aakkem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zengyu714 zengyu714 merged commit 323c020 into awslabs:master Feb 17, 2023
@zengyu714 zengyu714 deleted the list-shards branch February 17, 2023 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants