Producer Version 0.15.X is broken on streams > 1000 shards

[vmwarealthomas]

We have no problem using the Version 0.15.X when writing to streams < 1000 shards but once we try doing large streams it gives this error of unable to update shard maps:

Shard map update for stream "<stream_name>" (arn: "arn:.......:stream/stream_name failed. Code: InvalidArgumentException Message: NextToken and StreamARN cannot be provided together.; retrying in 1000 ms

Encountered AWSError 
HTTP response code: 400
Resolved remote host IP address: <ip>
Request ID: d0ac795e-c6ae-8664-8cc4-086e20aaa2ab
Exception name: InvalidArgumentException
Error message: NextToken and StreamARN cannot be provided together.
6 response headers:
connection : close
content-length : 102
content-type : application/x-amz-json-1.1
date : Thu, 16 Feb 2023 17:59:12 GMT

This is the same problem as seen in this issue in the botocore library: boto/botocore#2009.

Since the 0.15 release notes have this line KPL 0.15.0 now incorporates StreamARN in the Kinesis requests, such as PutRecords and ListShards, to take advantage of Kinesis Data Streams (KDS) enhanced availability I think the underlying ListShards call is broken for large streams.

[zengyu714]

Hi @vmwarealthomas, we appreciate you letting us know about this fix, and we'll get to work on it this week.