Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate source and streaming rate limit #16466

Closed
2 tasks
kwannoel opened this issue Apr 24, 2024 · 3 comments · Fixed by #17796
Closed
2 tasks

Separate source and streaming rate limit #16466

kwannoel opened this issue Apr 24, 2024 · 3 comments · Fixed by #17796
Assignees

Comments

@kwannoel
Copy link
Contributor

kwannoel commented Apr 24, 2024

source_rate_limit will limit the upstream side of the downstream executors.
streaming_rate_limit will limit the backfill rate.

  • Update the session variable setting
  • Update the parser to handle the with options.
@github-actions github-actions bot added this to the release-1.9 milestone Apr 24, 2024
@st1page
Copy link
Contributor

st1page commented Apr 24, 2024

To be more precise, source_rate_limit will control the source rate limit with connections for this source/table in relation to the external system's source rate limit.
Meanwhile, streaming_rate_limit will control the rate limit between the MV/sink being created and its upstream, which also includes the situation when an MV is created on a source. In this case, I believe that a minimum value should be taken between streaming_rate_limit and the source_rate_limit. This is because when the Kafka source's read.offset=oldest, the scenario is the same as with backfill.

@fuyufjh fuyufjh removed this from the release-1.10 milestone Jul 10, 2024
@fuyufjh
Copy link
Member

fuyufjh commented Jul 10, 2024

Either LGTM

@kwannoel
Copy link
Contributor Author

Separate into backfill_rate_limit and streaming_rate_limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants