Tracking: Critical Performance & Stability Issues

[fuyufjh]

This is a tracking issue for the critical bugs found during manual, performance or longevity tests, including severe performance downgrading, unlimited resource usage, deadlocks, crashes, etc.

As we know that stability & reliability is one of the most important objectives of this quarter, so let's take them as the first priority.

Please feel free to append your issues to this list.

Stability:

• bug: barrier_inflight_latency is too high and keeps growing  #6571
• nexmark q5 long running OOM due to unawareness of container memory limit #6615
• streaming: frequent timeout in end-to-end tests #6617
• drop materialized view command got stuck  #6639
• bug: hash agg state fetching panics at vnode should not be accessed in longevity test #6642
• creation of materialise view stuck with "table is in creating procedure" error #6684
• bug: recovery failed due to panic in local barrier manager #6715
• indeterministic result from ch benchmark batch query q4 #6875
• frontend: batch runner panics at "Receiver should always exist!" #6883
• longevity panic: unwrap() in source_executor.rs #6931
• bug(streaming): memory leaks in HashJoinExecutor #6942
• bug: panicked at 'no block iter' #6972
• batch: lookup join accesses invalid epoch #6979
• Failed in get: Hummock error: ObjectStore failed with IO error  #7002
• Risingwave is not able to parse the some INTERVAL query #7059
• S3 csv source lost data #7063
• nexmark_chaos_q102_multiple: mem-table operation conflicts #7103
• bug: range scan datum deserialization panicked at 'assertion failed: self.remaining() >= dst.len()' #7158
• bug: hummock wait for current epoch panicked at current epoch can't read, because the epoch in storage is not updated #7169
• bug: hummock uploader assertion failure when seal_epoch #7208
• bug: hummock error NoSuchKey #7232
• etcdserver: request is too large and grpc: received message larger than max #7315
• Compute node will restart frequently when running nexmark q16 #7394
• frontend: Incorrect use of block_on() on plan_fragmenter  #7439
• bug: hummock panic at pinning an older version #7472
• append-only kafka sink: Transaction error: Failed to initialize Producer ID: Local: Timed out #7499
• deterministic test: kafka producer panicked at 'called Option::unwrap() on a None value #7633
• bug: stuck when dropping sink #7639
• bug: meta does not stop as expected #7654
• bug: meta doesn't unpin epoch when creating MV fails #7657
• panicked at vnode 151 should not be accessed by this table #7698
• memory leak in allocation_stat_for_batch #7694
• batch task memory leak #7696
• Meta node will restart when running nexmark q9 #7753
• OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again. #7781
• Longevity job is getting error "Storage error: Hummock error: ReadCurrentEpoch error Cannot read when cluster is under recovery" #7841
• rwc-3-longevity-20230212-170319 fails #7851
• rwc-3-longevity-20230211-170317 fails #7852
• bug(storage): memory leaks #8125
• bug(mv-on-mv): creating count(*) mv cause barriers pile-up #8177
• bug: MaterializeBuffer index out of bounds: the len is 0 but the index is 7 #8136
• bug: watermark buffer panics when scaling  #8203
• scaling test panicked at 'overwrites an existing key' #8199
• bug: dropping nexmark q4 blackhole sink stuck #8474
• bug: Crash loop due to OOM  #8693
• bug: limit 1 batch query caused OOM #8721
• bug: longevity test OOM my-only no mv-on-mv #9279
• bug(storage): panic: inserted new level is not the newest #9655
• bug(batch): distributed query holds hummock version/snapshot for long time #9732
• bug: barrier stuck for minutes  #9723
• bug(meta): meta crashloop due to panic #10173
• bug: compaction task stuck for hours #10209
• bug: barrier stuck for more than 40mins #10210
• bug(streaming): hash join panic loop #10212
• bug(compaction group): expired task is not correctly rejected #10215
• meta node panicked at 'should find the level to insert into when applying compaction generated delta' #10538
• compute node OOM while create ch benchmark q3 with 100 warehouses #10899
• bug(compaction): unexpected l0->lbase task including more l0 inputs than level0_max_compact_file_number #11154
• panic: pk should be in the cache #11170
• bug: move table to an existing compaction group can violate can_concat for non-overlapping level #11703

Performance:

• bug: nexmark q4 performance degrade #6619
• perf: sysbench throughput is much lower than expection #7165
• perf: nexmark q7 become slower and slower #7244
• discussion: nexmark q16 low performance #7271
• perf: sysbench: select_random_ranges #7365