Recursive CTE didn't track memory accurately

[guo-shaoge]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb/pkg/util/cteutil/storage.go

Lines 156 to 169 in 6695d5e

	// Reopen impls Storage Reopen interface.
	func (s *StorageRC) Reopen() (err error) {
	if err = s.rc.Close(); err != nil {
	return err
	}
	s.iter = 0
	s.done = false
	s.err = nil
	// Create a new RowContainer.
	// Because some meta infos in old RowContainer are not resetted.
	// Such as memTracker/actionSpill etc. So we just use a new one.
	s.rc = chunk.NewRowContainer(s.tp, s.chkSize)
	return nil
	}

L167 create a new row container after each CTE iteration. But it didn't attach memtracker to session tracker, so the memory control will not work correctly.

Specifically, the memory control for resTbl of CTE is as expected, but after one iteration of the CTE computation, the memory usage of iterInTbl and iterOutTbl cannot be accurately tracked (and they also cannot be spilled to disk) because of above bug.

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

[yibin87]

Reproduce case:
drop table if exists tbl_0;
create table tbl_0(a int);
insert into tbl_0 values(100);
insert into tbl_0 values(200);
insert into tbl_0 values(300);
insert into tbl_0 values(400);
set cte_max_recursion_depth = 10000000;
set tidb_mem_quota_query = 10000;
explain analyze with recursive cte_0 (col_10,col_11,col_12,col_13,col_14,col_15,col_16,col_17,col_18) AS ( select 1, 2,3,4,5,6,7,"A", "B" from tbl_0 UNION all select col_10 + 1,col_11 + 1,col_12 + 1 , col_13+1, col_14+1, col_15+1, col_16+1, concat(col_17, "B"), concat(col_18, "C") from cte_0 where col_10 < 19800000 ) select * from cte_0 order by col_18, col_14 limit 4;

The max memory usage tracked is 7-8k bytes. However, the heap profile info shows the CTE operator uses at least 160M.

[JasonWu0506]

/severity major

[guo-shaoge]

todo: Also remember to check if memory/cpu is back to normal when this sql is killed!