query stuck when trying to kill it because of OOM

[guo-shaoge]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. hack tidb code, delete following code, so we can set tidb_mem_quota_query a very small value to help reproduce this bug

   tidb/pkg/sessionctx/variable/sysvar.go

   Lines 1127 to 1130 in 199bf90

   	if intVal > 0 && intVal < 128 { // 128 Bytes
   	s.StmtCtx.AppendWarning(ErrTruncatedWrongValue.FastGenByArgs(TiDBServerMemoryLimitSessMinSize, originalValue))
   	intVal = 128
   	}

2. build tidb and start tiup playround
3. download 10-tidb-slow.log and set @@tidb_slow_query_file = '/home/guojiangtao/10-tidb-slow.log'; (remember to change the slow query file path)
4. set @@tidb_mem_quota_query = 10;
5. run select time,host host_ip,Query_time as exec_max_time,parse_time,compile_time,Query as sql_text,Digest as sql_id,is_internal,succ, Plan as plan_text,mem_max as mem_max,User as parse_user,DB as database_name,total_keys,request_count,process_time,process_keys from information_schema.SLOW_QUERY order by time desc;

2. What did you expect to see? (Required)

query is canceled

3. What did you see instead (Required)

query is stucked

4. What is your TiDB version? (Required)

master 560e92e

[yibin87]

/assign @yibin87

[yibin87]

Selecting plan from slow_query table without limit restriction is not recommanded behavior. And this is a long existing issue. Down to major.

[yibin87]

/remove severity-critical

[yibin87]

/remove-severity critical

[yibin87]

/severity major

[yibin87]

For version 6.5, 7.5, the query met illegal nil pointer: ERROR 1105 (HY000): runtime error: invalid memory address or nil pointer dereference

[yibin87]

v8.1.0 stuck also

[yibin87]

The plan is Sort <=== Projection <==== MemTableReader
After debug, located the goroutine that get stuck:
First Projection executor tries to alloc memory:

tidb/pkg/executor/projection.go

Line 289 in a187e9d

e.memTracker.Consume(outputChk.MemoryUsage())

Then, it triggers the memory action panic:

tidb/pkg/util/memory/action.go

Line 173 in a187e9d

panic(err)

After it enters runtime.gopanic, and it invokes the recover function in executor::Next:

tidb/pkg/executor/internal/exec/executor.go

Line 435 in a187e9d

err = util.GetRecoverError(r)

Finally, it got stuck when invokes runtime mcall(recovery)

[yibin87]

The plan is Sort <=== Projection <==== MemTableReader After debug, located the goroutine that get stuck: First Projection executor tries to alloc memory:

tidb/pkg/executor/projection.go

Line 289 in a187e9d

e.memTracker.Consume(outputChk.MemoryUsage())

Then, it triggers the memory action panic:

tidb/pkg/util/memory/action.go

Line 173 in a187e9d

panic(err)

After it enters runtime.gopanic, and it invokes the recover function in executor::Next:

tidb/pkg/executor/internal/exec/executor.go

Line 435 in a187e9d

err = util.GetRecoverError(r)

Finally, it got stuck when invokes runtime mcall(recovery)

It turned out to be debug only stuck on mac OS, the real stuck happen on the following stack:

[yibin87]

And check the source code, found the unlock may not be invoked:

tidb/pkg/util/execdetails/execdetails.go

Line 1303 in a187e9d

e.mu.Lock()

[yibin87]

After fix this, the stuck disappeared, still not know what's actually happening when the panic happen.

[yibin87]

/label affects-8.1

[yibin87]

/label affects-7.5

[yibin87]

/label affects-7.1

[yibin87]

/label affects-6.5

[yibin87]

/label affects-6.1

[yibin87]

/remove-impact leak

[yibin87]

/label affects-5.4