Ingest writers leak when cancelling adding index job

[tangenta]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Set tidb_enable_dist_task = on.
2. Run add index.
3. Cancel add index through admin cancel ddl jobs X.

2. What did you expect to see? (Required)

Add index job is canceled normally, no other side effects.

3. What did you see instead (Required)

DDL worker panic. Ingest writers leak.

[2024/06/05 07:06:10.237 +00:00] [ERROR] [misc.go:114] ["panic in the recoverable goroutine"] [label=ddl-worker] [funcInfo="worker 3, tp add index runDDLJob"] [r="close of closed channel"] [stack="[github.com/pingcap/tidb/pkg/util.Recover\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/misc.go:118\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nruntime.closechan\n\t/usr/local/go/src/runtime/chan.go:365\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).closeEngine\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine_mgr.go:373\ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).CloseEngine\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/local.go:793\ngithub.com/pingcap/tidb/pkg/lightning/backend.engine.unsafeClose\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/backend.go:345\ngithub.com/pingcap/tidb/pkg/lightning/backend.(*OpenedEngine).Close\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/backend.go:291\ngithub.com/pingcap/tidb/pkg/ddl/ingest.(*engineInfo).Clean\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/engine.go:116\ngithub.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtx).UnregisterEngines\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/engine_mgr.go:107\ngithub.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtxMgr).Unregister\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/backend_mgr.go:300\ngithub.com/pingcap/tidb/pkg/ddl.convertAddIdxJob2RollbackJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/rollingback.go:95\ngith](http://github.com/pingcap/tidb/pkg/util.Recover/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/misc.go:118/nruntime.gopanic/n/t/usr/local/go/src/runtime/panic.go:914/nruntime.closechan/n/t/usr/local/go/src/runtime/chan.go:365/ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*engineManager).closeEngine/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/engine_mgr.go:373/ngithub.com/pingcap/tidb/pkg/lightning/backend/local.(*Backend).CloseEngine/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/local/local.go:793/ngithub.com/pingcap/tidb/pkg/lightning/backend.engine.unsafeClose/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/backend.go:345/ngithub.com/pingcap/tidb/pkg/lightning/backend.(*OpenedEngine).Close/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/lightning/backend/backend.go:291/ngithub.com/pingcap/tidb/pkg/ddl/ingest.(*engineInfo).Clean/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/engine.go:116/ngithub.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtx).UnregisterEngines/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/engine_mgr.go:107/ngithub.com/pingcap/tidb/pkg/ddl/ingest.(*litBackendCtxMgr).Unregister/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ingest/backend_mgr.go:300/ngithub.com/pingcap/tidb/pkg/ddl.convertAddIdxJob2RollbackJob/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/rollingback.go:95/ngith)

Here is the heap profile when there is no DDL running job.

4. What is your TiDB version? (Required)

f1ec74b

[tangenta]

There are concurrent close operations:

[2024/06/05 07:06:07.811 +00:00] [INFO] [backend.go:344] ["engine close start"] [category=ddl-ingest] [engineTag=:8] [engineUUID=faa79115-fb21-579d-a6d9-5c344586b1de]
[2024/06/05 07:06:09.824 +00:00] [INFO] [backend.go:344] ["engine close start"] [category=ddl-ingest] [engineTag=:8] [engineUUID=faa79115-fb21-579d-a6d9-5c344586b1de]
...
[2024/06/05 07:06:10.237 +00:00] [INFO] [backend.go:346] ["engine close completed"] [category=ddl-ingest] [engineTag=:8] [engineUUID=faa79115-fb21-579d-a6d9-5c344586b1de] [takeTime=2.425636049s] []

One of them is from pipeline.Close() and the other is from unregister() caused by cancelling.

[tangenta]

Introduced by #53233, it does not affect 8.1.