Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fatal error: concurrent map writes, HashJoin on temporary table #55337

Closed
tiancaiamao opened this issue Aug 9, 2024 · 1 comment · Fixed by #55338
Closed

fatal error: concurrent map writes, HashJoin on temporary table #55337

tiancaiamao opened this issue Aug 9, 2024 · 1 comment · Fixed by #55338
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. impact/crash crash/fatal severity/major sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.

Comments

@tiancaiamao
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Not sure how to reproduce.. here is the panic stack:

fatal error: concurrent map writes

goroutine 16084384107 [running]:
github.com/pingcap/tidb/sessionctx/variable.(*SessionVars).GetTemporaryTable(0x402d991408?, 0x40008fc700)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessionctx/variable/session.go:2298 +0xd4
github.com/pingcap/tidb/table/tables.(*TableCommon).Allocators(0x4001cc4b40, {0x531e3f8, 0x46251bf680})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/table/tables/tables.go:1550 +0x7c
github.com/pingcap/tidb/executor.getAutoIncrementID({0x531e3f8, 0x46251bf680}, 0x4339266a80, 0x40008fc700)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:389 +0xdc
github.com/pingcap/tidb/executor.(*memtableRetriever).setDataFromTables(0x42a5a7baa0, {0x52aca90?, 0x44f1754a50?}, {0x531e3f8, 0x46251bf680?}, {0x45d7eb6800, 0xe, 0x1?})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:689 +0x690
github.com/pingcap/tidb/executor.(*memtableRetriever).retrieve(0x42a5a7baa0, {0x52aca90, 0x44f1754a50}, {0x531e3f8, 0x46251bf680})
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:111 +0x1e8
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0x405ebfdc20, {0x52aca90, 0x44f1754a50}, 0x4069e0fc20)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/memtable_reader.go:118 +0x210
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b17e0, 0x405ebfdc20}, 0x4069e0fc20)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*SelectionExec).Next(0x5ae73ec460, {0x52aca90, 0x44f1754a50}, 0x4558ba57c0)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1596 +0xbc
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b1c60, 0x5ae73ec460}, 0x4558ba57c0)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*buildWorker).fetchBuildSideRows(0x4069e0f900, {0x52aca90, 0x44f1754a50}, 0x4124a3a240, 0x4124a3a2a0, 0x44fdfb5140)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:332 +0x114
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndBuildHashTable.func2()
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:1178 +0xa8
github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover.func1()
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:51 +0x64
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:43 +0x9c

goroutine 1 [chan receive, 68928 minutes]:
github.com/pingcap/tidb/server.(*Server).Run(0x4007395b00)
        /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:402 +0x20c
main.main()

goroutine 110 [select]:
github.com/tikv/client-go/v2/internal/locate.(*RegionCache).cacheGC(0x40014274a0)
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_cache.go:2013 +0x144
created by github.com/tikv/client-go/v2/internal/locate.NewRegionCache
        /go/pkg/mod/github.com/tikv/client-go/[email protected]/internal/locate/region_cache.go:433 +0x350

goroutine 101 [select]:
go.opencensus.io/stats/view.(*worker).start(0x4000686100)
        /go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x88
created by go.opencensus.io/stats/view.init.0
        /go/pkg/mod/[email protected]/stats/view/worker.go:34 +0xa4

goroutine 49 [select]:
github.com/tikv/pd/client.(*baseClient).memberLoop(0x4000c534a0)
        /go/pkg/mod/github.com/tikv/pd/[email protected]/base_client.go:139 +0x140
created by github.com/tikv/pd/client.(*baseClient).init
        /go/pkg/mod/github.com/tikv/pd/[email protected]/base_client.go:109 +0x164

goroutine 114 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
        /go/pkg/mod/github.com/golang/[email protected]/glog.go:882 +0x5c
created by github.com/golang/glog.init.0
        /go/pkg/mod/github.com/golang/[email protected]/glog.go:410 +0x200

goroutine 321 [chan receive, 822 minutes]:
gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun(0x4000fc3f20)
        /go/pkg/mod/gopkg.in/natefinch/[email protected]/lumberjack.go:379 +0x44
created by gopkg.in/natefinch/lumberjack%2ev2.(*Logger).mill.func1
        /go/pkg/mod/gopkg.in/natefinch/[email protected]/lumberjack.go:390 +0xa4

2. What did you expect to see? (Required)

no panic

3. What did you see instead (Required)

panic

Master should have the same bug.

tidb/pkg/sessionctx/variable/session.go

Line 2775 in 240702e

tempTables[tblInfo.ID] = tempTable

4. What is your TiDB version? (Required)

v6.5.6-20240517-4b81451
@tiancaiamao tiancaiamao added the type/bug The issue is confirmed as a bug. label Aug 9, 2024
@lcwangchao
Copy link
Collaborator

Seems information_schema.table is joining it self and caused the data race:

goroutine 16084384107 [running]:
github.com/pingcap/tidb/sessionctx/variable.(*SessionVars).GetTemporaryTable(0x402d991408?, 0x40008fc700)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessionctx/variable/session.go:2298 +0xd4
github.com/pingcap/tidb/table/tables.(*TableCommon).Allocators(0x4001cc4b40, {0x531e3f8, 0x46251bf680})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/table/tables/tables.go:1550 +0x7c
github.com/pingcap/tidb/executor.getAutoIncrementID({0x531e3f8, 0x46251bf680}, 0x4339266a80, 0x40008fc700)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:389 +0xdc
github.com/pingcap/tidb/executor.(*memtableRetriever).setDataFromTables(0x42a5a7baa0, {0x52aca90?, 0x44f1754a50?}, {0x531e3f8, 0x46251bf680?}, {0x45d7eb6800, 0xe, 0x1?})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:689 +0x690
github.com/pingcap/tidb/executor.(*memtableRetriever).retrieve(0x42a5a7baa0, {0x52aca90, 0x44f1754a50}, {0x531e3f8, 0x46251bf680})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:111 +0x1e8
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0x405ebfdc20, {0x52aca90, 0x44f1754a50}, 0x4069e0fc20)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/memtable_reader.go:118 +0x210
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b17e0, 0x405ebfdc20}, 0x4069e0fc20)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*SelectionExec).Next(0x5ae73ec460, {0x52aca90, 0x44f1754a50}, 0x4558ba57c0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1596 +0xbc
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b1c60, 0x5ae73ec460}, 0x4558ba57c0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*buildWorker).fetchBuildSideRows(0x4069e0f900, {0x52aca90, 0x44f1754a50}, 0x4124a3a240, 0x4124a3a2a0, 0x44fdfb5140)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:332 +0x114
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndBuildHashTable.func2()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:1178 +0xa8
github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover.func1()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:51 +0x64
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:43 +0x9c



goroutine 16084384202 [runnable]:
github.com/pingcap/tidb/statistics.PseudoTable(0x40008fcc40)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/statistics/table.go:1117 +0x50
github.com/pingcap/tidb/table/tables.TempTableFromMeta(0x40008fcc40)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/table/tables/tables.go:2025 +0x20
github.com/pingcap/tidb/sessionctx/variable.(*SessionVars).GetTemporaryTable(0x404bb6d438?, 0x40008fcc40)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/sessionctx/variable/session.go:2297 +0xb4
github.com/pingcap/tidb/table/tables.(*TableCommon).Allocators(0x4001cc4ea0, {0x531e3f8, 0x46251bf680})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/table/tables/tables.go:1550 +0x7c
github.com/pingcap/tidb/executor.getAutoIncrementID({0x531e3f8, 0x46251bf680}, 0x4339266a80, 0x40008fcc40)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:389 +0xdc
github.com/pingcap/tidb/executor.(*memtableRetriever).setDataFromTables(0x42a5a7ba40, {0x52aca90?, 0x44f1754a50?}, {0x531e3f8, 0x46251bf680?}, {0x444393d880, 0xe, 0x0?})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:689 +0x690
github.com/pingcap/tidb/executor.(*memtableRetriever).retrieve(0x42a5a7ba40, {0x52aca90, 0x44f1754a50}, {0x531e3f8, 0x46251bf680})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/infoschema_reader.go:111 +0x1e8
github.com/pingcap/tidb/executor.(*MemTableReaderExec).Next(0x405ebfdb80, {0x52aca90, 0x44f1754a50}, 0x4069e0fbd0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/memtable_reader.go:118 +0x210
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b17e0, 0x405ebfdb80}, 0x4069e0fbd0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*SelectionExec).Next(0x5ae73ec380, {0x52aca90, 0x44f1754a50}, 0x40bb626820)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:1596 +0xbc
github.com/pingcap/tidb/executor.Next({0x52aca90, 0x44f1754a50}, {0x52b1c60, 0x5ae73ec380}, 0x40bb626820)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:328 +0x41c
github.com/pingcap/tidb/executor.(*probeSideTupleFetcher).fetchProbeSideChunks(0x43468aa940, {0x52aca90, 0x44f1754a50}, 0x400)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:253 +0x134
github.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndProbeHashTable.func1()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/join.go:390 +0x84
github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover.func1()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:51 +0x64
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).RunWithRecover
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:43 +0x9c

@lcwangchao lcwangchao mentioned this issue Aug 9, 2024
Merged
13 tasks
@lcwangchao lcwangchao added sig/sql-infra SIG: SQL Infra affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. and removed may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 labels Aug 9, 2024
@ti-chi-bot ti-chi-bot bot closed this as completed in 2675c66 Aug 9, 2024
This was referenced Aug 9, 2024
Merged
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Aug 12, 2024
@ti-chi-bot
variable: fix data race in GetTemporaryTable (#55338) (#55340)
df0aeb2 
close #55337
ti-chi-bot bot pushed a commit that referenced this issue Aug 12, 2024
@ti-chi-bot
variable: fix data race in GetTemporaryTable (#55338) (#55339)
3ea0661 
close #55337
ti-chi-bot bot pushed a commit that referenced this issue Aug 12, 2024
@ti-chi-bot
variable: fix data race in GetTemporaryTable (#55338) (#55341)
0ff1860 
close #55337
ti-chi-bot bot pushed a commit that referenced this issue Aug 14, 2024
@ti-chi-bot
variable: fix data race in GetTemporaryTable (#55338) (#55342)
f440415 
close #55337
@jebter jebter added the impact/crash crash/fatal label Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. impact/crash crash/fatal severity/major sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

variable: fix data race in GetTemporaryTable lcwangchao/tidb
3 participants
@lcwangchao @tiancaiamao @jebter