Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flacky test because of the location mismatch #54860

Closed
YangKeao opened this issue Jul 24, 2024 · 1 comment · Fixed by #54913
Closed

Flacky test because of the location mismatch #54860

YangKeao opened this issue Jul 24, 2024 · 1 comment · Fixed by #54913
Assignees
@YangKeao
Labels
component/test feature/developing the related feature is in development severity/minor sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.

Comments

@YangKeao
Copy link
Member

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Run any tests with testkit, it will have chance to trigger this issue. However, it's not easy to reproduce.

2. What did you expect to see? (Required)

panic: assert failed, location mismatch, ctxLoc: Local, varsLoc: Local, stmtLoc: Local [recovered]
	panic: assert failed, location mismatch, ctxLoc: Local, varsLoc: Local, stmtLoc: Local

goroutine 1691 [running]:
github.com/pingcap/tidb/pkg/executor.(*Compiler).Compile.func1()
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/executor/compiler.go:57 +0x491
panic({0x5a741a0?, 0xc00753c5d0?})
	/usr/lib/go/src/runtime/panic.go:770 +0x132
github.com/pingcap/tidb/pkg/util/intest.doPanic(...)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/util/intest/assert.go:59
github.com/pingcap/tidb/pkg/util/intest.Assert(...)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/util/intest/assert.go:27
github.com/pingcap/tidb/pkg/expression/context.AssertLocationWithSessionVars(0xa5a4be0, 0xc0063f2008)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/expression/context/context.go:232 +0x1f9
github.com/pingcap/tidb/pkg/expression/contextsession.(*SessionEvalContext).TypeCtx(0xc0071943f0)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/expression/contextsession/sessionctx.go:209 +0xbd
github.com/pingcap/tidb/pkg/expression.typeCtx(...)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/expression/context.go:57
github.com/pingcap/tidb/pkg/expression.(*Constant).Equal(0xc0063f8240, {0x70d3798, 0xc0071943f0}, {0x7116908?, 0xc0063f8240})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/expression/constant.go:437 +0xcb
github.com/pingcap/tidb/pkg/expression.(*baseBuiltinFunc).equal(0xc007194630, {0x70d3798, 0xc0071943f0}, {0x70ef260?, 0xc007194630?})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/expression/builtin.go:400 +0xc3
github.com/pingcap/tidb/pkg/expression.(*ScalarFunction).Equal(0xc007179ab0, {0x70d3798, 0xc0071943f0}, {0x7116a70?, 0xc007179ab0})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/expression/scalar_function.go:372 +0xc2
github.com/pingcap/tidb/pkg/planner/cardinality.getMaskAndRanges({0x70d9520, 0xc00753c0f0}, {0xc00753c550, 0x1, 0x1}, 0x90?, {0x0?, 0x63bc201?, 0xc001aab428?}, 0x0, ...)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/cardinality/selectivity.go:730 +0x4c6
github.com/pingcap/tidb/pkg/planner/cardinality.Selectivity({0x70d9520, 0xc00753c0f0}, 0xc007622a20, {0xc00753c4b0, 0x1, 0x1}, {0xc00753c1d0, 0x1, 0x1?})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/cardinality/selectivity.go:130 +0xbf1
github.com/pingcap/tidb/pkg/planner/core.(*DataSource).deriveStatsByFilter(0xc001ccc588, {0xc00753c4b0, 0x1, 0x1}, {0xc00753c1d0, 0x1, 0x2})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/core/stats.go:197 +0x159
github.com/pingcap/tidb/pkg/planner/core.(*DataSource).DeriveStats(0xc001ccc588, {0x0?, 0xc001ccc588?, 0xc006e303c0?}, 0xc001aac150?, {0x1e704fb?, 0x0?, 0xc001aac118?}, {0x0, 0x0, ...})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/core/stats.go:400 +0x59e
github.com/pingcap/tidb/pkg/planner/core/operator/logicalop.(*BaseLogicalPlan).RecursiveDeriveStats(0xc001ccc5a8, {0x0, 0x0, 0x0})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/core/operator/logicalop/base_logical_plan.go:188 +0x29b
github.com/pingcap/tidb/pkg/planner/core/operator/logicalop.(*BaseLogicalPlan).RecursiveDeriveStats(0xc006e303c0, {0x0, 0x0, 0x0})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/core/operator/logicalop/base_logical_plan.go:181 +0x16f
github.com/pingcap/tidb/pkg/planner/core/operator/logicalop.(*BaseLogicalPlan).RecursiveDeriveStats(0xc0065202a0, {0x0, 0x0, 0x0})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/core/operator/logicalop/base_logical_plan.go:181 +0x16f
github.com/pingcap/tidb/pkg/planner/core/operator/logicalop.(*BaseLogicalPlan).RecursiveDeriveStats(0xc006e302f0, {0x0, 0x0, 0x0})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/core/operator/logicalop/base_logical_plan.go:181 +0x16f
github.com/pingcap/tidb/pkg/planner/core.physicalOptimize({0x71124c0, 0xc006e302d0}, 0xc007310240)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/core/optimizer.go:1046 +0x12d
github.com/pingcap/tidb/pkg/planner/core.doOptimize({0x708f408, 0xc00a26e410}, {0x70d9520, 0xc00753c0f0}, 0xf134a, {0x71124c0, 0xc006e302d0})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/core/optimizer.go:308 +0x128
github.com/pingcap/tidb/pkg/planner/core.DoOptimize({0x708f408, 0xc00a26e410}, {0x70d9520, 0xc00753c0f0}, 0xf134a, {0x71124c0, 0xc006e302d0})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/core/optimizer.go:355 +0x133
github.com/pingcap/tidb/pkg/planner.optimize({0x708f408, 0xc00a26e410}, {0x70d9520, 0xc00753c0f0}, {0x709e2d0, 0xc0085b3560}, {0x70de2d0, 0xc0070be600})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/optimize.go:525 +0xa16
github.com/pingcap/tidb/pkg/planner.Optimize({0x708f408, 0xc00a26e410}, {0x711b948, 0xc00741e508}, {0x709e2d0, 0xc0085b3560}, {0x70de2d0, 0xc0070be600})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/planner/optimize.go:356 +0x1937
github.com/pingcap/tidb/pkg/executor.(*Compiler).Compile(0xc0069a9a48, {0x708f408?, 0xc00a26e410?}, {0x70a81e0, 0xc0085b3560})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/executor/compiler.go:99 +0x48e
github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt(0xc00741e508, {0x708f408?, 0xc00a26e410?}, {0x70a81e0, 0xc0085b3560})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/session/session.go:2098 +0x89a
github.com/pingcap/tidb/pkg/session.(*session).ExecuteInternal(0xc00741e508, {0x708f408, 0xc00a26e410}, {0xc006ede040, 0x3c}, {0x0, 0x0, 0x0})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/session/session.go:1523 +0x245
github.com/pingcap/tidb/pkg/ddl/session.(*Session).Execute(0xc00753c100, {0x708f408, 0xc00a26e410}, {0xc006ede040, 0x3c}, {0x63de7d9, 0xe})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/ddl/session/session.go:85 +0x1d0
github.com/pingcap/tidb/pkg/ddl/systable.(*manager).GetMinJobID.func1(0xc00753c100)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/ddl/systable/manager.go:120 +0x9b
github.com/pingcap/tidb/pkg/ddl/systable.(*manager).withNewSession(0xc002016348, 0xc00141adf8)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/ddl/systable/manager.go:71 +0xd1
github.com/pingcap/tidb/pkg/ddl/systable.(*manager).GetMinJobID(0xc00a64c1e0?, {0x708f408?, 0xc00a26e410?}, 0xc009276d80?)
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/ddl/systable/manager.go:118 +0x66
github.com/pingcap/tidb/pkg/ddl/systable.(*MinJobIDRefresher).refresh(0xc00a0e5c98, {0x708f408?, 0xc00a26e410?})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/ddl/systable/min_job_id.go:66 +0x4b
github.com/pingcap/tidb/pkg/ddl/systable.(*MinJobIDRefresher).Start(0xc00a0e5c98, {0x708f408, 0xc00a26e410})
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/ddl/systable/min_job_id.go:54 +0x3d
github.com/pingcap/tidb/pkg/ddl.(*ddl).Start.func3()
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/ddl/ddl.go:869 +0x29
github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1()
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/util/wait_group_wrapper.go:157 +0x52
created by github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run in goroutine 87
	/home/yangkeao/Project/github.com/YangKeao/tidb/master/pkg/util/wait_group_wrapper.go:155 +0x73
FAIL	github.com/pingcap/tidb/pkg/executor	0.554s
FAIL

It's quite strange. The logic of AssertLocationWithSessionVars shows the ctxLocStr is different with varsLocStr (or stmtLocStr). But they are the same in the log:

// AssertLocationWithSessionVars asserts the location in the context and session variables are the same.
// It is only used for testing.
func AssertLocationWithSessionVars(ctxLoc *time.Location, vars *variable.SessionVars) {
	ctxLocStr := ctxLoc.String()
	varsLocStr := vars.Location().String()
	stmtLocStr := vars.StmtCtx.TimeZone().String()
	intest.Assert(ctxLocStr == varsLocStr && ctxLocStr == stmtLocStr,
		"location mismatch, ctxLoc: %s, varsLoc: %s, stmtLoc: %s",
		ctxLoc.String(), ctxLocStr, stmtLocStr,
	)
}

3. What did you see instead (Required)

Test successfully.

4. What is your TiDB version? (Required)

master-04e5ff999ac117552e144070a8ac440484fc44da
@YangKeao YangKeao added type/bug The issue is confirmed as a bug. sig/sql-infra SIG: SQL Infra labels Jul 24, 2024
@YangKeao YangKeao self-assigned this Jul 24, 2024
@Defined2014
Copy link
Contributor

Defined2014 commented Jul 29, 2024 •
edited
Loading

Same as #54868, maybe fixed by #54913. PTAL @YangKeao

@YangKeao YangKeao mentioned this issue Jul 29, 2024
Merged
13 tasks
@ti-chi-bot ti-chi-bot bot closed this as completed in 108e98a Jul 29, 2024
hawkingrei pushed a commit to hawkingrei/tidb that referenced this issue Aug 1, 2024
@D3Hunter @hawkingrei
ddl: init global vars from system tables before start domain (pingcap…
df0a776 
…#54913)

close pingcap#54860, close pingcap#54868
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@YangKeao YangKeao

Labels
component/test feature/developing the related feature is in development severity/minor sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ddl: init global vars from system tables before start domain D3Hunter/tidb
4 participants
@D3Hunter @jebter @YangKeao @Defined2014