Use Operator upgrade TiDB Cluster Failed

[apollodafoni]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Use Operator upgrade cluster from v6.1.7 to master version

2. What did you expect to see? (Required)

upgrade successfully

3. What did you see instead (Required)

1. one tidb pod always in restart status

NAME READY STATUS RESTARTS AGE
cluster-discovery-67b84dc545-mc2pp 1/1 Running 0 140m
cluster-monitor-0 3/3 Running 0 140m
cluster-pd-0 1/1 Running 0 138m
cluster-tidb-0 1/1 Running 0 139m
cluster-tidb-1 0/1 Running 14 (6m1s ago) 137m
cluster-tikv-0 1/1 Running 0 137m
sdkserver-0 1/1 Running 0 140m

2. tidb.log has some fatal error

[2024/07/18 02:11:24.586 +00:00] [INFO] [ddl.go:1228] ["start DDL job"] [category=ddl] [job="ID:78, Type:create view, State:queueing, SchemaState:none, SchemaID:3, TableID:0, RowCount:0, ArgLen:3, start time: 2024-07-18 02:11:24.542 +0000 UTC, Err:, ErrCount:0, SnapshotVersion:0, LocalMode: false"] [query="CREATE OR REPLACE SQL SECURITY INVOKER VIEW mysql.tidb_mdl_view as (\n\t\tSELECT tidb_mdl_info.job_id,\n\t\t\tJSON_UNQUOTE(JSON_EXTRACT(cast(cast(job_meta as char) as json), "$.schema_name")) as db_name,\n\t\t\tJSON_UNQUOTE(JSON_EXTRACT(cast(cast(job_meta as char) as json), "$.table_name")) as table_name,\n\t\t\tJSON_UNQUOTE(JSON_EXTRACT(cast(cast(job_meta as char) as json), "$.query")) as query,\n\t\t\tsession_id,\n\t\t\tcluster_tidb_trx.start_time,\n\t\t\ttidb_decode_sql_digests(all_sql_digests, 4096) AS SQL_DIGESTS\n\t\tFROM mysql.tidb_ddl_job,\n\t\t\tmysql.tidb_mdl_info,\n\t\t\tinformation_schema.cluster_tidb_trx\n\t\tWHERE tidb_ddl_job.job_id=tidb_mdl_info.job_id\n\t\t\tAND CONCAT(',', tidb_mdl_info.table_ids, ',') REGEXP CONCAT(',(', REPLACE(cluster_tidb_trx.related_table_ids, ',', '|'), '),') != 0\n\t);"]
[2024/07/18 02:11:25.466 +00:00] [WARN] [runaway.go:166] ["failed to start runaway manager loop, please check whether the bootstrap or update is finished"] [error="[schema:1146]Table 'mysql.tidb_runaway_watch' doesn't exist"]
[2024/07/18 02:11:26.466 +00:00] [INFO] [info.go:1153] [SetTiFlashGroupConfig]
[2024/07/18 02:11:48.834 +00:00] [INFO] [manager.go:363] ["get owner"] ["owner info"="[log-backup] /tidb/br-stream/owner ownerManager 408520bb-3dc8-472b-9e5e-8a0d25f2ff4c"] ["owner key"=/tidb/br-stream/owner/19d090c3673c16a7] [ownerID=408520bb-3dc8-472b-9e5e-8a0d25f2ff4c] [op=none]
[2024/07/18 02:11:48.834 +00:00] [INFO] [manager.go:223] ["become owner"] ["owner info"="[log-backup] /tidb/br-stream/owner ownerManager 408520bb-3dc8-472b-9e5e-8a0d25f2ff4c"]
[2024/07/18 02:12:00.469 +00:00] [INFO] [owner_daemon.go:56] ["daemon became owner"] [id=408520bb-3dc8-472b-9e5e-8a0d25f2ff4c] [daemon-id=LogBackup::Advancer]
[2024/07/18 02:12:00.470 +00:00] [INFO] [advancer.go:523] ["Subscription handler spawned."] [category="log backup subscription manager"]
[2024/07/18 02:16:24.440 +00:00] [INFO] [domain.go:3014] ["refreshServerIDTTL succeed"] [serverID=1547] ["lease id"=19d090c3673c1698]
[2024/07/18 02:20:03.590 +00:00] [INFO] [ddl.go:1343] ["DDL job is failed"] [category=ddl] [jobID=78]
[2024/07/18 02:20:03.590 +00:00] [INFO] [tidb.go:269] ["rollbackTxn called due to ddl/autocommit failure"]
[2024/07/18 02:20:03.590 +00:00] [WARN] [session.go:2162] ["run statement failed"] [schemaVersion=43] [error="[meta:1050]DDL job rollback, error msg: table already exists"] [session="{\n "currDBName": "",\n "id": 0,\n "status": 2,\n "strictMode": false,\n "user": null\n}"]
[2024/07/18 02:20:03.590 +00:00] [FATAL] [bootstrap.go:3337] ["mustExecute error"] [error="[meta:1050]DDL job rollback, error msg: table already exists"] [stack="github.com/pingcap/tidb/pkg/session.mustExecute\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/bootstrap.go:3337\ngithub.com/pingcap/tidb/pkg/session.upgradeToVer94\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/bootstrap.go:2514\ngithub.com/pingcap/tidb/pkg/session.upgrade\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/bootstrap.go:1428\ngithub.com/pingcap/tidb/pkg/session.runInBootstrapSession\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:3670\ngithub.com/pingcap/tidb/pkg/session.bootstrapSessionImpl\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:3435\ngithub.com/pingcap/tidb/pkg/session.BootstrapSession\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:3395\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/cmd/tidb-server/main.go:420\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/cmd/tidb-server/main.go:326\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"] [stack="github.com/pingcap/tidb/pkg/session.mustExecute\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/bootstrap.go:3337\ngithub.com/pingcap/tidb/pkg/session.upgradeToVer94\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/bootstrap.go:2514\ngithub.com/pingcap/tidb/pkg/session.upgrade\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/bootstrap.go:1428\ngithub.com/pingcap/tidb/pkg/session.runInBootstrapSession\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:3670\ngithub.com/pingcap/tidb/pkg/session.bootstrapSessionImpl\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:3435\ngithub.com/pingcap/tidb/pkg/session.BootstrapSession\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:3395\nmain.createStoreAndDomain\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/cmd/tidb-server/main.go:420\nmain.main\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/cmd/tidb-server/main.go:326\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"]
command terminated with exit code 137

4. What is your TiDB version? (Required)

Release Version: v6.1.7
Edition: Community
Git Commit Hash: 613ecc5
Git Branch: heads/refs/tags/v6.1.7
UTC Build Time: 2023-07-04 12:00:13
GoVersion: go1.19.10
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

[apollodafoni]

/impact crash

[apollodafoni]

/component ddl

[apollodafoni]

/assign @tangenta