Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

admin check failed with “Error 8223 (HY000): data inconsistency in table” after kill one of tikv or tikv rolling restart during adding index #51407

Closed
Lily2025 opened this issue Feb 29, 2024 · 2 comments · Fixed by #51897
Assignees
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.0 affects-8.1 This bug affects the 8.1.x(LTS) versions. component/ddl This issue is related to DDL of TiDB. severity/critical type/bug The issue is confirmed as a bug.

Comments

@Lily2025
Copy link

Lily2025 commented Feb 29, 2024

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、tidb_enable_dist_task='on'
2、run sysbench
3、add index for one of table
4、kill one of tikv
5、admin check index when adding index finish

tidb logs:
endless-ha-test-add-index-tps-7080337-1-324.tar.gz

2. What did you expect to see? (Required)

admin check success

3. What did you see instead (Required)

admin check failed(Error 8223 (HY000): data inconsistency in table: sbtest1, index: index_test_1709003362736, handle: 35087534, index-values:"handle: 35087534, values: [KindString 54150009827-63531154822-50885588575-38796535820-59831497760-15874376114-47900593283-86891964691-89920570946-88503431724]" != record-values:"handle: 35087534, values: [KindString 48350368445-67037925709-53068813339-93858222535-70857817307-21035926610-73758438108-82851278774-47393726843-33708297691]")
operatorLogs:
[2024-02-27 11:09:22] ###### start adding index
alter table sbtest1 add index index_test_1709003362736 (c)
[2024-02-27 11:09:22] ###### wait for ddl job finish
[2024-02-27 11:17:30] ###### ddl job finished
select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1709003362736 (c)'
jobId: 449, job type: add index /* ingest */, state: synced
add index done, it takes: 8m7.626657145s
[2024-02-27 11:17:30] ###### start admin check
admin check index sbtest1 index_test_1709003362736

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: c1b70d5
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-02-26 11:45:04
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-02-27T10:57:36.218+0800

@Lily2025 Lily2025 added the type/bug The issue is confirmed as a bug. label Feb 29, 2024
@Lily2025
Copy link
Author

/assign tangenta

@seiya-annie seiya-annie added component/ddl This issue is related to DDL of TiDB. and removed may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 labels Mar 5, 2024
@Lily2025 Lily2025 changed the title admin check failed with “Error 8223 (HY000): data inconsistency in table” after add index and kill one of tikv admin check failed with “Error 8223 (HY000): data inconsistency in table” after kill one of tikv during adding index Mar 6, 2024
@Lily2025
Copy link
Author

Lily2025 commented Mar 6, 2024

admin check failed after rolling restart tikv during adding index

admin check failed(Error 8223 (HY000): data inconsistency in table: sbtest1, index: index_test_1709662618059, handle: 42670109, index-values:"handle: 42670109, values: [KindString 79793922415-96432231064-49793791306-09392438963-57303495845-77266668293-27635913748-54072365834-99544797245-07250190896]" != record-values:"handle: 42670109, values: [KindString 06357938616-54130163307-25396091638-95567474086-75208163438-20698888865-07940411079-68156310255-39716096584-08207716142]")
operatorLogs:
[2024-03-06 02:16:58] ###### start adding index
ALTER TABLE sbtest1 ADD INDEX index_test_1709662618059(c)
[2024-03-06 02:16:58] ###### wait for ddl job finish
[2024-03-06 02:19:58] ###### ddl job finished
select job_id, job_type, state from information_schema.ddl_jobs where query = 'ALTER TABLE sbtest1 ADD INDEX index_test_1709662618059(c)'
jobId: 579, job type: add index /* ingest */, state: synced
add index done, it takes: 3m0.357638174s
[2024-03-06 02:19:58] ###### start admin check
admin check index sbtest1 index_test_1709662618059

Analysis from @tangenta
It appears that the MDL did not take effect and did not block the DDL advance, resulting in a very old transaction being committed
img_v3_028n_ab9015b4-7ef0-4978-a1fb-7cd04c4a9b0g

tidb version:
./tidb-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: 5ac8a5b
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-03-04 11:47:38
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-03-05T20:02:37.101+0800

@Lily2025 Lily2025 changed the title admin check failed with “Error 8223 (HY000): data inconsistency in table” after kill one of tikv during adding index admin check failed with “Error 8223 (HY000): data inconsistency in table” after kill one of tikv or tikv rolling restart during adding index Mar 6, 2024
@wjhuang2016 wjhuang2016 added type/cherry-pick-for-release-6.5 This PR is cherry-picked to release-6.5 from a source PR. type/cherry-pick-for-release-7.1 This PR is cherry-picked to release-7.1 from a source PR. type/cherry-pick-for-release-7.5 This PR is cherry-picked to release-7.5 from a source PR. labels Mar 20, 2024
ti-chi-bot bot pushed a commit that referenced this issue Apr 7, 2024
@ti-chi-bot ti-chi-bot added the affects-8.1 This bug affects the 8.1.x(LTS) versions. label Apr 9, 2024
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue Apr 28, 2024
@jebter jebter added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. and removed type/cherry-pick-for-release-6.5 This PR is cherry-picked to release-6.5 from a source PR. type/cherry-pick-for-release-7.1 This PR is cherry-picked to release-7.1 from a source PR. type/cherry-pick-for-release-7.5 This PR is cherry-picked to release-7.5 from a source PR. labels May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.0 affects-8.1 This bug affects the 8.1.x(LTS) versions. component/ddl This issue is related to DDL of TiDB. severity/critical type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants