Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dumpling is slow in 1M tables case #57902

Closed
lance6716 opened this issue Dec 3, 2024 · 0 comments · Fixed by #57894
Closed

dumpling is slow in 1M tables case #57902

lance6716 opened this issue Dec 3, 2024 · 0 comments · Fixed by #57894
Labels
affects-8.5 This bug affects the 8.5.x(LTS) versions. component/dumpling This is related to Dumpling of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@lance6716
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

prepare a large cluster with many database, each database has a small number of tables. Then use dumpling to dump a database with 2 tables

2. What did you expect to see? (Required)

dump is fast, because only 2 tables will be dumped

3. What did you see instead (Required)

tiup dumpling:nightly -u root -P 4000 -h [10.104.100.98](http://10.104.100.98/) --filetype csv -t 16 -r 200000 -F 256MiB --database sb10wnormal1
[2024/12/02 06:50:54.641 +00:00] [INFO] [dump.go:152] ["begin to run Dump"] [conf="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"session-token\":\"\",\"provider\":\"\",\"force-path-style\":true,\"use-accelerate-endpoint\":false,\"role-arn\":\"\",\"external-id\":\"\",\"object-lock-enabled\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"azblob\":{\"endpoint\":\"\",\"account-name\":\"\",\"account-key\":\"\",\"access-tier\":\"\",\"sas-token\":\"\",\"encryption-scope\":\"\",\"encryption-key\":\"\"},\"SpecifiedTables\":false,\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoSequences\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":false,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"[10.104.100.98](http://10.104.100.98/)\",\"Port\":4000,\"Threads\":16,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"info\",\"LogFile\":\"\",\"LogFormat\":\"text\",\"OutputDirPath\":\"./export-2024-12-02T06:50:54Z\",\"StatusAddr\":\":8281\",\"Snapshot\":\"454327600314580994\",\"Consistency\":\"snapshot\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\",\",\"CsvDelimiter\":\"\\\"\",\"CsvLineTerminator\":\"\\r\\n\",\"Databases\":[\"sb10wnormal1\"],\"Where\":\"\",\"FileType\":\"csv\",\"ServerInfo\":{\"ServerType\":3,\"ServerVersion\":\"8.5.0-alpha-256-gec288d95a3\",\"HasTiKV\":true},\"Rows\":200000,\"ReadTimeout\":900000000000,\"TiDBMemQuotaQuery\":0,\"FileSize\":268435456,\"StatementSize\":1000000,\"SessionParams\":{\"tidb_snapshot\":\"454327600314580994\"},\"Tables\":{},\"CollationCompatible\":\"loose\",\"CsvOutputDialect\":0,\"IOTotalBytes\":null,\"Net\":\"\"}"]
...
[2024/12/02 07:05:42.661 +00:00] [ERROR] [main.go:78] ["dump failed error stack info"] [error="sql: SELECT COUNT(1) as c FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='SEQUENCE': Error 9006 (HY000): GC life time is shorter than transaction duration, transaction starts at 2024-12-02 06:50:38.718 +0000 UTC, GC safe point is 2024-12-02 06:50:54.618 +0000 UTC"] [errorVerbose="Error 9006 (HY000): GC life time is shorter than transaction duration, transaction starts at 2024-12-02 06:50:38.718 +0000 UTC, GC safe point is 2024-12-02 06:50:54.618 +0000 UTC\nsql: SELECT COUNT(1) as c FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='SEQUENCE'\[ngithub.com/pingcap/tidb/dumpling/export.CheckIfSeqExists\n\tgithub.com/pingcap/tidb/dumpling/export/sql.go:932\ngithub.com/pingcap/tidb/dumpling/export.prepareTableListToDump\n\tgithub.com/pingcap/tidb/dumpling/export/dump.go:1153\ngithub.com/pingcap/tidb/dumpling/export.(*Dumper).Dump\n\tgithub.com/pingcap/tidb/dumpling/export/dump.go:218\nmain.main\n\t./main.go:75\nruntime.main\n\truntime/proc.go:272\nruntime.goexit\n\truntime/asm_amd64.s:1700](http://ngithub.com/pingcap/tidb/dumpling/export.CheckIfSeqExists/n/tgithub.com/pingcap/tidb/dumpling/export/sql.go:932/ngithub.com/pingcap/tidb/dumpling/export.prepareTableListToDump/n/tgithub.com/pingcap/tidb/dumpling/export/dump.go:1153/ngithub.com/pingcap/tidb/dumpling/export.(*Dumper).Dump/n/tgithub.com/pingcap/tidb/dumpling/export/dump.go:218/nmain.main/n/t./main.go:75/nruntime.main/n/truntime/proc.go:272/nruntime.goexit/n/truntime/asm_amd64.s:1700)"]

a query is slow, which meet other error and fails the dump

4. What is your TiDB version? (Required)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-8.5 This bug affects the 8.5.x(LTS) versions. component/dumpling This is related to Dumpling of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant