The timeout is too short for fetching data of INFORMATION_SCHEMA.TIFLASH_TABLES

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

An issue reported in the asktug (https://asktug.com/t/topic/1036157/1)

mysql> select * from INFORMATION_SCHEMA.TIFLASH_TABLES  limit 1\G
ERROR 1105 (HY000): rpc error: code = DeadlineExceeded desc = context deadline exceeded

2. What did you expect to see? (Required)

3. What did you see instead (Required)

The timeout is only 1 second for fetching the system table from tiflash instance. When there are thousand of table with tiflash replica, the query easily run into timeout.

https://github.com/pingcap/tidb/blob/v7.5.4/pkg/executor/infoschema_reader.go#L3081-L3083

Timeout is 1 second since #42720

There are about 5000 tables/partition in the cluster.

mysql> select count(*) from INFORMATION_SCHEMA.TIFLASH_REPLICA;
+----------+
| count(*) |
+----------+
|     3090 |
+----------+
1 row in set (0.04 sec)

mysql> select count(*) from INFORMATION_SCHEMA.TIFLASH_REPLICA t join INFORMATION_SCHEMA.partitions p on t.table_schema = p.table_schema and t.table_name= p.table_name;
+----------+
| count(*) |
+----------+
|     4892 |
+----------+
1 row in set (0.30 sec)

4. What is your TiDB version? (Required)

v7.5.4

[JaySon-Huang]

/label affects-8.5

[JaySon-Huang]

/label affects-8.1

[JaySon-Huang]

/label affects-7.5

[JaySon-Huang]

/label affects-7.1

[JaySon-Huang]

/label affects-6.5

[JaySon-Huang]

Under non-disaggregated arch, when there are about 4900 table with tiflash replicas on 1 tiflash instance, it takes about 0.26 second to query the INFORMATION_SCHEMA.TIFLASH_TABLES table.

> select count(*) from information_schema.tiflash_tables;
+----------+
| count(*) |
+----------+
| 9        |
+----------+
1 row in set
Time: 0.012s

> select count(*) from information_schema.tiflash_tables;
+----------+
| count(*) |
+----------+
| 4969     |
+----------+
1 row in set
Time: 0.261s

Under TiFlash Disaggregated Storage and Compute Architecture, it could take more time to get the info from INFORMATION_SCHEMA.TIFLASH_TABLES. Because some metadata need to access files on S3.

[JaySon-Huang]

/label severity/moderate

[ti-chi-bot]

@JaySon-Huang: The label(s) severity/moderate cannot be applied. These labels are supported: fuzz/sqlancer, fuzz/comp, challenge-program, compatibility-breaker, first-time-contributor, contribution, good first issue, correctness, duplicate, proposal, security, needs-more-info, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, needs-cherry-pick-release-8.1, needs-cherry-pick-release-8.5, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, affects-8.1, affects-8.4, affects-8.5, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5, may-affects-8.1, may-affects-8.5.

In response to this:

/label severity/moderate

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[JaySon-Huang]

/severity moderate

[seiya-annie]

/report community

[JaySon-Huang]

/remove-label affects-6.5