Global sort IMPORT INTO stuck after region size is larger than 96MB

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、tidb_enable_dist_task='on' and enable global sort
2、run import into

2. What did you expect to see? (Required)

import into can success

3. What did you see instead (Required)

import into blocked

the status of import job is not finished after 2h0m0s (now: 2024-08-12 23:45:55, jobId: 60004, step: importing, status: running)
operatorLogs:
[2024-08-12 21:45:54] ###### start import into
import into user_data1 from 's3://qe-testing/brie/lightning-mhy-100G-csv/sysbench.user_data1.000001*.csv?access-key=xxxxxx&secret-access-key=xxxxxx&endpoint=http%3a%2f%2fks3-cn-beijing-internal.ksyuncs.com&force-path-style=false&region=Beijing&provider=ks' WITH DETACHED,thread=16,skip_rows=1
[2024-08-12 21:45:54] ###### wait for import job to finish
[2024-08-12 23:45:55] ###### wait for import job to finish timeout (2h0m0s)
select id, table_schema, table_name, step, status,parameters from mysql.tidb_import_jobs where start_time >= '2024-08-12 21:45:54' and table_schema = 'importtest1' and table_name = 'user_data1'
jobId: 60004, table_schema: importtest1, table_name: user_data1, step: importing, status: running

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v8.3.0-alpha
Edition: Community
Git Commit Hash: 6df7aba
Git Branch: heads/refs/tags/v8.3.0-alpha
UTC Build Time: 2024-08-10 11:52:15
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-08-12T21:45:47.369+0800

[Lily2025]

/assign D3Hunter
/severity critical

[D3Hunter]

duplicate with #51734

[lance6716]

The reason is tikv's default region size is larger than 96MB after tikv/tikv#17311, and we hardcoded some calculations which assume the region size.

So IMPORT INTO generates large tasks and can't allocate the memory for it.

[lance6716]

Mark affects-8.1 because user may manually changed the region parameters in old versions

[lance6716]

close it because cherry-pick to 8.1.2 is done