lightning failed with error “tidb lightning encountered error: failed to create remote duplicate kv stream: type:UNKNOWN message:"invalid store ID 0, not found” when kill pd leader

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run lightning
2、kill pd leader

2. What did you expect to see? (Required)

lightning can cuccess

3. What did you see instead (Required)

lightning failed with error “tidb lightning encountered error: failed to create remote duplicate kv stream: type:UNKNOWN message:"invalid store ID 0, not found” when kill pd leader

Run Lightning failed.
cmd start at 2024-01-15 23:06:42
cmd failed at 2024-01-15 23:24:34
stdout:
Verbose debug logs will be written to /tmp/lightning.log.2024-01-15T15.06.42Z

+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| # | CHECK ITEM | TYPE | PASSED |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 1 | Source data files size is proper | performance | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 2 | the checkpoints are valid | critical | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 3 | table schemas are valid | critical | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 4 | Cluster version check passed | critical | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 5 | Lightning has the correct storage permission | critical | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 6 | local disk resources are rich, estimate sorted data size 58.08GiB, local available is 3.263TiB | critical | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 7 | The storage space is rich, which TiKV/Tiflash is 5.398TiB/0B. The estimated storage space is 174.2GiB/0B. | performance | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 8 | Cluster doesn't have too many empty regions | performance | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 9 | Cluster region distribution is balanced | performance | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+
| 10 | no CDC or PiTR task found | critical | true |
+----+-----------------------------------------------------------------------------------------------------------+-------------+--------+

tidb lightning encountered error: failed to create remote duplicate kv stream: type:UNKNOWN message:"invalid store ID 0, not found"

ha-test-lightning-tps-5970173-1-664.tar.gz

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: 3a3237e
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-01-15 11:42:58
GoVersion: go1.21.5
Race Enabled: false
Check Table Before Drop: false
Store: unistore

./tidb-lightning -V
Release Version: v8.0.0-alpha
Git Commit Hash: 3a3237e
Git Branch: heads/refs/tags/v8.0.0-alpha
Go Version: go1.21.5
UTC Build Time: 2024-01-15 11:39:25
Race Enabled: false

[Lily2025]

/severity major
/assign lance6716