Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

br: backup may fail for some of tidb-operator deployed clusters #53029

Closed
YuJuncen opened this issue May 6, 2024 · 4 comments
Closed

br: backup may fail for some of tidb-operator deployed clusters #53029

YuJuncen opened this issue May 6, 2024 · 4 comments
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/br This issue is related to BR of TiDB. report/customer Customers have encountered this bug. severity/major type/bug The issue is confirmed as a bug.

Comments

@YuJuncen
Copy link
Contributor

YuJuncen commented May 6, 2024

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Run a cluster with tidb-operator. And then do some scale-out or something for making CoreDNS responses no such host.

2. What did you expect to see? (Required)

BR should be able to retry this error.

3. What did you see instead (Required)

BR directly fails.

4. What is your TiDB version? (Required)

v6.5.3

Note

Though it seems retrying host name not found errors isn't a good enough idea, in some internal network where DNS name registering is asynchronous and in that case, we may need to wait the service name registration finish. The side-effect is that when we trying to backing up from some host manually and we made a typo, we may need to wait for a long time before BR fails.

@BornChanger
Copy link
Contributor

/label affects-6.5

@ti-chi-bot ti-chi-bot bot added the affects-6.5 This bug affects the 6.5.x(LTS) versions. label May 6, 2024
@BornChanger
Copy link
Contributor

/label affects-7.5

@ti-chi-bot ti-chi-bot bot added the affects-7.5 This bug affects the 7.5.x(LTS) versions. label May 6, 2024
@BornChanger
Copy link
Contributor

/label affects-7.1

@ti-chi-bot ti-chi-bot bot added the affects-7.1 This bug affects the 7.1.x(LTS) versions. label May 6, 2024
@jebter jebter added the component/br This issue is related to BR of TiDB. label May 6, 2024
ti-chi-bot bot pushed a commit that referenced this issue May 6, 2024
@YuJuncen
Copy link
Contributor Author

YuJuncen commented May 6, 2024

Note: this should be fixed in master due to the new pdhttp client, where we use the retry.InitialBackoffer, it will retry all kind of errors:

CleanShot 2024-05-06 at 16 50 47@2x

YuJuncen added a commit to YuJuncen/tidb that referenced this issue May 6, 2024
mittalrishabh pushed a commit to mittalrishabh/tidb that referenced this issue May 30, 2024
@ti-chi-bot ti-chi-bot added affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. labels May 30, 2024
@ti-chi-bot ti-chi-bot added the affects-8.1 This bug affects the 8.1.x(LTS) versions. label May 30, 2024
@seiya-annie seiya-annie added the report/customer Customers have encountered this bug. label Jun 3, 2024
@BornChanger BornChanger removed the affects-8.1 This bug affects the 8.1.x(LTS) versions. label Jul 24, 2024
YuJuncen added a commit to YuJuncen/tidb that referenced this issue Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. component/br This issue is related to BR of TiDB. report/customer Customers have encountered this bug. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

No branches or pull requests

5 participants