Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: single node CN shutdown now take quite long #20731

Open
1 task done
fengttt opened this issue Dec 12, 2024 · 0 comments
Open
1 task done

[Bug]: single node CN shutdown now take quite long #20731

fengttt opened this issue Dec 12, 2024 · 0 comments
Assignees
Labels
kind/bug Something isn't working severity/s1 High impact: Logical errors or data errors that must occur
Milestone

Comments

@fengttt
Copy link
Contributor

fengttt commented Dec 12, 2024

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

2.0-dev, main

Commit ID

42141a8

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

Hit ^C to stop a one node deployment. It takes quite sometime to shutdown. Some sample log. At the end the system WILL correctly shutdown, but given that a single node instance has really nothing to migrate, it should be faster.

{"level":"ERROR","time":"2024/12/11 17:15:36.252835 -0800","name":"cn-service.lock-client","caller":"morpc/backend.go:251","msg":"connect to remote failed","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","remote":"127.0.0.1:19002","backend-id":"0193b86f-83bb-795a-bcd8-047caf5c8748"}
{"level":"ERROR","time":"2024/12/11 17:15:36.252853 -0800","name":"cn-service.lock-client.lock-client","caller":"morpc/client.go:515","msg":"create backend failed","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","backend":"127.0.0.1:19002","error":"can not connect to remote backend, dial tcp4 127.0.0.1:19002: connect: connection refused"}
{"level":"ERROR","time":"2024/12/11 17:15:36.252880 -0800","name":"cn-service.lockservice","caller":"lockservice/lock_table_keeper.go:197","msg":"failed to keep lock table bind","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","error":"can not connect to remote backend, dial tcp4 127.0.0.1:19002: connect: connection refused"}
{"level":"ERROR","time":"2024/12/11 17:15:37.254325 -0800","name":"cn-service.lock-client","caller":"morpc/backend.go:882","msg":"init remote connection failed, retry later","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","remote":"127.0.0.1:19002","backend-id":"0193b86f-87a5-7785-8ec0-2baabf4742f8","can-retry":false,"error":"dial tcp4 127.0.0.1:19002: connect: connection refused"}
...

{"level":"INFO","time":"2024/12/11 17:15:43.468137 -0800","name":"cn-service.lockservice","caller":"lockservice/deadlock.go:130","msg":"dead lock checker","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","cost":"28m25.433718667s"}
{"level":"INFO","time":"2024/12/11 17:15:43.473217 -0800","name":"cn-service.lockservice","caller":"lockservice/lock_table_keeper.go:83","msg":"keep lock table bind task","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","cost":"28m25.437829875s"}
{"level":"INFO","time":"2024/12/11 17:15:43.473246 -0800","name":"cn-service.lockservice","caller":"lockservice/lock_table_keeper.go:103","msg":"keep remote locks task","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","cost":"28m25.437857875s"}
{"level":"INFO","time":"2024/12/11 17:15:43.514240 -0800","name":"cn-service.gossip","caller":"gossip/node.go:195","msg":"leaving gossip cluster","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","node ID":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf"}
{"level":"INFO","time":"2024/12/11 17:15:43.524881 -0800","name":"cn-service","caller":"logutil/action.go:38","msg":"close components completed","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","action":"close","components":"cnservice","cost":"1m52.866069625s"}
{"level":"INFO","time":"2024/12/11 17:15:43.525003 -0800","caller":"mo-service/main.go:471","msg":"Shutdown service complete."}
{"level":"INFO","time":"2024/12/11 17:15:43.799432 -0800","caller":"motrace/trace.go:225","msg":"Shutdown trace complete."}
{"level":"INFO","time":"2024/12/11 17:15:43.803384 -0800","caller":"mo-service/main.go:126","msg":"Shutdown complete"}

Expected Behavior

No response

Steps to Reproduce

Start single node instance, hit ^C to shutdown.

Additional information

No response

@fengttt fengttt added kind/bug Something isn't working needs-triage severity/s1 High impact: Logical errors or data errors that must occur labels Dec 12, 2024
@fengttt fengttt added this to the 2.1.0 milestone Dec 12, 2024
@sukki37 sukki37 modified the milestones: 2.1.0, 2.0.2 Dec 13, 2024
@aressu1985 aressu1985 modified the milestones: 2.0.2, 2.1.0 Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working severity/s1 High impact: Logical errors or data errors that must occur
Projects
None yet
Development

No branches or pull requests

5 participants