Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1.2 Miner&Worker using Http should have some problems #4981

Closed
Xib1uvXi opened this issue Nov 24, 2020 · 0 comments · Fixed by #5015
Closed

v1.2 Miner&Worker using Http should have some problems #4981

Xib1uvXi opened this issue Nov 24, 2020 · 0 comments · Fixed by #5015

Comments

@Xib1uvXi
Copy link

Describe the bug

In our big cluster, we can often find in the log that the worker cannot connect to the miner

2020-11-24T14:22:11.520+0800    ^[[33mWARN^[[0m auth    auth/handler.go:39      JWT Verification failed (originating from 10.0.8.101:34388): RPC client error: sendRequest failed: Post "http://10.0.8.101:2345/rpc/v0": dial tcp 10.0.8.101:2345: connect: connection refused
2020-11-24T14:26:40.953+0800    ^[[33mWARN^[[0m auth    auth/handler.go:39      JWT Verification failed (originating from 10.0.8.101:46968): RPC client error: sendRequest failed: Post "http://10.0.8.101:2345/rpc/v0": dial tcp 10.0.8.101:2345: connect: connection refused
2020-11-24T14:26:40.953+0800    ^[[33mWARN^[[0m auth    auth/handler.go:39      JWT Verification failed (originating from 10.0.8.101:47060): RPC client error: sendRequest failed: Post "http://10.0.8.101:2345/rpc/v0": dial tcp 10.0.8.101:2345: connect: connection refused
2020-11-24T14:26:43.926+0800    ^[[33mWARN^[[0m auth    auth/handler.go:39      JWT Verification failed (originating from 10.0.8.101:48918): RPC client error: sendRequest failed: Post "http://10.0.8.101:2345/rpc/v0": dial tcp 10.0.8.101:2345: connect: connection refused

We have checked the network connection between worker and miner, The network connection is no problem

# in our worker 
telnet 10.0.8.101 2345

# output
Trying 10.0.8.101...
Connected to 10.0.8.101.
Escape character is '^]'.
^C
Connection closed by foreign host.

Then I realized that the TCP-Backlog queue may be full

# in our miner
   netstat -s | grep LISTEN
    193 SYNs to LISTEN sockets dropped
# in our miner
   netstat -s | grep TCPBacklogDrop
    TCPBacklogDrop: 81

At the same time, I also found that there will be a large number of TIME_WAIT

   netstat -na |grep TIME_WAIT | wc -l
89542

I review the code of go-jsonrpc about httpClient, I found that connection pool is not used, Why not use go http Transport, I think it should be able to improve the problem of excessive http connections

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant