Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why does connection to localhost fail on local ? #7429

Closed
manuraj17 opened this issue Jul 21, 2024 · 28 comments
Closed

Why does connection to localhost fail on local ? #7429

manuraj17 opened this issue Jul 21, 2024 · 28 comments
Assignees
Labels
Area: Client Includes Channel/Subchannel/Streams, Connectivity States, RPC Retries, Dial/Call Options and more. stale Status: Requires Reporter Clarification Type: Bug

Comments

@manuraj17
Copy link

manuraj17 commented Jul 21, 2024

I faced this issue when I was testing out a simple application; Not able to put a finger on what exactly am missing.

My client code is returning with this error

could not greet: rpc error: code = DeadlineExceeded desc = context deadline exceeded

I have my server running as

const (
	port = ":50051"
)

// server is used to implement helloworld.GreeterServer.
type server struct {
	helloworld.UnimplementedGreeterServer
}

// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *helloworld.HelloRequest) (*helloworld.HelloReply, error) {
	log.Printf("Received: %v\n", in.GetName())
	return &helloworld.HelloReply{Message: "Hello " + in.Name}, nil
}

func main() {
	lis, err := net.Listen("tcp", port)
	if err != nil {
		log.Fatalf("failed to listen: %v", err)
	}
	s := grpc.NewServer()
	helloworld.RegisterGreeterServer(s, &server{})
	log.Printf("server starting at %v\n", lis.Addr())
	if err := s.Serve(lis); err != nil {
		log.Fatalf("failed to serve: %v", err)
	}
}

Client connecting to localhost as

const (
	address = "localhost:50051"
	// address     = "[::1]:50051"

	// address     = "127.0.0.1:50051"
	defaultName = "world"
)

func main() {
	// Set up a connection to the server.
	conn, err := grpc.NewClient(address, grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		log.Fatalf("did not connect: %v", err)
	}
	defer conn.Close()
	c := helloworld.NewGreeterClient(conn)

	// Contact the server and print out its response.
	name := defaultName
	if len(os.Args) > 1 {
		name = os.Args[1]
	}
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	defer cancel()
	r, err := c.SayHello(ctx, &helloworld.HelloRequest{Name: name})
	if err != nil {
		log.Fatalf("could not greet: %v", err)
	}
	log.Printf("Greeting: %s", r.GetMessage())
}

This code fails.

But when I connect with either of

	address     = "[::1]:50051"
	address     = "127.0.0.1:50051"

It works. I am trying to understand what am missing here. Can anyone help me understand? TIA!

Full application code available here

UPDATE (More Info):

Some more information
Version

google.golang.org/grpc v1.65.0

I am able to connect if I am using 127.0.0.0 and [::1]
The issue specifically seems to occur when localhost is used.

etc/hosts is below

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost
127.0.0.1       mail.local
255.255.255.255 broadcasthost
::1             localhost

I am running the server at add = :50051 and initialising like

	lis, err := net.Listen("tcp", port)

It works with Dial

When I use Dial it works

	conn, err := grpc.DialContext(ctx, addr,
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithBlock(),
	)

I have tried all variations to connect without using Dial, and everything except localhost works. The resolution of localhost also seems to be working correctly. I get the following logs

 localhost resolves to: [::1 127.0.0.1]

NOTE: IPV6 gets resolved first.

Though I am not able to figure what failed here. I will have end up not using localhost for now. Though, curious what is happening here.

@manuraj17 manuraj17 changed the title Why am I not able to connect to localhost on local ? Why does connection to localhost fail on local ? Jul 21, 2024
@purnesh42H purnesh42H self-assigned this Jul 21, 2024
@purnesh42H
Copy link
Contributor

purnesh42H commented Jul 22, 2024

@manuraj17 I was also able to repro on my local with localhost. It looks like when using localhost with grpc.NewClient address resolution takes more than 5s which is why we get Deadline exceeded in the example. If you try with 10s timeout, rpc will succeed.

Will discuss with team and update on what is the best course of action here. Thanks once again for catching this.

@manuraj17
Copy link
Author

@purnesh42H Thanks for checking on this. Will wait for the update 👍🏽

@purnesh42H
Copy link
Contributor

Client logs with 1 second timeout when using localhost with grpc.NewClient

2024/07/23 21:08:08 INFO: [core] original dial target is: "localhost:50051"
2024/07/23 21:08:08 INFO: [core] [Channel #1]Channel created
2024/07/23 21:08:08 INFO: [core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"dns", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/localhost:50051", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2024/07/23 21:08:08 INFO: [core] [Channel #1]Channel authority set to "localhost:50051"
2024/07/23 21:08:08 INFO: [core] [Channel #1]Channel exiting idle mode
2024/07/23 21:08:13 could not greet: rpc error: code = DeadlineExceeded desc = context deadline exceeded
exit status 1

@manuraj17
Copy link
Author

@purnesh42H
Could share what changed between from being able to reproduce to now? Just curious on what happened earlier and what is happening now.

@purnesh42H
Copy link
Contributor

purnesh42H commented Jul 25, 2024

@manuraj17 the problem seems to be when using grpc.NewClient. The address resolution is taking longer when using localhost and that's why you experience timeout. If you try the same example with 10s timeout, rpc will succeed. We are still debugging the root cause.

@purnesh42H
Copy link
Contributor

purnesh42H commented Jul 25, 2024

this call is blocked for longer when using localhost

if err := cc.waitForResolvedAddrs(ctx); err != nil {

@purnesh42H
Copy link
Contributor

@manuraj17 I am not able to repro this anymore and localhost seems to work with existing example. For now, we will keep this open but won't investigate further since its not a widespread issue. Feel free to add any new information here and we can re-evaluate the priority

@dfawley
Copy link
Member

dfawley commented Jul 29, 2024

Are you still able to reproduce this? Can you run with full debug logging enabled and include the logs?

https://github.com/grpc/grpc-go#how-to-turn-on-logging

@manuraj17
Copy link
Author

@dfawley Will check and update; Though any idea on what is happening in this scenario?
@purnesh42H was able to identify and figure it's a resolution issue. Curious as to what information you folks have regarding this.

@dfawley
Copy link
Member

dfawley commented Jul 30, 2024

@dfawley Will check and update; Though any idea on what is happening in this scenario?

No idea. If your machine can't resolve localhost, or properly connect to an address it returns, then that sounds like a configuration issue to me.

@manuraj17
Copy link
Author

No idea. If your machine can't resolve localhost, or properly connect to an address it returns, then that sounds like a configuration issue to me.

If it's my machine issue wouldn't it fail for

  • Dial method
  • every other app that is running in localhost

Not sure what what you are on to here coz it still doesn't clariy how @purnesh42H was able to reproduce it for a while; did he have some configuration issue?

@purnesh42H
Copy link
Contributor

@manuraj17 can you pull the latest and then try running with grpc.NewClient and localhost with the logging turned on https://github.com/grpc/grpc-go#how-to-turn-on-logging? Since, this is not always reproducible, we need more information to verify if its a library or configuration issue.

To clarify, even for me the resolution was just taking a bit longer with localhost but that's not happening anymore

Copy link

github-actions bot commented Aug 6, 2024

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

@github-actions github-actions bot added the stale label Aug 6, 2024
@manuraj17
Copy link
Author

manuraj17 commented Aug 10, 2024

@purnesh42H

can you pull the latest

Are you suggesting the latest from the grpc main branch?

@github-actions github-actions bot removed the stale label Aug 10, 2024
@purnesh42H
Copy link
Contributor

Are you suggesting the latest from the grpc main branch?

Yes

Copy link

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

@github-actions github-actions bot added the stale label Aug 16, 2024
@ciaphas01
Copy link

ciaphas01 commented Aug 21, 2024

We ran into this issue running code using the latest grpc-go release as of 8 August 2024. While I unfortunately cannot provide any logs or code, I can say we only started seeing the issue when switching from using Dial to NewClient, as previously commented, and only when all nameservers listed in /etc/resolv.conf were unreachable. (Figuring this part out took a while!)

Similar to the reporter, changing from localhost to 127.0.0.1 for the server and client also avoided the error, even with all invalid/unreachable nameservers in /etc/resolv.conf.

(Note: while the nameservers in /etc/resolv.conf were invalid, we could still e.g. ping localhost and get a response; it was only grpc-go that seemed to suffer any issues (beyond what you'd naturally expect when your DNS config is awry, anyway).)

(Also, localhost was listed in /etc/hosts similar to the reporter.)

@ciaphas01
Copy link

Apologies for the double post, forgot the @purnesh42H and I'm not sure editing mentions into comments actually works lol

@github-actions github-actions bot removed the stale label Aug 21, 2024
@ciaphas01
Copy link

Update: curiously, the problem seems to stop manifesting immediately when the machine is physically disconnected from all networks. Perhaps somewhere there's code that's causing this DNS record lookup to get skipped if all network interfaces are down, allowing the localhost entry in /etc/hosts to take over and the health check to proceed?

Copy link

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

@github-actions github-actions bot added the stale label Aug 27, 2024
@purnesh42H
Copy link
Contributor

In new grpc.NewClient the default name resolution is dns as opposed to passthrough in deprecated grpc.Dial. Try NewClient with passthrough as the resolver and it should behave the same.

This type of issue with dns resolution is more likely related to system configuration rather than a problem with the gRPC library itself as it relies on the underlying system for DNS resolution. If the system takes time to resolve localhost, this delay will be reflected in gRPC client behavior as well.

@eshitachandwani eshitachandwani added the Area: Client Includes Channel/Subchannel/Streams, Connectivity States, RPC Retries, Dial/Call Options and more. label Sep 4, 2024
@ClaudioAlbertin
Copy link

We are seeing the same issue, and indeed using passthrough works. But even with passthrough net.Dial will eventually perform the DNS resolution, and this seemingly works just fine, so I think there is an open question about what the grpc-go DNS resolver does differently to provoke this issue (both use net.Resolver eventually).
I'm not sure it should just be written off to system misconfiguration. Even if that was a factor, the people encountering this seem to have no other issues with DNS resolution (we don't) and are slowed down while trying to switch to NewClient with the recommended default resolution scheme. It'd be great to at least find a more satisfying explanation.

@arjan-bal
Copy link
Contributor

@ClaudioAlbertin can you provide the gRPC logs when the DNS resolver fails?

@purnesh42H
Copy link
Contributor

@ClaudioAlbertin can you provide the gRPC logs when the DNS resolver fails?

@ClaudioAlbertin ping on this

@ClaudioAlbertin
Copy link

Unfortunately I can only reliably reproduce the issue on our CI server in some tests with very noisy output, but I'll see what logs I can get out of it.

It may be worth noting for @manuraj17 that the grpc-go DNS resolver also performs a lookup for a TXT record (in the hope of finding a Service Config), which does not happen if an IP address or the passthrough resolver is used. This lookup contacts an actual DNS server, whereas resolving localhost otherwise just involves reading local files. This could contribute to the RPC taking much longer with grpc.NewClient vs grpc.Dial. You can test whether this is your issue by disabling service configs and by extension the TXT lookup using the grpc.WithDisableServiceConfig() dial option. This did not fix the issue for us, however.

@purnesh42H purnesh42H removed their assignment Sep 27, 2024
@arjan-bal
Copy link
Contributor

arjan-bal commented Sep 27, 2024

@manuraj17 as suggested by @ClaudioAlbertin, can you try disabling service config using the grpc.WithDisableServiceConfig() dial option and see if localhost resolution works with grpc.NewClient? If it does, it means that all the nameservers present in the /etc/resolv.conf are un-reachable.

@arjan-bal
Copy link
Contributor

arjan-bal commented Sep 27, 2024

@ClaudioAlbertin to be sure, are you seeing the issue with localhost or another hostname?

Also, do you happen to be using an http proxy? If yes, there is a known issue (#7556) which causes DNS resolution to happen on the client instead of the proxy which could cause timeouts if the client can't reach the DNS itself.

Copy link

github-actions bot commented Oct 3, 2024

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Client Includes Channel/Subchannel/Streams, Connectivity States, RPC Retries, Dial/Call Options and more. stale Status: Requires Reporter Clarification Type: Bug
Projects
None yet
Development

No branches or pull requests

7 participants