net: pure Go DNS resolver hangs with ipv6 link-local address on macOS

[mattrobenolt]

What version of Go are you using (go version)?

$ go version
go version go1.18 darwin/amd64

This is also reproduced against go1.17.8, so doesn't appear to be new to go1.18.

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

macOS 12.3, Intel (amd64)

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/xxx/Library/Caches/go-build"
GOENV="/Users/xxx/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/xxx/go/pkg/mod"
GOOS="darwin"
GOPATH="/Users/xxx/go"
GOPROXY="https://proxy.golang.org/,direct"
GOROOT="/usr/local/Cellar/go/1.18/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.18/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/pb/fk1_tmbn1nq32bgpd6njflcw0000gn/T/go-build3259567292=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Making a DNS lookup against an ipv6 link-local address hangs and fails on macOS when using thenative (non-CGO) resolver.

This appears to manifest within the non-CGO based DNS resolver, but I believe it's more generally applicable to the network stack itself, it's just easier to reproduce through DNS.

The simplest reproduction we could come up with was this compiled with CGO_ENABLED=0:

package main

import (
	"context"
	"fmt"
	"net"
	"os"
	"time"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	ips, err := net.DefaultResolver.LookupIP(ctx, "ip", "google.com")
	if err != nil {
		fmt.Fprintf(os.Stderr, "Could not get IPs: %v\n", err)
		os.Exit(1)
	}
	for _, ip := range ips {
		fmt.Printf("google.com. IN A %s\n", ip.String())
	}
}

Paired with an /etc/resolv.conf of:

nameserver fe80::a04e:cfff:fe2f:ad64%en0
nameserver 10.0.0.1

I've deduced this down to explicitly any link-local address (fe80::*) and not generically to ipv6. Other ipv6 addresses, both public and private network appear fine.

This behavior is especially concerning because it seems to cascade into larger macOS failures. All DNS seems to start failing on the whole system requiring ultimately a reboot to restore stability.

Given this, I believe this is also a bug within macOS that Go happens to be exploiting somehow.

I personally don't have a macOS machine to iterate on this, but we've deduced this both with customers running our CLI (https://github.com/planetscale/cli) as well as employees internally.

After a bunch of deducing, we've narrowed it down to the link-local address.

Again, when compiled with CGO_ENABLED=1, everything works correctly and is fine. The issue only occurs when compiled with CGO_ENABLED=0, triggering the pure Go DNS resolution path.

We originally thought this was related to arm64 (M1 Macs), but was able to finally reproduce it on Intel.

What did you expect to see?

Fast DNS resolution

What did you see instead?

Program hangs, eventually times out, in extreme cases, causing entire OS to become unstable.

For additional context: planetscale/discussion#181

[heschi]

cc @ianlancetaylor @neild

[dannyfallon]

We reproduced this with Go 1.18.4 and Go 1.16.5 on M1 and Intel machines running OS X 12.3-12.5 with two different Go applications. All affected users had the fe80::* IPv6 DNS address in their nameservers. The behaviour was always what the OP described as the extreme circumstances - hung process that ignores signals, eventual network disconnection and then a frozen machine needing forced reboots.

Changing DNS servers on the network connection to drop the fe80::* address resolved it. As a mitigation we also started to build OS X binaries on OS X machines with Cgo enabled to force the use of the Cgo resolver. We lost several days debugging this ourselves and when I came to report this I found this issue already. Is there any update or ETA on when investigation might take place? This seems fairly critical.

[apparentlymart]

One possible way to confound a reproduction on darwin_amd64 is if a program is being built for both darwin_amd64 and darwin_arm64 on a build worker whose native toolchain is the darwin_arm64 one. In that situation the default for CGO_ENABLED is different between the two targets: darwin_arm64 is considered to be a cross-compile and therefore CGO_ENABLED defaults to off.

I wrote some more about this in a comment on the general issue about the pure Go resolver on macOS. The short version is that if you want to reproduce it reliably it's important to explicitly set CGO_ENABLED=0 or CGO_ENABLED=1 consistently across builds for both architectures, and not rely on the default. That will then ensure that both of your builds are using the same resolver implementation.

[silvestriluca]

I'm experiencing this issue too using Terraform (which is based on GoLang).
Very very bad bug. All my network stack crashes and I'm forced to:

• force close the apps that are using nework (slack, browser, terminal...)
• Reboot mac

Check hashicorp/terraform#31467 (comment)

It is worth to get in touch with macOS network devs, because even if there is some issue in go implementation, network stack shouldn't crash so dramatically, to the point that reboot is the only option.

[heschi]

cc @golang/darwin

[zyxkad]

I'm using mac M1 Ventura 13.5 (22G74), the local dns issue is actually affected my nslookup command, but it didn't affect the go program you provide with whether CGO enabled or not.

$ go version
go version go1.20.4 darwin/arm64

go env

GO111MODULE=""
GOARCH="arm64"
GOBIN="/Users/ckpn/Mine/projects/golang/bin"
GOCACHE="/Users/ckpn/Library/Caches/go-build"
GOENV="/Users/ckpn/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/ckpn/Mine/projects/golang/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/ckpn/Mine/projects/golang"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.20.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.4/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.4"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="0"
GOMOD="/Users/ckpn/Mine/projects/golang/test/dns/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/3g/rxdhvbj96lv68y35t2cvghkr0000gn/T/go-build2449011362=/tmp/go-build -gno-record-gcc-switches -fno-common"

[lukasmalkmus]

I'm seeing the same behaviour: go version go1.22.2 darwin/arm64!