cmd/go: document that module names without dots are reserved

[mvdan]

Intuitively, most Go developers publish Go code under import paths like foo.com/bar, not foo/bar. Often, this is just to make them reachable on the internet.

However, some have avoided the .tld entirely when working with local-only GOPATHS and modules. And this mostly works, as long as you're careful to not clash with an existing standard library package.

For example, I recall a recent issue in this tracker (which I can't find now!) where their GOPATH build broke when upgrading the Go version, as the root path element was plugin, which had been added to the standard library since.

An instance of this discussion happened last September, as pointed out by @bcmills. Quoting @rsc:

We want to allow experimentation, so we're not categorically cutting off all module statements in all contexts without dots, but anything that has to get resolved through downloading needs a dot and more generally should have a fully qualified domain name. That's how we've carved the name space between standard library and external things. I understand that pedantically the RFCs don't require dots but that's the separation we've used from very very early on with goinstall and later go get.

However, this separation isn't well documented anywhere. Even if we consider it as a warning or recommendation more than a strict rule, I think we should still mention it somewhere.

I also think we should encourage the use of .tld even if the module is for local use only, because of potential breakages with future Go builds. Perhaps a suggestion can be made, like .local.

Another advantage of strongly encouraging the use of .tld is that it's much easier to tell if a package is from the standard library or not. Without the separation, one would have to keep a list of all standard library package paths, which easily gets out of date. Or run go list std, which adds work.

Note that this doesn't need to change how cmd/go behaves; it's fine if a module name without a dot happens to build today.

[mvdan]

The title contains "modules", but I assume this would apply to the GOPATH mode too. That is, $GOPATH/src/foo/bar would be discouraged just like module foo/bar.

[andybons]

@bcmills @jayconrod

[mvdan]

Small detail I forgot - it's a dot in the first path element. foo.com/bar is fine as a non-reserved path, but foo/bar.baz technically isn't.

[mvdan]

Another small detail. It's not correct to say "all import paths without dots are reserved by the standard library", because there are a few counter-examples. Use go list foo.go, and you'll see command-line-arguments. Or do go build -x foo.go, and you'll see [...]/compile -p main [...].

Both main and command-line-arguments are reserved, but they don't belong to the standard library. Perhaps it's correct to say that they belong to ad-hoc packages, those constructed via lists of Go files alone.

I'm also a bit surprised that the two commands above give different package paths. I think that's counter-intuitive, but maybe there's a good reason for it.

[bcmills]

Import paths without dots are reserved for the standard library and toolchain.

command-line-arguments and main are both synthesized by the toolchain: if you try to import them from some other package you're gonna have a bad time.

[mvdan]

Thanks, I like that wording. It's clear in my mind now. I'll have a look at sending a CL before the freeze is over :)

[mvdan]

I'd like to work on this this cycle. Any thoughts on where such a piece of documentation could go? I'm thinking either go help modules or https://golang.org/ref/mod. I imagine the latter might be best, because we don't want go help modules to become too long for terminals.

[jayconrod]

https://golang.org/ref/mod would be best. Ideally go help modules will be significantly shortened in 1.16, and it refer to that, plus some introductory docs.

[mvdan]

A separate but still related issue: tinygo is a different implementation of Go, and it adds its own standard library packages, much like syscall/js exists for gc's port to js/wasm. For example, they add packages like machine and device/arm, which I think is entirely reasonable. When one builds with tinygo, tinygo is what owns the standard library and toolchain, so it's within its right to include these extra packages.

However, go mod tidy gets upset, as it's not aware of those packages:

github.com/tinygo-org/tinygo/src/device/arm imports
  runtime/volatile: package runtime/volatile is not in GOROOT (/home/mvdan/tip/src/runtime/volatile)
github.com/tinygo-org/tinygo/src/device/nxp imports
  device/arm: package device/arm is not in GOROOT (/home/mvdan/tip/src/device/arm)
github.com/tinygo-org/tinygo/src/examples/adc imports
  machine: package machine is not in GOROOT (/home/mvdan/tip/src/machine)

I think, to allow modules to play nicer with alternative Go implementations, we should allow it to ignore reserved package paths that it doesn't know about.

[bcmills]

@mvdan, machine and device are reserved for the Go standard library, not for other related standard libraries. TinyGo should arguably be using paths like tinygo.org/machine and tinygo.org/device/arm rather than machine and device/arm, and at that point they could provide dummy or polyfill packages that could be fetched by go mod tidy when using the regular go tool.

[mvdan]

machine and device are not reserved

Hm, doesn't that contradict what we said earlier in the thread?

Import paths without dots are reserved for the standard library and toolchain.

[bcmills]

Ah, yeah. I see what you mean. They are reserved for the Go standard library — not for adjacent projects.

(Edited my reply above.)

[aykevl]

Hi all! Author of TinyGo here. You might want to read my previous comment on the issue: #43079 (comment)

I am well aware of this issue and I like to change it, but it will be a major breaking change so I'd like to provide a soft transition path. The proposal in #34055 would be a big help to allow that to happen, and luckily it appears to lean towards being accepted. These reserved paths made sense when I started the project but especially now with Go modules it's becoming a compatibility issue (something which I've wanted to avoid).

EDIT: although I should add that it's still important that if you put "machine" or "tinygo.org/machine", it has to be the local path. go list (when called from TinyGo with the custom GOROOT) should not try to go through the Go proxy to resolve them. But this appears to be the case, so that's good.

[mvdan]

An upcoming change in go mod tidy also seems to align with this documentation change: https://go-review.googlesource.com/c/go/+/298749/

[gopherbot]

Change https://golang.org/cl/359594 mentions this issue: _content/ref/mod: document dotless module paths

[mvdan]

$ go list -e -f '{{.Standard}} -- {{.Error}}' future/std
false -- package future/std is not in GOROOT (/home/mvdan/tip/src/future/std)

I wonder if Standard should be true for all import paths without dots, even those which are missing packages, as they are technically reserved for the standard library - even if they are not part of it today.

I ran into this today as I assumed Standard to be true for arena - but one of the Go versions I was testing with was 1.19, where the package doesn't exist yet, so... it was false there.

[bcmills]

I don't think we should bother setting Standard for packages that don't exist. Despite the names being reserved, some user code does use dotless package paths, and I would rather not explicitly set a field we're not sure about for about a package we don't know anything about.

[mvdan]

Yeah, that's fair. I was a bit on the fence about that, and I guess I can implement the "reserved" logic myself if I want to.