unable to go run / go install directly

[flokli]

Describe the bug

I tried to execute the jwx binaries (from cmd/jwx) directly, through go run:

❯ go version
go version go1.19 linux/amd64

To Reproduce / Expected behavior

❯ go run github.com/lestrrat-go/jwx/v2/cmd/jwx@latest
go: github.com/lestrrat-go/jwx/v2/cmd/jwx@latest: module github.com/lestrrat-go/jwx/v2@latest found (v2.0.6), but does not contain package github.com/lestrrat-go/jwx/v2/cmd/jwx

Additional context
I think it's due to the funky go.mod file in the cmd/jwx folder.

I read https://go.dev/doc/modules/managing-source, and it looks like it's not really supported to have nested go modules?

I tried removing cmd/jwx/go.{sum,mod}, but then the build fails due to github.com/urfave/cli/v2 not in the top-level go.mod.

What do you think about removing cmd/jwx/go.{sum,mod}, and adding the cmd/jwx dependencies into the root go.{sum,mod}, or if that's not an option, moving the cli tools into a separate repo?

[lestrrat]

TBH I haven't taken a look at this tool for sometime now, but the thing is I kind of got tired of having to sync the cli and the library.

So for starters: I didn't want to include the CLI with the library because pulling the library should not have to include CLI tools.

Then regardless of its location, it needs to be a go module.
Now we are left with jwx the library and jwx the CLI.
jwx the CLI will have to depend on a particular version of jwx the library.
When a new version of the library is released, the CLI will need to be released, and so forth.

I grew tired of this, and at some point I snuck in that go.mod with a replace directive so that the CLI automatically depends on the library it came with.

However, this implies that you would need the following operations to install the library:

git clone https://github.com/lestrrat-go/jwx.git
cd /path/to/jwx/cmd/jwx
go install .

Anyways, that's the current situation.

---

I'm willing to look into better ways, but what I do NOT want to do are:

• Force jwx (library) users to install jwx (CLI) dependencies, and
• Have to maintain a separate project for jwx

So unfortunately I do not want to do either of your suggestions :/

[flokli]

Hm. I can understand you don't want to add the CLI parser to the go.mod of the library, if you want to keep the dependency closure small.

However, the root go.mod also propagates multiple versions of github.com/stretchr/testify to library consumers, so not sure how bad urfave/cli/v2 on top would be… ;-) It shouldn't end up in any consumer binaries, as testify doesn't.

If you want to keep two go modules, you could move things around, have the library and cli be their own independent modules at the repo root, as described here, but I'm not sure if local relative replacements would work for the go run usecase, and due to the structural changes, it would be a major incompatibility breakage for library consumers.

[lestrrat]

TBH I don't see jwx as the main dish of this module -- it's more like "Hey, you could do this if you want" type of tool.
So I'm ok with it being second-class citizen in the jwx family, possibly by instructing users to perform the three step install I described above (of course, it could be a bit more automated), and not allowing go install.

I believe I do owe a better/up-to-date (and correct) explanation in the README, but I'm not sure I want to do anything with jwx the CLI.

Please correct me if I'm wrong and there's a cool way to solve all of my pain points while allowing a clean go install procedure

[flokli]

Understood. But if this is meant more as an example on how to use the library, I'd really suggest to make it self-consistent (without the relative import replacement), so it can be copied around by people adopting it.

The go.mod doesn't need to be always bumped when you do a new release, only if you really actively want to land a fix in the CLI, or add a new feature there.

I checked other projects, https://github.com/philippgille/gokv/blob/master/examples/go.mod for example has a nested go.mod too, and go run github.com/philippgille/gokv/examples@latest seems to build just fine.

[flokli]

Ah, and apparently even with the approach currently used in the repo, you still need to bump go.sum whenever there's a change in the root go.sum, don't you?

[lestrrat]

No, I want the jwx CLI to always be in sync. Otherwise it would confuse me when bug reports arrive. Trust me, I've been doing this OSS maintenance for 20 years.

Ah, and apparently even with the approach currently used in the repo, you still need to bump go.sum whenever there's a change in the root go.sum, don't you?

That can be semi-automated by including a check in the CI (the CI also checks for any new generated content)

[flokli]

I'd argue bumping a go.mod could also be automated by GH actions, but hey, it's your decision. Thanks for exchanging opinions on this.

[lestrrat]

If you are expecting me to write the code and maintain it, then right. It's my decision, and end of story.

But if you have an actual implementation that fulfills what I want, and are willing to help me maintain it, please feel free to enlighten and embarrass me for not being able to pull this off. Again, I'm not against the idea. I'm against doing anything more than minimal work to keep maintaining it.

[flokli]

If the "two repositories" proposal would have been an option, I would have helped with the PR in this repo, and the new structure in the other repo. There could have been some dependabot GH action automation in place that'll open a PR for you whenever you do a new release of the library, keeping the maintenance effort quite low, while still allowing go run/go install.

But yeah, so far all the suggestions were turned down, so I guess we'll just leave it like that.

[lestrrat]

I hear you. I know my requirements are pretty hard, but I was wondering if there was a ingenious way to do it.

One thing that I want to stress is that my requirements basically boil down to ">>I<< don't want to do maintenance". So if you are up to long term commitment, you can always volunteer to maintain jwx the CLI yourself, and my requirements are moot points. So if you ever feel like taking the reigns and maintain it in a separate repo (you could use github.com/jwx-go org) that's an option too. JFYI

[flokli]

Understood. But if this is meant more as an example on how to use the library, I'd really suggest to make it self-consistent (without the relative import replacement), so it can be copied around by people adopting it.

The go.mod doesn't need to be always bumped when you do a new release, only if you really actively want to land a fix in the CLI, or add a new feature there.

I checked other projects, https://github.com/philippgille/gokv/blob/master/examples/go.mod for example has a nested go.mod too, and go run github.com/philippgille/gokv/examples@latest seems to build just fine.

Apparently, #809 did now implement this "independent go.mod file approach", and the new scripts/release.sh bumps the dependency. Thanks!

[lestrrat]

Hmm? So you mean it's not that we had multiple go.mods, but it was the replace directive?
I didn't really intend #809 to solve your problems (in my rudimentary testing it didn't quite work), so am just curious.

And speaking of which, the release.sh script is half-baked because I abandoned the idea of doing it right... I need to fix it
(it's not committing the result of go mod edit before tagging)

[flokli]

The root-level go.mod still seems to be a problem for go run/go install, so no, I was wrong:

❯ go run github.com/lestrrat-go/jwx/v2/cmd/jwx@19cd6e04796a754a5c74964e64ba1df62d172105
go: downloading github.com/lestrrat-go/jwx/v2 v2.0.4-0.20220829235828-19cd6e04796a
go: github.com/lestrrat-go/jwx/v2/cmd/jwx@19cd6e04796a754a5c74964e64ba1df62d172105: module github.com/lestrrat-go/jwx/v2@19cd6e04796a754a5c74964e64ba1df62d172105 found (v2.0.4-0.20220829235828-19cd6e04796a), but does not contain package github.com/lestrrat-go/jwx/v2/cmd/jwx

However, the CLI now is self-contained - it contains a pin of github.com/lestrrat-go/jwx/v2, meaning if I move this directory anywhere else on my hard drive, I can still go build it.

[sjwl]

I was able to fix this on my fork by simply:

1. deleting cmd/jwx/go.mod and cmd/jwx/go.sum
2. global replace github.com/lestrrat-go/jwx with github.com/sjwl/jwx in all *.go and go.mod files <== THIS IS ONLY NEEDED FOR MY FORK
3. running go mod tidy

Such that this command worked for me

% go install github.com/sjwl/jwx/v2/cmd/[email protected]
go: downloading github.com/sjwl/jwx/v2 v2.0.8-sjwl-3
% jwx
NAME:
   jwx - Tools for various JWE/JWK/JWS/JWT operations

USAGE:

% go version
go version go1.20 darwin/arm64

this golang issue seems related: golang/go#34055