k3s: package wg-add.sh script

[simon-weber]

Describe the bug
k3s provides a wg-add.sh script that is used when --flannel-backend=wireguard is passed to the server. Since it's not included in the k3s package, k3s errors out during startup when wireguard support is enabled.

I managed to get it working by downloading and packaging it like this:

pkgs.runCommandLocal "my-script.sh" {
  script = ./wg-add.sh;
  nativeBuildInputs = [ pkgs.makeWrapper ];
} ''
  makeWrapper $script $out/bin/wg-add.sh \
  --prefix PATH : ${lib.makeBinPath [ pkgs.bash pkgs.kmod ]}

And then added it to my nixos config via an overlay:

systemd.services.k3s.path = [pkgs.bash pkgs.wireguard pkgs.wireguard-tools pkgs.wgAdd];

I'm not actually sure which (if any) of those dependencies are necessary -- my first guess worked, so I stopped messing with it 😁

See also #98090, which is a similar script (I copied the issue template from there).

Notify maintainers
@euank

[stale]

I marked this as stale due to inactivity. → More info

[simon-weber]

Still an issue, I think.

[flokli]

Seems flannel-io/flannel#1230 got resurrected, which might obsolete this issue.

[genofire]

must still package correct for nixos ...

Mar 31 12:58:35 pavo k3s[3531044]: time="2022-03-31T12:58:35+02:00" level=fatal msg="flannel exited: failed to run command: wg genkey | tee /var/lib/rancher/k3s/agent/etc/flannel/privatekey | wg pubkey Err: exec: \"sh\": executable file not found in $PATH Output: "

[flokli]

cc @Mic92

[euank]

The upstream PR for native flannel wireguard (linked above) is merged and released, woo!

However, k3s is still not quite using it. The upstream k3s setup still uses wg-add.sh + the flannel "extension" type, not the flannel "wireguard" type: https://github.com/k3s-io/k3s/blob/313aaca547f030752788dce696fdf8c9568bc035/pkg/agent/flannel/setup.go#L69-L76

The main downside this has is dual-stack support: k3s-io/k3s#4364

I think we have two options for this issue:

1. Wait on upstream k3s (and possibly help upstream) to use the native flannel wireguard setup, which is better
2. Package wg-add.sh now, and understand that it has no dual-stack support and may need to be updated soon with upstream changes

Given my reading of k3s-io/k3s#4365, it sounds like switching between the wg-add.sh extension setup and the native wireguard support will be a breaking change, and it also sounds like it's pretty imminent, so maybe it's easier to just wait a little longer on that.

[euank]

Upstream k3s is updating to the native flannel wireguard backend (k3s-io/k3s#5391)
There hasn't been a release with it yet, but it's been backported to all the supported release branches, and seems like it should be released pretty soon.

This new native flannel wireguard backend doesn't require the wg-add.sh script, and from my testing will just work with the current packaging.

At this point, my preference is for us to not package wg-add.sh for nixpkgs (since it'll be deprecated and in general worse starting with the next k3s release), and to mark this issue as resolved after we update to one of the upcoming k3s releases that have --flannel-backend=wireguard-native.

[euank]

As of k3s 1.23.6 (updated to here), wireguard can be used with the --flannel-backend=wireguard-native flag. This doesn't use the wg-add.sh script, and I've verified the nixpkgs k3s package functions correctly with that backend.

Closing this issue as fixed by using the native backend instead; please do drop a word if you run into any issues there!

[flokli]

SGTM 👍