Can Systemd 247 "credentials" logic solve secret management for us?

[davidak]

Issue description

Configuring credentials in NixOS is an unsolved problem, since saving them in the nix store is a bad idea (it is world-readable on a system).

A solution would be to store it outside the nix store, as state on disk (like linux user passwords), in best case with permissions, so only the service that needs it can access it.

Maybe the new "credentials" logic from Systemd 247 can solve it for us?

A new "credentials" logic has been added to system services. This is
a simple mechanism to pass privileged data to services in a safe and
secure way. It's supposed to be used to pass per-service secret data
such as passwords or cryptographic keys but also associated less
private information such as user names, certificates, and similar to
system services. Each credential is identified by a short user-chosen
name and may contain arbitrary binary data. Two new unit file
settings have been added: SetCredential= and LoadCredential=. The
former allows setting a credential to a literal string, the latter
sets a credential to the contents of a file (or data read from a
user-chosen AF_UNIX stream socket). Credentials are passed to the
service via a special credentials directory, one file for each
credential. The path to the credentials directory is passed in a new
$CREDENTIALS_DIRECTORY environment variable. Since the credentials
are passed in the file system they may be easily referenced in
ExecStart= command lines too, thus no explicit support for the
credentials logic in daemons is required (though ideally daemons
would look for the bits they need in $CREDENTIALS_DIRECTORY
themselves automatically, if set). The $CREDENTIALS_DIRECTORY is
backed by unswappable memory if privileges allow it, immutable if
privileges allow it, is accessible only to the service's UID, and is
automatically destroyed when the service stops.

Source: https://github.com/systemd/systemd/blob/6706384a89ae0c462e7172588c80667190c4d9e2/NEWS#L320

Update: Turns out this feature was added because of the discussions in NixOS/rfcs#59 (comment). So the answer is probably yes!

cc @flokli @Mic92 @aanderse @arianvp @edolstra @shlevy @d-goldin @globin

Thanks to @poettering for making this possible!

Related:

• Provide options for storing secrets outside the Nix store #24288
• nixos/ldap: unify secrets handling #64951
• systemd: 246.6 -> 247 #102355
• [RFC 0005] Nix encryption rfcs#5
• [RFC 0059]: Systemd Service Secrets rfcs#59
• [RFC 0059]: Systemd Service Secrets rfcs#59 (comment)
• credentials logic to pass privileged data to services systemd/systemd#16568
• RFE: per-service credentials system systemd/systemd#15778

[davidak]

I thought about the issue for 7 minutes.

• When Nix is a package manager, why put your credentials into "packages"? A debian package maintainer wouldn't put his login into the package, so every debian user is logged in as him. Why can't Nix differentiate between packages and configuration?
• When Nix could put configuration files in the nix store with access rights only for the service that needs it (and root for debugging) AND not share those files, the problem would be also solved.

Then the credentials won't have to be state and the configuration is reproducible again.

Related:

• Support private files in the Nix store nix#8

[xaverdh]

also related: #93659

[Mic92]

Also related: Mic92/sops-nix#39 (review)

[happysalada]

I did some testing with a conf akin to the following

        serviceConfig = {
          LoadCredential = "secret_key_base:/run/secrets/unionBlue.secret_key_base";
        };
        environment = {
          SECRET_KEY_BASE = "$CREDENTIALS_DIRECTORY/secret_key_base";
        };

having the /run/secrets being populated with agenix.
Having confirmed that the /run/secrets/unionBlue.secret_key_base was populated, I still get the following error

72670]: union-blue.service: Failed to set up mount namespacing: /run/systemd/unit-root/run/credentials/union-blue.service: No such file or directory
72670]: union-blue.service: Failed at step NAMESPACE spawning /nix/store/sxpb7vbxx1kwprnhsvr2cn7l0vssl9r2-union-0.0.1/bin/union: No such file or directory

I'm not sure how to proceed next. I'm interested in testing if anybody has ideas.

[davidak]

@happysalada you could ask upstream (systemd) how the feature should be used in this case if the documentation is not clear. maybe something is still missing which they can add

[happysalada]

submitted systemd/systemd#19604
Will update when I have a working example

[happysalada]

just to give a summary of what happened.
It seems that the loadCredential feature doesn't work with systemd 247, but people have reported that it's fixed with 248. There is a PR pending that updates to systemd 248 #123476

the systemd people advise not using environment variables for secrets at all. Meaning that any application written to get secrets (database password for example) from environment variables should be modified to read it from a file instead. To me it seems that this is going to need some work with upstream packages to change the behavior. It seems that some settings like db passwords are most often read from env vars, while things from cert files is most often a file. I'm curious to see how upstream maintainers will react to those proposed changes.

Regarding usage it's a little odd to me, but it seems the way to use this will be pass a file coming from run/secrets. Meaning doing traditional secret management to have secrets be available in /run/secrets/my_secret and then having
serviceConfig.LoadCredential = "my_secret:/run/secrets/my_secret";
then inside your application you would have
my_secret = File.read("$CREDENTIALS_DIRECTORY/my_secret")

Please correct me, If you feel I misunderstood something.

[davidak]

It seems that some settings like db passwords are most often read from env vars

That is very common in Docker, maybe they made it popular.

e.g. https://github.com/docker-library/docs/blob/6b9b7ee305a055ecfe0c9c0b772bc2da176d736e/postgres/README.md#start-a-postgres-instance

it might be OK inside a container, but we should use files

thank you for bringing this topic forward!

[flokli]

just to give a summary of what happened.
It seems that the loadCredential feature doesn't work with systemd 247, but people have reported that it's fixed with 248. There is a PR pending that updates to systemd 248 #123476

Please correct me, If you feel I misunderstood something.

At least with current systemd in unstable (systemd-247.6), It doesn't seem to be broken anymore.

I created /run/secrets/foo in a directory only readable by root, and am able to pass this down to a restricted, DynamicUser:

root@tp:/run/secrets|⇒  systemd-run -p Type=oneshot -p DynamicUser=yes -p User=test -p LoadCredential=foo:/run/secrets/foo -t /bin/sh
Running as unit: run-u715.service
Press ^] three times within 1s to disconnect TTY.
sh-4.4$ /run/current-system/sw/bin/env   
LANG=en_US.UTF-8
INVOCATION_ID=a368227532684b0ea1ed234df7bee687
USER=test
PWD=/
TERM=xterm-256color
SHLVL=1
LOGNAME=test
PATH=/nix/store/ss3pnkjknb8bc9ljdyy9wn00n0y7bxhp-systemd-247.6/bin/
CREDENTIALS_DIRECTORY=/run/credentials/run-u715.service
_=/run/current-system/sw/bin/env
sh-4.4$ /run/current-system/sw/bin/cat /run/credentials/run-u715.service/foo
bar

[happysalada]

very silly question, how do you pass several credentials ?
it seems to me that systemd wants you to define several LoadCredential directives.
It won't work with nix right? having several LoadCredential attributes as part of serviceConfig will generate an error right?

[arianvp]

You can assign a list

LoadCredential = [ a b c ]. We convert that to multiple directives

[dotlambda]

At least with current systemd in unstable (systemd-247.6), It doesn't seem to be broken anymore.

It is still broken. The systemd version hasn't changed since I posted systemd/systemd#19604 (comment).

[flokli]

@dotlambda you can't reproduce the command output I posted in my comments (here and here)?

I think your issue is that in your example you want systemd to do the substitution, not a wrapper script.

[dotlambda]

@flokli Please read my comment. You have to use DynamicUser, RuntimeDirectory, and ExecStartPre to trigger the bug.

[flokli]

Let's carry the discussion over to systemd/systemd#19604 (comment) - that's unrelated to whether systemd 247's credential logic can be used to solve secret management.

It can, as demonstrated in #102397 (comment), it just requires some wrapper scripts for services not supporting reading secrets from files, and assembling these files with environment variables passed at runtime - so closing this issue.

[happysalada]

How about we leave it open until we manage to use it somewhere in nixpkgs ?

[happysalada]

Just an additional note that I tried to run this on the systemd 247.6 without DynamicUser. ExecStartPre and LoadCredentials seem to be enough to trigger the bug (for me).
I just wanted to test passing several LoadCredentials to a service.
It seems that the credentials don't show when you run systemctl show my-service perhaps it's just because of the bug. I thought I might just share this.
Just as an aside, I submitted a PR upstream to test this on plausible analytics. The PR has been merged, but they haven't released a new version. the systemd PR update on nixpkgs is not merged either, so no rush I think, just sharing my plan.

[happysalada]

Systemd update to 249.4 just landed on unstable.
There are a couple of gotchas with using LoadCredentials but my simple testing reveal it to work well.

The gotchas are

• credentials are not available in ExecPreStart
• if used inside ExecStart the environment variable $CREDENTIALS_DIRECTORY should be used between {} like ${CREDENTIALS_DIRECTORY}/my_credential. Since in systemd unlike in bash / is not a variable separator.

I was thinking of writing some documentation for it, but I don't know at all where it would belong. If people know, I'm interested.

PS: working on a PR in nixpkgs that uses that. However it needs a bit more work. I will update when it's close to merging

[arianvp]

A systemd section in the nixos manual might be worth it? If that's too high of a bar we could create a nixos.wiki page

…

On Tue, 5 Oct 2021, 06:49 Raphael Megzari, ***@***.***> wrote: Systemd update to 249.4 just landed on unstable. There are a couple of gotchas with using LoadCredentials but my simple testing reveal it to work well. The gotchas are - credentials are not available in ExecPreStart - if used inside ExecStart the environment variable $CREDENTIALS_DIRECTORY should be used between {} like ${CREDENTIALS_DIRECTORY}/my_credential. Since in systemd unlike in bash / is not a variable separator. I was thinking of writing some documentation for it, but I don't know at all where it would belong. If people know, I'm interested. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#102397 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZNIZUASFLEOA3RH4WFPDUFJ7UFANCNFSM4TGZHCAQ> .

[flokli]

@happysalada @arianvp NixOS manual should be better for discoverability. I'd be more than happy to review a PR for this.

There's already quite some markdown files in nixos/doc/manual/, so the exposure to docbook should be fairly small ;-)

[happysalada]

I couldn't find the manual directory, but there is a using directory with some markdown file.
I'm going to have a look at a rough summary soon. (thinking about adding a systemd.chapter.md file in the using directory, let me know if you think there is a better solution.).

[happysalada]

Ok, very early design for a systemd section, feedback welcome
#142282

[YoshiRulz]

How is this going to work on non-NixOS hosts?

[flokli]

Non-NixOS hosts are not affected by this. We render systemd units only for NixOS machines. The systemd credentials system simply exposes files in a $CREDENTIALS_DIRECTORY, which the service is able to access.

[bjornfor]

I think this can be closed now?

(Note that unfortunately we cannot access credentials in ExecStartPre yet: systemd/systemd#19604)

[happysalada]

I agree, I think this can be closed. the credentials logic has been implemented in several modules.
Anyone can feel free to re-open if they feel this hasn't been addressed properly

[asymmetric]

the credentials logic has been implemented in several modules.

@happysalada could you link to some of them?

[happysalada]

plausible.nix and lemmy.nix are two of them.
if you look for CREDENTIALS in nixpkgs, you can find other occurences of it

[con-f-use]

So just out of interest because I'm window-shopping for a secrets solution: now that the issue is closed, can we get a summary?

• Is this used somewhere?
• How is it used, what do I need to do to make the secrets available to my user/server?
• How do I get the secrets to the machine that needs them?
• Where is the documentation? (I found a draft but that hasn't been merged for 1.5 years)

[happysalada]

If you search for loadcrediential in nixpkgs youll see where it s used.
The secrets management is a whole other rabbit hole, personally i just go with agenix (it has decent documentation ).

The one problem you might run into though is that if upstream us just putting secets in env vars. Then there isnt any point in using loadcredential, whatever you do it will end up in the nix store. The best is if upstream enables loading a secret through a path, then loadcredential is the perfect solution for it. If you are packaging your own software, then using env vars that read a path is the way to go.

[davidak]

It would be great to have the best practices of credential handling in nixos modules documented und used in all where it is possible, to have a consistent experience.

[flokli]

@arianvp recently also mentioned the concept of /etc/credstore (or /run/credstore).

If you don't specify an absolute path to the secret, but only an identifier, that's where systemd credentials logic looks things up. Maybe we could align our recommendations towards that?

Is someone willing to take over @happysalada 's PR at #142282 (or at least the credentials-related parts)?