lib/types: Allow paths as submodule values

[infinisil]

Motivation for this change

Because allowing paths as submodule values isn't always backwards compatible, that change from #75031 had to be reverted.

This restores that behavior but also fixes the backwards incompatibility in nixpkgs and adds a release note about it.

Ping @roberth @danbst @grahamc

Things done

• Updated docs with this change
• Ran lib/tests/modules.sh successfully (again, the revert actually broke a test for this very thing in there)

[roberth]

Can't we solve the problem by flipping the either arguments where necessary?
In the long run parameters like this add up to a significant overhead for users and maintainers, in particular

• documentation becomes harder to use
• complexity of the module system logic
• code breaking when eventually switching from submodule to submoduleWith

We should lean towards simplification and document any required changes to user modules in the release notes.

[infinisil]

@roberth Tried that already, but unfortunately not as easy as that. Main problem is that types.path.check fails for things that can't be coerced to strings, and I don't see an easy way to fix this (we'd need some builtins.isCoercibleToString function). This is why it's in the second place in that either.

If you have a decent implementation of types.path.check then we could maybe use that instead and hope to fix all such either instances instead. But we then still can't fix either occurrences outside of nixpkgs.

[roberth]

Blocked on #76977 it seems. The builtin doesn't solve the other problem with path.check, which is adding intermediate cleanSourceWith sources to the store.

But we then still can't fix either occurrences outside of nixpkgs.

Can be done in the release notes. I don't expect it to be a big problem (famous last words?).
Alternatively we can document the temporary nature of the new parameter and deprecate submodule.

[infinisil]

Now that we could switch all either submodule path to either path submodule, a problem is that either submodule path won't work with future versions of nixpkgs, and either path submodule won't work with past versions of nixpkgs. This means projects defining such a type can't accommodate all different nixpkgs versions if they use such a type.

[infinisil]

Uses are probably few though, so it might not matter much

[roberth]

This can be further remedied by backporting your path.check fix #77133 if necessary.

[infinisil]

Removed the allowsPathsAsValues again, fixed the either in certmgr, backported the path.check fix and added a release note for the incompatibility.

[danbst]

I'd say this is bad. Can't we already use types.submodule { imports = [ ./path-to-options ]; }? If shortcut brings such tricky behavior, maybe its not worth adding it.

[roberth]

@danbst It's for values, which occur more frequently than option/type declarations. (I think you mistyped?)
The real trickiness is in either, not submodule.
I think it's worth it because it makes the syntax for modules more consistent and therefore easier to understand.

[infinisil]

It allows e.g.

some-submodule = mkMerge [
  ./some-module.nix
  (mkIf some-condition ./other-module.nix)
]

instead of

some-submodule = mkMerge [
  { imports = [ ./some-module.nix ]; }
  (mkIf some-condition { imports = [ ./other-module.nix ]; })
]

It's not that big of a benefit, but I don't think we should not allow that just because of a slight inconvenience with the either type in a very special case.

[danbst]

@infinisil I don't think it is possible to mkMerge sub modules. Will it be possible after this change?

nix-repl> (evalModules { 
  modules = [ 
   { 
    options.bla = mkOption { 
        type = with types; submodule (mkMerge [ { options.foobar = mkOption {}; } ]); 
    }; 
   }  
   { bla = {}; } 
  ]; }
).config.bla
error: The option `bla.options' defined in `<unknown-file>' does not exist.

However, if I remove mkMerge [], it works.

[danbst]

@roberth this also adds an exception, when order of items in either matters. ADT with exceptions :)

I can't build of top of my head a counterexample (when either is "ADT with exceptions" with current types), maybe it is flawed already. In that case, adding another "exception" is fine.

[roberth]

@danbst as much as I'd like either to be an algebraic sum type, it isn't. It's an untagged union with untagged union problems.

Also, because a mathematical union discards duplicates, if more than one field of the union can take on a single common value, it is impossible to tell from the value alone which field was last written.
-- wikipedia

either should have been called union.
"Either" is commonly used for tagged unions, in Haskell, Scala, Purescript and a bunch of languages and libraries. It should have been avoided.

[infinisil]

Oh yeah, what I showed above (the non-path version) doesn't work, it needs to be this (if you don't use submoduleWith) which is even worse:

some-submodule = mkMerge [
  ({ ... }: { imports = [ ./some-module.nix ]; })
  (mkIf some-condition ({ ... }: { imports = [ ./other-module.nix ]; }))
]

Compare that to the version when paths are allowed

[danbst]

@infinisil still nope. I've cloned your branched and checked that paths don't work either:

[danbst@station:~/dev/nixpkgs-pr-76861]$ nix eval '(with import ./lib; (evalModules { 
  modules = [ 
   { 
    options.bla = mkOption { 
        type = with types; submodule (mkMerge [ ./dummy.nix ]); 
    }; 
   }  
   { bla = {}; } 
  ]; }
).config.bla
)'
error: value is a path while a set was expected, at /home/danbst/dev/nixpkgs-pr-76861/lib/modules.nix:208:25
(use '--show-trace' to show detailed location information)

And yes, if I remove mkMerge it works.

[infinisil]

Ah I didn't pay attention to the code, no that doesn't work because mkMerge isn't supposed to be used for option types, it's only for the config section. Your example without mkMerge already worked previously indeed. This change is only for the cases like the example I showed (assigning submodules in the config section)

[infinisil]

Here's a simple full example of what's now possible (Edit: Which is pretty much what the test case does)

config.nix:

{ config, lib, pkgs, ... }: {
  options.some-submodule = lib.mkOption {
    type = lib.types.submodule [];
  };

  config.some-submodule = ./some-module.nix;
}

some-module.nix:

{ lib, ... }: {
  options.foo = lib.mkOption {
    default = false;
  };
}

$ nix-instantiate --eval nixos --arg configuration ./config.nix -A config.some-submodule.foo
false

[danbst]

@roberth okay, you are right. either (attrsOf str) (attrsOf int) behaves differently depending on order of type arguments....

[infinisil]

@roberth Good to merge?