Fix Julia packages with binary blobs failures and add withPackages attribute

[fedeinthemix]

Description of changes

Currently many Julia packages installed with the native package manager Pkg fail to work. In this PR we fix this by

1. Adding a Julia build system to patch the binary blobs and make them available in nixpkgs with the help of the withPackages passthru attribute. A utility julia2nix is also added to facilitate adding packages to nixpkgs.
2. Put the julia compiler in a buildFHSUserEnv thereby providing the interpreter in the expected location and defining some required environment variables.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

@ninjin @7c6f434c @NickCao @benneti @pasqui23 @florianjacob @mayl @schlichtanders

[NickCao]

Overall promising, a bit skeptical about the dependency handling part.

[benneti]

Hey, thanks for another try! It seems like julia2nix -f -o env.nix -- "IJulia" is broken however piping to the file still works (i.e. I think "-o" is broken).

Additionally what is the intended way to loag a custom env

julia-bin.withPackages
  (ps: [ ps.Plots ] ++ (map fixpkgs
         (import ./env.nix {juliaPkgs = ps; inherit fetchurl; })))

does not seem to work as dependencies only in the env seem to be unavailable.

[fedeinthemix]

You call it like this julia2nix -f -o env.nix IJulia (no "--"). But with this variant you don't need it anymore. For packages not in nixpkgs you just use Pkg as normal.

[benneti]

Ah sorry I am just so used to -- being able to separate the "rest" of the options.
I think it would be nice to still have the option for including a custom env file, to have a standard julia environment for the system (for example IJulia and other pkgs where I might want to regenerate the configuration when the path to julia changes) or simply packages I always want to have anyway.
But there is probably a way with the current implementation, I'll try it later and provide some feedback.

[benneti]

I don't know if this is accepted in nix, but maybe something like julia.withPackagesList that takes a generated packages list and automates

let
fixpkgs = pkgs: ...
inherit (julia-bin.juliaPackages) juliaPackagesFromList buildJuliaPackage;
juliaPkgs = juliaPackagesFromList (map fixpkgs juliaPkgsList);
juliaPkgsList = import ./env.nix { inherit juliaPkgs fetchurl; };
in julia-bin.withPackages (ps: (lib.mapAttrsToList (name: val: val) juliaPkgs))

but as this works in someway I am already happy.

Also I don't understand what triggers the precompiler sometimes it works and sometimes it does not for no apparent reason (and for the same packages no less).
PS: what makes this even more strange is, if I use home-manager to recursively symlink

  home.file.".julia" = {
    # HACK seems to enhance the chance of precompilation being picked up
    source = "${julia}/share/julia";
    recursive = true;
  };

precompilation seems to not happen at all. Maybe julia gets confused if some directories exists in ~/.julia but not all.
PPS: symlinking is no reliable solution as it breaks some packages (for me SymPy)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/1761

[fedeinthemix]

Sorry for the incremental changes. I used the new "Add to batch commit" feature, but then didn't realize that I didn't commit all of them...

[fedeinthemix]

would be glad if anyone with commit access could comment on what's missing to get this merged (apart from fixing the rl-2305 conflict).

[NickCao]

I have nothing against this. cc @GTrunSec (an active julia user) for a review.

[fedeinthemix]

ZHF: #230712

[fedeinthemix]

Not sure why ofbor-eval-package-list-no-aliases fails. The package producing the error works on my system

[benneti]

I am travelling right now so i can not check in detail, but it might be that there is no graphical environment available in the ofborg script making plots fail.

…

On 13 May 2023 12:33:49 CEST, fedeinthemix ***@***.***> wrote: Not sure why ofbor-eval-package-list-no-aliases fails. The package producing the error works on my system  -- Reply to this email directly or view it on GitHub: #210685 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[fedeinthemix]

I am travelling right now so i can not check in detail, but it might be that there is no graphical environment available in the ofborg script making plots fail.

I don't think this is the issue: the evaluation was successful until yesterday when I fixed two merge conflicts

1. A release note entry (no code change)
2. A couple of days ago julia_19 was added to master. So I wrapped it as the others

+  julia_19-bin = wrapJulia (callPackage ../development/compilers/julia/1.9-bin.nix { });

Also, there's no plot involved in the evaluation.

Can it be a false failure?

[RaitoBezarius]

I am travelling right now so i can not check in detail, but it might be that there is no graphical environment available in the ofborg script making plots fail.

I don't think this is the issue: the evaluation was successful until yesterday when I fixed two merge conflicts

1. A release note entry (no code change)
2. A couple of days ago julia_19 was added to master. So I wrapped it as the others

+  julia_19-bin = wrapJulia (callPackage ../development/compilers/julia/1.9-bin.nix { });

Also, there's no plot involved in the evaluation.

Can it be a false failure?

I don't believe so, let's try again.

@ofborg eval

[fedeinthemix]

I've found out that buildFHSUserEnv was deprecated in favor of buildFHSEnv.

[RaitoBezarius]

Result of nixpkgs-review pr 210685 run on x86_64-linux 1

2 packages blacklisted:

• nixos-install-tools
• tests.nixos-functions.nixos-test

1 package failed to build:

• julia2nix

7 packages built:

• cantor (libsForQt5.cantor ,plasma5Packages.cantor)
• julia (julia-stable ,julia_19)
• julia-bin (julia-stable-bin ,julia_19-bin)
• julia-lts (julia-lts-bin ,julia_16-bin)
• julia_18
• julia_18-bin
• labplot

[RaitoBezarius]

Seems like not ready yet, also, please fix the title and ensure you follow CONTRIBUTING for commit messages.

[RaitoBezarius]

Result of nixpkgs-review pr 210685 run on aarch64-darwin 1

4 packages marked as broken and skipped:

• julia-bin
• julia-stable-bin
• julia_18-bin
• julia_19-bin

[RaitoBezarius]

Result of nixpkgs-review pr 210685 run on x86_64-darwin 1

5 packages marked as broken and skipped:

• julia-bin
• julia-stable-bin
• julia_18-bin
• julia_19-bin
• labplot

[fedeinthemix]

closing in favor of #225513