Julia Pkgs in NixOS

[nico202]

Issue description

It would be nice to have julia pkgs easily installable the way we can install them in python on other languages

The packages resides in a git project (https://github.com/JuliaLang/METADATA.jl), every package has it's (git) url, every version hashes and dependencies

The variable JULIA_PKGDIR defines a folder in which packages get installed, I think we can symlink from the store to this folder (not sure if it needs to be ro/rw)

[vcunat]

/cc maintainer @7c6f434c.

[7c6f434c]

Is Julia package manager behaviour more stable than the language features now? I mean, Pkg is already quite fine, and for example opam2nix is harder to use for an outsider (me) than making opam work. So I am afraid that a simple approach to redoing Julia's Pkg will be hard to use and immediately out of date, if I do it myself… Maybe there is a sane plan, though — it's just my lack of clue.

[FRidh]

As soon as they document their environment variables I'm up for having a further look at this.

[kbdave314]

I'd like an interface to the native package manager as well.

@FRidh: cf. pull request JuliaLang/julia#20162

[tkelman]

The Julia package manager is due for a rewrite, any integration you do with it now would have to be redone.

[FRidh]

Yesterday I had a short look at it.
https://github.com/FRidh/nixpkgs/tree/julia

Implemented thus far is

• a function for building a Julia package. It will definitely need some more work, but the basic thing is there.
• fix-point combinator based package set
• buildEnv, withPackages. We might want to use JULIA_HOME for this.
• some other helper functions.
• a script to take the official packages index and create JSON.
• uses JULIA_LOAD_PATH to build up a path with Julia modules.

To do:

• have a function for building official packages
• parse the data in the JSON further so we can extract platform-dependent dependencies and version constraints.
• decide where packages should live. The current julia.site seems to me like the proper location
• decide what to do with precompiled modules. The first time you import a module with Julia it precompiles it (like Python's .pyc). Ideally we would put this in the same derivation as the actual module code.

What's most important I think though is to decide on how we expect it to work with impure paths. Do we want to be able to also install packages using Pkg.add() or not? And if so, where should it put it?

Note the code is quite a mess at this point, and I just dumped everything in one commit. I won't have time to look at this for at least another week, and I might not even continue since I don't use Julia myself.

[nico202]

Hi, how is it going? Anybody has hints?

[AlexanderKoshkarov]

I know it may be a wrong place to ask, but for this day, is there a way to at least install Julia packages (IJulia.jl, Plots.jl, PyPlot.jl, DifferentialEquations.jl, Sundials.jl and so on)? It is the only thing which keeps me from trying NixOs one more time (last time all was good except I could not make Julia with packages work, it was a year ago)

maybe you can comment here
https://www.reddit.com/r/NixOS/comments/6btseq/nixos_and_julia/

Sorry for the wrong place again, but I cannot find people who would share their experience with Julia under NixOs

[FRidh]

@AlexanderKoshkarov no, there is no "finished" implementation yet for installing Julia packages with Nix. The closest there is right now is, as far as I know, the branch I mentioned in #20649 (comment).

[nico202]

I'm using it, you just need to figure out the dependency of a package. I create a nix-shell env with Julia and the dependency the package needs, and set the Julia pkg dir to a subfolder of the project directory. I can upload an example tomorrow, not at my desk right now Il 15 giugno 2017 18:19:26 CEST, AlexanderKoshkarov <[email protected]> ha scritto:

…

I know it may be a wrong place to ask, but for this day, is there a way to at least install Julia packages (IJulia.jl, Plots.jl, PyPlot.jl, DifferentialEquations.jl, Sundials.jl and so on)? It is the only thing which keeps me from trying NixOs one more time (last time all was good except I could not make Julia with packages work, it was a year ago) maybe you can comment here [https://www.reddit.com/r/NixOS/comments/6btseq/nixos_and_julia/](url) Sorry for the wrong place again, but I cannot find people who would share their experience with Julia under NixOs -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #20649 (comment)

-- Inviato dal mio dispositivo Android con K-9 Mail. Perdonate la brevità.

[AlexanderKoshkarov]

@nico202, Thanks for the assurance that such people exist and it is possible! Yes, the example would be awesome, I already forgot a lot about nix, so it would be very helpful.

[nico202]

@AlexanderKoshkarov like @FRidh said, my solution is not that nix-y; nonetheless, if you are on NixOS and you want to use julia, this is the only way I found to install packages.

Here's a gist with an example
https://gist.github.com/nico202/9377b3b378eab950dc05566e733ea320
The hard part is figuring out what dependencies are needed by the packages you want to install, but I successfully installed GTK.jl, Plots.jl and others.
To figure out the dependencies I just enter the env, try to install the package, see any error message, if any just delete the .julia_pkgs dir (just to be sure to have a clean env), add the dependency and start over.
For example, Requests neets MBedTLS, which requires mbedtls and Libz, so in my LD_LIBRARY_PATH for projects needing Requests.jl I add ${pkgs.zlib}/lib:${pkgs.mbedtls}/lib

[lesscodeless]

How does current Julia-NixOS-users deal with hard-coded references to /sbin/ldconfig when attempting to use Julia packages? E.g. try a Pkg.add("RCall") followed by using RCall to see what I mean.

Greping for ldconfig in a ~/.julia folder with some packages in it shows that this is a recurring pattern.

[nico202]

@lesscodeless see my gist in the previous comment. Just add to ld library path ${R}/lib/

[mnacamura]

Many Julia packages use BinDeps.jl in their deps/build.jl. BinDeps often calls yum, apt, pacman, or homebrew to install dependencies. How to manage this in Nix might be another concern.

[nico202]

BinDeps calls the package manager only if it's not able to find a suitable dependecy already installed. So we don't need to manage it

[FRidh]

BinDeps calls the package manager only if it's not able to find a suitable dependecy already installed. So we don't need to manage it

It may be good to "disable" it anyway for non-sandboxed builds.

[mnacamura]

I see. How does BinDeps identify whether a library is available or not?

[mnacamura]

In my case on Darwin, I had to delete all dependency to Homebrew.jl to install a package using BinDeps. Here is an example: https://gist.github.com/mnacamura/ce3b8e6640d54158df1863cb54e94ff8

[nico202]

How does BinDeps identify whether a library is available or not?

Setting LD_LIBRARY_PATH works

[mnacamura]

Setting LD_LIBRARY_PATH works

Thanks for the answer. Even if I set LD_LIBRARY_PATH and delete a line @osx Homebrew in REQUIRE, Pkg.resolve() starts installing Homebrew.jl. Any suggestion?

[mnacamura]

Pkg.resolve() starts installing Homebrew.jl. Any suggestion?

Oh I misunderstood. deps/build.jl contains using Homebrew so that the build just failed.

[mnacamura]

Based on @FRidh's work (#20649 (comment), thanks @FRidh), I implemented this more: https://github.com/mnacamura/nixpkgs/tree/julia-modules. I'm still puzzled how to treat Homebrew on Darwin (mnacamura@e78f629).

[nico202]

@mnacamura Wow, thanks! Some question:

• Are you experiencing improve / offer better alternative to realpath JuliaLang/julia#34135?
• Next julia will have Pkg3: in their juliacon 2017 talk they said part of the idea was taken from nixos, @tkelman do you know something more?

However on my config i'm using the following packages, with corresponding ld library path

# IJulia.jl
ijulialdpath = "${pkgs.mbedtls}/lib:${pkgs.zeromq3}/lib";
 
 # ImageMagick.jl
imagemagickldpath = "${pkgs.imagemagickBig}/lib";
 
# HDF5.jl
hdf5ldpath = "${pkgs.hdf5}/lib";

# Cairo.jl
cairojlldpath = "${pkgs.cairo}/lib:${pkgs.gettext}/lib:${pkgs.pango.out}/lib:${pkgs.glib.out}/lib";

# Gtk.jl
gtkjlldpath = "${pkgs.gtk3}/lib:${pkgs.gdk_pixbuf}/lib";
  # GZip.jl # Required by DataFrames.jl
  gzipldpath = "${pkgs.gzip}/lib:${pkgs.zlib}/lib";

# GR.jl # Runs even without Xrender and Xext, but cannot save files, so those are required
grldpath = "${pkgs.xorg.libXt}/lib:${pkgs.xorg.libX11}/lib:${pkgs.xorg.libXrender}/lib:${pkgs.xorg.libXext}/lib";
glfwldpath = "${pkgs.glfw}/lib";
freetypeldpath = "${pkgs.freetype}/lib";
glvisualizeldpath = "${glfwldpath}:${freetypeldpath}";

If you are adding the dependencies manually, this might help

edit: added GZip.jl

[mnacamura]

Are you experiencing JuliaLang/julia#34135?

Hmm... I've never seen that segfault.

If you are adding the dependencies manually, this might help

Thank you, it is helpful. Currently ~120 packages seem to use BinDeps. Adding their native dependencies manually is tedious. I wish Pkg3 could provide information of native dependencies in a proper format...

So, what's the difference of Pkg3 from the current Pkg (Pkg2?)?

[nico202]

1. Ok the fact that you have not seen the segfault is interesting, are you on linux 64bit? Are you using julia 0.6.2? does it work (non-segafualt) even in a pure nix-shell?
   The shortest way to reproduce is:
   nix-shell --pure -p julia --run 'julia -e "error()"'

could you try?

2. dependencies should be defined in a toml file like here. Every package has it's own unique id (if it happens that multiple packages have the same name, it's not a problem, and renaming a package does not change much since the uuid stays the same). Also if I understood/remember well, the LOAD_PATH variable is changed, and it's replaced to a data structure containing multiple paths for system-level packages and official julia pkgs, user packages...
   Pinning a version should be easier (ie. add JSON @0.10), tarball are now preferred to git clone because of space/speed, and I even think better precompilation

edit: notes: I'm not involved in the dev of Pkg3, I just saw a youtube video weeks ago, so if anybody knows better if it will help the integration with nixpgks it would be useful to know :D

[mnacamura]

% nix-shell --pure -p julia --run 'julia -e "error()"'
Stacktrace:
 [1] error() at ./error.jl:30

I use Julia 0.6.2 on x86_64-darwin.

Many important changes... I'm looking forward to seeing its release, and going to wait and see.

[FRidh]

Adding their native dependencies manually is tedious.

Yep, it is, but it is manageable, at least as long as we have say one version per package.

dependencies should be defined in a toml file like here. Every package has it's own unique id (if it happens that multiple packages have the same name, it's not a problem, and renaming a package does not change much since the uuid stays the same)

That's interesting. So, that means we can translate every package and every version to a Nix expression of which the key is the uuid, and the value the derivation. Expressions can then refer to each other via the uuid. Furthermore, we could have a mapping of name to uuid corresponding to the latest or stable release. That basically allows including every package at every version in Nixpkgs, although it becomes unmanageable because of the earlier comment regarding native dependencies. Also, how will upstream deal with security issues? Using uuid to refer to dependencies is the same as using git submodules, or Python's pip freeze which should absolutely not be done for libraries.

they said part of the idea was taken from nixos

Nix uses a hash for the outputs. Pkg3 uses them for the inputs, for which Nix uses variables.

[nico202]

Also, how will upstream deal with security issues? Using uuid to refer to dependencies is the same as using git submodules, or Python's pip freeze which should absolutely not be done for libraries.

Sorry I can't follow you here. uuids are for package names, not package versions, so you can upgrade.

(The youtube video I was referring to is this, this is a slide with the feature highlight )

they said part of the idea was taken from nixos

Nix uses a hash for the outputs. Pkg3 uses them for the inputs, for which Nix uses variables.

They say this at 7:15. I think they say this just because once a package is installed is never touched again, not sure if there's anything more than this.

edit: maybe we can get in touch with @KristofferC

[tbenst]

@danieldk needed to add appropriate paths to LD_LIBRARY_PATH (per @nico202 https://gist.github.com/nico202/9377b3b378eab950dc05566e733ea320). I revised the comment above and now works with Cario

[danieldk]

@tbenst No dice. LD_LIBRARY_PATH already contains /lib in the chroot, which has symlinks for all libraries, so all libraries should be visible. The problem on my machine (NixOS unstable) is that Julia calls ldconfig -p, which fails due to ld.so.cache not being available. I now have the following big hack to create ld.so.cache in the chroot, and building Julia packages works:

self: super: {

julia-fhs =
let
  ldconfigWrapper = with super; stdenv.mkDerivation {
    name = "ldconfig-env";

    nativeBuildInputs = [ makeWrapper ];

    phases = [ "installPhase" "fixupPhase" ];

    installPhase = ''
      makeWrapper ${glibc.bin}/bin/ldconfig $out/sbin/ldconfig \
        --add-flags "-C /usr/ld.so.cache"
    '';
  };
in
  super.buildFHSUserEnv {
    name = "julia-fhs";
  
    targetPkgs = pkgs: with pkgs; [
      ldconfigWrapper
  
      julia_11
  
      autoconf
      binutils
      coreutils
      gnumake
      m4
      pkgconfig
      stdenv.cc
      utillinux
  
      curl
      git
  
      cairo
      fontconfig
      freetype
      gettext
      glib
      libintl
      libpng
      libffi
      librsvg
      pango
      pixman
      zlib
    ];
  
    extraBuildCommands = with super; ''
      # Cannot write to /etc?
      echo "$out/lib" > $out/usr/ld.so.conf
      ldconfig -f $out/usr/ld.so.conf -C $out/usr/ld.so.cache
    '';

    runScript = "julia";
  };
}

[tbenst]

@danieldk yes that comes from https://github.com/JuliaPackaging/BinDeps.jl/blob/382082a30de7186d6e27880807f6be353fcbd593/src/dependencies.jl#L398 which if I recall correctly only gets called if a .so is not found. I could reproduce your error before adding the LD_LIBARARY_PATH line, and verified that it could build afterwards. I’m also on unstable so would be surprised indeed if the code in #20649 (comment) doesn’t work for building Cairo.

In an case great to have your solution for ldconfig -p!

[RCHG]

@danieldk, I was trying to understand your nice piece of code but I am just a nix/nixos beginner, so I don't know how to manage overlays in a step by step guide (I am yet learning derivations...). Could you kindly indicate where I could find examples to implement your overlay at /etc/nixos/configuration.nix level? Thanks in advance!!

[couchemar]

@RCHG For adding the overlay you need to add function from snippet to nixpkgs.overlays (https://nixos.org/nixpkgs/manual/#chap-overlays). Example content of the configuration.nix:

{ config, pkgs, lib, ... }:

let

juliaOverlay = self: super: {
  julia-fhs = ...
};

in {
...
nixpkgs.overlays = [
  juliaOverlay
];
...
}

[cstich]

The closest I've gotten to getting the Julia ecosystem working on NixOS is the following. Unfortunately, while I can get CUDA working, CUDNN still does not, so not very useful for ML packages like CuArrays or Flux.

For what it is worth test CuArrays passes almost all tests. Does that mean CUDNN is working?

[schlichtanders]

what is the current status of packaging Julia with Nix?

[tbenst]

@schlichtanders I'm not aware of any efforts. Seems like this is possible today however, using the approach in #20649 (comment). I don't think the lack of determinism in precompilation is a deal-breaker. From a practical standpoint, nix hashes the inputs not the outputs, so the ecosystem already works with non-deterministic builds: https://r13y.com/.

When upstream fixes reproducibility, great, we'll reap the benefits, but I don't think that should stop us from having a workable solution for now. I think we just need someone to spearhead the effort. I'd certainly be happy to help with specific packages I care about (e.g. Flux.jl).

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/system-with-nixos-how-to-add-another-extra-distribution/5767/11

[573]

With overlay by @danieldk and expression by @tbenst tried and got
builderror.log

[thomasjm]

Hi all,

I've been working on a tool for creating Julia environments in Nix now that the new and improved Pkg3 and Pkg.Artifacts systems exist. It's brand new and I've only tried it with a small number of packages, but it seems to work. I'd love any feedback or bug reports. You can find it here: https://github.com/thomasjm/julia2nix

[nbren12]

FWIW, guix has recently been adding support for several julia packages (example). Can their approach be emulated?

[nico202]

@nbren12 that's the simple build system I wrote, which is enough to get packages working.
The julia binary is wrapped like this so that JULIA_LOAD_PATH and JULIA_DEPOT_PATH of installed julia packages is respected

jll packages have their .so file path fixed like this

[nbren12]

Cool @nico202! Would it be worth porting to nix?

[nico202]

The build system is working ok, with a few problems

1. packages are precompiled twice, once during build and once again when the user import the package, and
2. packages are not deterministic (but that's a more a Julia problem indeed)
3. It relies on the existence of Project.toml (which some old packages do not have, for which we have a procedure that creates that file but increases a bit the maintenance burden)

(1. can probably be fixed, I just to have to find some time to look at it. I also made some progress on 3 here but I'm stuck and nobody gave me support 😞 )

However, if anybody is willed to have give a try, I guess it's worth it (packages are working and it should also be pretty straightforward to write an importer for them).

(but I switched to guix so I'm sorry I won't be able to add the build system myself)

[stale]

I marked this as stale due to inactivity. → More info

[samrose]

I am investigating how guix community accomplishes Julia packaging now + julia2nix

[fedeinthemix]

Hi @samrose, did you make any progress on this?

[fedeinthemix]

I've created PR #208379 that addresses this. Please help to review and check it.

[thomasjm]

Hi all, I've just pushed a new attempt at this: #225513.

I think this version is exciting because it lets you build arbitrary and up-to-date package sets in a nice way, without adding a ton of stuff to the Nixpkgs repo. There is some IFD though; see the PR for more discussion. Please feel free to comment or review!

[shivaraj-bh]

@thomasjm I was trying out juliaWithPackages to use the package RDatasets and it fails with the following error:
Overrides.toml:310:42 error: key already has a value because there are two entries with same key 20780fe7a8774baffbeecc2f9a7367d4e8162500 and different values.

[thomasjm]

Good point @sbh69840 , I just added deduplication for Overrides.toml keys and it works now.

[shivaraj-bh]

Thanks! That works now.
There seems to be another problem, PyPlot is using Python from Conda instead of using the one packaged in Nix

[thomasjm]

@sbh69840 fixed! Now we have Python provided to Julia earlier in the process, and also a mapping to provide extra Python packages for Julia packages which need them. There's some related discussion in the PR thread. Please keep the feedback coming :)

[thomasjm]

Closing this now since julia.withPackages landed in #225513.