Python ecosystem improvements (placeholder issue)

[domenkozar]

This is a placeholder for general improvements to Python packaging infrastructure in Nixpkgs.

Documentation

• why do we use propagatedBuildInputs?
• how to generate PYTHONPATH out of packages? (makePythonPath in nix, and toPythonPath in bash)
• explain a bit in more detail how python setup hook works and that packages not using buildPythonPackage need to add python to propagatedBuildInputs (Python: some packages don't propagate python causing "nix-shell -p" to fail #17679, Python: split buildPythonPackage into two functions #18143)
• wrapPython
• how to use virtualenv with Nix? (Better documentation for python.buildEnv / virtualenv #5623, Document how to customize modules available to virtualenv #66)

Python modules

• add support for modules to all python interpreters (Python 3 dependency bloat #12333, Python: use separate output for tkinter #19309)
• enable readline by default (adds 16M to the closure (nixpkgs python repl isn't very terminal-friendly #17607, Python: use separate output for tkinter #19309)
• enable ncurses by default (Python 2.7 Full: sqlite3 module doesn't work in virtualenv #492 (comment), Python: use separate output for tkinter #19309)
• enable sqlite by default (Python 2.7 Full: sqlite3 module doesn't work in virtualenv #492 (comment), Python: use separate output for tkinter #19309)
• tkinter.tix doesn't yet function with $ nix-shell -p 'python.withPackages(ps: [ps.tkinter])' --run "python -c 'import Tix; Tix.Tk()'". Python tix module #21643 (review)

Build functions

• join python and buildPythonPackage setup-hooks
• namePrefix should be default empty and prefixed with pythonX.X- only inside python-packages.nix (buildPythonApplication: use new function for Python applications #13124)
• refactor buildPythonPackage into python.mkDerivation (similar to cabal.mkDerivation)
• helpers for python software that doesn't use distutils/setuptools (Python: split buildPythonPackage into two functions #18143)
• convert all mkDerivation using distutils to buildPythonPackage
• convert all buildPythonPackage not using distutils to mkDerivation Typically buildPythonPackage or buildPythonApplication with format = "other"; should be used. See the docs.
• replace magicalSedExpression (Replace magic sed for Python argv[0] with a python wrapper. #11168)
• buildPythonPackage shouldn't use postFixup (buildPythonPackage uses postFixup for itself #9204, 85a87f5)

Packages

• modernize packages in python-modules folder to be either a module or an application (versioned pythonX.Y- prefix #1468)
• pypi2nix: a tool to generate Nix expressions for Python packages.
• fixed-point combinator for python-packages.nix (Python: Fixed-point combinator #17428)
• autogenerated data with urls to and hashes for all PyPI packages (Python: buildPyPIPackage and generated data #16005, Autogenerated data in Nixpkgs #15480).

Tests

• it should be easier to run different test runners (setup.py / nosetests / py.test) (python buildPythonPackage: check and run correct test command #11715)
• hook for testing importing modules (f7e28bf)
• import all modules after building (python improvement: import all modules after the build is finished #3821, Python: default test runner fails often because it imports all modules #14849)
• reduce closure sizes by not installing tests (Python: reduce closure sizes by not installing tests #12379)

Issues to resolve

• Depend on a Python application without adding the libraries to PYTHONPATH (Depending on a python package without adding it to PYTHONPATH #11423)
• Tox (tox package broken #15224)
• virtualenv should use pythonFull (Make virtualenv use pythonFull, not python #7134, Python: use separate output for tkinter #19309)
• unify Python interpreters (together with pypy) (Python: move interpreters #17339)
• Nix replacement for ctypes.util.find_library ([python] ctypes.util.find_library should return full path to the library #7307)
• setuptools namespaces are broken (give collisions) (python: setuptools namespaces are broken (give collisions) #2412, python-wrapper broken by recent re-factoring #1826)
• data_files upstream issue (Patch upstream to correctly handle install_data/data_files in Python #4968)
• zc.recipe.egg is broken
• Circular dependencies
• pypy PYTHONPATH is incomplete (pypyPackages: wrapped scripts are missing paths from PYTHONPATH #12948)
• Python on Nix is not compatible with manylinux1 (Python on Nix claims to be manylinux compatible #18484, Add support for PEP 513 "manylinux1" wheels #71935)

[domenkozar]

cc @chaoflow @garbas @cillianderoiste @offlinehacker @matejc (anyone else)?

[domenkozar]

cc @rach

[shlevy]

Do you think it makes sense to make separate issues for more of these?

[domenkozar]

For some todos, if more then one line is needed to describe what's needed to be done (and if some discussion is needed).

[kevinburke]

Readline support would be very nice. Currently arrow keys in the nix python interpreter don't work.

[domenkozar]

@kevinburke could you describe exact steps to reproduce that? Stable or unstable channel?

[domenkozar]

Updated TODOs to match current state in nixpkgs.

[domenkozar]

@kevinburke once #4495 is merged, using pythonFull will mean readline support. For now you can use python and specify python.modules.readline as buildInput for your expressions.

[joelmo]

I think this could be added under issues to resolve:
Import test for all derivations that add packages to site-packages folder. It would then be easier to detect if packages propagate all other packages needed for the basic import.

The test itself is very simple (python -c 'import matplotlib, pkg1, pkg2'), but it requires the correct environment to be set up after the build.

[domenkozar]

See #3821

[jagajaga]

P-p-p-ping :)

[datakurre]

I'm in need of "improved" solution for managing a lot of in-house packages in setup, where we have normal python package development and custom nix channel.

I'd prefer to have default.nix in each package repository to easily get development environments for each package. Yet, I would not like to manage duplicate buildPythonPackage-derivation in our in-house channel repository.

Have you ever considered somehow supporting explicit release.nix / default.nix in Python source packages? Could it be any more dangerous than what setup.py already is?

For my own usage, I might simply check if the package has default.nix, expect it to have buildPythonPackage derivation and call that.

[FRidh]

You could keep a default.nix file in the repo, and use src = ./.; , and
then fetch the repo and import it or use callPackage. However, you will
then import from another derivation, and this is currently not supported on
Hydra.

[RonnyPfannschmidt]

@datakurre have you considered always having your custom channel, and each default.nix overriding the src attribute of a derivation with the path ./.

[datakurre]

@FRidh @RonnyPfannschmidt Thanks for those ideas!

I was aware that Hydra-compatibility is an issue (didn't know that limitation of derivation not being able to call derivation though), so I probably should try @RonnyPfannschmidt's approach at first. Adding new packages and adding new dependencies for developed package would be a bit inconvenient, but for existing packages it would make default.nix look less scary (for non-nixers).

I hope to end up with something that I could document as an example workflow with nix.

[stale]

I marked this as stale due to inactivity. → More info

[OliverEvans96]

I think #91916 should be added to this list.