installation: Switch to UV for dependency management

[EdmundGoodman]

Summary

Adopt uv for dependency management, replacing the existing pip approach.

Motivation

uv is developed by Astral, who also wrote ruff which is already in use in xDSL CI pipelines. It offers the benefits of much faster dependency resolution and installation, along with leveraging the more modern pyproject.toml configuration format defined in PEPs 518 and 618.

Requested by @superlopuh

Changes

This replacement requires the following changes:

• Modification to the venv target of the Makefile to create the virtual environment
• Update CI flows to install with uv as opposed to pip, including testing
  • ci-core.yaml
  • ci-mlir.yaml
  • ci-notebooks.yaml
  • ci-pyright-fails.yaml
  • code-formatting.yaml
  • jupyterlite.yaml (cannot easily be spawned due to workflow triggers)
  • pythonpublish.yaml (cannot easily be spawned due to required secrets)
  • release-notes.yaml (requires no changes)
• Update documentation to reflect these changes
• Test across operating systems
  • Windows
  • MacOS
  • NixOS
• Resolve changes required to Dependabot

Post-merge checklist

Once merged, we should check the following things:

• The release flow worked correctly (including versioneer)
• Users are able to switch to uv and instructions are clear
• The update-bot workflow creates helpful and correct PRs in a timely manner

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.15%. Comparing base (e719c7f) to head (70d46ee).
Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3294      +/-   ##
==========================================
- Coverage   90.48%   90.15%   -0.33%     
==========================================
  Files         472      463       -9     
  Lines       59276    58720     -556     
  Branches     5637     5627      -10     
==========================================
- Hits        53633    52937     -696     
- Misses       4205     4345     +140     
  Partials     1438     1438              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[superlopuh]

Could you please also migrate the makefile? Maybe with a phony target that checks whether uv is installed, and directs developers to installation instructions. The goal would be for make tests to be runnable outside of an activated venv

[math-fehr]

Thanks a lot for the PR!
A few quick questions/requests I have:

• Can we move the removed requirements.txt in the pyproject.toml? That way we can still track them with dependabot, and also this is the only change that's missing to still support virtualenv. I'd personally would like to still continue supporting venv if possible, as for now it is still way more popular than uv.
• Do you think we can automate the update of uv.lock with dependabot? Currently, we get daily automated updates of dependencies with dependabot, and it would quite annoying to have to update the uv.lock manually every time.
• We will probably have to test pythonpublish after the PR gets merged. Just ping me and I'll try to run it to see if it works or not, but looking at it this should be fine I think.

[math-fehr]

Are these dependencies moved to uv.lock?
I feel that this should rather go to the .toml file so that we can track them with dependabot, it was also bad before when they were here, but at least we had a list of them.

[superlopuh]

No, these are in the pyproject.toml, and are installed via --all-extras

[alexarice]

We've been using uv for our quantum stack recently and it seems to work nicely. I have a couple of comments:

• When to use uv run. It seems in the test ci you are activating the venv anyway so I don't see the reason to us uv run everywhere.
• If you did want to use uv run everywhere, you would need to make the filecheck tests use it (i.e. every xdsl-opt should be uv run xdsl-opt). You can get most of these by modifying lit.cfg. I'm not sure if this is actually worth it?
• I've always had problems with the wgpu package and had to manually remove it. Will there be an easy way to do that with the new set up?
• uv should be added to flake.nix (I can do this afterwards if necessary)

[EdmundGoodman]

Have just resolved the merge conflicts, revealing two final issues:

• flake.lock will need to be regenerated to include both nodejs and uv
• Failing pipeline runs for wgpu tests (https://github.com/xdslproject/xdsl/actions/runs/12200467352/job/34036682236?pr=3294) to be resolved

[alexarice]

* `flake.lock` will need to be regenerated to include both `nodejs` and `uv`

If you have nix installed, you can run nix flake update, otherwise I can do it and commit to this branch.

[superlopuh]

Almost there!

The three things I see that are still not working are:

1. the Python-based Testing CI seems not to install pytest in the preparation step, should be easy enough to debug locally
2. the ONNX type errors are similarly because onnx is not installed in that CI action, this is weird to me because it's the same command as the MLIR-based Testing one... There are some differences in the steps taken before the install command, one of which changes the location of the ci, maybe that affects whether uv finds the lockfile, or something like that? Just an idea
3. The test failure to do with a mismatching string is a little weird, because we already test 3.10, 3.11, and 3.12, so it looks like the uv-distributed Pythons have a slightly different error message, it should be easy enough to add it to one of the options in the regex in the test

If you give me commit access I'm happy to do point 3. quicky

[superlopuh]

Thanks again @EdmundGoodman !