What would be best practice for using Pixi in a shared environment setting? #2681
Replies: 2 comments 1 reply
-
could never find a solution that solves all the issues of working on HPC the closest idea was to have a shared repository of pixi manifests (initially figured one manifest with any number of environments but that gets difficult to manage) |
Beta Was this translation helpful? Give feedback.
-
This is a valid question, we indeed don't support a workflow that fits this directly. We have lots of floating ideas on improving this workflow. Currently, the project workflow is heavily focused on reproducibility of a project. But that is hard to combine with a centralized set of dependencies. There are a few ideas that might help you:
We're trying to get a good picture of what the HPC community would need. Some parts we already know of are:
If you have more ideas, let us know. |
Beta Was this translation helpful? Give feedback.
-
Background
Our group uses a Github project template which includes launch scripts to run various workflows. For the analyses we do, we make a new repository from the template, and then run that on our HPC. The launch scripts all use the same shared conda environment. Pixi, however, implements a project centered approach which would mean we would have the same environment recreated for every new analysis. This is problematic as we have file and file number quotas on our project directories and generally, using conda has been a major culprit in running out of space fast.
Question
So what would be the best practice here in regards to using Pixi? On the one hand each analysis maintains it's own lock for reproducibility, but on the other, for every new project we end up duplicating the same files we already use for other projects and this counts unproductively to our quota. If I understand correctly, the detached-environments doesn't solve this, but just provides an alternate path to store project environments. They wouldn't be shared across analyses from different repositories.
Beta Was this translation helpful? Give feedback.
All reactions