OS error, too many open files while running pixi on a project first time

[adrinjalali]

On this PR (skops-dev/skops#451), if I remove pixi.lock and .pixi/envs and try to run pixi run tests I get the following error:

$ pixi run -e ci-sklearn15 tests
  ⠒ dev:linux-64         [00:00:03] [━━━━━━━━━━━━━━╾─────]  157/223  extracting pypi packages
  ⠂ creating python environment to solve pypi packages for 'ci-sklearn17'
    ✔ validate cache       168 packages in 587ms
    ⠂ download & extract   [00:00:00] [━╾──────────────────] 5.57 MiB @ 5.17 MiB/s pillow (+8)
    ⠂ installing packages  [00:00:00] [━━━━━━━━━━━━────────]   103/178 qt6-main (+64)
  ⠒ creating python environment to solve pypi packages for 'docs'
    ✔ validate cache       173 packages in 379ms
    ⠒ download & extract   [00:00:00] [╾───────────────────] 286.54 KiB @ 334.99 KiB/s scipy (+1)
    ⠒ installing packages  [00:00:00] [━───────────────────]     6/184 qt6-main (+166)
  ⠙ creating python environment to solve pypi packages for 'ci-sklearn15'
    ✔ validate cache       162 packages in 16ms
    ⠁ download & extract   [00:00:00] [────────────────────]      0 B @ 0 B/s    matplotlib
    ⠙ installing packages  [00:00:00] [────────────────────]     0/181 qt6-main (+161)
  ⠙ creating python environment to solve pypi packages for 'ci-sklearn16'
    ✔ validate cache       162 packages in 73ms
    ⠉ installing packages  [00:00:00] [────────────────────]     0/180 qt6-main (+161)
    ⠉ validate cache       [00:00:00] [━━━━━━━━━━━━━━━━━━━━] python (+1)
    ⠉ download & extract   [00:00:00] [────────────────────]      0 B @ 0 B/s    catboost (+6)
    ⠉ installing packages  [00:00:00] [────────────────────]     0/199 qt6-main (+83)                                                                                                           × failed to download pypi name mapping
  ├─▶ Failed to create a temp file at /home/adrin/.cache/rattler/cache/conda-pypi-mapping/tmp
  ╰─▶ Too many open files (os error 24) at path "/home/adrin/.cache/rattler/cache/conda-pypi-mapping/tmp/.tmpgUt6Ip"

pixi version:

$ pixi --version
pixi 0.38.0

So I'd need to generate the lock file step by step, bu commenting out most in this section

[tool.pixi.environments]
lint = ["lint"]
docs = ["docs"]
dev = ["docs","tests", "lint", "dev", "rich"]
ci-min-deps = ["rich", "tests", "lint", "min-deps"]
ci-sklearn11 = ["rich", "tests", "lint", "sklearn11"]
ci-sklearn12 = ["rich", "tests", "lint", "sklearn12"]
ci-sklearn13 = ["rich", "tests", "lint", "sklearn13"]
ci-sklearn14 = ["rich", "tests", "lint", "sklearn14"]
ci-sklearn15 = ["rich", "tests", "lint", "sklearn15"]
ci-sklearn16 = ["rich", "tests", "lint", "sklearn16"]
ci-sklearn17 = ["rich", "tests", "lint", "sklearn17"]

and then iteratively uncommenting them and re-running the pixi command to have them added to the lock file.

cc @glemaitre

[moritzwilksch]

workaround that has helped me before is increasing the ulimit: ulimit -n 10000

[adrinjalali]

I think with newer versions of pixi (which I cannot easily use since it's not released in archlinux yet) one can set concurrency params in .pixi/config.toml and that might help with the issue. But it's still nice to catch this error and automatically limit the open files count no matter what the user settings are.

[baszalmstra]

We are already trying to increase the ulimit at startup. If you run pixi -v you should see at the start what is happening there.

[traversaro]

fyi @xela-95 this is probably related to the issue you were experiencing in ami-iit/comodo#23 (comment) .

[adrinjalali]

We are already trying to increase the ulimit at startup. If you run pixi -v you should see at the start what is happening there.

I don't think pixi should try to change ulimit. There's probably a good reason in many cases to have the values as they are. As a sysdamin, I don't think I'd like a process to change those limits.

[baszalmstra]

As far as I understand it there is a hard and soft limit. Pixi will never be able to set the value higher than the hard limit.