Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poetry artifact cache breaks pip's wheel cache #3439

Closed
3 tasks done
maksbotan opened this issue Dec 1, 2020 · 8 comments
Closed
3 tasks done

Poetry artifact cache breaks pip's wheel cache #3439

maksbotan opened this issue Dec 1, 2020 · 8 comments
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged

Comments

@maksbotan
Copy link
Contributor

  • I am on the latest Poetry version.

  • I have searched the issues of this repo and believe that this is not a duplicate.

  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

  • OS version and name: debian:buster in Docker

  • Poetry version: 1.1.3

Issue

Hi,

Starting with some version poetry started to store pypi artifacts in ~/.cache/pypoetry/artifacts, including wheels and source .tar.gz distributions when there are no wheels on pypi.

Later poetry calls pip install with an absolute path to those downloaded artifacts. However, in this mode pip does not look into its own cache for already built wheels and always rebuilds from source (it does store the resulting wheel in cache though).

When the package in question contains C extensions, for example, this behavior causes long build times. In my particular scenario, I run poetry install a lot in CI jobs and I'd like to benefit from caching the already built wheels.

Is there anything that can be done to mitigate this?

@maksbotan maksbotan added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Dec 1, 2020
@1ace
Copy link
Contributor

1ace commented Jan 19, 2021

I second that; it's extra annoying in a CI where you run the same poetry install thousands of times and you have to recompile the same packages every time instead of installing the one that's right there in the cache.

@1ace
Copy link
Contributor

1ace commented Jan 19, 2021

I've been digging through the code, and it's a bit unclear to me if poetry is supposed to do something differently, or if pip is the one who's supposed to figure out it needs to check its cache before re-building a wheel?

I think it's the latter, in which case this issue can be closed and one should be open in https://github.com/pypa/pip/issues (if there isn't already one, but I haven't found one)

@sdispater any thoughts?

@1ace
Copy link
Contributor

1ace commented Jan 19, 2021

Something like this might do it:

diff --git a/poetry/installation/pip_installer.py b/poetry/installation/pip_installer.py
index df1249737a1fe21b9122..ff54ac6d8589e16a58f3 100644
--- a/poetry/installation/pip_installer.py
+++ b/poetry/installation/pip_installer.py
@@ -6,6 +6,11 @@
 
 from clikit.api.io import IO
 
+try:
+    from pip._internal.utils.appdirs import user_cache_dir
+except ImportError:  # pip < 10
+    from pip.utils.appdirs import user_cache_dir
+
 from poetry.core.pyproject.toml import PyProjectTOML
 from poetry.repositories.pool import Pool
 from poetry.utils._compat import encode
@@ -34,6 +39,6 @@ def install(self, package, update=False):
 
-        args = ["install", "--no-deps"]
+        args = ["install", "--no-deps", "--find-links", user_cache_dir('wheel')]
 
         if (
             package.source_type not in {"git", "directory", "file", "url"}
             and package.source_url

@Aloisius
Copy link

Aloisius commented Feb 26, 2021

#3732 should fix the issue. The problem is that pip will not check its wheel cache of previously built packages unless you pass in a requirements.txt file since the logic for it is in the link resolver. Poetry simply does a pip install --no-deps cache/artifacts/.../package-1.0.0.tar.gz which bypasses it.

The patch creates a temporary requirements.txt file and does a pip install --no-deps -r temp_requirements.txt on it. It seems to solve the problem for me, but if someone could test it to be sure.

$ poetry install
$ rm -rfi `poetry env info -p`
$ export PIP_LOG=$PWD/pip.log
$ poetry install

That should show it installing wheels it had previously built in the pip.log file.

@ironhalik
Copy link

Any news regarding this issue? Crypto is moving away from proof-of-work, and so could we stop building wheels constantly :)

@dimbleby
Copy link
Contributor

#6205 renders pip's cache irrelevant

@Secrus
Copy link
Member

Secrus commented Sep 25, 2024

We are no longer using pip as an installer, so this is no longer valid.

@Secrus Secrus closed this as completed Sep 25, 2024
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants