Cache git dependencies as wheels

[maksbotan]

Currently, poetry install will clone, build and install every git dependency when it's not present in the environment. This is OK for developer's machines, but not OK for CI - there environment is always fresh, and installing git dependencies takes significant time on each CI run, especially if the dependency has C extensions that need to be built.

This commit builds a wheel for every git dependency that has precise reference hash in lock file and is not required to be in editable mode, stores that wheel in a cache dir and will install from it instead of cloning the repository again.

Pull Request Check List

Resolves:

• Added tests for changed code.
• Updated documentation for changed code.

I've checked locally that this code works, can you give me an advice on how test could look like?

Should this approach also be extended to handle 'tar.gz'-only distributions? See #3439 for rationale.

[neersighted]

Overall the approach looks decent, I think it needs some refinement but this is a productive path. However, this is pretty low priority in the review backlog -- it will probably take some time to give detailed feedback.

[maksbotan]

Thanks! In the meantime I'll try to extend the approach to tar.gz distributions and write a test.

Could you please set hacktoberfest-accepted label on this pr?

[dimbleby]

I'd be much less enthusiastic about this, it brings complexity to poetry - and increases reliance on pip - for what seems like a niche use case.

If you're repeatedly using the same commit of a project, perhaps it would be better simply to build a wheel from that commit, and rely on that wheel.

[neersighted]

Poetry is not participating in Hacktoberfest; participation is opt-in this year (labeled repositories only) and we have not chosen to opt in.

@dimbleby I agree it adds complexity and I think the implementation will need to change a decent amount to land this. However, I also think that this does not increase reliance on pip, as we were building those git clones like directory deps when they were non-Poetry projects anyway, and the way this PR is architected makes it easy to swap in build later when we start removing pip.

That being said, there's nothing that stops us from incrementally introducing build here -- would you be interested in attempting a version of this PR making use of build, @maksbotan?

[maksbotan]

@neersighted re Hacktoberfest — got it, no problem.

As for the use case, I don't think it's niche. IMO it affects everyone who runs CI jobs for projects with git dependencies and/or tar.gz only distributions.

I'll try to update the PR following your suggestion about build. You mean the code that is used to install packages from directories? To change it to build wheels?

BTW, what is the point of removing pip?

[neersighted]

Your PR has moved from using the code we use for directory deps to a standalone solution for building and caching wheels; I was explaining that to @dimbleby. I would suggest that you preserve the scope of this PR, and experiment with build as the build tool (via the Python API) instead of pip.

We are working to extract pip from Poetry for a variety of reasons. These include:

• pip is an implementation detail and users try to configure it because they are mislead by thinking Poetry's architecture/design is similar to Pipenv
• pip has a massive surface area that frequently causes bugs
• It would be nice to not require pip to be present in every environment Poetry interacts with
• pip is a major performance limitation
• The ecosystem strongly benefits from multiple implementations of standards

You can see the WIP several maintainers are working on #6205.

[maksbotan]

Thank you for the response.

I still don't quite get what you mean by build. Stuff like EditableBuilder, SdistBuilder, etc?

[neersighted]

Thank you for the response.

I still don't quite get what you mean by build. Stuff like EditableBuilder, SdistBuilder, etc?

I mean the PyPI package, which has an API.

[maksbotan]

Ah, thanks, now I see what you mean. I've never heard of that think before, will take a look.

[maksbotan]

Hi @neersighted!

I finally had time for this PR again. I've reworked my changes on top of PR #6205 which you mentioned. There is a lot of work done there, so it was actually pretty easy to add caching of wheels for git dependencies.

I've set this PR's base branch to PR #6205, so that it can be incorporated into that feature once reviewed.

Please tell me what do you think about this.

[maksbotan]

@neersighted hi! Can I ping you on this one? This is an useful feature IMO, and PR code is now actually pretty concise.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.