Poetry fails in CI/CD with ERROR: In --require-hashes mode, all requirements must have their versions pinned

[shadycuz]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04.5 LTS
• Poetry version: 1.1.4
• Link of a Gist with the contents of your pyproject.toml file: Feature: Unit Test Cloudformation Templates DontShaveTheYak/cloud-radar#5

Issue

My builds were working fine locally and remotely. Here is an example of the first 12 or so lines...

2020-11-28T13:25:00.7733264Z ##[group]Run nox
2020-11-28T13:25:00.7733666Z �[36;1mnox�[0m
2020-11-28T13:25:00.7772013Z shell: /bin/bash -e {0}
2020-11-28T13:25:00.7772325Z env:
2020-11-28T13:25:00.7772772Z   pythonLocation: /opt/hostedtoolcache/Python/3.8.6/x64
2020-11-28T13:25:00.7773219Z ##[endgroup]
2020-11-28T13:25:00.8815564Z nox > Running session lint-3.9
2020-11-28T13:25:00.8867576Z nox > Session lint-3.9 skipped: Python interpreter 3.9 not found.
2020-11-28T13:25:00.8869448Z nox > Running session lint-3.8
2020-11-28T13:25:00.8870400Z nox > Creating virtual environment (virtualenv) using python3.8 in .nox/lint-3-8
2020-11-28T13:25:01.6319387Z nox > poetry export --dev --format=requirements.txt --output=/tmp/tmp2554a7nu
2020-11-28T13:25:04.4828117Z nox > pip install --constraint=/tmp/tmp2554a7nu flake8 flake8-black flake8-bugbear flake8-import-order
2020-11-28T13:25:11.0176722Z nox > flake8 src tests noxfile.py

I added a package with poetry add cfn-flip

My builds still work locally but fail in github actions:

2020-12-10T12:32:26.6270480Z ##[group]Run nox
2020-12-10T12:32:26.6270955Z �[36;1mnox�[0m
2020-12-10T12:32:26.6313567Z shell: /bin/bash -e {0}
2020-12-10T12:32:26.6313932Z env:
2020-12-10T12:32:26.6314480Z   pythonLocation: /opt/hostedtoolcache/Python/3.8.6/x64
2020-12-10T12:32:26.6315036Z ##[endgroup]
2020-12-10T12:32:26.7301178Z nox > Running session lint-3.9
2020-12-10T12:32:26.7352891Z nox > Session lint-3.9 skipped: Python interpreter 3.9 not found.
2020-12-10T12:32:26.7354096Z nox > Running session lint-3.8
2020-12-10T12:32:26.7357585Z nox > Creating virtual environment (virtualenv) using python3.8 in .nox/lint-3-8
2020-12-10T12:32:27.5320470Z nox > poetry export --dev --format=requirements.txt --output=/tmp/tmp2k2e6e06
2020-12-10T12:32:30.7810961Z nox > pip install --constraint=/tmp/tmp2k2e6e06 flake8 flake8-black flake8-bugbear flake8-import-order
2020-12-10T12:32:32.4291735Z nox > Command pip install --constraint=/tmp/tmp2k2e6e06 flake8 flake8-black flake8-bugbear flake8-import-order failed with exit code 1:
2020-12-10T12:32:32.4293643Z Collecting flake8
2020-12-10T12:32:32.4300409Z ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
2020-12-10T12:32:32.4303405Z     flake8 from https://files.pythonhosted.org/packages/d4/ca/3971802ee6251da1abead1a22831d7f4743781e2f743bd266bdd2f46c19b/flake8-3.8.4-py2.py3-none-any.whl#sha256=749dbbd6bfd0cf1318af27bf97a14e28e5ff548ef8e5b1566ccfb25a11e7c839

Since my builds still worked locally I thought it might be an issue with PyPi, so I waited a day, but they are still failing. Up above I have linked my PR so you should be able to see the changes to the toml and the lock file. I'm very unsure what to try next except maybe deleting the lock file and having poetry recreate it again.

Locally and in the CI/CD I'm running the same Nox and Poetry versions. I think this might be related to python-poetry/poetry-plugin-export#38 and possibly python-poetry/poetry-plugin-export#145 since both of those are about poetry export.

[trishankatdatadog]

I am having the same issue. Not sure how to fix.

[shadycuz]

@trishankatdatadog I had more time to investigate and while I haven't found any fixes I do think I have started narrowing down the problem space. I still can't rule out poetry for sure but it seems like it's a pip or python issue from a change to the underlying Github actions servers. You can see a link to my issue with them above ^.

[trishankatdatadog]

@shadycuz let me know what you find, because I gave up, would rather focus on my code right now

[Corfucinas]

I'm having the same problem.

This is my log

nox > Command pip install --constraint=/tmp/tmp2khsb66t xdoctest failed with exit code 1:
Collecting xdoctest
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    xdoctest from https://files.pythonhosted.org/packages/36/59/e70a32f7d5e2c4933bda64991ce5254f553c31ddd918f7b9a7e732cdd1e2/xdoctest-0.15.0-py2.py3-none-any.whl#sha256=695ea04303a48cbb319709270d43f7bae7f3de3701aec73f09d90a216499992e
nox > Session xdoctest-3.9.1 failed.

I do have my hashes in requirements.txt

xdoctest==0.15.0 \
    --hash=sha256:695ea04303a48cbb319709270d43f7bae7f3de3701aec73f09d90a216499992e \
    --hash=sha256:7f0a184d403b69b166ebec1aadb13c98c96c59101e974ae2e4db4c3a803ec371

[sinoroc]

1. Why is it filed as a poetry issue?

2. Read pypa/pip#8792 (and probably some other pip tickets as well)

[sinoroc]

Looks like you are all using the same pattern, but I am not sure I understand it and whether or not it makes sense. Would you mind walking me through it with a minimal reproducible example?

---

My gut feeling is that it boils down to pip install Alpha Bravo==1.2.3 --hash=xyz:abc, and pip is very clear that you can't have requirements both with hash and without hash in the same installation command. On the other hand you seem to mean that it used to work, and/or does still work in some cases...

[trishankatdatadog]

Would you mind walking me through it with a minimal reproducible example?

This is the smallest reproducible example I can find right now. Does this help?

[sinoroc]

I'd prefer you walk me through the thinking process that lead you to using this pattern.

• Why do you do what you do?
• Why do you export into requirements.txt then use this as a constraint file to install development dependencies?
• What are the other things you have tried and did not work?
• Why don't you install all development dependencies?
• And so on

I have my answers for most of these questions, but they probably don't match yours. Point is: maybe there is a better alternative.

[shadycuz]

@Corfucinas

I do have my hashes in requirements.txt

xdoctest==0.15.0 \
    --hash=sha256:695ea04303a48cbb319709270d43f7bae7f3de3701aec73f09d90a216499992e \
    --hash=sha256:7f0a184d403b69b166ebec1aadb13c98c96c59101e974ae2e4db4c3a803ec371

Thanks for posting this. I was running my builds in CI/CD and did not have access to the contents of the exported file. It's great that we can confirm the hashes are present.

[trishankatdatadog]

I'd prefer you walk me through the thinking process that lead you to using this pattern.

All good questions. It very largely came from hypermodern Python. I'll dig deeper and see what broke.

What's strange is that the nox setup works locally on my machine but not remotely on GitHub.

As for hashes in requirements.txt, I can confirm that when I rerun the offending command on my machine, I see the hashes.

[shadycuz]

@sinoroc

1. Why is it filed as a poetry issue?

2. Read 2020 resolver does not consider hashes in the constraints file pypa/pip#8792 (and probably some other pip tickets as well)

I filed this as a poetry issue because I use poetry to manage my dependencies and not pip. I understand it's most likely an issue with something other than poetry since I did not change the version of poetry when this problem occured, but since this issue affects other users of poetry it was probably a good place to create it in the end.

I'd prefer you walk me through the thinking process that lead you to using this pattern.

@trishankatdatadog and I both got the pattern from the excellent blog series Hypermodern Python. I'm not sure about @Corfucinas

• Why do you do what you do?
• Why do you export into requirements.txt then use this as a constraint file to install development dependencies?
• What are the other things you have tried and did not work?
• Why don't you install all development dependencies?
• And so on

To be honest I do it because the blog told me so. I didn't spend much time thinking about it. I know the blog has a companion repo. The companion repo seems to have evolved beyond what was instructed in the blog posts. It seems they found better ways to do things. I unfortunately didn't realize this until after following the now outdated blog posts. I can say that up until this issue I was extremely happy with my new development workflow compared to just using pipand virtualenv in the past.

I have my answers for most of these questions, but they probably don't match yours. Point is: maybe there is a better alternative.

If you have a fix I would be eager to test it, even if the fix is just changing how we install our development dependencies for nox.

This workflow still works fine locally. I'm guessing its something else that changed not related to poetry. I have confirmed that the github actions did change the underlying image used for it's runners. I have issue with them as well.

[shadycuz]

@Corfucinas @trishankatdatadog

Seems like this was already discussed and solved in the Hypermodern repo.

[shadycuz]

@sinoroc Yes you linked directly to the issue =) with pip.

[shadycuz]

I feel good enough about closing this now. The issue is with the change to how pip handles the constraints flag. pip maintainers have confirmed it's not a bug but a design choice based on the new pip resolver.

Workarounds:

• For Hypermodern python users pass this flag in your poetry export command
  --without-hashes,
• Stop passing in packages with pinned hashes using the pip --contstraints flag.
• If you are using pip directly then pin it to a version before 20.3
  python -m pip install --upgrade pip==20.2.4
• If you are using something else like virtualenv that depends on pip, make sure you pin it.
  python -m pip install --upgrade virtualenv==20.0.26
  Or use a env var VIRTUALENV_PIP=20.2.4

[sinoroc]

@shadycuz Be careful with pinning too eagerly. I guess you all know that by now. :)
Additionally I do not believe pinning virtualenv has decisive influence on the pip version used.

I do not know if the message went through to you all, so I will repeat it here. From what I understood, the issue is triggered by having in the same pip install command requirements both with hashes and without hashes. So things like pip install Something --constraint export.txt will always fail if export.txt contains hashes, since Something has no hash on the command line. So indeed you can:

• either export without hashes
• or remove the Something and install everything with pip install --requirement export.txt

Both have their pros and cons. Altogether the integration between poetry and tox (nox) could be improved. I started a plug-in to attempt to fix this integration (tox-poetry-dev-dependencies), but it is only half way there.

Anyway, looks like for now the best compromise for your use cases might be to export without hashes.

Some things to look for in the future:

• Support dependencies groups (#1644) scheduled for v1.2
• https://github.com/python-poetry/poetry/issues/2010

[alexifm]

Wanted to quickly chime in that nox-poetry has just been updated to address this: https://github.com/cjolowicz/nox-poetry/releases/tag/v0.7.0

[remram44]

This happened to me because of environment markers, I was running pip on a version of Python that didn't match the tool.poetry.dependencies.python set in pyproject.toml. Therefore some dependencies were not specified for that Python version in requirements.txt, and pip would fail (it would implicitly look for a version of that lib to satisfy dependencies of other requirements, then fail with this obtuse message about hashes).

E.g. I have python = "^3.7" and my requirements.txt contains:

advocate==1.0.0
urllib3==1.26.4; python_version >= "3.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4" and python_version >= "3.7"

When running that through pip on Python 3.6, no version of urllib3 is pinned, but advocate depends on urllib3. Pip would grab some version of urllib3 that satisfies advocate's dependencies and then complain that urllib3 does not specify hashes (the requirement it grabbed automatically has no hashes, even though the requirements.txt hash hashes).

Collecting urllib3<2.0,>=1.22
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    urllib3<2.0,>=1.22 from https://files.pythonhosted.org/packages/5f/64/43575537846896abac0b15c3e5ac678d787a4021e906703f1766bfb8ea11/urllib3-1.26.6-py2.py3-none-any.whl#sha256=39fb8672126159acb139a7718dd10806104dec1e2f0f6c88aab05d17df10c8d4 (from advocate==1.0.0->-r requirements.txt (line 1))

Hopefully that helps somebody else.

[abend-arg]

pypa/pip#9243 was reopened so I think this ticket should be reopened again. I ran into this same issue while trying to automatize one pipeline today.

[shadycuz]

Reopening as the original fix has been reverted but is being tracked here pypa/pip#11019

[dimbleby]

What is this issue saying that poetry is doing wrong or should do differently? It's a long and confusing thread.

Is #5537 relevant? That fixed an issue where poetry export for a package with extras resulted in the same error message in this issues title (when trying to pip install the exported requirements.txt)

[shadycuz]

To be fair @dimbleby , I dont know and I never knew or if I did know, I dont anymore ;)

It's been referenced a lot and has had a lot of activity. I know that when pip changed something. Poetry went from working to not working. Then pip was fixed and it started working again. I think this issue was just to point people to a workaround and possible action from poetry maintainers if they needed to do something to poetry internals.

I haven't had the problem reoccur but it's also a very complicated problem. With the next release of pip. I'm guessing my original problem comes back. So I reopened the issue.

If contributors think this issue is no longer useful, we can close it again. That way people experiencing issues could create new issues with less chatter and hopefully, someone can describe their problem in a much more elegant way than I ever could.

Special shoutout to Poetry and its contributors, who have made creating and maintaining python applications an enjoyable experience. I really feel bad for all the projects still using setup.py and requirements.txt files.

[dimbleby]

I agree that lots of people seem to think that Something is Wrong, but no-one seems to have identified any relevant way in which poetry could be improved. In that context I don't see that an open issue here is doing any good - what should the maintainers to do with it? Indeed, if there's not a poetry problem: why should poetry maintainers do anything with it at all?

(So far as being a pointer to a workaround goes: a closed issue works just as well as an open issue.)

If you or anyone else can make the case that your error is attributable to poetry then please do: that would be very helpful. Otherwise I'd suggest that this is better closed.

[shadycuz]

The problem with a closed issue is that it signals that the problem doesn't exist or was fixed. This current problem doesn't exist currently, but it will for sure soon.

While I agree their doesn't seem to be much the poetry maintainers can do if it's a third-party issue, it is still relevant to the users of poetry who will encounter this issue and by default, the search on a repo does not look at closed issues.

But I think its time to close this, as its long and convoluted and I dont remember any of the details of how to reproduce.

[graingert]

I can reproduce this issue, but it's only related to the poetry export requirements.txt file and the specific use in nox (which was worked around by disabling security)

This isn't something that can be fixed in peotry, but it could be something that peotry can provide a workaround for eg a new command like poetry contrained-venv-install ./venv/ flake8 flake8-black flake8-bugbear that uses the poetry lock file to update the specified venv with a subset of deps

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.