How to add a dependency with --no-binary option

[dazza-codes]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Question

How is a dependency added so that poetry will run something like this:

pip3 install -U 'numpy >=1.17,<2.0' --no-binary numpy

What does that look like in the pyproject.toml file?

PS. See a work-around using poetry export in #365 (comment)

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[chipfranzen]

This is still an issue. The spatial library shapely requires installing with a no-binaries flag on macOS. See this issue: shapely/shapely#416

[geckon]

I'm also interested in this.

[Eisbrenner]

Cartopy is another library which requires the no binary flag, since I work with geospatial data I hardly want to miss out on it and would really appreciate an implementation into poetry.

And further, it is possible to install such packages with poetry, using something like NO_BINARY_SOMETHING=true poetry install? Is this in the documentation somewhere?

[abn]

This is an interesting case. I think the more, elegant solution would be to be able to specify a per-dependency requirement. The issue however, is that this is ususally a PEP 517 frontend task, which generally falls outside the scope of poetry's core features. That said, since we are moving more towards being more independent of tools like pip for installation, specifying formats for a dependency to be installed might become more relevant.

We might need to discuss where we draw the line first, before enabling a feature like this. I do think think this is unavoidable.

For now, a workaround might be to use a url dependency pointing to an sdist along with a platform marker.

[sondrelg]

I'm having this same issue.

@abn would you be able to provide a working example of your suggested workaround?

[sbreidbach-edr]

We're in the same position with pygeos - we've tried setting PIP_NO_BINARY as suggested in #365 but still face the same errors.

[abn]

@abn would you be able to provide a working example of your suggested workaround?

shapely = { url = "https://files.pythonhosted.org/packages/42/f3/0e1bc2c4f15e05e30c6b99322b9ddaa2babb3f43bc7df2698efdc1553439/Shapely-1.7.1.tar.gz" }

We're in the same position with pygeos - we've tried setting PIP_NO_BINARY as suggested in #365 but still face the same errors.

This will nto work with 1.1.0 as we no longer use pip as the backend to discover and download artifacts.

[fgimian]

I just came across this with confluent-python. In order to install it with SASL support, you must supply the --no-binary flag so it compiles against the librdkafka headers and libraries.

In fact, their full install command is:

pip install --no-binary :all: confluent-kafka

Sadly this doesn't seem to be supported by my beloved Poetry 😄

[ziotom78]

We are facing a similar problem in our package (https://github.com/litebird/litebird_sim). We depend on a numerical C++ package that requires significant memory to be built, and sometimes this makes Readthedocs builds crash. What we would like to do is to have the following situation:

1. No binary wheels should be used when installing the package, as the C++ dependency detects the features of the host CPU at compile time and enables a number of optimizations that can significantly improve its runtime performance;

2. When building the docs, the performance of C++ code is not relevant and we can stick to binary packages to reduce the memory load and make Readthedocs happy.

We manage package builds using Poetry and documentation builds using a plain old requirements.txt file. However, we do not know how to tell Poetry to avoid using binary wheels.

[JordyScript]

Trying to install Kivy on Pop_OS (Linux) using the wheel that Pypi provides results in broken audio. The fix is to install the relevant dependencies and install kivy with the --no-binary. I can't use poetry to install kivy to the virtual environment because of this issue.

It would be great if the --no-binary option could be integrated into the CLI of poetry.

Just to be able to type:
poetry add --no-binary kivy kivy

Would be a huge improvement, but this script is a good workaround so far.

[remram44]

I am running into that. Currently using sed to remove the binary wheels from poetry.lock:

sed -i '/{file = "pmdarima.*\(32\|64\|686\|386\)\.\(whl\|egg\)"/d' /src/poetry.lock

[sneakers-the-rat]

+1 for this, big problem for constrained OSes like raspiOS where wheels are often incompatible with system packages (eg. numpy wheels all require GLIBC >= 2.29 and RaspiOS buster, which is still required for lots of embedded applications, ships with 2.28)

[vivienmast]

I am having the same issue, as I need argon2-cffi-bindings for my project and there seems to be no wheel compatible with my specific MacOS platform.

[rupurt]

+1 to this being an issue. Having a lot of trouble installing tensorflow-macos for M1 and it's dependency h5py

[sneakers-the-rat]

think this is part of a cluster of issues, tried to start a discussion here: #4137 (comment)

[perlow]

Having the same issue.

[rphel]

Same issue here. It's impossible to use geospatial libraries as a compilation of Shapely/Cartopy is needed. All our efforts to migrate from pip to poetry are wasted by this issue. Is there a way to help?

[markthor]

+1 on this being an issue. Trying to make our packages fit the AWS lambda size limit of 250mb. This issue is currently making all our efforts of migrating to poetry a waste.

[remram44]

@markthor I don't think this issue affects any size limit. Whether installing from binary or source, the size of the final installation should be the same.

[markthor]

@remram44 I am not an expert on this subject, so I might have misunderstood something. But when i do poetry run pip install --force-reinstall numpy --no-binary numpy the size of my .venv is reduced by approximately ~60MB, and the code runs exactly the same. Might be a macOS specific thing.

[sneakers-the-rat]

@remram44 I am not an expert on this subject, so I might have misunderstood something. But when i do poetry run pip install --force-reinstall numpy --no-binary numpy the size of my .venv is reduced by approximately ~60MB, and the code runs exactly the same. Might be a macOS specific thing.

this basically overrides the lockfile. poetry run executes a command inside the venv shell, which in this case is forcing a reinstall of numpy from a wheel specified by the repositories available in the venv (which as far as I can tell is the same as those configured for pip globally, rather than conditioned in any way by the pyproject.toml file)

[agates]

@remram44 I am not an expert on this subject, so I might have misunderstood something. But when i do poetry run pip install --force-reinstall numpy --no-binary numpy the size of my .venv is reduced by approximately ~60MB, and the code runs exactly the same. Might be a macOS specific thing.

this basically overrides the lockfile. poetry run executes a command inside the venv shell, which in this case is forcing a reinstall of numpy from a wheel specified by the repositories available in the venv (which as far as I can tell is the same as those configured for pip globally, rather than conditioned in any way by the pyproject.toml file)

It is in fact bypassing the wheel, the very purpose of --no-binary, which many of us need to do for various reasons as the wheel often does not include features we need, or vice versa.

This is the entire point of this issue. He's bypassing the lockfile because poetry doesn't support --no-binary.

[remram44]

@markthor I just tested this and got the opposite result, my build from source (using pip) is much bigger than the binary wheel installation. Every .so is 4 times bigger.

Either way, my current workaround of changing the files in the poetry.lock [metadata.files] section has worked great for me, I can remove files given any criteria I want, and have the CI check that it's been done. Probably worth having Poetry support such a common case, but consider this approach if it's blocking you.

[sneakers-the-rat]

@remram44 I am not an expert on this subject, so I might have misunderstood something. But when i do poetry run pip install --force-reinstall numpy --no-binary numpy the size of my .venv is reduced by approximately ~60MB, and the code runs exactly the same. Might be a macOS specific thing.

this basically overrides the lockfile. poetry run executes a command inside the venv shell, which in this case is forcing a reinstall of numpy from a wheel specified by the repositories available in the venv (which as far as I can tell is the same as those configured for pip globally, rather than conditioned in any way by the pyproject.toml file)

It is in fact bypassing the wheel, the very purpose of --no-binary, which many of us need to do for various reasons as the wheel often does not include features we need, or vice versa.

This is the entire point of this issue. He's bypassing the lockfile because poetry doesn't support --no-binary.

sorry ya i was just speaking loosely which in retrospect is the opposite of the meaning i intended. wheel -> tarball is what i meant. basically i agree and also think it's a thing poetry should support, and have tried to sketch some ideas about how that could work over here in the context of specifying sources/architectures ( #4137 (comment) ) because from what i can tell about the library it could be approached as a general means of passing arguments from pyproject to the solver to the lockfile

edit: spelling

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.