Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User-specified arguments for pip wheel #268

Closed
seoester opened this issue Feb 8, 2020 · 9 comments
Closed

User-specified arguments for pip wheel #268

seoester opened this issue Feb 8, 2020 · 9 comments

Comments

@seoester
Copy link

seoester commented Feb 8, 2020

I would like to see a way to specify additional arguments to pip wheel.

pip wheel acts as the build frontend here and the only "official" way to pass on arguments to the backend, e.g. a setuptools powered setup.py, is through the --build-option and --global-option flags.

More importantly, pip wheel builds the wheel in an isolated environment as suggested in PEP 517 ("Recommendations for build frontends"). In order to have any custom (i.e. not specified in pyproject.toml) build-time dependencies the --no-build-isolation argument of pip wheel has to be used.

I don't understand how the CIBW_BEFORE_BUILD can even be used to install build requirements as suggested in the documentation. pip wheel will creates a new virtualenv where the installed packages should not be available. Perhaps somebody can clarify?

(I'm currently looking into cibuildwheel in order to prepare wheels for a Cython-based package of mine. It looks like a great way to save boilerplate build scripts!)

EDIT: Fixed paragraph related to build isolation; Added second pip wheel option flag.

@joerick
Copy link
Contributor

joerick commented Feb 9, 2020

You could be right on this, it's something that comes up relatively frequently. Previously we've discussed the possibility of allowing the user to customise the whole pip wheel command, but it was dismissed as too cumbersome, too much internal logic making its way into the config files.

A previous discussion on this #120 also considered it, but in the end pyproject.toml had the features required. Specifically, the build-system/requires option might be of interest to you.

The other thing I can recommend is to try setting the PIP_NO_BUILD_ISOLATION environment variable.

Our examples, which install deps in CIBW_BEFORE_BUILD, don't assume the use of pyproject.toml. Build isolation is disabled if a pyproject.toml isn't found. Although maybe pyproject.toml use is going to increase - is that the recommended way to build a Cython project these days?

@seoester
Copy link
Author

seoester commented Feb 9, 2020

Thanks for the helpful response!

PIP_NO_BUILD_ISOLATION sounds sufficient for the purposes of using CIBW_BEFORE_BUILD. Perhaps the documentation can be updated with your notes on building with PEP 517 / PEP 518? I think many people will stumble upon this issue as the PEPs (hopefully) become the de facto way of building new and existing projects.

I found the state of documentation concerning setuptools, Cython and the PEPs unsatisfactory. Many recommendations are still aimed at old tooling and the official documentation does not inform well about the role of pyproject.toml.
I've been attempting to bridge the gap and write up a sensible configuration plus setup.py script.

The problem with Cython is that two kinds of procedures are required for the same project:

  • During packaging: Cython is required, files are transpiled (cythonize()). Sources (.pyx and the transpilation output .cpp) are included in the sdist. The extension is freshly built for the wheel.

    For these builds I need to pass extra options to pip wheel: I'm implementing an additional command transpile_cython which ensures that the .cpp files are up-to-date outputs of transpiling .pyx files. (Still WIP, there may be some issues with incomplete meta-data.)
    Notable projects using Cython, e.g. pandas, simply transpile whenever Cython is available (^= importable).
    But I believe an additional command is the way to go as it is more explicit and leads to increased visibility => reproducibility (this is also in line with recommendations by the Cython documentation).

  • During install from sdist: Cython should not be required, as the extension can be built from .cpp files alone.

    For this, I'm planning to remove Cython from pyproject.toml in the sdist command (during packaging).
    (However, I also see the argument that with build isolation build-system/requiring Cython does not matter, as it is installed only in the temporary build environment. Cython is simply not required during building.)

So that is a scenario where it is important to pass further options to pip wheel to be passed on to setup.py. I think it is an example of a specific procedure required during packaging vs. normal install.
While environment variables can be used instead, --build-option and --global-option are the proper way of passing information to setup.py.

@joerick
Copy link
Contributor

joerick commented Feb 10, 2020

Thank you for this. I think I see why CIBW_BEFORE_BUILD coupled with build isolation is causing you problems. Couple questions (I'm not a Cython user so forgive me if these are confused):

  • Your project manifest includes the transpiled .cpp files, right? Because they're included in the sdist. So they'd be copied over into the isolated build before the wheel is built. So what's wrong with ensuring these files are up-to-date in by running cythonize in CIBW_BEFORE_BUILD?

  • For your setup.py transpile_cython option, is there a reason why you'd much rather use a command line argument over an environment variable? Something like PROJECT_NAME_TRANSPILE_CYTHON=1 would seem to work, unless I'm missing something? Or just cleanliness?

On a side note, you've clearly thought a lot about this! Once you get it figured out, would you be interested in writing a guide that we could host in our docs of a 'recommended' modern way to build Cython wheels?

@seoester
Copy link
Author

  • Your project manifest includes the transpiled .cpp files, right? Because they're included in the sdist. So they'd be copied over into the isolated build before the wheel is built. So what's wrong with ensuring these files are up-to-date in by running cythonize in CIBW_BEFORE_BUILD?

cythonize() is "awkwardly" integrated with the whole distutils / setuptools process. I considered running it before passing over control to pip wheel, but this already requires executing setup.py (the CLI cythonize does not perform quite in the same way). That means a "pre-build" environment must be set-up. I feel like this violates the concept of build isolation and would require another set of tools essentially replicating what build-system/requires is supposed to achieve.

I'm the first to admit this is somewhat esoteric 😄

  • For your setup.py transpile_cython option, is there a reason why you'd much rather use a command line argument over an environment variable? Something like PROJECT_NAME_TRANSPILE_CYTHON=1 would seem to work, unless I'm missing something? Or just cleanliness?

It's possible, but I think it's less clean: PEP 517 defines a backend agnostic way of passing configuration on to the backend (c.f. "Config Settings"). This is implemented by pip wheel through --build-option and --global-option. I think including a way to pass this configuration should be reconsidered in cibuildwheel. Environment variables rely on implementation specifics and - according to my interpretation - circumvent the build isolation somewhat.

In addition, I want to make transpile_cython a command, not just a flag. That means I would have to parse the environment variable and then insert transpile_cython in some arbitrary place in sys.argv to be picked up (setup.py clean transpile_cython build_ext respects command order).

On a side note, you've clearly thought a lot about this! Once you get it figured out, would you be interested in writing a guide that we could host in our docs of a 'recommended' modern way to build Cython wheels?

I was planning to write a blog post anyway, I'd love to contribute back to the documentation :)

@YannickJadoul
Copy link
Member

It would be good if we can fully support this build isolation, though? I actually still haven't figured this out the difference with the old way of doing things, but PIP_NO_BUILD_ISOLATION feels a bit like a hack?

Do PIP_BUILD_OPTION and PIP_GLOBAL_OPTION work, by the way? I thought most of the pip flags can also be passed as environment variables?

@seoester
Copy link
Author

Do PIP_BUILD_OPTION and PIP_GLOBAL_OPTION work, by the way? I thought most of the pip flags can also be passed as environment variables?

That works, good catch!

@seoester
Copy link
Author

seoester commented Feb 25, 2020

However, the pip wheel frontend for PEP 517 does not support passing configuration to the PEP 517 backend yet pypa/pip#5771 :(
That includes --build-option and --global-option. Pip stops with an error when specified.

Until config passing is supported, this issue is essentially blocked. The general design of the integration with cibuildwheel could already be discussed.

@joerick
Copy link
Contributor

joerick commented Feb 26, 2020

Ah! That's a bummer. However I like the idea of a cibuildwheel option like CIBW_BUILD_CONFIG that might be used like CIBW_BUILD_CONFIG={"transpile_cython": true} - a JSON object that's passed to build backends via pip as config_settings.

That would allow arbitrary config of a build while cibuildwheel maintains control over how pip wheel is called.

But I guess that would have to wait for that feature to be implemented in pip.

All that said, if you're still interested in doing a Cython demo project/docs page using one of the workarounds above, I'm sure it would be of great value to Cython users out there :)

@henryiii
Copy link
Contributor

Config options were added in cibuildwheel 2.10! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants