-
Notifications
You must be signed in to change notification settings - Fork 151
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
arm64-only package selects universal2
as the arch tag on macOS 11
#406
Comments
This code seems to be poorly structured and is probably the root cause. |
@henryiii When you have a chance, I would be curious to get your perspective on this ? |
To my knowledge, the code you refer to in packaging is the one used to know which platforms tag are acceptable to be installed (i.e. the other way around). is your CPython a universal2 one ?
Edit: I guess you're not using a "raw" setuptools build system which would have build with |
Thanks for getting to this, @mayeut! Still working though a messaging backlog. Maybe you are using Scikit-Build? I don't think we've worked through the details there yet, ideally it should respect the settings that control setuptools, but for now, maybe the info in pypa/cibuildwheel#582 is helpful? |
scikit-build with |
Yes.
Hrm, that's unfortunate.
Is this documented anywhere? I assume
This project (VTK) isn't doing universal builds yet (I mean it could, but the per-wheel size would be quite large by that point).
No, it's a CMake build which generates the
I'd really rather override just the architecture and not have to guess at what the rest of the platform name would be otherwise. |
This has come up often enough that I've started implementing #161. The question is, should allow any arbitrary tags or should they be checked against the list of supported tags? |
This could land in the next release but I need some kind of feedback! |
@brettcannon or @pfmoore might have opinions here? Hard for me to say without seeing the interface - I'm mostly interested in building wheels with the standard tools ( Scikit-build 0.12 should fully support targeting different platforms, and the competitor example, |
Sorry, I have no opinion here. I'm not a Mac user, personally. |
For implementing #407, be sure to look at https://github.com/scikit-build/cmake-python-distributions/blob/master/scripts/convert_to_generic_platform_wheel.py and/or https://github.com/scikit-build/ninja-python-distributions/blob/master/scripts/convert_to_generic_platform_wheel.py (which should be basically the same). The interface is different, but the core work is likely similar. If you are going to work on it, let me know - I'll try to ping you when I start working on it, as well - it would be good to have a basic mutex here. :) |
Is there a way to detect that the compiled files are ARM64 only? All this mac specific stuff was contributed by others so I'm pretty much in the dark here myself. |
Generally, ARCHFLAGS should be set correctly, and if so, I think the wheels come out with the correct names? If you are using CMake, you should either use scikit-build 0.12, which does this correctly, or use something like this if doing it by hand: https://github.com/pybind/cmake_example/blob/f6539807e764707f2ee2b9d7ba0c8d017d685853/setup.py#L93-L97 The same tooling that used to check for FAT binaries should work for the new arch, too, see https://stackoverflow.com/questions/1085137/how-do-i-determine-the-target-architecture-of-static-library-a-on-mac-os-x |
My case is that I have libraries built completely outside of In addition (though not in my case), some bits may be x86_64-only as some modules cannot be built for arm64 yet (e.g., due to the lack of a Fortran compiler meaning numpy and scipy are not yet available). Basically, I've been able to hack it up using |
I think it depends on the user experience you want. If you want to prevent potential typos, then check. But obviously complete flexibility regardless of what the tools we can all think of expect says let people type whatever. I would personally check and if it turns out not to be flexible enough then stop doing that. |
This is exactly the sort of thing #407 would be great for, I think. I'm probably partially off tomorrow then mentoring for a workshop the rest of the week, so I can probably start on #407 next week unless @agronholm has already started. |
I haven't started yet; I'm busy with another project currently. |
Have to do cross-compilation, hope to deliver by the morning. |
Apple clang is too randomly broken (no operator<=> for std::string, seriously?), so this needs GCC. Also, there were some non-fatal warnings about a mismatch of "deployment target", so I figured out that I probably need to use the latest and greatest to limit the blast size when stuff breaks. Fingers crossed; these blind builds really take a leap of faith. Since Apple clang is not enough and the bundled GCC is not a cross-compiler (and I don't really feel like bootstrapping one today), we cannot build arm64 binaries on Mac OS yet. That required another fair amount of hoop jumping due to pypa/wheel#406 and/or pypa/setuptools#2520.
Apple clang is too randomly broken (no operator<=> for std::string, seriously?), so this needs GCC. Also, there were some non-fatal warnings about a mismatch of "deployment target", so I figured out that I probably need to use the latest and greatest to limit the blast size when stuff breaks. Fingers crossed; these blind builds really take a leap of faith. But the resulting CPython module loads successfully, so I suppose this might actually work? Since Apple clang is not enough and the bundled GCC is not a cross-compiler (and I don't really feel like bootstrapping one today), we cannot build arm64 binaries on Mac OS yet. But the Python version that is driving this build is the `universal2` fat binary thingy, and due to pypa/wheel#406 and/or pypa/setuptools#2520, this required another fair amount of hoop jumping. Finally, the BSD-ish userland comes with its set of peculiarities.
Apple clang is too randomly broken (no operator<=> for std::string, seriously?), so this needs GCC. Also, there were some non-fatal warnings about a mismatch of "deployment target", so I figured out that I probably need to use the latest and greatest to limit the blast size when stuff breaks. Fingers crossed; these blind builds really take a leap of faith. But the resulting CPython module loads successfully, so I suppose this might actually work? Since Apple clang is not enough and the bundled GCC is not a cross-compiler (and I don't really feel like bootstrapping one today), we cannot build arm64 binaries on Mac OS yet. But the Python version that is driving this build is the `universal2` fat binary thingy, and due to pypa/wheel#406 and/or pypa/setuptools#2520, this required another fair amount of hoop jumping. Finally, the BSD-ish userland comes with its set of peculiarities.
macos12 python 3.11 x86_64-only wheel is incorrectly marked as universal2: As we can see from the uploaded pypi wheel, the whl only contains an x86_64 so file, and even after set
also didn't work |
"universal" in bdist_wheel refers to pure Python wheels that support Python 2 and Python 3 at the same time, it's a historical option that has nothing to do with universal2, Apple's way of shipping x86 & arm binaries. I'd recommend looking at https://github.com/pybind/python_example - you based this on a very old version of this example (including the incorrect statement that Python 3.6 has I'd also use setuptools_scm instead of rolling your own git tag system. I'd also recommend https://github.com/pypa/cibuildwheel if you want to build redistributable wheels, you'll be stuck with CPython's compiled macOS version, for example, if you roll wheel building yourself. You can retag yourself as a quick fix using the new |
thanks @henryiii. It seems the grpc project is also facing similar issue, they have a dirty fix here: https://github.com/grpc/grpc/blob/2845a248d6e1f617bbe0ffcd851cb1818725cda3/tools/run_tests/artifacts/build_artifact_python.sh#L160 |
wheel tags --platform-tag=macosx_11_0_arm64 --remove <old_wheel_name> is a "correct" fix that handles the dist-info. It's a bit of a hack; I think it's because you aren't handling the compiling correctly via setuptools, but it's a correct hack. |
I have a wheel I'm trying to build with C++ code in it. It contains only arm64 binaries for the non-Python code, so
universal2
is the wrong arch tag to use. Is there a way to suppress this and instead usearm64
?The text was updated successfully, but these errors were encountered: