Rebuild for protobuf423 with additional fixes #363

traversaro · 2023-06-04T17:22:28Z

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

conda-forge-webservices · 2023-06-04T17:22:34Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

traversaro · 2023-06-04T23:55:27Z

The additional fixes seems to be working fine.

traversaro · 2023-06-04T23:56:52Z

@conda-forge-admin, please rerender

hmaarrfk · 2023-06-05T03:03:18Z

recipe/build.sh

+sed -i.bak "s/CXX_STANDARD 11/CXX_STANDARD 17/" cmake/OpenCVDetectCXXCompiler.cmake
+sed -i.bak "s/CXX_STANDARD 11/CXX_STANDARD 17/" cmake/OpenCVPluginStandalone.cmake
+sed -i.bak "s/CXX_STANDARD 11/CXX_STANDARD 17/" modules/objc/test/cmakelists.template
+sed -i.bak "s/CXX_STANDARD 11/CXX_STANDARD 17/" modules/objc/generator/templates/cmakelists.template


is this stuff necessary still?

I removed and almost all jobs are passing, so I guess no.

yeah, i'm somewhat worried it is necessary for osx lol ;) you never know!

i'm somewhat worried it is due to the order of arguments.

I think protobuf now expeoses abseil (right???) and thus needs a consistent version of c++ between libraries.

this is why I originally had added it (before your patch)

it may be that for all other platforms except for osx arm64, the order of the c++ flags makes it so that it is c++17 and not c++11

The osx-arm64 failure is due to the use of the host protoc instead of the build one:

2023-06-05T08:59:56.3610410Z [552/1488] cd $SRC_DIR/build/modules/dnn && $PREFIX/bin/protoc-23.2.0 --cpp_out :$SRC_DIR/build/modules/dnn -I $SRC_DIR/modules/dnn/src/caffe -I $SRC_DIR/modules/dnn/src/onnx -I $SRC_DIR/modules/dnn/src/tensorflow $SRC_DIR/modules/dnn/src/caffe/opencv-caffe.proto 2023-06-05T08:59:56.3612450Z FAILED: modules/dnn/opencv-caffe.pb.h modules/dnn/opencv-caffe.pb.cc $SRC_DIR/build/modules/dnn/opencv-caffe.pb.h $SRC_DIR/build/modules/dnn/opencv-caffe.pb.cc 2023-06-05T08:59:56.3614230Z cd $SRC_DIR/build/modules/dnn && $PREFIX/bin/protoc-23.2.0 --cpp_out :$SRC_DIR/build/modules/dnn -I $SRC_DIR/modules/dnn/src/caffe -I $SRC_DIR/modules/dnn/src/onnx -I $SRC_DIR/modules/dnn/src/tensorflow $SRC_DIR/modules/dnn/src/caffe/opencv-caffe.proto 2023-06-05T08:59:56.3615580Z /bin/sh: $PREFIX/bin/protoc-23.2.0: Bad CPU type in executable

see conda-forge/conda-forge-pinning-feedstock#4075 (comment) for a possible patch. I do not think there is any problem of c++ flags order, if you look to the compiler invocation each invocation either has -std=c++11 or -std=c++17. As protobuf or abseil are not exposed in public headers, there is no risk of this creating problems. However, if we want to just play safe, we can re-add the patch to ensure that all of opencv compilers with C++17.

The osx-arm64 failures hopefully will be fixed by 2436f99 .

if you look to the compiler invocation each invocation either has -std=c++11 or -std=c++17

nope nope nope nope, please let's not play this game of whack-a-mole with abseil (which only supports a consistent C++ standard). 🙏

As protobuf or abseil are not exposed in public headers

They've mentioned they plan to use abseil in their public API, so this will happen eventually.

They've mentioned they plan to use abseil in their public API, so this will happen eventually.

Ah sorry, I got mixed up here. Protobuf will expose abseil in its API, opencv may choose not to do that. Still, I'd strongly advise that we use a uniform C++ standard version here.

if you look to the compiler invocation each invocation either has -std=c++11 or -std=c++17

nope nope nope nope, please let's not play this game of whack-a-mole with abseil (which only supports a consistent C++ standard). 🙏

Sure, I was just replying to @hmaarrfk on the point of the order of the arguments, not endorsing mixing standard versions in the same library (that unfortunatly it is what happens if one set manually CMAKE_CXX_STANDARD and/or CXX_STANDARD .

traversaro · 2023-06-05T07:05:55Z

Windows failures were probably fixed by conda-forge/libprotobuf-feedstock#162 .

traversaro · 2023-06-05T17:28:20Z

I think the PR is ready for review, the only two remaining failures are temporary network failures.

hmaarrfk · 2023-06-05T22:15:31Z

Hmmm. Seems good to me

h-vetinari

One small thing, otherwise LGTM!

recipe/build.sh

h-vetinari · 2023-06-06T07:21:28Z

Completely unrelated, but small enough to consider it nevertheless: could you add dev_url: https://github.com/opencv/opencv under extras:? I find it nice to be able to navigate to the upstream repo directly from the feedstock README.

traversaro · 2023-06-06T07:27:11Z

Completely unrelated, but small enough to consider it nevertheless: could you add dev_url: https://github.com/opencv/opencv under extras:? I find it nice to be able to navigate to the upstream repo directly from the feedstock README.

Done: 0011f24 .

traversaro · 2023-06-06T07:28:24Z

@conda-forge-admin, please rerender

recipe/bld.bat

mshabunin · 2023-06-06T16:55:57Z

I'm wondering whether it is really necessary to raise C++ standard level. Conda packages can be used by C++ developers and sudden change of C++ ABI can break their applications. Does conda-forge have predefined set of toolchains which have stable ABI for all packages in the repository? OpenCV wouldn't override C++ standard, but it will set it to 11 if it was not explicitly defined.

It means that OpenCV have failed to find Lapack-compatible library (OpenBLAS, MKL, ...):

CMake Error at cmake/OpenCVUtils.cmake:797 (message):
  Some dependencies have not been found or have been forced, unset
  ENABLE_CONFIG_VERIFICATION option to ignore these failures or change
  following options:

  WITH_LAPACK

Technically Lapack is not required and can be disabled if it gives troubles on this platform (explicitly set WITH_LAPACK=OFF). Package dependencies will be changed in this case though, I'm not sure if it's a good thing.

h-vetinari · 2023-06-06T20:50:24Z

I'm wondering whether it is really necessary to raise C++ standard level.

Yes, because of abseil, see here.

Conda packages can be used by C++ developers and sudden change of C++ ABI can break their applications.

Where we know of ABI breaks based on the standard version (like abseil), we take great care to rebuild all dependent packages so that everything in conda-forge uses a consistent ABI. If people compile C++ on top of conda-forge dependencies and update any ABI-relevant library in their environment (basically anything in the global pinning), it's up to them to recompile (effectively a local execution of our migration machinery).

Does conda-forge have predefined set of toolchains which have stable ABI for all packages in the repository?

The answer is yes, but (aside from compilers and fundamental support libraries) that baseline is constantly in flux, as new versions with new ABIs get released, and we migrate all dependents over to that, before we upgrade the baseline to that. You can see the ongoing migrations here.

As I mentioned, abseil is special here. And the problem is made worse by protobuf using abseil in their public API. This also makes the protobuf migration special - because we expect problems along the way, we actually double the build matrix on all feedstocks, until we can build everything with protobuf 4 (normal migrations rely on the assumption that they can be done relatively quickly across the ecosystem, which is not the case here)

mshabunin · 2023-06-07T07:28:55Z

Thank you for reply. Currently I can see that the only issue is missing Lapack dependency, it could be that try_compile in the OpenCV here is missing <LANG>_STANDARD <std> or can not be compiled using C++ 17. I suggest removing all sed patching and checking <build>/CMakeFiles/CMakeError.log to determine what is wrong with Lapack. Alternatively we can disable Lapack temporarily and reintroduce it later.

h-vetinari · 2023-06-07T07:34:36Z

Alternatively we can disable Lapack temporarily and reintroduce it later.

I think we should get lapack to work again. Not sure what broke though. In any case, Lapack is Fortran (and a C interface), so shouldn't be affected by the C++ standard. More than that, it shouldn't be rebuilt at all (if we're talking about the actual Lapack and not OpenCV's interface to it), but taken from the one we have already.

mshabunin · 2023-06-19T19:26:07Z

FYI, I've been able to build OpenCV with protobuf@v3.23.3 with minor OpenCV patching and raising CXX_STANDARD to 17 (opencv/opencv#23791 (comment)). Official new protobuf support in the upstream will require slightly more effort though.

traversaro · 2023-06-19T21:17:50Z

FYI, I've been able to build OpenCV with protobuf@v3.23.3 with minor OpenCV patching and raising CXX_STANDARD to 17 (opencv/opencv#23791 (comment)). Official new protobuf support in the upstream will require slightly more effort though.

A related patch can be found in https://github.com/conda-forge/opencv-feedstock/blob/1925cac9c12099a6c710eee3f79c847f76166f75/recipe/fix_protobuf23.patch .

h-vetinari · 2023-06-27T03:01:37Z

Here's the aggregate of my changes. Hopefully the commits make sense individually. I've picked a few obvious improvements from MicrosoftDocs/cpp-docs#365 as well.

h-vetinari · 2023-06-27T03:31:00Z

Not sure where this is coming from now; perhaps because I'm triggering not just on error-code 1, but NEQ 0? 🤔

Verifying WITH_IPP=ON => 'HAVE_IPP'=FALSE

Co-Authored-By: H. Vetinari <[email protected]>

Even though opencv appears to hard-code C++11 in some places, those shouldn't be relevant for us, and it should respect the user-provided CMAKE_CXX_STANDARD, see opencv/opencv#23752 Co-Authored-By: Mark Harfouche <[email protected]> Co-Authored-By: Silvio Traversaro <[email protected]>

…nda-forge-pinning 2023.06.27.00.31.06

h-vetinari · 2023-06-27T04:57:25Z

Pretty sure that wasn't me (osx / win):

  File "C:\Miniforge\lib\site-packages\conda_index\index\__init__.py", line 347, in _get_resolve_object
    sd._process_raw_repodata(repodata_copy)  # type: ignore
TypeError: SubdirData._process_raw_repodata() missing 1 required positional argument: 'state'

h-vetinari

Aside from the frankly unsustainable build matrix (luckily, we can improve this soon with conda-forge/conda-forge-pinning-feedstock#4605 and/or #365), this LGTM now.

Great sleuthing on the C++17 vs. LAPACK issue, resp. the one about finding protobuf!

PS. Happy to give this some more time if someone wants to check my patch rebase butchery (though hopefully, the outcome is tasty 😋).

hmaarrfk · 2023-06-27T11:54:28Z

Alright great! Thanks all! very exciting!

traversaro requested review from hajapy, hmaarrfk, jakirkham, msarahan, ocefpaf, patricksnape, xhochy and zym1010 as code owners June 4, 2023 17:22

traversaro mentioned this pull request Jun 4, 2023

Next round of abseil / grpc / protobuf upgrades conda-forge/conda-forge-pinning-feedstock#4075

Open

3 tasks

hmaarrfk reviewed Jun 5, 2023

View reviewed changes

hmaarrfk approved these changes Jun 5, 2023

View reviewed changes

h-vetinari requested changes Jun 6, 2023

View reviewed changes

recipe/build.sh Show resolved Hide resolved

h-vetinari mentioned this pull request Jun 6, 2023

Make libopencv independent of python version #364

Open

h-vetinari reviewed Jun 6, 2023

View reviewed changes

recipe/bld.bat Show resolved Hide resolved

traversaro mentioned this pull request Jun 22, 2023

Revisiting OpenCV with CUDA build #328

Open

h-vetinari force-pushed the patch-5 branch from 732e2d7 to 7a1a46b Compare June 27, 2023 03:00

h-vetinari force-pushed the patch-5 branch 2 times, most recently from f77689b to 9aa86a3 Compare June 27, 2023 03:05

hmaarrfk mentioned this pull request Jun 27, 2023

Soft close ffmpeg 6 migration stopping the dual builds conda-forge/conda-forge-pinning-feedstock#4605

Merged

5 tasks

regro-cf-autotick-bot and others added 15 commits June 27, 2023 14:35

Rebuild for protobuf423

c30243b

simplify python-include logic in build.sh

d98b752

Co-Authored-By: H. Vetinari <[email protected]>

renormalize patches & move them to separate folders

7044ab8

Create fix_protobuf23.patch

bbe4604

rely on run-export of freetype

0f5c688

sort host dependencies

7591c0d

solidify errorlevel-handling in bld.bat

069bfc5

build script cosmetics

5356b3c

Fix LAPACK-detection with C++17

4ecaf9b

Update project metadata

5d77871

add dummy tests for opencv output

536c484

Add hdf5 migration

5998cbb

add host-dependency on libiconv (required by libopencv_wechat_qrcode)

d69bb32

MNT: Re-rendered with conda-build 3.25.0, conda-smithy 3.23.1, and co…

9effa89

…nda-forge-pinning 2023.06.27.00.31.06

h-vetinari force-pushed the patch-5 branch from 9aa86a3 to 9effa89 Compare June 27, 2023 03:35

This was referenced Jun 27, 2023

Update conda and mamba conda-forge/miniforge#466

Merged

Builds on osx & win failing conda-forge/status#147

Closed

h-vetinari approved these changes Jun 27, 2023

View reviewed changes

hmaarrfk merged commit cc19341 into conda-forge:main Jun 27, 2023

This was referenced Jun 27, 2023

Rebuild for hdf51141 #367

Closed

Rebuild for protobuf423 #362

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rebuild for protobuf423 with additional fixes #363

Rebuild for protobuf423 with additional fixes #363

traversaro commented Jun 4, 2023 •

edited

Loading

conda-forge-webservices bot commented Jun 4, 2023

traversaro commented Jun 4, 2023

traversaro commented Jun 4, 2023

hmaarrfk Jun 5, 2023

traversaro Jun 5, 2023

hmaarrfk Jun 5, 2023

hmaarrfk Jun 5, 2023

hmaarrfk Jun 5, 2023

traversaro Jun 5, 2023

traversaro Jun 5, 2023

h-vetinari Jun 6, 2023 •

edited

Loading

h-vetinari Jun 6, 2023

traversaro Jun 6, 2023

traversaro commented Jun 5, 2023

traversaro commented Jun 5, 2023

hmaarrfk commented Jun 5, 2023

h-vetinari left a comment

h-vetinari commented Jun 6, 2023

traversaro commented Jun 6, 2023

traversaro commented Jun 6, 2023

mshabunin commented Jun 6, 2023

h-vetinari commented Jun 6, 2023

mshabunin commented Jun 7, 2023

h-vetinari commented Jun 7, 2023

mshabunin commented Jun 19, 2023

traversaro commented Jun 19, 2023

h-vetinari commented Jun 27, 2023 •

edited

Loading

h-vetinari commented Jun 27, 2023 •

edited

Loading

h-vetinari commented Jun 27, 2023

h-vetinari left a comment

hmaarrfk commented Jun 27, 2023

Rebuild for protobuf423 with additional fixes #363

Rebuild for protobuf423 with additional fixes #363

Conversation

traversaro commented Jun 4, 2023 • edited Loading

conda-forge-webservices bot commented Jun 4, 2023

traversaro commented Jun 4, 2023

traversaro commented Jun 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari Jun 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

traversaro commented Jun 5, 2023

traversaro commented Jun 5, 2023

hmaarrfk commented Jun 5, 2023

h-vetinari left a comment

Choose a reason for hiding this comment

h-vetinari commented Jun 6, 2023

traversaro commented Jun 6, 2023

traversaro commented Jun 6, 2023

mshabunin commented Jun 6, 2023

h-vetinari commented Jun 6, 2023

mshabunin commented Jun 7, 2023

h-vetinari commented Jun 7, 2023

mshabunin commented Jun 19, 2023

traversaro commented Jun 19, 2023

h-vetinari commented Jun 27, 2023 • edited Loading

h-vetinari commented Jun 27, 2023 • edited Loading

h-vetinari commented Jun 27, 2023

h-vetinari left a comment

Choose a reason for hiding this comment

hmaarrfk commented Jun 27, 2023

traversaro commented Jun 4, 2023 •

edited

Loading

h-vetinari Jun 6, 2023 •

edited

Loading

h-vetinari commented Jun 27, 2023 •

edited

Loading

h-vetinari commented Jun 27, 2023 •

edited

Loading