Poll on Meson build

[mikkelfj]

Is there any interest in support for the Meson build system?
If so, please write a comment explaining how / why, or why not.

Meson makes it possible to have dependencies on schema files and cause code to recompile partially and properly when changing a schema.

It requires Ninja and Python 3.4. Python is not exposed in build scripts and Meson has no dependencies on other Python libraries.

Meson install on Ubuntu:

apt-get update
apt-get install python3-pip
pip3 install meson

MacOS:

brew update
brew install meson

build:

git clone myproject
cd myproject
mkdir build
meson . build

I already ported it to flatcc 0.4.3 with the exception of a proper AppVeyor build for Windows CI (anyone up for fixing this?)

https://github.com/dvidelabs/flatcc/tree/meson

This branch is not maintained but could be integrated into the main project with sufficient interest.

A demo project using Meson to build a custom project, including a setup script to install Meson is found here: https://github.com/dvidelabs/flatcc-meson-sample

CMake likely would have to be supported as well, or are there no strong feelings about this?

[mikkelfj]

The current CMake / Ninja build does not always rebuild test cases when using ninja -C build/debug <target>, and there are tests disabled because dependencies are broken when using Ninja build with CMake.

Meson has a lot of work going on with cross compilation and ARM target support.

However, Meson still needs some work before maturing for automated CI builds such as building 32-bit binaries on 64-bit MSVC2015 systems, and generally detecting the MSVC compiler environment.

mesonbuild/meson#2134

[mikkelfj]

For anyone following along - there is now a functional Appveyor build for Windows that can build both meson and cmake.

Meson 0.44.0 was the last release supporting python 3.4 which is as high as travis CI goes. Current Meson 0.45.1 required python3.5. This Python jump is a bit of a concern - resulting in the same issues as CMake has with need to run old versions, and one reason this project probably won't switch. However, there is a meson branch that is occasionally updated, at least for now.

The main problem with CMake is that dependencies are broken and especially so with the Ninja backend which is the only way to have fast builds. Due to the code generating nature of flatcc, this is a real problem - the benchmarks can't build with CMake using Ninja. Meson has no issues dealing with this correctly.

However, the primary concern is to build flatcc and libflatccrt.a and the dependency issue is more relevant to tests and derived projects.

[mikkelfj]

v0.5.2-meson has been tagged and meson branch supports flatcc version 0.5.2.

[barcharcraz]

This is relevant to my interests. Being able to just add flatcc as a subproject is great.

[mikkelfj]

Thanks for the feedback.

Does the meson branch work for you as it is?
https://github.com/dvidelabs/flatcc/tree/v0.5.2-meson

[kragniz]

I'm using meson with wrap-git and v0.5.2-meson. It seems to work well so far! It'd be great to have it merged into master.

[mikkelfj]

Thanks. I would probably want Appveyor to build meson on Linux in place of Travis before merging to master. Contributions are welcome.

[kragniz]

What's the reason for Appveyor over some other CI service? I can't say I've had great experiences with it in the past.

[mikkelfj]

In the past Appveyor was very slow and it's only saving grace was that it supported Windows. Later they significantly upgraded their build performance and added improved build matrix support. Later again they added support for Linux builds, but I have not tested that.

Meanwhile Travis started to get slower and randomly started to not support some gcc versions and generally only support fairly old OS. In particular I wasted 1-2 days trying to get Travis to support a recent but not bleeding edge version of Python needed by Meson. Additionally, I frequently have users reporting issues with recent gcc versions that also cannot be supported on Travis.

And finally, having only one CI framework would be an improvement. Since Travis does not support Windows and flatcc does, Travis cannot be that CI solution.

[mikkelfj]

I do have issues with Appveyor not mixing the latest MSVC version with older versions - I somehow have to choose. I worry this will be more difficult when also trying to support Linux builds. This is why I could use some help setting this up. It is rather time consuming.

[kragniz]

I've recently been using builds.sr.ht, which gives access to up to date linux distro images. Maybe using that might be better suited?

[mikkelfj]

Currently they do not support Windows and appears to be in the early stages, and Travis works well enough to not warrant an immediate migration without additional benefits such as also supporting Windows. Meson Travis builds work by using an older Meson version and accepting those limitations, for now.

[ddevault]

Hi, sr.ht guy here. builds.sr.ht doesn't support Windows, but we do support lots of other useful things, like *BSD and multi-arch builds. builds.sr.ht is also quite a lot faster than Travis, almost all builds start within 10 seconds of submission.

[derosier]

I'm going to give my opinion. Note that it's my opinion and is specifically around my use-cases for flatcc and comes from 20+ years of working with and building Linux, cross-platform and embedded build systems.

Personally, I would have negative interest in having flatcc have Meson support. And by that, I mean I'd rather it didn't get it, for a very specific reason - it would likely make future decisions on how flatcc is built depend on things that Meson can do. Other build systems that are currently supported would likely atrophy. Even in a best-intentioned scenario, "Hey we'll add Meson and still continue to maintain and support these others", the other build systems would end up getting the short-end of the stick for the new shiny system.

Most plain-vanilla systems, most embedded build systems, integration with other systems, etc suffer when you choose to use the new and non-standard build system. In the embedded world, even embedded linux, one doesn't have all these other packages to use. And even if they could be made available, I don't want to spend weeks tracking down dependencies trying to get Meson up and running on my specialized build platform.

As it is, I have to disable the default Ninja configuration that flatcc uses so I can use standard CMake Makefiles output. It's not a big deal and is only a couple of lines of patch work, but increasing the effort by having to do that reduces the utility of flatcc.

Flatcc's primary use, IMHO, is in embedded devices. Otherwise, why wouldn't you just use regular FlatBuffers that support C++, python, etc...? As such, it's critical to minimize the necessary tool dependancies.

For me it comes down to the minimal toolchain and resources necessary to use flatcc. Adding Meson would mean more resources necessary to build flatcc, and it would mean more developer-resources absorbed by something that's not the main purpose of flatcc. And what happens when the next shiny new build system comes out, do we jump on that too, further reducing the effort that goes into maintaining and adding to flatcc's core code? Once you add a feature to an OSS project, you've got to maintain it. That's where the most critical thinking on the question of "add Meson?" needs to happen.

[mikkelfj]

Thanks, Steve, input appreciated, and you are right, CMake would eventually atrophy. And, as you must have noticed, an eventual move to Meson is not a rushed decision.

However, Meson has great support for cross compiling and users occassionally add support for various obscure platforms. So overall I think it would be a net win. My major gripe is that Meson uses relatively new Python and then advances to an even newer Python. I'd really like to see a C version of Meson.

CMake isn't working that great for more complex builds (especially the Ninja backend) and having CMake running on obscure embedded platforms is probably more of a challenge than adding a Meson cross compile configuration file (not that I have done that).

But there are many counter arguments to using Meson as you have pointed out.

[mikkelfj]

See also

• https://groups.google.com/d/msg/mesonbuild/45q0a4W1oCM/8vnzx73ZBAAJ
• https://hwengineer.github.io
• https://hwengineer.github.io/discovery/

[ddevault]

I don't think it's wise to put your faith in CMake. If you want to maximize your software's compatability and lifetime you should just write straight-up Makefiles. Almost no one likes using CMake and the community's response to Meson has been nothing short of astronomical. I don't see CMake living even another 5 years. Support for embedded use-cases is even better in Meson.

The only issue is that Meson is written in Python. At some point I hope to write a C implementation of Meson to fix that issue. Otherwise, it's really good.

[derosier]

Thanks. Interesting links.

Well, CMake's been around for a long time and I don't see it going away anytime soon. Perhaps "the community" I work in is different than yours. That said, I actually would prefer straight-up Makefiles. That's the lowest-common-denominator and is compatible with all build systems. It's way easier to integrate Makefiles.

Simple fact is, a move to Meson means a dependency on both Meson and Python. Ultimately it means I will be unable to use flatcc out-of-the-box in certain products if it goes that way (actually I realize that I already don't use it out-of-the-box: I always have to kill the Ninja dependency). Replacing flatcc would be easier than adding support for Meson into major embedded build platforms that I work with.

Do what you think is best. As I am currently dependent on this project (and about to become more so), I just wanted to put my $0.02 before you chose something because you didn't think anyone had any problems with it. If flatcc goes with Meson, I'll deal with it in whatever way is necessary to get my projects done. There's a large set of possibilities. That's the beauty of OSS.

[ddevault]

Perhaps "the community" I work in is different than yours

I doubt it.

[mikkelfj]

@derosier you are welcome to contribute the flatcc/scripts dir to provide a more out of the box environment for Makefiles.

For me build.sh and test.sh has a tremendious value in using ninja as it compiles very fast during development. It would be faster still had I been using Meson.

Meson also allows me to include multiple levels of schema files, make a single change in the deepest nested file, and I have a minimal delta compilation automatically detected. It also allows me to compile benchmarks with Ninja backend, which isn't possible with CMake because dependencies simply do not work correctly. Thus, benchmarks are compiled using shell scripts and therefore rarely. And therefore bugs are occasionally found here.

For what it is worth, I have been working on very large concurrent Makefile driven projects for embedded systems, which takes hours to build and congests the network drives holding source code. I have also developed Makefile code generators to handle a great number of build variants in display sizes, language, etc. which blew up the build times even further. So no, Makefiles are not the way to go when you scale up.

[ghost]

Please do not use Meson/Ninja. I have some experience using it for embedded projects, and honestly, I wish I didn't. The dumbed-down Python like language is weak and insufficiently expressive. It's model of dependencies, build targets, etc is weak. Its model of subprojects results in duplication of code and is incompatible with git submodules. Attempting to release projects using subprojects requires manually tuning/locking all of the git sha's in all of the .wrap files. It has no support for common embedded C compilers, particularly ARMCC, which for many is a non-starter. Please just use make. No, not cmake. Just plain old make.

[mikkelfj]

@scribibble sorry to hear that you had a bad experience. A Makefile is hard to get right in the general case, but it is easy to write you own to build just the flatcc runtime, or just use a shell script as several tests do.

I agree that that the module system in meson has some limitations as you point out. I personally use a deps directory with subdirs that each has a meson file that imports from a dependency. Then I can copy those dirs around as needed.

Meson also doesn't help updating branches, but if you have a stable git tag or branch, it seems to be working. See also: mesonbuild/meson#4162

I work with code generation at multiple levels at once, and the only build system that largely works for that kind of complexity is Meson. Perhaps tux could work, but it has other issues.

As the scripting power, I see both pros and cons. Is it something you cannot handle with python or shell scripting?

I have not tried ArmCC myself (in a while, or ever with Meson), but it does not seem difficult to support new compilers via a cross file?
https://github.com/mesonbuild/meson/blob/master/cross/armcc.txt
Or Arduino with Cobol?
https://www.youtube.com/watch?v=PcII9W6NK4Q

My biggest issue is dependencies on projects that already have subprojects. I might get into some problems there, and I hope Meson improves in this area over time.

[ghost]

I agree that I can roll my own makefiles, which is what I will likely end up doing if I use flatcc. Using cross files to enable ARMCC was not possible out of the box prior to 1.46, but is now via patches submitted by a 3rd party. So maybe that issue is resolved.

Yes, I can use extra-Meson-icular technologies to do things that are difficult in Meson. I would prefer not to have to, as it just adds dependencies.

In any case, let me rephrase my original comment. Change "Please do not use Meson/Ninja." to "Please keep it easy to use make with this" (per your comment, maybe you have already done so.). That, at least in my personal opinion, might encourage adoption by embedded projects.

In any case, @mikkelfj, thanks for the quick response, and @derosier, your comments well stated and are appreciated. Cheers.

[mikkelfj]

You are welcome.

You can use the script

 scripts/setup.sh -x -b -s ../foobar
 cd ../foobar
 scripts/build.sh

to build flatcc, and create a test project with build scripts and example code, then enter the project and build it with a simple shell script. You can use the script as a starting point for a Makefile.
You can also link directly with 0 or more flatcc/src/runtime/*.c files depending on what you do, instead of using a library.

[ghost]

Ok, thanks. That sounds reasonable. I appreciate the suggestion.

[mikkelfj]

I just disabled a test case that was broken for Make builds. It was already broken for Ninja. The test is perfectly fine (and quite simple): it attempts to compile an object file from a generated c file to test that flatcc code generation works.

e280ce1

This might be due to the C compiler call the CMake uses for custom builds. The result is that <assert.h> cannot be found. Manually compiling the generated file with an -I path to the flatcc include dir works perfectly.

However, at this stage I have given up on using CMake with generated code. There are already numerous tests that are not running.

In comparison, I am building projects with meson where changes to a FlatBuffers .fbs schema triggers all the correct builds throughout the chain, with much much less time spent trying to understand the build system.

Since this project is about code generation, I have decided to eventually move to a meson build system.

I listen to the critique raised and CMake will continue to be supported for the foreseeable future for the core build distribution products, but only very basic tests will be driven by CMake. I will also make Makefiles the default build for CMake at that point.

Meson builds will be merged to the main project in some future release and drive the majority of test cases. There are still concerns about CI builds, so I am not rushing into this.

[mikkelfj]

@scribibble also note that I recently added the following introduction, based on your feedback:
https://github.com/dvidelabs/flatcc#introduction

[fractalb]

I want to use meson build in a new project. I'd like to know what the current status of the meson build support for this project?

[mikkelfj]

@fractalb Meson build is not tested regularly, but it should be working, notably on the meson branch. If you find any issues it will be fixed to any reasonable extent. If you need a merge of a more recent version, we can look into that, but you should also be able to merge yourself.

[fractalb]

@mikkelfj Thanks for the info. I'll give it a try and see.

[dontlaugh]

Count my vote 👍 for meson support

[eli-schwartz]

My major gripe is that Meson uses relatively new Python and then advances to an even newer Python. I'd really like to see a C version of Meson.

By the way, these days this exists and we @mesonbuild are big fans of it. :)
(It's also doing some really cool innovations, like adding an interactive debugger, a code formatter, and a static analyzer.)

https://mesonbuild.com/FAQ.html#but-i-really-want-a-version-of-meson-that-doesnt-use-python
https://sr.ht/~lattis/muon/

It is written in c99 and goes well with samurai, a ninja reimplementation that also uses c99 (ninja needs C++11). In fact, muon can even optionally build a copy of samurai into itself and run it as muon samu.

In general, muon has no dependencies if you don't want dependencies. It bootstraps from a shellscript that simply runs $CC upon an amalgam.c. It can be optionally built with:

• libpkgconf, for finding pkg-config dependencies
• libcurl, for trying to download subprojects when they aren't cached
• libarchive, for trying to extract subprojects when they are cached, but not unpacked

Muon is generally feature-complete for C/C++, but meson's modules system is still WIP (high-profile ones like pkgconfig, fs are already implemented).
Muon can successfully build and run tests on the meson branch here.

[mikkelfj]

@eli-schwartz interesting.
I actually developed a build engine with support for meson, called pion, developed in C89 with zero dependencies other than the compiler itself, and it can do compiler discovery also for old Windows compilers. It's really fast to execute. But I had to shelf that for a while.
Unlike meson, it implements a full build engine like ninja. However, recently I'm thinking it would be uphill to chase meson and would prevent pion from doing other stuff against the meson philosphy, so I might go in a slightly other direction.
As to flatcc itself, I'm not really sure where it goes, there seems to be a strong interest in just keeping CMake.

[mikkelfj]

As to libraries like libarchive etc. I'm thinking of shelving out such things through discovery such that the build engine itself can always be trusted to compile with no configuration.

[dontlaugh]

In lieu of direct support of building with Meson, perhaps we can work out how to include flatcc sources and runtime libraries with Meson's subprojects. Has anyone attempted this?

[mikkelfj]

I'm not really opposed to adding support for meson build, but I am not going to do it in the foreseeable future, and CMake needs to stay. So if anyone wants to do that and keep up with changes to the CMake, I'm open to that.
I already have proof of concept branch doing that.
Also note that github actions build would, probably the Weekly build, would have to be extended with meson build tests, just as both ninja and make are being testet as backends to the current CMake build.

[mikkelfj]

Oh, and a muon samuray build would also be of interest since flatcc is a pure C build.

[eli-schwartz]

@mikkelfj

As to flatcc itself, I'm not really sure where it goes, there seems to be a strong interest in just keeping CMake.

Based on my reading of the comments here, it seems like there was two general sentiments:

• "please, whatever else you do, don't add python as a build dependency"
• "I don't really like either cmake or meson, please make sure it is still possible to use Makefiles"

I didn't see any strong sentiment in favor of CMake, but maybe I'm missing something.

In any event -- I have come here to point out that if people do have an interest in Meson, but are concerned about the python dependency, that is no longer a problem. Do with that information as you wish. :)

[amcn]

It would be desirable to me to have access to flatcc from a meson context and I wish to contribute to making that possible.

I'm not sure what the best way to enable this is though. Either writing a wrap and maintaining it in mesonbuild/wrapdb, updating it to reflect changes in upstream, or making Meson a first class build system here upstream?

I am prepared to work on it in either case but I am wondering which would be preferable to people here?

[mikkelfj]

@amcn If you are willing to maintain it (I can make trivial changes as needed) I think it should be in the main repo rather than a wrap. Documentation would mention something along the lines of meson build support is dependent on community support. We would need to include at least some build targets with meson the Github Actions build. Perhaps make it a separate build from the others so it can fail in isolation.

[amcn]

@mikkelfj Thanks for your feedback. I will start to work on this here upstream. I have taken a look at your Meson branch from a few years ago and will use it as a starting point. Once it is ready I will send a PR and work with you to get it merged.

I intend then to make a very small addition to mesonbuild/wrapdb in order that users can take advantage of the new upstream meson support via meson wrap install flatcc and get access to the generators.

[mikkelfj]

Thanks for working on this. Please remember to add a hook so scripts/build.sh continues to work with meson builds also.

[mikkelfj]

@amcn has made a PR that will be integrated after next release which I hope will follow soon. Anyone interested in the meson build might want to look at the above PR and suggest improvements where relevant. CMake will not be replaced and the meson build will not be used to test older build targets.