[tests] Consider compiling and running tests with (possibly all) combinations of supported flags

[piotrpawelczyk]

Not really a bug, nor a feature request, just a way to improve the chances of catching bugs.

While running tests for PHP7 target, trying to check what's the cause of #6387 issue, I found out that all (unit) tests are run with fixed flags. For those specific flags it's impossible to reproduce the error.

I think it would allow for early detection of such errors if tests where compiled dynamically, e.g. with generated combinations of flags (DCE, optimizations and everything that makes sense).

[Simn]

This always sounds nice, but then you realize that it grows exponentially over the number of flags.

[piotrpawelczyk]

Yes, it does. That's what CI servers are for - you run locally only a subset of unit tests and then on the server all possible tests are run, leveraging multithreading, if setup allows.

Anyway, it's just an idea to consider. :)

[Simn]

You must be new to free open source CI servers. :D

[piotrpawelczyk]

I am, actually. ;)

But people who have access to really powerfull servers are still reluctant to using its potential - some things have nothing to do with available resources. I've seen a lot of this at my previous jobs. :)

[Simn]

Coming back to this, I still don't think this is practical. We could think about it if we had the necessary CI resources, but as it stands there is nothing we can do.