[Feature] - Support for benchmarks

[acki-m]

I think and also many others, would like to see an easy possibility to benchmark our code inside the testing framework of catch.
At the moment there is nothing in it. I found an already existing issue, but it seems to be not done:
#167

I would really like to see something like in the Qt Test Framework:
http://doc.qt.io/qt-5/qttestlib-tutorial5-example.html

In my opinion it would be very cool feature. Then catch would be for me the complete testing package.
Anyway, I really appreciate your work in this project. I like catch.

[jbrwilkinson]

Catch already has a '--durations yes' command line option which can output the duration time of a test, albeit in humanly-readable format. Would this be enough for you? If not, what is missing/different?

[dan-ryan]

"configData().showDurations = Catch::ShowDurations::Always" did the trick for me. Though breaking it down by using class names titles would make it easier to read.

A total tally would also be nice.

[acki-m]

I am looking for a real benchmark tool.
Like this:
https://bruun.co/2012/02/07/easy-cpp-benchmarking
Or take a look of all the features of the Qt benchmark suite:
https://www.youtube.com/watch?v=IPnzBO0dbrw

This is not some minor feature, but it will be a great enhancement to catch.
Most of the benchmark libs are not header only, when phil will get it to work also header only, that would be nice. But yeah, phil has to say something about this topic, if something like this will ever go into catch.
@philsquared
Will it?

[jbrwilkinson]

Although the hayai library looks really easy to use, I don't think you'll get accurate performance timing due to caching and branch prediction effects (see http://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-an-unsorted-array for a wonderful explanation).

A more realistic approach would be to run all your tests lots of times and randomly pick which test is next until they've all been tested the desired number of times. Then output the stats.

We've seen performance stats lurch wildly between great and awful based on what ran just before it.

[acki-m]

What are you trying to tell me, that benchmarking libraries are useless?
Unfortunately the maintainer doesn't want to say anything to this topic...so I have to look for other libraries which fill the gap.

[refi64]

@acki-m The maintainer is also very, very busy and will leave issues for sometimes over a month before even noticing they're there.

[horenmar]

Since this is not going into Catch Classic, but might get into Catch 2 at some point, I am going to close it in favour of #852.