Investigate code coverage with multithreaded tests

[gilles-peskine-arm]

This is an investigation task. How reliable is our code coverage measurement when there are multithreaded tests? Do we need to change or configure our tooling?

Currently, we do code coverage measurements by running tests/scripts/basic-build-test.sh on Ubuntu 16.04. It works in the following way:

1. Compile with GCC's GCOV (gcc --coverage) to generate code usage counter. This code might not be thread-safe, which is apparently the cause of a problem we hit with lcov.
2. Run tests.
3. Use lcov to process the data and output human-readable reports.

From the problem that #8926 fixes, we know that GCOV is producing some bad data. How bad? Are modern versions better? Would switching to Clang be better? Should we exclude multithreaded tests from code coverage measurement?

If we dig into the logs of the code-coverage job, lcov: WARNING: negative counts found in tracefile Coverage/tmp/tests.info indicates that there was bad data from GCOV. The absence of this message doesn't necessarily indicate that the data is good: there might be bad data that's not so bad that a hit count becomes negative.

Prerequisite: we should probably switch to modern tooling first. There's no point in investigating problems that other people have already fixed.

Goal of this task: evaluate the reliability of our code coverage measurement where it involves multithreaded tests, and decide on follow-up actions if any.