testing.md

Testing

This project uses a lot of tools and tests to ensure and enforce the best code quality and squash bugs. This file lists the tools used within this project, and describes how to use them.

Table of Contents

1. Automatic testing
2. Run-time tools
   1. Valgrind
3. Static Analyzers
   1. Clang-tidy
   2. Cppcheck
   3. Infer
   4. Flint++
4. Other tools
   1. Clang-format

Automatic testing

Whenever a changelog is commited and pushed, Travis CI and GitHub Actions will initiate a fresh build each time, so that breaking changes will be catched and resolved. These make use of BuildCache, which contains some binary files to massively increase the build speeds, which is essential, because tests should be fast.

CodeFactor is also used to analyze the code, mainly for styling issues. This process is also fully automatic, and an A+ score is aimed for.

Run-time tools

Use these tools for debugging the binary executable for finding run-time bugs.

Valgrind

Even though modern C++ is used, a memory bug can still occur. Valgrind runs executables in a controlled environment, which exposes memory leaks, undefined behavior and other memory errors. Click here for source code tarballs with instructions.

To install Valgrind on Debian/Ubuntu:

$ sudo apt install -y valgrind

A Makefile target is also included for running Valgrind with recommended settings:

$ make memory

Static Analyzers

Use the following tools to analyze the source code for possible bugs and recommendations for best practices. Multiple static analyzers are used to ensure no problems are missed.

Clang-tidy

Clang-tidy is a tool made by the LLVM development team and is part of the LLVM Project and Clang Tool Suite. Clang is the modern compiler we use, and is made & supported by respected people, like Apple, ARM, Google, Mozilla, Sony etc.

Clang-tidy is a clever analyzer which makes sure modern C++ is used, copy reduction is at its highest, best practices are followed, etc. To install the tool, visit the website or install it using your package manager.

There isn't an easy command to use clang-tidy yet on multiple files, but you can use the following command to run it on a single file:

$ clang-tidy <filename> -- -I. -std=c++17

Some projects and companies have included their own set of rules and recommendations, which often are non-applicable, duplicate, or unnecessary. You can omit certain tools by adding filters to the .clang-tidy configuration file.

Cppcheck

Cppcheck is an open-source tool for finding bugs by looking at the source code. Visit the website for download instructions. A Makefile target is included for running cppcheck with recommended settings:

$ make cppcheck

This will output details about the analysis, but some of the output is ignored because it contains some false positives.

Infer

Infer is another open-source analyzer developed by Facebook. To install the tool, binary builds are recommended for people not experienced with the Opam/OCaml tooling environment, but you can still build it yourself.

To run Infer, use the following Makefile target:

$ make infer

This will run Infer on every file that has been changed, but if you want to scan a clean build (recommended), use the following target:

$ make infer-clean

Which effectively does make clean infer.

Flint++

Flint++ is another open-source static analysis tool originally developed by Facebook. The tool is in form of a linter, which may not cleverly analyze code as other tools, but is still found very useful. Visit the GitHub page for more information and build instructions. Flint++ is also incorporated within the Makefile:

$ make flint

Other tools

clang-format

To enforce styling rules throughout the source code, clang-format is used. This will reformat the code wherever necessary. Note that this tool isn't applied automatically, and the tool should definitely be ran after a number of changes/commits. Visit the website for install instructions.

To run the tool throughout the codebase, a useful command is made:

$ tools/formatter.sh

This will run clang-format using the rules of .clang-format for all .hpp and .cpp files. Please note however that this tool is being executed automatically by JanitorBot, and manually running this tool is often unnecessary, because it might affect files that you weren't working on. It is recommended to run the tool if the code before your changes was passing the formatter, but after the changes not. If this is the case, you should preferably run the tool before commiting.