Descartes evaluates the capability of your test suite to detect bugs using extreme mutation testing.
Descartes is a mutation engine plugin for PIT which implements extreme mutation operators as proposed in the paper Will my tests tell me if I break this code?.
To target a Maven project, PIT has to be configured and Descartes should be declared as a dependency and as the mutation engine to use.
In the pom.xml file include the following:
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.4.7</version>
<configuration>
<mutationEngine>descartes</mutationEngine>
</configuration>
<dependencies>
<dependency>
<groupId>eu.stamp-project</groupId>
<artifactId>descartes</artifactId>
<version>1.2.5</version>
</dependency>
</dependencies>
</plugin>Then, execute the regular mutation coverage goal in the folder of the project under test:
cd my-project-under-test
mvn clean package # ensures clean state
mvn org.pitest:pitest-maven:mutationCoverage -DmutationEngine=descartes
For more information and other options, see section "Running Descartes on your project".
Mutation testing allows you to verify if your test suites can detect possible bugs. Mutation testing does it by introducing small changes or faults into the original program. These modified versions are called mutants. A good test suite should able to kill or detect a mutant. Read more. Traditional mutation testing works at the instruction level, e.g., replacing ">" by "<=", so the number of generated mutants is huge, as the time required to check the entire test suite. That's why the authors of Will my tests tell me if I break this code? proposed an Extreme Mutation strategy, which works at the method level.
In Extreme Mutation testing, the whole logic of a method under test is eliminated.
All statements in a void method are removed. In other case the body is replaced by a return statement. With this approach,
a smaller number of mutants is generated. Code from the authors can be found in their GitHub repository.
The goal of Descartes is to bring an effective implementation of this kind of mutation operator into the world of PIT and check its performance in real world projects.
The goal of extreme mutation operators is to replace the body of a method by one simple return instruction or just remove all instructions if is possible. The tool supports the following mutation operators:
This operator accepts a void method and removes all the instructions on its body. For example, with the following class as input:
class A {
int field = 3;
public void Method(int inc) {
field += 3;
}
}the mutation operator will generate:
class A {
int field = 3;
public void Method(int inc) { }
}This operator accepts a method with a reference return type and replaces all instructions
with return null. For example, using the following class as input:
class A {
public A clone() {
return new A();
}
}this operator will generate:
class A {
public A clone() {
return null;
}
}This is a special operator which targets methods that return arrays. It replaces the entire
body with a return statement that produces an empty array of the corresponding type.
For example, the following class:
class A {
public int[] getRange(int count) {
int[] result = new int[count];
for(int i=0; i < count; i++) {
result[i] = i;
}
return result;
}
}will become:
class A {
public int[] getRange(int count) {
return new int[0];
}
}This operator accepts any method with primitive or String return type. It replaces the method body
with a single instruction returning a defined constant.
For example, if the integer constant 3 is specified, then for the following class:
class A {
int field;
public int getAbsField() {
if(field >= 0)
return field;
return -field;
}
}this operator will generate:
class A {
int field;
public int getAbsField() {
return 3;
}
}New in version 1.2.6
This operator accepts any method whose return type has a constructor with no parameters and belongs to a java package.
It replaces the code of the method by a single instruction returning a new instance.
For example:
class A {
int field;
public ArrayList range(int end) {
ArrayList l = new ArrayList();
for(int i = 0; i < size; i++) {
A a = new A();
a.field = i;
l.add(a);
}
return l;
}
} is transformed to:
class A {
int field;
public List range(int end) {
return new ArrayList();
}
} This operator handles the following special cases:
| Return Type | Replacement |
|---|---|
Collection |
ArrayList |
Iterable |
ArrayList |
List |
ArrayList |
Queue |
LinkedList |
Set |
HashSet |
Map |
HashMap |
This means that if a is supposed to return an instance of Collection the code of the mutated method will be
return new ArrayList();.
This operator is not enabled by default.
New in version 1.2.6
This operator accepts any method whose return type is java.util.Optional.
It replaces the code of the method by a single instruction returning an empty instance.
For example:
class A {
int field;
public Optional<Integer> getOptional() {
return Optional.of(field);
}
} is transformed to:
class A {
int field;
public Optional<Integer> getOptional() {
return Optional.empty();
}
} This operator is not enabled by default.
Descartes avoids some methods that are generally not interesting and may introduce false positives such as simple getters, simple setters, empty void methods or methods returning constant values, delegation patterns as well as deprecated and compiler generated methods. Those methods are automatically detected by inspecting their code. A complete list of examples can be found here. The exclusion of stop methods can be configured. For more details see section: "Running Descartes on your project".
PIT reporting extensions work with Descartes and include XML, CSV and HTML format. The HTML format is rather convinient since it also shows the line coverage.
Descartes also provides two new reporting extensions:
- a general reporting extension supporting
JSONfiles. It works also with Gregor, the default mutation engine for PIT. To use just setJSONas report format for PIT. - a reporting extension designed for Descartes that generates a JSON file with information about pseudo and partially tested methods. To use just set
METHODas report format for PIT. - Descartes can generate a human readable report containing only the list of methods with testing issues by using the
ISSUESformat.
Examples of these reporting extensions for Apache Commons-CLI can be checked here.
For more details on how to use and configure these reporting extensions please check section: "Running Descartes on your project".
Stable releases of Descartes are available from Maven Central. Descartes is a plugin for PIT so they have to be used together. PIT integrates with majors test and build tools such as Maven, Ant and Gradle.
Then, configure PIT for the project and specify descartes as the engine inside a mutationEngine tag in the pom.xml file.
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.4.7</version>
<configuration>
<mutationEngine>descartes</mutationEngine>
</configuration>
<dependencies>
<dependency>
<groupId>eu.stamp-project</groupId>
<artifactId>descartes</artifactId>
<version>1.2.5</version>
</dependency>
</dependencies>
</plugin>With PIT and Descartes configured, just run the regular mutation coverage goal in the folder of the project under test:
cd my-project-under-test
mvn clean package # ensures clean state
mvn org.pitest:pitest-maven:mutationCoverage -DmutationEngine=descartes
The operators to be used must be specified in the pom.xml. Each operator identifier should be added
to the mutators element inside the configuration element. void and null operators are
identified by void and null respectively.
For the constant mutation operator, the values can be specified using the regular literal notation
used in a Java program. For example true, 1, 2L, 3.0f, 4.0, 'a', "literal",
represent boolean, int, long, float, double, char, string constants.
Negative values and binary, octal and hexadecimal bases for integer constants are also supported
as stated by the language specification.
In order to specify a byte or short value, a cast-like notation can be used: (short) -1, (byte)0x1A.
The following configuration:
<mutators>
<mutator>void</mutator>
<mutator>4</mutator>
<mutator>"some string"</mutator>
<mutator>false</mutator>
</mutators>will instruct the tool to use the void operator and the constant operator will replace the body of
every int returning method with return 4; and will use "some string" and false for every string and boolean method.
If no operator is specified, the tool will use void and null by default. Which is equivalent to:
<mutators>
<mutator>void</mutator>
<mutator>null</mutator>
</mutators>If no operator is specified Descartes will use the following configuration:
<mutators>
<mutator>void</mutator>
<mutator>null</mutator>
<mutator>true</mutator>
<mutator>false</mutator>
<mutator>empty</mutator>
<mutator>0</mutator>
<mutator>1</mutator>
<mutator>(byte)0</mutator>
<mutator>(byte)1</mutator>
<mutator>(short)0</mutator>
<mutator>(short)1</mutator>
<mutator>0L</mutator>
<mutator>1L</mutator>
<mutator>0.0</mutator>
<mutator>1.0</mutator>
<mutator>0.0f</mutator>
<mutator>1.0f</mutator>
<mutator>'\40'</mutator>
<mutator>'A'</mutator>
<mutator>""</mutator>
<mutator>"A"</mutator>
</mutators>
All the goals defined by the pitest-maven plugin should run in the same way without any
issues, see http://pitest.org/quickstart/maven/.
To configure the stop methods under consideration Descartes provide a STOP_METHODS feature.
This feature is enabled by default. The parameter exclude can be used to prevent certain methods to be treated as stop methods and bring them back to the analysis. This parameter can take any of the following values:
exclude |
Method description | Example |
|---|---|---|
empty |
void methods with no instruction. |
public void m() {} |
enum |
Methods generated by the compiler to support enum types (values and valueOf). |
|
to_string |
toString methods. |
|
hash_code |
hashCode methods. |
|
deprecated |
Methods annotated with @Deprecated or belonging to a class with the same annotation. |
@Deprecated public void m() {...} |
synthetic |
Methods generated by the compiler. | |
getter |
Simple getters. | public int getAge() { return this.age; } |
setter |
Simple setters. Includes also fluent simple setters. | public void setX(int x) { this.x = x; } and public A setX(int x){ this.x = x; return this; } |
constant |
Methods returning a literal constant. | public double getPI() { return 3.14; } |
delegate |
Methods implementing simple delegation. | public int sum(int[] a, int i, int j) {return this.adder(a, i, j); } |
clinit |
Static class initializers. | |
return_this |
Methods that only return this. |
public A m() { return this; } |
return_param |
Methods that only return the value of a real parameter | public int m(int x, int y) { return y; } |
kotlin_setter |
Setters generated for data classes in Kotlin (New in version 2.1.6) |
So, for example, if we don't want to exclude deprecated methods and mutate them the following snippet should be added under the configuration element:
<features>
<feature>
<!-- This will allow descartes to mutate deprecated methods -->
+STOP_METHODS(except[deprecated])
</feature>
</features>More than one group can be excluded at the same time:
<features>
<feature>
<!-- This will allow descartes to mutate toString and enum generated methods -->
+STOP_METHODS(except[to_string] except[enum])
</feature>
</features>The feature can be completely disabled:
<features>
<feature>
<!--No method is considered as a stop method and therefore all of them will be mutated -->
-STOP_METHODS()
</feature>
</features>As said before, there are several reporting options provided by Descartes:
JSONfor a general mutation testing report using that file format. It can be used with Gregor.METHODSthat produces amethods.jsonfile with the list of all methods analyzed and categorized according to the mutation testing result.ISSUESa human readable report containing only the methods with testing issues.
They can be configured and combined as regular PIT report formats:
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.4.7</version>
<configuration>
<outputFormats>
<value>JSON</value>
<value>METHODS</value>
<value>ISSUES</value>
</outputFormats>
<mutationEngine>descartes</mutationEngine>
</configuration>
<dependencies>
<dependency>
<groupId>eu.stamp-project</groupId>
<artifactId>descartes</artifactId>
<version>1.2.5</version>
</dependency>
</dependencies>
</plugin>From version 1.2.6 Descartes includes a new feature AVOID_NULL which prevents the null operator to mutate a method marked with @NotNull.
The feature is active by default.
It can be removed by adding the following configuration:
<features>
<feature>
-AVOID_NULL()
</feature>
</features>Follow the instructions to set up PIT
for a project that uses Gradle.
In the build.gradle file add the local Maven repository to the buildscript block and set
a pitest configuration element inside the same block. In the dependencies block put the artifact information:
pitest 'eu.stamp-project:descartes:1.2.5'
then specify descartes in the mutationEngine option inside the plugin configuration.
An example of the final configuration could be:
buildscript {
repositories {
mavenCentral()
mavenLocal()
}
configurations.maybeCreate("pitest")
dependencies {
classpath 'info.solidsoft.gradle.pitest:gradle-pitest-plugin:1.4.0'
pitest 'eu.stamp-project:descartes:1.2.5'
}
}
apply plugin: "info.solidsoft.pitest"
pitest {
targetClasses = ['my.package.*'] //Assuming all classes in the project are located in the my.package package.
mutationEngine = "descartes"
pitestVersion = "1.4.7"
}
The pitestVersion property has to be specified to avoid version issues with the default version shipped with the gradle plugin.
At last, run the pitest task for the project under test.
gradle pitest
Descartes can be used when invoking PIT from the command line. To do this,
follow the instructions for running PIT,
include Descartes in the classpath specification and add --mutationEngine=descartes.
In a terminal clone the repository:
git clone https://github.com/STAMP-project/pitest-descartes.git
switch to the cloned folder:
cd pitest-descartes
and install Descartes using the regular Apache Maven commands:
mvn install
After installing the package, PIT should be able to find the Descartes mutation engine.
A comparison on the number of mutants created and execution time between Descartes and Gregor, the default mutation engine for PITest is available here.
Articles mentioning Descartes:
Descartes is published under LGPL-3.0 (see LICENSE.md for further details).
Issues, pull requests and other contributions are welcome.
Give us your feedback! Please take 5’ of your time to fill in this quick questionnaire.
This is important for us. As a recognition for your feedback, you will receive a limited edition STAMP Software Test Pilot gift and be recognized as a STAMP contributor.
This campaign will close on 31 September, 2019. You will be contacted individually for a customized gift and for contribution opportunities.
Descartes is partially funded by research project STAMP (European Commission - H2020)
