Mutation testing is hard... But totally worth it!

I recently learned about the concept from a colleague. For those who might not know what it is:

Testing by mutation calculates the durability of your unit tests against mutated versions of your source code. These mutant versions are designed to be the result of common programming errors. Since there are many different types of errors, this means that normally a large number of mutants are created and tested. These mutants would be extremely difficult to create by hand, so mutants are generally automatically generated from changing the source code using common error patterns.

Error pattern examples may include switching conditional, logical, and arithmetic operators. Possibly even changing the value of literals within the code. These are just a few ideas, and I am hoping one day we can have a more standardized set of patterns to use.

How it works

A (hopefully obvious) perquisite to mutation testing is that your unit tests need to pass against your original source code. Also, the faster your unit tests are, the easier mutation testing will be.

So, step by step:

1. A baseline unit test is run against your original source code
2. Mutated versions of your code (mutants) are generated by changing the code slightly
3. Each mutant is ran against the same unit tests
4. If a mutant passes the test suite, it is considered killed
5. Your score is calculated by how many mutants were killed against how many were created

The goal is to try to kill all the mutants.

By example...

Here we have a very simple Javascript module:

//Bank Account Module
module.exports = function() {
	var balance = 0;
	this.getBalance = function() {
		return balance;
	};
	this.deposit = function(amount) {
		balance += amount;
	};
	this.withdraw = function(amount) {
		if (balance >= amount) {
			balance -= amount;
			return true;
		} else {
			return false;
		}
	};
	return this;
};

Easy mutation could be to change the balance literal to initialize to -1.

If our unit tests started each check with a clean Account instance and assumed that the balance should be 0, our tests would fail.

//Unit Test (Wrong)
it('should deposit', function() {
	account.deposit(50);
	account.getBalance().should.equal(50);
});

The corrected unit test might look something like this:

//Unit Test (Corrected)
it('should deposit', function() {
	var initial = account.getBalance();
	account.deposit(50);
	account.getBalance().should.equal(initial + 50);
});

This is one example where mutation testing could help.

So I created something

In order to better understand and learn about mutation testing, I wanted to write my own Javascript mutation tester. I found one project for mutation testing in Javascript, and although I am normally a fan of "don't re-invent the wheel". In this case, the tool was only usable for grunt built projects. I wanted to create a more general use tool which was (build system) agnostic.

Big hurdle I hit with writing this was actually "node-specifc" issues. While mutating source code, loading and running tests, I kept getting the same results. After hours of beating my head against the wall, I tried renaming the files before loading them in. Sure enough, it worked. So in doing this project I learned a bit more about how the node.js require method works and how it caches files.

Here is the project.