Skip to main content.

In-Class Lab: Using JUnit to Catch Mutants

Objectives:

Background

We generally assess the functionality of the code we write by running it and writing tests. But how do we assess the quality of the tests? Does test suite quality and coverage matter? Yes! In some cases, knowing that code is bug-free is literally a matter of life and death.

One way that test suites are evaluated is called Mutation Testing. The idea is to have a piece of code that you believe works. Then you introduce "mutant" varieties of the code by changing one small thing - like removing a line of code, changing an operator, or changing a literal. After creating mutant code, you run the test suite on the mutants. A mutant is "killed" if a test case fails--therefore flagging the bug. The total number of mutants you successfully kill gives a measure of how good the test suite coverage is--the higher the number, the better.

For this lab, I will provide you with a method specification, one correct implementation, and 15 mutant implementations. Your job is to write a test suite that does NOT flag the correct implementation but does successfully reveal each mutant.

This is not the way you will typically write JUnit test cases -- it's just meant to help you practice and understand if you're writing good JUnit test cases.

Set Up

Clone the repository. In Eclipse, you'll do this in the Git repository view.

Now, we have to do a little set up. (If you choose to use a different IDE than Eclipse, you'll have to figure out how to do the set up in your IDE.)

  1. In the Git repository view, expand your repository, which is named something like catch-the-mutants-username.
  2. Right-click on Working Tree and select Import Project. Follow through that dialog and create the project
  3. If there is a J on the project--meaning that it's a Java project, fantastic. If not
    1. Right-click the project and select Properties
    2. Select Project Facets
    3. If necessary, click "Convert to faceted form"
    4. Select "Java" facet
    5. Click OK
  4. Now, we need to set up the classpath. Review: what is the classpath?
    1. Right-click the project, select Build Path -> Configure Build Path
    2. In the Libraries tab, click on Classpath, and then click on "Add Library...". Select "JUnit", click Next, and then choose JUnit 5 from the dropdown, and click "Finish".
    3. In the Source tab, if it's not already there, click "Add Folder" and add the src folder. This tells Eclipse to tell Java that that's where the source code is
    4. In the Libraries tab, check if mutants.jar is in the Classpath. If not, click on Classpath, and then click on "Add Jars...". Navigate and select your project -> lib -> mutants.jar and click OK.

Hopefully, there are no longer any red x's on your project/on the Java code files.

Understanding the Provided Code

Three important files:

revealer.MutantRevealer.java:
This is the test file that you will complete.
mutants.Mutant.java:
This is the interface that all mutant code implements. It provides the code’s problem description (also found below).
testthetests.RevealingMutantsEvaluator.java:
This is the file that you should RUN. You will run it as a Java Application, NOT as a JUnit file.

Mutant Goals

All Mutants implement this method: 

        /**
	 * Returns the third shortest String in the array
	 * 
	 * If there are fewer than 3 words in the array or if the array is null, the method
	 * should throw an IllegalArgumentException.
	 * If there is a tie for the third shortest, any of the tied strings is valid.
	 * If there is a tie for shortest or second shortest, the duplicates do not
	 * affect the calculation of the third shortest.
	 * If there is no third shortest word, then the method returns null.
	 * The original array should not be altered.
	 *  
	 * Examples:
	 *    thirdShortest(["a", "ab", "abc"]) returns "abc"
	 *    thirdShortest(["a", "b", "bc", "ab", "bye", "and"]) returns "bye" or "and" because
	 *        “a” and “b” are the shortest, “ab” and “bc” are the second shortest, and “bye” and “and” are the third shortest.
	 *    thirdShortest(["a"]) should throw an IllegalArgumentException
	 * 
	 * @param words an array of Strings
	 * @return a String giving the third shortest String from the array if it exists; otherwise, returns null
	 * @throws IllegalArgumentException if array is null or if there are fewer than 3 words in the array
	 */
         public String thirdShortest(String[] words);

Your Mission

Write your tests for the thirdShortest method in the MutantRevealer.java file. Don’t delete the code given to you in the method marked @BeforeAll. That method lets my code load a new Mutant to test. Also, don’t rename or move the class.

Then you will see how good your test code is by running RevealingMutantsEvaluator. Let me reiterate: DO NOT run your MutantRevealer class using the Run As JUnit option. This would run the file on one mutant only. You need to run the RevealingMutantsEvaluator file as a Java Application. This code in turn will run JUnit on your code in a loop for all the mutants.

When your test suite accurately kills all mutants except Wolverine, you will see the victory message: “Good testing! YOU CAUGHT ALL THE BAD MUTANTS!”

How Does RevealingMutantsEvaluator work?

  1. First, MutantRevealer will be run on Wolverine, who is a good mutant. This mutant should pass all the tests, as it is a correct implementation of the problem.
  2. Then the MutantRevealer will be run on 15 mutants, which have all incorrectly implemented the method. If your test accurately shows that the mutant program is incorrect (by at least one of the tests failing), then you have “killed” or "revealed" that mutant.

What kind of asserts can I write?

The Assertions class has all the different assert methods you can call.

Post-Mortem