In-Class Lab: Using JUnit to Catch Mutants

Objectives:

Practice creating good JUnit test cases
Practice using JUnit framework within Eclipse

Background

We generally assess the functionality of the code we write by running it and writing tests. But how do we assess the quality of the tests? Does test suite quality and coverage matter? Yes! In some cases, knowing that code is bug-free is literally a matter of life and death.

One way that test suites are evaluated is called Mutation Testing. The idea is to have a piece of code that you believe works. Then you introduce "mutant" varieties of the code by changing one small thing - like removing a line of code, changing an operator, or changing a literal. After creating mutant code, you run the test suite on the mutants. A mutant is "killed" if a test case fails--therefore, revealing the bug. You only need one test case to fail to reveal the fault. The total number of mutants you successfully kill gives a measure of how good the test suite coverage is--the higher the number, the better.

For this lab, I will provide you with a method specification, one correct implementation, and 15 mutant implementations. Your job is to write a test suite where all the test cases pass for the correct implementation and at least one test case fails for each mutant.

This is not the way you will typically write JUnit test cases. You don't know all the possible bugs. It's just meant to help you practice writing JUnit test cases and help you to understand if you're writing good JUnit test cases.

Set Up

Clone the repository. In Eclipse, you'll do this in the Git repository view.

Now, we have to do a little set up.

In the Git repository view, expand your repository, which is named something like catch-the-mutants-username.
Right-click on Working Tree and select Import Project. Follow through that dialog and create the project
If there is a J on the project--meaning that it's a Java project, fantastic. If no J on the project, then
1. Right-click the project and select Properties
2. Select Project Facets
3. If necessary, click "Convert to faceted form"
4. Select "Java" facet
5. Click "Apply and Close"
Now, we need to configure the project.
1. Check which version of Java is in use by looking for the JRE System Library in the project. It should be 21. If it's something else, you need to change the settings:
  1. On a Mac, go to Eclipse -> Preferences. On Windows, go to Window -> Preferences.
  2. Expand Java and then click on "Installed JREs".
  3. Hopefully JDK 21 is there. If not, add it. You may need to search for it. On Windows, check this location: C:\Program Files\Java\
  4. Check the box for JDK 21. Click Apply.
  5. Look for a warning -- something about the Java version not being bound and click the link and select the JDK with the number 21 in it.
  6. Now click on Java Compiler. You should see your installed JREs. JDK 21 is likely the default. If not, choose it. Then Apply and Apply and Close
2. Now, we need to set up the classpath. Review: what is the classpath?
  1. Right-click the project, select Build Path -> Configure Build Path
  2. In the Libraries tab, click on Classpath, and then click on "Add Library...". Select "JUnit", click Next, and then choose JUnit 5 from the dropdown, and click "Finish".
  3. In the Libraries tab, check if mutants.jar is in the Classpath. If not, click on Classpath, and then click on "Add Jars...". Navigate and select your project -> lib -> mutants.jar and click OK.
  4. Still in the Libraries tab, check that you're using Java 21 as the Java Runtime Environment. If not, click on the JRE and select "Edit". Then, select "Workspace Default" (which you set up earlier). Then click "Finish".
  5. In the Source tab, if it's not already there, click "Add Folder" and add the src folder. This tells Eclipse to tell Java that that's where the source code is.
Hopefully, there are no longer any red x's on your project/on the Java code files.
Understanding the Provided Code

Three important files:

revealer.MutantRevealer.java:
This is the test file that you will complete.

mutants.Mutant.java:
This is the interface that all mutant code implements. It provides the code’s problem description (also found below).

testthetests.RevealingMutantsEvaluator.java:
This is the file that you should RUN. You will run it as a Java Application, NOT as a JUnit file. (Again, this is not how it will typically work when you write JUnit test cases. This is just meant to be a good way to practice before getting to the "real" stuff.)

Mutant Goals

All Mutants implement this method:
```
        /**
	 * Returns the third shortest String in the array
	 * 
	 * If there are fewer than 3 words in the array or if the array is null, the method
	 * should throw an IllegalArgumentException.
	 * If there is a tie for the third shortest, any of the tied strings is valid.
	 * If there is a tie for shortest or second shortest, the duplicates do not
	 * affect the calculation of the third shortest.
	 * If there is no third shortest word, then the method returns null.
	 * The original array should not be altered.
	 *  
	 * Examples:
	 *    thirdShortest(["a", "ab", "abc"]) returns "abc"
	 *    thirdShortest(["a", "b", "bc", "ab", "bye", "and"]) returns "bye" or "and" because
	 *        “a” and “b” are the shortest, “ab” and “bc” are the second shortest, and “bye” and “and” are the third shortest.
	 *    thirdShortest(["a"]) should throw an IllegalArgumentException
	 * 
	 * @param words an array of Strings
	 * @return a String giving the third shortest String from the array if it exists; otherwise, returns null
	 * @throws IllegalArgumentException if array is null or if there are fewer than 3 words in the array
	 */
         public String thirdShortest(String[] words);
```
Your Mission

Write your tests for the thirdShortest method in the MutantRevealer.java file. Don’t delete the code given to you in the method marked @BeforeAll. That method lets my code load a new Mutant to test. Also, don’t rename or move the class.
Then you will see how good your test code is by running RevealingMutantsEvaluator. Let me reiterate: DO NOT run your MutantRevealer class using the Run As JUnit option. This would run the file on one mutant only. You need to run the RevealingMutantsEvaluator file as a Java Application. This code in turn will run JUnit on your code in a loop for all the mutants.
When your test suite accurately kills all mutants except Wolverine, you will see the victory message: “Good testing! YOU CAUGHT ALL THE BAD MUTANTS!”
How Does RevealingMutantsEvaluator work?
1. First, MutantRevealer will be run on Wolverine, who is a good mutant. This mutant should pass all the tests, as it is a correct implementation of the problem.
2. Then the MutantRevealer will be run on 15 mutants, which have all incorrectly implemented the method. If your test accurately shows that the mutant program is incorrect (by at least one of the tests failing), then you have “killed” or "revealed" that mutant.
What kind of asserts can I write?

The Assertions class has all the different assert methods you can call.
Post-Mortem
- What are the benefits of unit testing/using JUnit? Consider if you were developing/maintaining the method: How would your testing/development process change?
- Why did the output come out in strange/unexpected orders sometimes?
- Is it okay that some mutants passed some of the test cases?
- Recall the characteristics of good unit tests. How did you achieve them in your testing?

In-Class Lab: Using JUnit to Catch Mutants

Background

Set Up

Understanding the Provided Code

Mutant Goals

Your Mission

How Does RevealingMutantsEvaluator work?

What kind of asserts can I write?

Post-Mortem