ICS 45C Spring 2022
Notes and Examples: Unit Testing
Includes a code example with the moniker UnitTesting
Background
What is unit testing?
Many times when we write larger programs built out of many smaller pieces, it's difficult to test the individual pieces when in the context of the entire program. When we write classes, for example, our goal is to write a reusable component — one that we could potentially use not only in many places within the program we're writing, but maybe in other programs where a similar need arose. It's often the case that our class will have features that aren't fully exercised in the code we're writing — or, at least, that don't get exercised in an obvious, easy-to-duplicate way — yet we'll still want to be sure that the features work the way they should.
In order to achieve our goal of making sure that the class itself works, we need to test it on its own, independent of the rest of the program. This will allow us to focus our attention on individual behaviors of the class without worrying about what complex interactions we need to set up in our whole program to cause these behaviors to occur. This kind of testing, where we focus on a single, small piece of a program — a single class, a single module, sometimes even a single function — in isolation from the others is called unit testing. A data structure implementation, like the one you're building in Project #3, is a great example of where this can be a really useful technique, though there are plenty of others. (There are some who would say that every line of code in a program should be covered by at least one unit test; this can be difficult to achieve in practice, but is a worthy goal if you can achieve it. It should be noted, though, that fully unit testing a program requires techniques we've not yet learned, so it's probably beyond the limits of what we can accomplish in this course. How you design a program is an important part of whether you can write unit tests for it.)
Unit testing frameworks
When you write unit tests, regardless of what programming language you're writing them in, you face a similar set of problems:
Because these are problems you will always face when writing unit tests, you will find that, in most programming languages, there are unit testing frameworks available that provide out-of-the-box solutions to all of these problems (and sometimes more). Once in a while, a language's standard library will include one — e.g., Python's standard library has a unittest module — but, more often than not, these are third-party libraries written by others, though they're generally provided in open-source form, which means that they're usually free and that you can participate in their development if you're so inclined.
C++, like many other programming languages, has no built-in unit testing framework in its standard library, but there are multiple projects available in open source that address it. One of them, called Google Test (because it was written by developers at Google, though it's available to the community as open-source software), is already installed on your ICS 45C VM, and is the one that we'll be using in our work this quarter. Google Test automates the parts of unit testing that would otherwise be tedious to write ourselves, letting us instead focus our attention on the details of the tests we want to write. We set up our scenario, specify what the results should be, and Google Test does the rest; it can find and run all of our tests automatically, report on which tests succeeded and which failed, and even show us information about each failure (what the expected and actual results were, what line of code in our test function contained the discrepancy, and so on).
As usual with any kind of library, we'll have to learn a little bit in order to use Google Test, but it requires surprisingly little knowledge to get off the ground and start writing our tests.
Google Test basics
Google Test is actually quite simple to use in practice. While there are more complex uses of it than I'm showing here, using it in a basic way is very simple indeed. Here's what you'll need to do.
Where should your tests be written?
In the gtest directory within your project directory, you would want to create a source file, in which you would write a set of related tests. A good rule of thumb is that you'd have one of these for each source file you want to test, though I sometimes break them up even further than that. (In Project #3 we've already provided this source file, so all you'll need to do is write test functions in it.)
You won't need to write any header files, assuming that code in one test doesn't need to call into code from any other — which it generally doesn't — and Google Test will automatically find and execute all of your tests for you, so you won't need to worry about writing a main() function. (There is a main() function required, but I've already written it and provided it in every project template in a source file called gtestmain.cpp, which is a short one that basically tells Google Test: "Find every test function in every source file and run them all automatically.")
What are tests?
You can think of each individual test as a C++ function, albeit one with a slightly peculiar-looking syntax. Tests are run in isolation from one another and need to be completely self-contained. There are no guarantees about the order in which the tests are going to be run, so you don't want to write a test that assumes that some other test has already been completed. Each test looks something like this:
TEST(TestCaseName, testName)
{
// code for the test
}
A test begins with the word TEST (all uppercase), followed by two names in parentheses, followed by the test's body between curly braces. So it looks like a function, though it has no declared return type, and the name of the test is actually specified within the parentheses. In every other way, you can think of these as C++ functions and, more or less, be right about it. (Technically, TEST is what's called a macro; it gets translated, behind the scenes, to a function and some other bits and pieces that help Google Test to find it and call it for you.)
Individual tests are grouped together into what are called test cases (a slightly odd name, but you can think of them as test groups if it helps). For each test you write, you specify the name of the test case first, then the name of the individual test. So the test above is a test called testName in a test case called TestCaseName.
What does each test test?
Each test you write is intended to focus on one behavior of the code you're testing. Note that I didn't say one function or one member function, for the simple reason that you quite often can't test one function without using others. For example, if you consider the Song class we wrote in a previous example, you wouldn't be able to test getArtist() by itself; what you would be testing is "When a Song is constructed and given a particular artist, a subsequent call to getArtist() returns the same artist" or "If a call to setArtist() is followed by a call to getArtist(), getArtist() will return whatever artist was passed to setArtist(), regardless of what was passed to the constructor when the Song was first created." These are what we call behaviors; behaviors are sequences of calls to functions for which there are known results. It's often the case that you need a sequence of calls before you have a result you can verify.
How much should each test test?
Each test should focus on a single behavior, not a single function. The goal is this: When a test fails, you should know exactly what scenario failed, rather than having to look through a long sequence of test code to try to decide what part of it is wrong. If you want to test three different behaviors involving the same function, write three separate tests.
Since each test has almost no overhead associated with it, it's quite easy to write many of them. If you find that lots of tests need to share the same setup, you can even use test fixtures (see the Google Test documentation for more details) to simplify them, though we'll skip that in the interest of simplicity for now; it's unlikely that test fixtures would simplify things in any meaningful way in your testing for Project #3.
Assertions and expectations
When writing a Google Test, there are two ways to specify results that should be checked by your test: assertions and expectations. They are nearly the same, with one key difference:
Which of these you want is mainly a matter of taste, but they both have the key characteristic that they are a way to say "This test won't succeed except under the following condition."
How you write them is to make calls to functions whose names begin with either ASSERT_ or EXPECT_. (Technically, these are macros, as well, but you can think of them as functions.) For example, if you want to write a test that asserts that two things are equal, you would call the function ASSERT_EQ like this:
ASSERT_EQ(0, a.size());
Note, too, that the order of these parameters is important: the first is the expected value and the second is the value we're testing. So the above is a way to say "I assert that the size of a should be 0." (It'll still work if you get them backward, but the report that you get back from Google Test will be written backward, too, which will be misleading when you try to figure out what's wrong. So it's best that we get the order right!)
Check out the Google Test documentation (linked below) for a list of what assertions and expectations are available; they're quite extensive (and there are even ways to create your own).
How to name your tests
If each test is intended to test some behavior, the best way to name our tests is to give them names that indicate what that behavior is. What does the test actually verify? The names are important because Google Test generates a report that shows the names of the tests that fail; good test names mean that I might well be able to understand what my problem is simply by reading the report and seeing what the actual and expected results were. If unit tests are a tool to make testing and debugging easier, we should do what we can to help them to do that job; naming plays a huge, vital role.
So, for example, if I was to write a test for the Song class from a previous code example, where I wanted to show that a Song's artist is whatever was given at the time it was constructed, I might write something like this:
TEST(SongTests, containTheArtistGivenWhenConstructed)
{
Song song{"Paul Simon", "So Beautiful or So What"};
ASSERT_EQ("Paul Simon", song.getArtist());
}
You might argue that you could also just as easily check the title in that same test, but I would argue that this is a different behavior, so it should be a different test.
TEST(SongTests, containTheTitleGivenWhenConstructed)
{
Song song{"Arcade Fire", "Afterlife"};
ASSERT_EQ("Afterlife", song.getTitle());
}
(And note, too, that I used different constants in each case. That's no accident; varying your test data is a good way to be sure that tests aren't passing because of some underlying assumption that doesn't generally hold true.)
Finding out more about Google Test
You would be well advised to spend a little bit of time reading through Google Test's documentation, particularly the Google Test Primer, so you can get a sense for what's available and how it works. This code example is not nearly all-inclusive, mainly because Google's documentation is already so complete; my goal here was to introduce you to the basic concepts, and also give you an idea of how to approach writing unit tests in a more general sense.
The documentation is linked below.
The code
The official moniker for this code example is UnitTesting, so your best bet is to do this:
Alternatively, you can click the link to the tarball below:
Note that this example has no code in its main() function in the app directory, but it does contain unit tests in its gtest directory. So, if you want to run the example, you would issue these commands in your project directory: