Informatics 43 Spring 2009, Course Project, Phase 4: Testing

Background

No matter how careful we are when we write code, mistakes are essentially unavoidable; there are just too many interlocking details that we need to get right. A good design (e.g., separation of concerns, classes with single responsibilities) can partially mitigate this issue, unit testing can certainly help, but we ultimately still need to test our system in whole, with all components completed and integrated together.

In your project this quarter, you've now completed at least some portion of the implementation and submitted it, so it's time to focus your efforts on testing, ensuring that the system has been built as specified in the Official Requirements Specification, using a combination of white-box and black-box testing techniques. This is a big job, much larger than you could reasonably be asked to complete in the time we have left this quarter, so you'll only be asked to test certain areas of functionality.

Also, because not everyone has completed the implementation — and because those of you who aren't finished haven't all completed on the same subset of it — you'll all test a separate, completed, "official" implementation instead of your own. (Of course, you're welcome to run your chosen test cases against your own implementation, if you'd like, but that won't be part of this phase of the project.)

Testing is at least partly driven by planning; deciding what test cases should be executed in order to gain confidence that a system is working as specified is often at least as much work as executing the tests you chose. As we discussed in lecture, this kind of planning can take many forms, with the goal being to find some systematic way of knowing that you've "covered all your bases." This phase of the project will provide you with practice in using some of these techniques.

In particular, you'll be working on two forms of testing:

Black box testing, which is focused on testing the behavior of the system under test, without considering (or perhaps even having access to) the source code.
White box testing, in which you use the structure of the source code to determine what to test and how to test it (e.g., by building control flow graphs and using coverage metrics to drive your test case selection).

The Official Implementation of the Triple P Enrollment System

The Official Implementation is a console-based implementation of the requirements that comprise Implementation Phase 1 of the Official Requirements Specification, along with a handful of the other requirements.

You can download the Official Implementation, in both compiled and source code forms, at the following link:

The Official Implementation of the Triple P Enrollment System

I should point out a few things about the Official Implementation:

It is not intended to be thought of as a model of what your implementation should have been. It is comprised of around 100 classes in 17 packages; that doesn't mean that your implementation should have been organized this way, or that there was anything wrong with having many fewer classes or avoiding the use of packages altogether. My goal was to demonstrate how far you can take ideas like separation of concerns, high cohesion, low coupling, and code reuse. For example, the menu system is generalized; different forms of output (lists, tables, "details") are generalized; formatters, converters, and validators are their own classes; every command on every menu is its own class; and so on. This kind of design affords you a good deal of flexibility. Adding the same command to multiple menus becomes simple — just create objects of that command's class in two places. Formatting output the same way (e.g., displaying a student's name in the form LastName, FirstName) in many places becomes simple; just create formatter objects in each place you need them.
As always, I had to balance my goal of a clean implementation against the time I had available to work on it, so I'm sure there are still areas where I could make it better; I've yet to ever write a substantially-sized program that I couldn't inspect afterward and find things I wish I'd done differently, even if I can't find bugs in the program. If you find areas that you think could be improved, let me know; I'd be glad to talk with you about them.
You'll find the main( ) method, which you can use to run the program, in the inf43.spring2009.tpes.Main class. The menus are fairly self-explanatory. You can get help at any prompt (when selecting commands or entering other input) by typing ? and pressing Enter.
The code you'll be concerned with in your tests is in the inf43.spring2009.tpes.model package. All of the code in other packages is related in some way to the user interface.
There are no comments in the program, though I've done my best to choose good identifier names, keep methods short and straightforward, and keep each class focused on one task. If there's anything about the code you're unsure of, please feel free to ask.
The system has been only lightly tested up to this point; while I didn't place any defects into it intentionally, I'm nearly certain that there are things wrong with it. This is part of the point, of course, given that I want you to be testing it.
It makes use of some Java features, particularly some of the thornier aspects of generics, that you likely haven't seen before. The areas of code where these new features arise are not going to be your focus, though I'm happy to explain some of the Java concepts demonstrated if you're interested. (If I see you in Informatics 101 in the Summer or Fall, you'll get a chance to learn more about these kinds of things.)

Test matrices

For this part of Phase 4, you'll be required to develop a set of test matrices, using the technique we discussed in lecture for choosing test cases:

Find a set of bases, each basis being a criterion for dividing possible test cases into subdomains. A basis can center on things like the size of a collection, whether a user is logged in, whether a particular command has been executed successfully, and so on.
For each basis, list all of the interesting subdomains that you can divide test cases into.

As an example, suppose you're testing a web site like Gmail, where users can write email messages, send them, search them, and so on. Suppose, in particular, that you're testing the login functionality. Some of the bases you might choose include:

Whether the username is valid
Whether the password matches against a user

Each basis divides test cases into subdomains, where all of the test cases in each subdomain share some interesting property. For example, the whether the entered password matches against users' passwords might yield three subdomains:

The password entered matches the password for the user specified
The password entered matches the password for a user other than the one specified
The password entered does not match the password for any user in the system

You then choose at least one test case in each subdomain, along with test cases that span multiple subdomains (if any). In the above example, you might choose these test cases:

There are two users registered in the system: Alex with password MyPassword1, Boo with password OtherPassword1. User enters username Alex with password MyPassword1. We expect login to be successful. [Subdomain 1]
There are two users registered in the system: Alex with password MyPassword1, Boo with password OtherPassword1. User enters username Alex with password OtherPassword1. We expect login to be unsuccessful. [Subdomain 2]
There are two users registered in the system: Alex with password MyPassword1, Boo with password OtherPassword1. User enters username Boo with password NoPassword1. We expect login to be unsuccessful. [Subdomain 3]
There are two users registered in the system: Alex with password MyPassword1, Boo with password MyPassword1. User enters username Boo with password MyPassword1. We expect login to be successful. [Subdomains 1 and 2]

Note that, for each test case, the expectation is included; it's not a test case if we don't know what behavior we expect.

We can display the bases, subdomains, and test cases in the form of a test matrix, as we saw in lecture. An example of a test matrix, detailing one basis and associated test cases in the example situation above, is available at the following link:

Example Test Matrix

Note the presence of an Outcome column in the example test matrix; as you execute tests, fill in the actual testing outcome that you observed (e.g., "performed as expected" or "login was successful even though it shouldn't have been").

Part 1: Black box testing (50 points)

For Part 1 of this project phase, you are required to plan and perform black box testing on part of the Official Implementation of TPES. In particular, you will separately develop and execute tests for two of the requirements from the Official Requirements Specification:

ES-6, which prohibits students from enrolling in more than one course that have overlapping meeting intervals.
CS-10, which describes how to calculate the "effective capacity" of a course.

For each of these requirements, determine at least three bases, making sure that each of them is a reasonable criterion that can be used to divide the domain of possible test cases into interesting subdomains. For each basis, list a reasonable number of subdomains (usually at least three) and one or two test cases for each subdomain, along with at least one test case that covers every possible combination of subdomains. (Note that not all subdomain combinations will be possible; sometimes, none of them will be possible.)

Record your test cases in a collection of test matrices, in the same basic form as the example linked in the previous section. Run the test cases against the Official Implementation of TPES and make a note of the outcome of each: note whether the test was succesful, along with a note about any incorrect behavior that was seen if the test failed.

Part 2: White box testing (50 points)

For Part 2 of this project phase, you are required to plan and perform white box testing on part of the Official Implementation of TPES. In our case, we're using the term "white box testing" to refer to whole-program testing driven by our knowledge of the structure of the source code; this form of testing is distinct from the unit testing you did in the previous phase, which focused on testing individual classes and methods completely out of the context of the rest of the program. You'll execute your white box tests by running the program; the difference is that you'll use the code to choose your test cases.

Line numbering and node coverage

Recall from lecture that we can build a control flow graph for methods in a Java program. The first step is to number each line of code, not including the signature or any line containing only a curly brace. Statements that are split into multiple lines — which I do so that my code can be read without the need for horizontal scrolling — should only be counted as a single line. For example, the following is an example of the numbering of lines in the Classroom.addCourseMeeting( ) method from the Official Implementation:

    public void addCourseMeeting(CourseMeeting courseToAdd)
    throws CourseMeetingOverlapsException
    {
1       if (courseMeetingToAdd.getClassroom() != this)
        {
            throw new IllegalArgumentException(
2               "courseMeetingToAdd refers to the wrong classroom");
        }
        
3       for (CourseMeeting courseMeeting : courseMeetings)
        {
4           if (courseMeeting.getMeetingInterval().overlaps(courseMeetingToAdd.getMeetingInterval())
            {
5               throw new CourseMeetingOverlapsException(courseMeeting, courseMeetingToAdd);
            }
        }
        
6       courseMeetings.add(courseMeetingToAdd);
    }

We won't build control flow graphs in this phase of the project — though you're free to draw them if you'd like — but we will use this line-numbering scheme to help track whether we've achieved the most code coverage we can, where every line of code that can be reached has been reached by at least one test case.

What you need to do in this part

First, number the lines of code in the following two methods in the Official Implementation. (You don't have to turn this part in, but you do need to number the lines so that you can refer to them in your test matrices.)

The overlaps method in the inf43.spring2009.tpes.model.MeetingInterval class
The enrollInCourse method in the inf43.spring2009.tpes.model.Student class

Next, determine whether there are any lines of code that are unreachable in the program as it is currently written. This will require you to look around other parts of the program that use these methods. (There are tools that can help; see below.) Make a note of any unreachable lines of code; you won't need to find tests that cover them.

Finally, you're ready to choose test cases and execute them, using the following procedure for each of the two methods that you've been asked to test:

Choose at least two bases that would allow you to divide the set of possible into a set of interesting subdomains.
Develop at least one test case covering each subdomain individually, as well as at least one test case covering each possible combinations of subdomains. (Note, as in your black box testing, that not all combinations of subdomains are possible.)
- Your test cases should be "whole-program" test cases, meaning that they're expressed in terms of commands executed in the user interface, as opposed to unit tests, where you're testing methods in isolation.
For each test case, include in your "expected output" a list of which lines of code will be reached in the method that you're testing.
Verify that there is at least one test case covering every reachable line of code in the method that you're testing.

Put together a test matrix as you did in Part 1 of this phase, using the same basic format shown in the example.

Navigating an unfamiliar code base

It's been said that the worst code in the world is "someone else's code." When you're working with code that you didn't write, it can be difficult to find your way around; even if you can follow the logic of short blocks of code, you won't necessarily have a feel for its overall structure.

Since you haven't seen the code in the Official Implementation before, it may be a bit difficult, at first, to find out whether a particular line of code can be reached via the user interface. For example, some error checks in the inf43.spring2009.tpes.model package are duplicated in the user interface code; in these cases, the exception will actually never be thrown by the model, so it won't be possible to use our form of white box testing to verify that they get thrown.

(Sidebar: You might be wondering why such checks are being done in more than one place. Why should the model check things that the user interface checks? Why should the user interface check things that the model checks? The reason is that they're each doing the checking for a different purpose. The user interface checks for things that will allow it to be construct a contextual error message. The model checks for things that will cause it to fail. The model operates, rightly, under the assumption that it could be surrounded by any user interface; given that, even if today's user interface does a particular check, tomorrow's user interface might not. So we're better off making the model as bullet-proof as we can, so that we'll be more resilient to change.)

Eclipse has built-in tools to lighten the burden of navigating someone else's code base. One useful one is the "References" tool, which asks Eclipse to search an entire project or even an entire workspace, looking for references to a particular class, method, variable, etc. Suppose, for example, that you'd like to find out where in the program the Course.addCourseMeeting( ) method is called. Here's how you can find out all the places where the method is called:

Right-click on the identifier addCourseMeeting in the declaration of the method in Course.java
In the pop-up menu, select References and then Project. This will tell Eclipse to look for all the places in the entire project where the method is called.
A Search area will appear somewhere within the Eclipse window. (For me, it appeared along the bottom edge of the window, but I can't guarantee that it'll show up in the same place for you.) In that window, you'll see packages, inside of which are classes, inside of which are methods; double-clicking the methods will take you to lines of code on which the Course.addCourseMeeting( ) method is called. (This particular method is only called from one other place in the program: line 99 of ScheduleCourseCommandHandler.java.)

Using tools like reference searching will make it easier to find your way around this obviously unfamiliar code base that I've given you.