(c) 2005 Marc Clifton All Rights Reserved.
In Part III, I'm going to introduce some extensions to unit testing that, in
my opinion, make unit testing more useful for the kind of work that I do.
I've extended the UI by adding tab pages for the pass, ignore, and fail
states. Test fixtures and their tests are now alphabetized (although not
necessarily run in alphabetical order--the sorting is done by the UI, not the
unit test library). And finally, I've added a few features that are
reflected in the UI as well.
Most of what I code (and I imagine what other people code) is two things:
- functions that do simple, specific, things
- processes that glue the functions together in a particular way
And a lot of times, the process is something that involves the user's
interaction. An example of a very simple linear process is a wizard
dialog, which prompts the user through a configuration in a very predictable and
regimented manner. The automatic billing case study that I've been using
in these articles is an example of a potentially less linear process.
Especially when there's a lot of user interaction, the software has to be made a
lot more robust (and flexible) in order to accommodate the different ways that
the user is going to interact with the program. The programming bug in the
Therac-25 that resulted in radiation overdose is a good example--the interface
involved a concurrent system and failed if the operator corrected an input error
within 8 seconds of entering the mistake. In this particular example:
The general consensus is that the Atomic Energy of Canada Limited is to
blame. There was only one person programming
the code for this system and he largely did all the testing. The machine was
tested for only 2700 hours of use, but for code which controls such a critical
machine, many more hours should have been put in to the testing phase. Also
Therac-25 was tested as a whole machine rather then in separate modules. Testing
in separate modules would have discovered many of the bugs. Also, if the AECL
believed that there were problems with the Therac-25 right after the first
incident then it is possible that most of the 5 other incidents could have been
avoided and possibly the 3 fatalities.1
On the other hand, the X-43A mishap, in which a hypersonic air-breathing
flight vehicle lost control in an unmanned test, was blamed not on the failure
to test separate modules, but the failure to properly test the system as a
whole:
The mishap board found the major contributors to the mishap were modeling
inaccuracies in the fin actuation system, modeling inaccuracies in the
aerodynamics, and insufficient variations of modeling parameters. The flight
mishap could only be reproduced when all of the modeling inaccuracies with
uncertainty variations were incorporated in the analysis.2
So here we have two different examples of how, while testing certainly was
done, it wasn't done in the right way--people lost their lives and taxpayers saw
their money going up in smoke.
Now, returning to the mundane, I have in the previous two articles on Unit
Testing:
- Written some unit tests in the test-first manner (Part I)
- Wrote stub code to verify that the tests failed (Part II)
- Corrected the unit tests (Part II)
- Implemented real functionality so that the unit tests passed (Part II)
But are my unit tests really all that good? The overall process
(automatic billing) consists of a lot of different steps, involves many
components, and involves a lot of user interaction:
There's a lot that can go wrong here, that must be done in a particular
order, and that are susceptible to the system changing because of the time it
takes to complete the process Consider that it can take weeks for a part
to arrive after it's been purchased, and several more weeks after that to
receive the invoice. Or, as sometimes happens, the invoice comes in before
the part has been received! Now, my case study is definitely a simplified
version of the purchasing/receiving/billing process at the boatyard where I've
written their yard management software, but even simplified, it is a good
illustration for the purposes of exploring unit testing.
There's a lot of repetition involved in my case study. For example,
testing whether the ClosePO function works involves setting up:
- two work orders
- three parts
- a vendor
- an invoice
- a charge
- and a partridge in a pear tree
But all this was already done as part of unit testing the individual work
orders, parts, vendors, invoices, and charges. Why not just combine these
steps into a single process?
A process is an ordered sequence of unit tests. As long as one test
passes, the next test is run. This requires several modifications to
MUTE:
- A test fixture must be designated as a process
- The tests themselves must have their order designated
- Tests that are not run because of a failure should be so indicated
- Running a process forward and in reverse
An in sequence process is one that runs in the order specified by the
programmer who wrote the unit tests. Each unit test typically builds on
information verified by the previous unit test. When a unit test fails,
the remaining unit tests in the sequence are designated as "not run" because it
would be pointless to run them. This is displayed with a blue circle for
each test not run. For example:
and the tests not run are listed in the "Not Run" tab:
The question then becomes, what do you test in order to ensure that the code
handles itself well when the user or the program does something in an unexpected
way (out of sequence)? Obviously, testing all the combinations is not
acceptable. The purchase order sequence test that I wrote involves 16
steps, and testing every combination of 16 steps is 16!, or 20,922,789,888,000
(that's almost 21 trillion cases!).
What does "out of sequence" mean? It means that a piece of code is run
before another piece of code. This clearly reduces the number of
combinations that have to be analyzed, because the total combinations includes
numerous combinations in which some code is still run in sequence, and we're not
interested in those because we know that the "in sequence" parts of the process
already pass! There is only one combination that runs all the code out of
sequence, and that's the combination in which the process is run in
reverse. So, there are only two tests that need to be performed--forward,
in which the process is run forward, and reverse, in which the process is run
backwards.
OK, this isn't entirely true. It is easily possible, for example, to
have a piece of code dependent upon two or more external objects. Testing
only in reverse order catches only the first dependency. Clearly, to catch
the second dependency, at least one predecessor (in sequence) must be run.
This condition is not handled in this version (yes, yes, I'll be adding it in
the next version as soon as I've put some thought into the implementation
issues).
This is something that is very worthy of additional unit test extensions, but
I'm not going to get into the issues involved at this point. Let's keep
things simple for now!
To support all this, we need some new attributes.
[AttributeUsage(AttributeTargets.Class, AllowMultiple=false, Inherited=true)]
public sealed class ProcessTestAttribute : Attribute
{
}
This attribute is attached to a test fixture (a class) to indicate to the
test runner that the tests should be run in the order specified by the
programmer. For example:
[TestFixture]
[ProcessTest]
public class POSequenceTest
{
...
}
[AttributeUsage(AttributeTargets.Method, AllowMultiple=false, Inherited=true)]
public sealed class SequenceAttribute : Attribute
{
private int order;
public int Order
{
get {return order;}
}
public SequenceAttribute(int i)
{
order=i;
}
}
This attribute is specified for each test case in the process test fixture,
numbered from 1 to the number of test cases. For example:
[Test, Sequence(1)]
public void POConstructor()
{
po=new PurchaseOrder();
Assertion.Assert(po.Number=="", "Number not initialized.");
Assertion.Assert(po.PartCount==0, "PartCount not initialized.");
Assertion.Assert(po.ChargeCount==0, "ChargeCount not initialized.");
Assertion.Assert(po.Invoice==null, "Invoice not initialized.");
Assertion.Assert(po.Vendor==null, "Vendor not initialized.");
}
[Test, Sequence(2)]
public void VendorConstructor()
{
vendor=new Vendor();
Assertion.Assert(vendor.Name=="", "Name is not an empty string.");
Assertion.Assert(vendor.PartCount==0, "PartCount is not zero.");
}
[Test, Sequence(3)]
public void PartConstructor()
{
part1=new Part();
Assertion.Assert(part1.VendorCost==0, "VendorCost is not zero.");
Assertion.Assert(part1.Taxable==false, "Taxable is not false.");
Assertion.Assert(part1.InternalCost==0, "InternalCost is not zero.");
Assertion.Assert(part1.Markup==0, "Markup is not zero.");
Assertion.Assert(part1.Number=="", "Number is not an empty string.");
part2=new Part();
part3=new Part();
}
...
The world is not perfect, and, when running our unit tests in reverse, we
don't want the unit test to fail, we want to see if the code being
tested fails. Therefore, there are cases when it is necessary to
execute some code earlier in the process in order to ensure that the unit test,
which depends on this code, doesn't break. This attribute handles
that. For example:
[Test, Sequence(4), Requires("PartConstructor")]
public void PartInitialization()
{
part1.Number="A";
part1.VendorCost=15;
Assertion.Assert(part1.Number=="A", "Number did not get set.");
Assertion.Assert(part1.VendorCost==15, "VendorCost did not get set.");
part2.Number="B";
part2.VendorCost=20;
part3.Number="C";
part3.VendorCost=25;
}
In order to initialize a part, well, that part has to be constructed
first! Therefore, this unit test requires that the constructor test be run
first.
It is very easy to fall into the idea that, for example, closing the PO
requires that parts and charges have been assigned to the PO. This is
NOT how the Requires attribute should be used, because all
this does is ensure that the process is run in a forward direction.
Rather, this attribute should be used to ensure that parameters that the unit
test code needs are already in existence. The only thing I've ever needed
the Requires attribute for is to guarantee that an object exists to
which the unit test is about to assign a literal. Contrast the above
example with the following code:
[Test, Sequence(15), Requires("POConstructor")]
public void AddInvoiceToPO()
{
po.Invoice=invoice;
Assertion.Assert(invoice.Number==po.Invoice.Number,
"Invoice not set correctly.");
}
Here, we do NOT require that the invoice object be constructed.
The property should validate this for itself. However, we DO
require that the purchase order object be prior constructed. A simple
"l-value" rule is sufficient to determine if the Requires attribute
needs to be used--if the object is on the left side of the equal sign, then
yes. If it is on the right side of the equal sign, then no.
Note that in the definition of the Requires attribute:
[AttributeUsage(AttributeTargets.Method, AllowMultiple=true, Inherited=true)]
public sealed class RequiresAttribute : Attribute
{
private string priorTestMethod;
public string PriorTestMethod
{
get {return priorTestMethod;}
}
public RequiresAttribute(string methodName)
{
priorTestMethod=methodName;
}
}
multiple attributes may be assigned to the same test. For example:
[Test]
[Sequence(16)]
[Requires("POConstructor")]
[Requires("WorkOrderConstructor")]
public void ClosePO()
{
...
}
[AttributeUsage(AttributeTargets.Method, AllowMultiple=false, Inherited=true)]
public sealed class ReverseProcessExpectedExceptionAttribute : Attribute
{
private Type expectedException;
public Type ExceptionType
{
get {return expectedException;}
}
public ReverseProcessExpectedExceptionAttribute(Type exception)
{
expectedException=exception;
}
}
In a regular unit test, the ExpectedException attribute is used
to ensure that the code under test throws the appropriate exception because the
unit test is setting up a failure case. Process tests are set up to
succeed--in other words, there shouldn't be any exceptions thrown when the
process is run in the forward direction (individual tests that throw exceptions
are still part of other unit tests). Testing a process in the reverse
direction may cause once working code to fail, hopefully with an exception
thrown by the code, not the framework. To test this, the
ReverseProcessExpectedException attribute has been added to make
sure that the code handles and out of order process.
Using the automatic billing case study I've been developing in Parts I and
II, I wrote a process test that goes through all the steps involved in getting
to the point where the PO can be closed. Compare this code to the
ClosePO unit test written in Part II:
[Test]
[Sequence(16)]
[Requires("POConstructor")]
[Requires("WorkOrderConstructor")]
public void ClosePO()
{
po.Close();
// one charge slip should be added to both work orders
Assertion.Assert(wo1.ChargeSlipCount==1,
"First work order: ChargeSlipCount not 1.");
Assertion.Assert(wo2.ChargeSlipCount==1,
"Second work order: ChargeSlipCount not 1.");
...
}
Note that all the setup stuff has already been done. A lot simpler,
isn't it?
Running this process in the forward direction, everyone is happy:
Now let's look at what happens when I run the process in reverse:
Yuck! Obviously, my code does not handle things being done in the wrong
order very well! Inspecting the failures:
makes it obvious that I'm not handling un-initialized objects very well at
all. Time to fix those.
An exception is thrown if the invoice does not already exist:
if (invoice==null)
{
throw(new InvalidInvoiceException());
}
This is an important issue to note to the user--a PO cannot be closed without
having an invoice against that PO!
This illustrates the usefulness of testing property assignments. It
could easily have been the unit test itself that was throwing an exception
because the the property assignment is not checking to see if the object being
passed to it is a valid object! To fix this, the assignment is
modified:
public Invoice Invoice
{
get {return invoice;}
set
{
if (value==null)
{
throw(new InvalidInvoiceException());
}
else
if (value.Number=="")
{
throw(new UnassignedInvoiceException());
}
// *** NO VENDOR TEST !!! ***
if (value.Vendor.Name != vendor.Name)
{
throw(new DifferentVendorException());
}
invoice=value;
}
}
Testing Properties
| Technically, the getter should also be tested in our unit tests, and
this raises the issue of whether or not the object returning the value
should test the value itself, or the object requesting the
value. |
This issue is complicated by the fact that it is often common practice to
"overload information". Meaning that, if the purchase order returns a
NULL, it means that the invoice hasn't yet been set. While this is easy
coding practice, it isn't a good practice. A method like:
public bool InvoiceExists(void) {return value != null;}
is a much better solution. Then, the getter can throw an exception if
the caller is about to get inappropriate data.
The same issues present themselves here and are easily corrected:
public void Add(Charge c)
{
if (c==null)
{
throw(new InvalidChargeException());
}
if (c.Description=="")
{
throw(new UnassignedChargeException());
}
charges.Add(c);
}
Validating Data Using The Has A Relationship
This brings up another design issue--if the Invoice class
were written in such a way that it merely returned a collection of charges
which the caller manipulates directly, then it would not be possible the
catch bad data exceptions. |
This points out the benefits of a "has a" relationship--the wrapping class
can perform data validation that would otherwise not be possible.
Here we have a case where the unit test is throwing an exception because the
Invoice class is not testing for valid data. This is easily
fixed:
public Vendor Vendor
{
get {return vendor;}
set
{
if (value==null)
{
throw(new InvalidVendorException());
}
vendor=value;
}
}
A couple data validation tests are added to fix this problem:
public void Add(Part p, WorkOrder wo)
{
if (p==null)
{
throw(new InvalidPartException());
}
if (wo==null)
{
throw(new InvalidWorkOrderException());
}
if (p.Number=="")
{
throw(new UnassignedPartException());
}
if (wo.Number=="")
{
throw(new UnassignedWorkOrderException());
}
if (!vendor.Find(p))
{
throw(new PartNotFromVendorException());
}
parts.Add(p, wo);
partsArray.Add(p);
}
More of the same...
public void Add(Part p)
{
if (p==null)
{
throw(new InvalidPartException());
}
if (p.Number=="")
{
throw(new UnassignedPartException());
}
if (parts.Contains(p.Number))
{
throw(new DuplicatePartException());
}
parts.Add(p.Number, p);
partsArray.Add(p);
}
Running the process is reverse now works, in the sense that all the bad data
is validated and the proper exceptions are thrown.
Unit testing really brings to the forefront the difference between using
assertions (or program by contract) and throwing exceptions (let the caller
handle the error). This doesn't mean that programming by contract requires
using assertions--rather, it means that programming by contract shoud not
use assertions but rather throw exceptions. The reason for this is
simple--the unit test itself uses assertions to validate data and expects
exceptions to be thrown if the unit under test detects a fault. The unit
test then verifies that the exception is expected or not.
Throwing exceptions results in more robust code. The exception tests
can (and should!) be left in production code, so that the higher level functions
can gracefully report problems to the user and take corrective actions.
Asserts, when they are removed in production code, simply result in program
crashes or erroneous operations when the unexpected happens (which inevitably
does).
Using unit testing principles therefore, asserts are quickly going to go the
way of the dinosaur. (Disagreements???)
- Unit tests themselves have bugs and therefore need testing.
- Writing "test first" code isn't as bad as I thought it would be.
- There's a lot of redundant setup code that gets written as the functions
being tested get higher up in the "object chain".
- Unit tests don't really ensure good coding and design. Bad code can
pass a unit test as easily as good code.
- Unit tests test, well, units, not processes.
- Testing property assignments is useful, especially to check if the
class is validating the value.
- Unit testing changes the paradigm of using asserts to throwing exceptions
(I imagine this statement will get lots of discussion)
As it currently stands, the code is not very robust. It doesn't verify
that
- the process sequence starts at 1
- increments by 1
- doesn't have any duplicates or skips
- the
Requires functions actually exist.
In other words, some unit tests really need to be written for this
thing! Well, in the next release, it'll be a bit more bullet proofed.
Well, part IV is not going to talk about scripting. Part IV is going to
look at some other useful additions to unit testing. Hopefully the next
part can wrap up those extensions (this one issue was worthy of an article in
itself, in my opinion), so hopefully Part V will cover scripted unit
testing.
Footnotes
1 - http://neptune.netcomp.monash.edu.au/
cpe9001/assets/readings/www_uguelph_ca_~tgallagh_~tgallagh.html
2 - http://spaceflightnow.com/news/n0307/23x43a/
References
Checking High-Level Protocols in Low-Level Software: http://research.microsoft.com/~maf/talks/Berkeley-VAULT.ppt
Programming By Contract: http://www.cs.unc.edu/~stotts/COMP204/contract.html