Test framework comparison
Here's a report of various testing frameworks which concentrates on that authors needs. A report which looks in detail at how each testing framework sandboxes each test would be nice.

How does CLUnit compare with XPTEST?

I tried using CLUnit for a little project recently, and played only a bit with XPTEST too. Here are my impressions:

  • In CLUnit, a test has a description and a category. Not a name. Or, sometimes the description is called its name instead. It turns out that you either have to give tests a very short description, or forget about (run-named-test), unless you want to type stuff like (run-named-test "Test whether foo-frobnicate works on Sunday afternoons when your cat stomped over your keyboard").
  • Both are not as polished as other xUnit frameworks. For example, both do not show anything green if you've been a nice guy (nor anything similar that would work without a GUI toolkit). Nor do they have all those convenient macros/functions to say whether a test failed or succeeded: CLUnit by default wants the output of a test function to be t (or eq[[ua]l[p]]) a given value), in XPTEST you have to call a function called FAILURE to indicate failure (or signal a TEST-FAILURE condition, or pass something false to TEST-ASSERT). Something like assert-equal, assert-not-equal, assert-condition etc. is missing.
  • XPTEST seems to be a little more like other xUnits. It is based on fixture classes, setup and teardown methods etc.

I am a little disappointed with CLUnit actually, and will look at XPTEST. It looks better, but then, I haven't used it in practice.

[Dunno who wrote the above --- rpg]

Comparison with RT

I have been using the venerable RT framework for testing a large program. It is simple, and seems to be very reliable.

On the other hand, I don't know how scalable it is. For a small system, you can define a bunch of tests, and then you call (DO-TESTS). That's fine. But I'm not sure how one should progress given a larger system, with multiple modules. In this case, I think one would like to have a framework that would permit one to group tests into "families" or modules, and only run the tests for a specific family.

I believe that some of these test frameworks provide this facility, but there are a bewildering number of them. I'm going to try to find a framework that will give me this test-grouping capability without causing too much upset when migrating from RT.

Quick follow-up: Seems like there's a modified version of RT that permits one to attach "notes" to entries (tests). Notes can contain disabling info. So it's possible to do a simple version of what I want using the latest version of RT, I think. On the other hand, the notes framework seems like a general-purpose, low-level tool to do the test grouping, and doesn't seem terribly well integrated into RT (not mentioned in docs, nor is it exercised in RT's self-test suite). Maybe it would be better to get a tool that does just this thing, and does it in a way that's integrated into the framework tightly.

Robert Goldman [rpg --- not that rpg]