dfregress - An automated test driver and framework

dfregress - An automated test driver and framework

 * dfregress is a simple test framework and test automation driver. It
   supports both normal userland testcases and kernel test cases.

 * It's key aim is to make it simple, if not outright dead easy, to
   write test cases. A test case is a simple program using
   printf/fprintf and exit - no magic needed. A kernel test case is a
   very small module that just needs to implement a few functions and
   call a logging and result function.

 * Sample output of the text frontend (Frontends are explained further
   down the text): http://leaf.dragonflybsd.org/~alexh/dfregress.txt

 * dfregress is very UNIXy, it uses makefiles to build the testcases,
   stdout/stderr redirection from the testcases (no fancy output
   functions needed in the testcases) and evaluates the normal return
   value of the testcase (no need to call fancy functions).

 * For kernel testcases it is a bit different - you do have to call
   functions to log output and to log the result, but it is very simple,
   hardly any overhead.

 * The test driver assumes that testcases are in the testcases/
   directory, but it supports several command line options.

 * The tests to run, including several options, are specified in the
   runlist file. An example runlist including all the testcases is
   included as config/runlist.run. Options that can be specified are:
   - timeout: (in seconds) after which the test is aborted if it hasn't
     finished.
   - test type (userland, kernel or buildonly)
   - make: which 'make' tool to use to build the test cases. Defaults to
     'make', but could also be 'gmake', etc.
   - nobuild: doesn't build the testcase and tries to directly execute
     it.
   - pre, post: external pre-run and post-run commands, e.g. to set up a
     vn device and to tear it down. This is to avoid duplication in test
     cases, the common setup can be factored out.
   - intpre, intpost: similar to the above, but it assumes that the
     testcase, when passed the parameter 'pre', will do the pre-setup,
     and when passed 'post' will do the post-test cleanup/setup/etc.
   - any number of command line arguments that are passed to the test
     case. (See the crypto/ test examples in the runlist).

 * A range of sample testcases are available in
   test/dfregress/testcases/sample, including a kernel testcase sample.

 * Note that many of the test cases in the testcases/ directory have
   been copied from elsewhere in the main repository and are,
   temporarily at least, duplicated.

 * The test driver is completely separated from the frontends. The test
   driver outputs the test run results in an XML-like format (plist)
   that can easily be parsed using proplib. Python and Ruby also have
   plist modules that could be used to parse the output.

 * The only available frontend is a simple C program that will parse the
   intermediate format to an easy to read plain text format. Additional
   frontends can be written in any language, as long as it is possible
   to load plists. Frontends are in the fe/ directory.

 * XXX: the default options (currently just the timeout) are still
   hardcoded in config.c.

 * The NOTES file gives details on how the test execution occurs and
   what result code can be raised at which point. This document, and a look
   at the generated plist, is all you need to write a new frontend that,
   for example, generates beautiful HTML output.
   For completeness sake, a part of NOTES is reproduced under the main commit
   message - specifically the part detailing the execution of a single test
   case.

======
Execution of a single test case:
======
1) chdir to testcase directory
        - if it fails, set RESULT_PREFAIL (sysbuf is of interest), goto (6)

2) build testcase (make) (unless nobuild flag is set).
        + build_buf is used for stdout/stderr
        - if there is an internal driver error (that leads to not running the
          build command), set RESULT_PREFAIL (sysbuf is of interest), goto (6)
        - if the build command has a non-zero exit value, set the result to
          BUILDFAIL, unless it's a buildonly test case, in which it is set to
          the actual result value (TIMEOUT, SIGNALLED, FAIL)
          goto (6)

3) run 'pre' command if intpre or pre is set.
        + precmd_buf is used for stdout/stderr
        - if there is an internal driver error (that leads to not running the
          command), set RESULT_PREFAIL (sysbuf is of interest), goto (6)
        - if the pre command has a non-zero exit value, set RESULT_PREFAIL and
          goto (6)

4) run actual testcase, depending on type
        + stdout_buf is used for stdout
        + stderr_buf is used for stderr
        - for BUILDONLY: set RESULT_PASS since the build already succeeded
        - for userland and kernel: run the testcase, possibly as a different
          user (depending on the runas option), set the result to the actual
          result value (TIMEOUT, SIGNALLED, FAIL, NOTRUN)
        - if there is an internal driver error (that leads to not running the
          command), RESULT_NOTRUN is set (sysbuf is of interest)

5) run 'post' command if intpost or post is set.
        + postcmd_buf is used for stdout/stderr
        - if there is an internal driver error (that leads to not running the
          command), set RESULT_POSTFAIL (sysbuf is of interest), goto (6)
        - if the post command has a non-zero exit value, set RESULT_POSTFAIL
          and goto (6)

6) clean testcase directory (make clean) (unless nobuild flag is set).
        + cleanup_buf is used for stdout/stderr and system (driver error) buffer
        - no further action.

7) results are saved.