3 .\" The DragonFly Project. All rights reserved.
5 .\" Redistribution and use in source and binary forms, with or without
6 .\" modification, are permitted provided that the following conditions
9 .\" 1. Redistributions of source code must retain the above copyright
10 .\" notice, this list of conditions and the following disclaimer.
11 .\" 2. Redistributions in binary form must reproduce the above copyright
12 .\" notice, this list of conditions and the following disclaimer in
13 .\" the documentation and/or other materials provided with the
15 .\" 3. Neither the name of The DragonFly Project nor the names of its
16 .\" contributors may be used to endorse or promote products derived
17 .\" from this software without specific, prior written permission.
19 .\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20 .\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21 .\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
22 .\" FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
23 .\" COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
24 .\" INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING,
25 .\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
26 .\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
27 .\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
28 .\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
29 .\" OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
37 .Nd an automation test driver and framework
43 .Op Fl p Ar prepost_dir
48 is a regression test framework and automation driver.
49 It executes a series of testcases as specified in the
51 and collects the results.
53 The path to the testcase collection is specified via the
55 argument, and similarly the path to the pre- and post commands is
60 only needs to be specified if the runlist contains custom pre or
63 The output is saved in plist format to the
66 Other tools, known as frontends, can parse the plist output into
67 an easier to read form.
69 The following is a summary of the arguments:
70 .Bl -tag -width indent
71 .It Fl o Ar output_plist
72 Specifies the file to which to write the results.
75 will be in plist format.
76 .It Fl t Ar testcase_dir
77 Specifies the directory in which to find the testcases specified in the runlist.
78 .It Fl r Ar runlist_file
79 Specifies the file containing the runlist to use for the test execution.
80 .It Fl p Ar prepost_dir
81 Specifies the directory in which to find the pre- and post commands used
83 This argument is only required when the runlist uses custom pre- or post
86 Prints a short synopsis.
89 Testcases are specified one testcase per line, with whitespace separated
91 Empty lines and lines beginning with a
95 Runlist lines are of the following format:
96 .Bd -literal -offset indent
97 .Ic testcase type options Cm arguments ...
102 part needs to be a relative path from the testcase base directory specified
105 argument to the resulting (after building the testcase) testcase executable.
106 The testcase will be executed with the
108 passed as command line arguments to it.
115 .Bl -tag -width indent -offset indent
117 A userland testcase is a normal userland executable that returns a non-zero
118 exit value on test failure.
120 A kernel testcase is run with the kernel test bridge inside the kernel.
122 A buildonly test passes when the build for the given testcase succeeds.
136 .Bl -tag -width indent -offset indent
138 This option does nothing.
139 All default options are applied.
140 .It Ic make Ar make_command
145 to build the testcase.
146 .It Ic timeout Ar timeout
147 Sets the timeout for the testcase after which the testcase will be aborted to
151 If this option is set, the build stage and cleanup stage of the test case
154 Runs the testcase as the user identified by the given
157 Executes the testcase command with the argument
159 during the pre-run command stage.
161 Executes the testcase command with the argument
163 during the post-run command stage.
165 Executes the command specified by
167 during the pre-run command stage.
168 .It Ic pre Ar postcmd
169 Executes the command specified by
171 during the post-run command stage.
173 .Sh TESTCASE EXECUTION
174 .Bl -enum -width 3n -offset indent
177 to the testcase directory.
178 If it fails, the result will be set to
182 buffer will provide further details.
185 the testcase using the command specified by the
187 option or, if not specified,
191 option was specified.
192 If there is an internal driver error, the result will be set to
194 the next step to be performed will be
198 buffer will provide further details.
199 If no internal error occurs, the
201 will contain the build log.
203 .Tn "RUN PRE COMMAND"
209 If there is an internal driver error, the result will be set to
211 the next step to be performed will be
215 buffer will provide further details.
216 If the pre command has a non-zero exit value, the result will be set to
219 next step to be performed will be
221 In this case and in the case where the command succeeds, the
223 will contain the execution log.
227 .Bl -tag -width 2n -compact
229 testcases will get their result set to
230 .Dv RESULT_PASS at this point, since the build must have succeeded already.
232 testcases will get executed in a separate child process, possibly as a
233 different user, depending on whether the
235 option was specified.
236 The result will be set to
238 if the timeout is reached before the testcase finishes,
240 if the testcase dies because of an unhandled signal (often due to crashing),
242 if the testcase could not be executed,
244 if the testcase completes but returns failure or
246 if the testcase completes and returns success.
248 testcases will be executed in kernel space by loading a given module and
249 running the testcase entry point function.
250 The result will be set to
252 if the testcase could not be executed.
253 Otherwise the result will be set according to the kernel test case to
260 The output will be logged separately for stdout and stderr to the
269 will contain more information.
271 .Tn "RUN POST COMMAND"
277 If there is an internal driver error, the result will be set to
278 .Dv RESULT_POSTFAIL ,
279 the next step to be performed will be
283 buffer will provide further details.
284 If the post command has a non-zero exit value, the result will be set to
287 next step to be performed will be
289 In this case and in the case where the command succeeds, the
291 will contain the execution log.
294 the testcase execution using the command specified by the
296 option or, if not specified,
302 option was specified.
303 If there is an internal driver error the
305 buffer will contain more information.
306 If no internal error occurs, the
308 will contain the cleanup log.
313 is in the Apple plist serialized object format.
314 This format can be easily parsed by using
317 Ruby and Python also have parsers for the plist format.
321 parses the intermediate output plist into a more easily readable format
322 such as plain text or websites.
329 .Sh HOW TO WRITE A TESTCASE
330 A userland testcase is a simple program that prints some relevant
331 information to stdout and stderr, both of which are captured by the test
332 driver, and returns an exit value of 0 if the test passed, or any other
333 non-zero exit value to signal a failure.
334 The exact exit value is recorded by
336 All signals/exceptions not explicitly caught by the testcase will abort
337 the execution of the testcase and the result will be set appropriately and
338 the signal number will be recorded.
340 A kernel testcase is a simple kernel module that defines two methods:
343 method as well as an optional
349 method will be run from a separate kernel thread.
350 The testcase will need to call
352 functions to record output and to notify of testcase completion.
355 man page for more details.
357 For all testcase types the build stage is common.
358 Every testcase should either have the
360 option set, or have a Makefile or similar in its directory.
363 assumes it is a standard
366 If that is not the case, the
368 option needs to be adjusted accordingly.
369 .Sh GENERAL ADVICE ON WRITING TESTCASES
370 Test only one thing at a time, it is not good practice to test multiple
371 things in the same testcase as it makes it less obvious what's going on.
373 Keep it short, simple and well documented on what the requirements are,
374 what the preconditions need to be, what exactly is being tested, ideally
375 with a reference to a particular bug if that exists, and most importantly
376 what the expected outcomes are.
378 Make sure your testcase doesn't depend on any other being run previously
379 as well as that it won't hinder any other testcase from running.
380 This effectively means that your testcase should make no assumptions as
381 to what has been run previously unless it has a registered pre-command
382 and that the system should be left as found before your testcase.
384 An example runlist can be found under
385 .Pa test/testcases/sample.run .
387 Several example testcases for both userland and kernel are under
388 .Pa test/testcases/sample .