2004 Free Software Foundation, Inc."> ]> The GNU Testing Framework 3 February 2004 &version New release Rob Savoye Free Software Foundation Ben Elliston Free Software Foundation Lisa Elliston

rob@welcomehome.org

&dejagnu-copyright; 0.7 2004-02-02 bje So much restructuring it's not funny .. 0.6.2 2002-07-16 rob Add new tutorial as a new chapter. 0.6.1 2001-02-16 rob Add info on the new dejagnu.h file. 0.6 2001-02-16 rob Updated for new release. 0.5 2000-01-24 rob Initial version after conversion to DocBook. This document describes the functionality of DejaGnu, the testing framework of the GNU Project. DejaGnu is a framework for testing other programs and many widely used GNU packages such as the GNU Compiler Collection (GCC) and the GNU Debugger (GDB) are tested using DejaGnu. Its purpose is to provide a uniform testing environment. Programs can be just as easily tested locally or on a remote computer of a different type than the computer running DejaGnu. Each program can have multiple testsuites, all supported by a single test harness. &dj; is written in Expect which in turn relies upon Tcl. &dj; is a framework for testing other programs. Its purpose is to provide a uniform testing environment. Think of it as a library of Tcl procedures to support writing a test harness. A test harness is the testing infrastructure created to support a specific program. Each program can have multiple testsuites, all supported by a single test harness. &dj; is written in Expect, which in turn relies upon Tcl. More information on Tcl and Expect can be found in . Julia Menapace first coined the term ``&dj;'' to describe an earlier testing framework she had written at Cygnus Support for testing GDB. When it was replaced it with the Expect-based framework, it was like &dj; all over again. More importantly, it was also named after Deja Snow Savoye (13 years old in September 2003), who was a toddler during &dj;'s beginnings. There are several advantages to testing with &dj;: The flexibility and consistency of the &dj; framework make it easy to write tests for any program, including batch-oriented and interactive programs. &dj; provides a layer of abstraction which allows you to write tests that are portable to any host or target where a program must be tested. For instance, a test for GDB can run from any supported host system on any supported target system. &dj; runs tests on many single board computers, whose operating software ranges from a simple boot monitor to a real-time OS. All tests have the same output format. This makes it easy to integrate testing into other software development processes. &dj;'s output is human readable, but can be easily parsed by other software. Using Tcl and Expect, it is easy to create wrappers for existing testsuites. By incorporating existing tests under &dj;, it is easier to have a single set of reporting programs. Running tests requires two things: the testing framework and the testsuites themselves. Tests are usually written in Expect using Tcl, but you can also use a Tcl script to run a testsuite that is not based on Expect. Expect scripts use .exp as a filename extension as a convention. Testsuites are typically kept in the source tree of the programs that they test. Version &version; has a number of substantial changes over version 1.3. The most visible change is that the version of Expect and Tcl included in the release are up-to-date with the current stable net releases. The biggest change is to the target configuration system. While this greatly improved cross-testing, is has made that subsystem very complicated. Other changes include: More built-in support for building target binaries with the correct linker flags. Currently this only works with GCC as the cross-compiler. Lots of bug fixes from years of heavy use at Cygnus Solutions. The use of Autoconf and Automake for building. Updated documentation in SGML format using the GNU DocBook tools. Beta level support for Windows, however this is still work in progress. This requires the Cygwin POSIX subsystem for Windows. To use &dj; on Windows, you need to first install the Cygwin system. Cygwin is a POSIX system for Windows. This covers both utility programs and a library that adds POSIX system calls to Windows. Among them is pseudo tty support for Windows that emulates the POSIX pty standard. The latest Cygwin is always available from the Cygwin home page. This works well enough to run "make check" on the GNU development tools after building them on a native Cygwin system. &dj; grew out of the needs of Cygnus Solutions, the company formerly known as Cygnus Support. Cygnus maintained and enhanced a variety of free programs in many different environments. A testing tool was needed that: was useful to developers while fixing bugs; automated running many tests during a software release process; was portable among a variety of host computers; supported cross-development testing; permitted testing interactive programs, like GDB; and permitted testing batch oriented programs, like GCC. Some of the requirements proved challenging. For example, interactive programs do not lend themselves very well to automated testing, but this capability is important. For instance, it is imperative to make sure that GDB works as well when cross-debugging as it does in a native configuration. Probably the greatest challenge was testing in a cross-development environment. Most cross-development environments are customized by each developer. Even when buying packaged evaluation boards there are many differences. The communication interfaces vary from a serial line to Ethernet. &dj; was designed with a modular communication setup, so that each mode of communication can be added as required and supported thereafter. Once a communication procedure is implemented, any testsuite can use it. Currently &dj; can use rsh, rlogin, telnet, tip, kermit and mondfe for remote communications. &dj; conforms to the POSIX 1003.3 standard for test frameworks. The POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at the time of writing there is only one other POSIX conforming test framework: TET. TET was created by Unisoft for a consortium comprised of X/Open, Unix International and the Open Software Foundation. The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says ``The sun shall shine'', a corresponding assertion might be ``The sun is shining.'' A test based on this assertion would pass or fail depending on whether it is day or night. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written. As there is no testsuite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages and their definitions. Four messages are supported for a required feature of POSIX conforming systems and a fifth for a conditional feature. &dj; supports the use of all five output messages. In this sense a testsuite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case: PASS A test has succeeded. That is, it demonstrated that the assertion is true. XFAIL POSIX 1003.3 does not incorporate the notion of expected failures, so PASS, instead of XPASS, must also be returned for test cases which were expected to fail and did not. This means that PASS is in some sense more ambiguous than if XPASS is also used. FAIL A test has produced the bug it was intended to capture. That is, it has demonstrated that the assertion is false. The FAIL message is based on the test case only. Other messages are used to indicate a failure of the framework. As with PASS, POSIX tests must return FAIL rather than XFAIL even if a failure was expected. UNRESOLVED A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion; this outcome requires a human being to go over the results and determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should be resolved to PASS or FAIL before a test run can be considered finished. Note that for POSIX compliance, each assertion must produce a test result code. If the test isn't actually run, it must produce UNRESOLVED rather than just leaving that test out of the output. This means that you have to be careful when writing tests to not carelessly use Tcl commands like return---if you alter the flow of control of the Tcl code you must insure that every test still produces some result code. Here are some of the ways a test may wind up UNRESOLVED: The execution of a test is interrupted. A test does not produce a clear result. This is usually because there was an ERROR from &dj; while processing the test, or because there were three or more WARNING messages. Any WARNING or ERROR messages can invalidate the output of the test. This usually requires a human being to examine the output to determine what really happened---and to improve the test case. A test depends on a previous test, which fails. The test was set up incorrectly. UNTESTED A test was not run. This is a place-holder, used when there is no real test case yet. UNSUPPORTED There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented. &dj; also uses this message when a testing environment (often a ``bare board'' target) lacks basic support for compiling or running the test case. For example, a test for the system subroutine gethostname would never work on a target board running only a boot monitor. &dj; uses the same output procedures to produce these messages for all testsuites and these procedures are already known to conform to POSIX 1003.3. For a &dj; testsuite to conform to POSIX 1003.3, you must avoid the setupxfail} procedure as described in the PASS section above and you must be careful to return UNRESOLVED where appropriate, as described in the UNRESOLVED section above. You can obtain DejaGnu from the DejaGnu web site at the Free Software Foundation, which is at www.gnu.org/software/dejagnu/ Once you have obtained the DejaGnu source archive and unpacked it, you must configure the package to specify where it is to be installed. You may then proceed with installing it. It is usually best to configure &dj; in a different directory than the source tree. To configure from a sibling build directory, execute the configure script from the source tree like this: ../dejagnu-&version/configure DejaGnu doesn't distinguish when installed whether it will be testing programs on a native system or a remote system. That is determined at runtime via its own configuration files. You may wish to use the configure option --prefix to specify where you want DejaGnu installed. The default installation prefix is under /usr/local, but you can select any alternate directory altdir by including --prefix{altdir}} on the configure command line. DejaGnu includes a small testsuite of own, but it does not include other testsuites; these are typically available with the packages that they test. Once a build directory is configured, installing &dj; is a simple matter of running: $ make install The make install output will show what was installed where, but the most important installation step to note is that the runtest command is installed into the $prefix/bin directory. This program is the driver for &dj;. &user; &ref;