diff options
| author | Ben Elliston <bje@gnu.org> | 2004-02-03 14:41:25 +0000 |
|---|---|---|
| committer | Ben Elliston <bje@gnu.org> | 2004-02-03 14:41:25 +0000 |
| commit | a2c66a5e8321016df4701484fce9bc750a5a7205 (patch) | |
| tree | 4e02d2a27141514587e2ff63a465d3230e666200 | |
| parent | d12323ba03dc5a0cb07d421de7233d4969415f5a (diff) | |
* doc/overview.sgml (revhistory): Update.
(install): Fold in top-level sections "obtaining" and
"installation" from ref.sgml.
* doc/user.sgml: Shuffle some chapters and top-level sections
about to make better structural sense.
* doc/ref.sgml (reference): Rename to "Library Reference".
(reference): Remove "obtaining" and "installation" sections.
Remove "filemap" section.
(unittestapi): Fold the contents of this chapter into the Unit
Testing chapter.
(appendix): New.
| -rw-r--r-- | ChangeLog | 14 | ||||
| -rw-r--r-- | doc/overview.sgml | 130 | ||||
| -rw-r--r-- | doc/ref.sgml | 310 | ||||
| -rw-r--r-- | doc/user.sgml | 2566 |
4 files changed, 1500 insertions, 1520 deletions
@@ -1,5 +1,19 @@ 2004-02-04 Ben Elliston <bje@wasabisystems.com> + * doc/overview.sgml (revhistory): Update. + (install): Fold in top-level sections "obtaining" and + "installation" from ref.sgml. + * doc/user.sgml: Shuffle some chapters and top-level sections + about to make better structural sense. + * doc/ref.sgml (reference): Rename to "Library Reference". + (reference): Remove "obtaining" and "installation" sections. + Remove "filemap" section. + (unittestapi): Fold the contents of this chapter into the Unit + Testing chapter. + (appendix): New. + +2004-02-04 Ben Elliston <bje@wasabisystems.com> + * doc/overview.sgml (dejagnu-copyright): Update copyright year. (dejagnu-code-copyright): Likewise. (<bookinfo>): Include myself as an author. diff --git a/doc/overview.sgml b/doc/overview.sgml index 1ab59c6..5ca5e2d 100644 --- a/doc/overview.sgml +++ b/doc/overview.sgml @@ -90,27 +90,33 @@ into another language, under the above conditions for modified versions. <!-- </legalnotice> --> <revhistory> <revision> + <revnumber>0.7</revnumber> + <date>2004-02-02</date> + <authorinitials>bje</authorinitials> + <revremark>So much restructuring it's not funny ..</revremark> + </revision> + <revision> <revnumber>0.6.2</revnumber> - <date>2002-7-16</date> - <authorinitials>rob@welcomehome.org</authorinitials> + <date>2002-07-16</date> + <authorinitials>rob</authorinitials> <revremark>Add new tutorial as a new chapter.</revremark> </revision> <revision> <revnumber>0.6.1</revnumber> - <date>2001-2-16</date> - <authorinitials>rob@welcomehome.org</authorinitials> + <date>2001-02-16</date> + <authorinitials>rob</authorinitials> <revremark>Add info on the new dejagnu.h file.</revremark> </revision> <revision> <revnumber>0.6</revnumber> - <date>2001-2-16</date> - <authorinitials>rob@welcomehome.org</authorinitials> + <date>2001-02-16</date> + <authorinitials>rob</authorinitials> <revremark>Updated for new release.</revremark> </revision> <revision> <revnumber>0.5</revnumber> - <date>2000-1-24</date> - <authorinitials>rob@welcomehome.org</authorinitials> + <date>2000-01-24</date> + <authorinitials>rob</authorinitials> <revremark>Initial version after conversion to DocBook.</revremark> </revision> </revhistory> @@ -137,7 +143,7 @@ into another language, under the above conditions for modified versions. <chapter id=overview xreflabel=Introduction> <title>Introduction</title> <sect1 id=whatis xreflabel="What is &dj;?"> - <title>What is &dj; ?</title> + <title>What is &dj;?</title> <para><productname>&dj;</productname> is a framework for testing other programs. Its purpose is to provide a uniform @@ -455,10 +461,108 @@ into another language, under the above conditions for modified versions. <chapter id=install> <title>Installation</title> - <sect1> - <title>Foo</title> - <para>Foo.</para> - </sect1> + + <sect1 id=obtaining xreflabel="Obtaining DejaGnu"> + <title>Obtaining DejaGnu</title> + + <para>You can obtain DejaGnu from the DejaGnu web site at the + <ulink URL="http://www.gnu.org">Free Software Foundation</ulink>, + which is at <ulink + URL="http://www.gnu.org/software/dejagnu/">www.gnu.org/software/dejagnu/ + </ulink></para> + </sect1> + + <sect1 id=installation xreflabel="Installation"> + <title>Installation</title> + + <para>Once you have the DejaGnu source unpacked and available, you must + first configure the software to specify where it is to run (and the + associated defaults); then you can proceed to installing it.</para> + + <sect2 id=configuring xreflabel="Configuring DejaGnu"> + <title>Configuring DejaGnu</title> + + <para>It is usually best to configure in a directory separate from the + source tree, specifying where to find the source with the optional + <emphasis>--srcdir</emphasis> option to + <emphasis>configure</emphasis>. DejaGnu uses the GNU + <emphasis>autoconf</emphasis> to configure itself. For more info on using + autoconf, read the GNU autoconf manual. To configure, execute the + <filename>configure</filename> program, no other options are + required. For an example, to configure in a seperate tree for objects, + execute the configure script from the source tree like this:</para> + + <screen> + ../dejagnu-&version/configure + </screen> + + <para>DejaGnu doesn't care at config time if it's for testing a native + system or a cross system. That is determined at runtime by using the + config files.</para> + + <para>You may also want to use the <command>configure</command> option + <emphasis>--prefix</emphasis> to specify where you want DejaGnu and its + supporting code installed. By default, installation is in subdirectories + of <filename>/usr/local</filename>, but you can select any alternate + directory <symbol>altdir</symbol> by including + <option>--prefix</option>{altdir}} on the + <command>configure</command> command line. (This value is captured in + the Makefile variables <emphasis>prefix</emphasis> and + <emphasis>exec</emphasis>prefix}.)</para> + + <para>Save for a small number of example tests, the DejaGnu distribution + itself does not include any testsuites; these are available + separately. Testsuites for the GNU development tools are included in + those releases. After configuring the top-level DejaGnu directory, unpack + and configure the test directories for the tools you want to test; then, + in each test directory, run <emphasis>make check</emphasis> to build + auxiliary programs required by some of the tests, and run the test + suites.</para> + + </sect2> + + <sect2 id=installing xreflabel="Installing DejaGnu"> + <title>Installing DejaGnu</title> + + <para>To install DejaGnu in your filesystem (either in + <filename>/usr/local</filename>, or as specified by your + <emphasis>--prefix</emphasis> option to <emphasis>configure</emphasis>), + execute.</para> + + <screen> + eg$ make install + </screen> + + <para><emphasis>make install</emphasis>does thes things for + DejaGnu:</para> + + <itemizedlist mark=bullet> + <listitem><para>Look in the path specified for executables + <symbol>$exec_prefix</symbol>) for directories called + <filename>lib</filename> and <filename>bin</filename>. If these + directories do not exist, <emphasis>make install</emphasis> creates + them.</para></listitem> + + <listitem><para>Create another directory in the + <filename>share</filename> directory, called + <filename>dejagnu</filename>, and copy all the library files into + it.</para></listitem> + + <listitem><para>Create a directory in the + <filename>dejagnu/share</filename> directory, called + <filename>config</filename>, and copy all the configuration files into + it.</para></listitem> + + <listitem><para>Copy the <emphasis>runtest</emphasis> shell script into + <filename>$exec_prefix/bin</filename>.</para></listitem> + + <listitem><para>Copy <filename>runtest.exp</filename> into + <filename>$exec_prefix/lib/dejagnu</filename>. This is the main Tcl + code implementing DejaGnu.</para></listitem> + + </itemizedlist> + </sect2> + </sect1> </chapter> <!-- include the user manual --> diff --git a/doc/ref.sgml b/doc/ref.sgml index 2fa363a..928ae05 100644 --- a/doc/ref.sgml +++ b/doc/ref.sgml @@ -1,113 +1,8 @@ <chapter id=reference> - <title>Reference</title> - - <sect1 id=obtaining xreflabel="Obtaining DejaGnu"> - <title>Obtaining DejaGnu</title> - - <para>You can obtain DejaGnu from the DejaGnu web site at the - <ulink URL="http://www.gnu.org">Free Software Foundation</ulink>, - which is at <ulink - URL="http://www.gnu.org/software/dejagnu/">www.gnu.org/software/dejagnu/ - </ulink></para> - - </sect1> - - <sect1 id=installation xreflabel="Installation"> - <title>Installation</title> - - <para>Once you have the DejaGnu source unpacked and available, you must - first configure the software to specify where it is to run (and the - associated defaults); then you can proceed to installing it.</para> - - <sect2 id=configuring xreflabel="Configuring DejaGnu"> - <title>Configuring DejaGnu</title> - - <para>It is usually best to configure in a directory separate from the - source tree, specifying where to find the source with the optional - <emphasis>--srcdir</emphasis> option to - <emphasis>configure</emphasis>. DejaGnu uses the GNU - <emphasis>autoconf</emphasis> to configure itself. For more info on using - autoconf, read the GNU autoconf manual. To configure, execute the - <filename>configure</filename> program, no other options are - required. For an example, to configure in a seperate tree for objects, - execute the configure script from the source tree like this:</para> - - <screen> - ../dejagnu-&version/configure - </screen> - - <para>DejaGnu doesn't care at config time if it's for testing a native - system or a cross system. That is determined at runtime by using the - config files.</para> - - <para>You may also want to use the <command>configure</command> option - <emphasis>--prefix</emphasis> to specify where you want DejaGnu and its - supporting code installed. By default, installation is in subdirectories - of <filename>/usr/local</filename>, but you can select any alternate - directory <symbol>altdir</symbol> by including - <option>--prefix</option>{altdir}} on the - <command>configure</command> command line. (This value is captured in - the Makefile variables <emphasis>prefix</emphasis> and - <emphasis>exec</emphasis>prefix}.)</para> - - <para>Save for a small number of example tests, the DejaGnu distribution - itself does not include any testsuites; these are available - separately. Testsuites for the GNU development tools are included in - those releases. After configuring the top-level DejaGnu directory, unpack - and configure the test directories for the tools you want to test; then, - in each test directory, run <emphasis>make check</emphasis> to build - auxiliary programs required by some of the tests, and run the test - suites.</para> + <title>Library Reference</title> - </sect2> - - <sect2 id=installing xreflabel="Installing DejaGnu"> - <title>Installing DejaGnu</title> - - <para>To install DejaGnu in your filesystem (either in - <filename>/usr/local</filename>, or as specified by your - <emphasis>--prefix</emphasis> option to <emphasis>configure</emphasis>), - execute.</para> - - <screen> - eg$ make install - </screen> - - <para><emphasis>make install</emphasis>does thes things for - DejaGnu:</para> - - <itemizedlist mark=bullet> - <listitem><para>Look in the path specified for executables - <symbol>$exec_prefix</symbol>) for directories called - <filename>lib</filename> and <filename>bin</filename>. If these - directories do not exist, <emphasis>make install</emphasis> creates - them.</para></listitem> - - <listitem><para>Create another directory in the - <filename>share</filename> directory, called - <filename>dejagnu</filename>, and copy all the library files into - it.</para></listitem> - - <listitem><para>Create a directory in the - <filename>dejagnu/share</filename> directory, called - <filename>config</filename>, and copy all the configuration files into - it.</para></listitem> - - <listitem><para>Copy the <emphasis>runtest</emphasis> shell script into - <filename>$exec_prefix/bin</filename>.</para></listitem> - - <listitem><para>Copy <filename>runtest.exp</filename> into - <filename>$exec_prefix/lib/dejagnu</filename>. This is the main Tcl - code implementing DejaGnu.</para></listitem> - - </itemizedlist> - </sect2> - </sect1> - - <sect1 id=builtins xreflabel="Builtin Procedures"> - <title>Builtin Procedures</title> - - <para>DejaGnu provides these Tcl procedures.</para> + <sect1 id=blah> + <title>Internal Procedures</title> <sect2 id=coreprocs xreflabel="Core Internal Procedures"> <title>Core Internal Procedures</title> @@ -4227,198 +4122,23 @@ </sect3> </sect2> - - </sect1> - - <sect1 id=filemap> - <title>File Map</title> - - <para>This is a map of the files in DejaGnu.</para> - - <itemizedlist> - <listitem><para>runtest</para></listitem> - <listitem><para>runtest.exp</para></listitem> - <listitem><para>stub-loader.c</para></listitem> - <listitem><para>testglue.c</para></listitem> - <listitem><para>config</para></listitem> - <listitem><para>baseboards</para></listitem> - <listitem><para>lib/debugger.exp</para></listitem> - <listitem><para>lib/dg.exp</para></listitem> - <listitem><para>lib/framework.exp</para></listitem> - <listitem><para>lib/ftp.exp</para></listitem> - <listitem><para>lib/kermit.exp</para></listitem> - <listitem><para>lib/libgloss.exp</para></listitem> - <listitem><para>lib/mondfe.exp</para></listitem> - <listitem><para>lib/remote.exp</para></listitem> - <listitem><para>lib/rlogin.exp</para></listitem> - <listitem><para>lib/rsh.exp</para></listitem> - <listitem><para>lib/standard.exp</para></listitem> - <listitem><para>lib/target.exp</para></listitem> - <listitem><para>lib/targetdb.exp</para></listitem> - <listitem><para>lib/telnet.exp</para></listitem> - <listitem><para>lib/tip.exp</para></listitem> - <listitem><para>lib/util-defs.exp</para></listitem> - <listitem><para>lib/utils.exp</para></listitem> - <listitem><para>lib/xsh.exp</para></listitem> - <listitem><para>lib/dejagnu.exp</para></listitem> - </itemizedlist> - - </sect1> - +</sect1> </chapter> -<chapter id=unittestapi xreflabel="Unit Testing API"> - <title>Unit Testing API</title> - - <sect1 id=cunit xreflabel="C Unit Testing API"> - <title>C Unit Testing API</title> - - <para>All of the functions that take a - <parameter>msg</parameter> parameter use a C char * that is - the message to be dislayed. There currently is no support for - variable length arguments.</para> - - - <sect2 id=passfunc xreflabel="pass function"> - <title>Pass Function</title> - - <para>This prints a message for a successful test - completion.</para> - - <funcsynopsis role="C"> - <funcdef><function>pass</function></funcdef> - <paramdef><parameter>msg</parameter></paramdef> - </funcsynopsis> - - </sect2> - - <sect2 id=failfunc xreflabel="fail function"> - <title>Fail Function</title> - - <para>This prints a message for an unsuccessful test - completion.</para> - - <funcsynopsis role="C"> - <funcdef><function>fail</function></funcdef> - <paramdef><parameter>msg</parameter></paramdef> - </funcsynopsis> - - </sect2> - - <sect2 id=untestedfunc xreflabel="untested function"> - <title>Untested Function</title> - - <para>This prints a message for an test case that isn't run - for some technical reason.</para> - - <funcsynopsis role="C"> - <funcdef><function>untested</function></funcdef> - <paramdef><parameter>msg</parameter></paramdef> - </funcsynopsis> - </sect2> - - <sect2 id=unresolvedfunc xreflabel="unresolved function"> - <title>Unresolved Function</title> - - <para>This prints a message for an test case that is run, - but there is no clear result. These output states require a - human to look over the results to determine what happened. - </para> - - <funcsynopsis role="C"> - <funcdef><function>unresolved</function></funcdef> - <paramdef><parameter>msg</parameter></paramdef> - </funcsynopsis> - </sect2> - - <sect2 id=totalsfunc xreflabel="totals function"> - <title>Totals Function</title> - - <para>This prints out the total numbers of all the test - state outputs.</para> - - <funcsynopsis role="C"> - <funcdef><function>totals</function></funcdef> - <paramdef><parameter></parameter></paramdef> - </funcsynopsis> - </sect2> - - </sect1> - - <sect1 id=cppunit xreflabel="C++ Unit Testing API"> - <title>C++ Unit Testing API</title> - - <para>All of the methods that take a - <parameter>msg</parameter> parameter use a C char * - or STL string, that is the message to be - dislayed. There currently is no support for variable - length arguments.</para> - - <sect2 id=passmeth xreflabel="pass method"> - <title>Pass Method</title> - - <para>This prints a message for a successful test - completion.</para> - - <funcsynopsis role="C++"> - <funcdef><function>TestState::pass</function></funcdef> - <paramdef><parameter>msg</parameter></paramdef> - </funcsynopsis> - </sect2> - - <sect2 id=failmeth xreflabel="fail method"> - <title>Fail Method</title> - - <para>This prints a message for an unsuccessful test - completion.</para> - - <funcsynopsis role="C++"> - <funcdef><function>TestState::fail</function></funcdef> - <paramdef><parameter>msg</parameter></paramdef> - </funcsynopsis> - </sect2> - - <sect2 id=untestedmeth xreflabel="untested method"> - <title>Untested Method</title> - - <para>This prints a message for an test case that isn't run - for some technical reason.</para> - - <funcsynopsis role="C++"> - <funcdef><function>TestState::untested</function></funcdef> - <paramdef><parameter>msg</parameter></paramdef> - </funcsynopsis> - </sect2> - - <sect2 id=unresolvedmeth xreflabel="unresolved method"> - <title>Unresolved Method</title> +<appendix id=appendix> + <title>Appendix</title> + <sect1> <title> Reporting Bugs </title> <para>Blah.</para> </sect1> + <sect1> <title> Tcl/Tk and Expect Resources </title> <para>Blah.</para> </sect1> - <para>This prints a message for an test case that is run, - but there is no clear result. These output states require a - human to look over the results to determine what happened. - </para> +<!-- Keep this stuff: - <funcsynopsis role="C++"> - <funcdef><function>TestState::unresolved</function></funcdef> - <paramdef><parameter>msg</parameter></paramdef> - </funcsynopsis> - </sect2> - - <sect2 id=totalsmeth xreflabel="totals method"> - <title>Totals Method</title> - - <para>This prints out the total numbers of all the test - state outputs.</para> - - <funcsynopsis role="C++"> - <funcdef><function>TestState::totals</function></funcdef> - <paramdef><parameter></parameter></paramdef> - </funcsynopsis> - </sect2> - - </sect1> +See the document "Debugger for + Tcl Applications" by Don Libes. (Distributed in PostScript + form with <emphasis>expect</emphasis> as the file + <filename>expect/tcl-debug.ps</filename>. +--> -</chapter> +</appendix> <!-- Keep this comment at the end of the file Local variables: diff --git a/doc/user.sgml b/doc/user.sgml index ffaa6a5..c711a10 100644 --- a/doc/user.sgml +++ b/doc/user.sgml @@ -1,18 +1,758 @@ - <chapter id=gettingup> - <title>Getting DejaGnu up and running</title> + <chapter id=runningtests> + <title>Running Tests</title> + + <para>There are two ways to execute a testsuite. The most common + way is to rely upon support in a <filename>Makefile</filename> for + a <emphasis>check</emphasis> target. The other way is to execute + the <command>runtest</command> program directly. To run + <command>runtest</command> directcly from the command line + requires either all the correct options, or the <xref + linkend=local> must be setup correctly. Automake can generate a + <filename>Makefile</filename> that does all of the right things + when the user invokes <command>make check</command> and is the + preferred approach. Both ways of executing a testsuite will be + covered in more detail below.</para> + + <sect1 id=makecheck xreflabel="Make Check"> + <title>make check</title> + + <para>To run tests from an existing collection, use + <command>configure</command> to configure a build directory and + then type:</para> + + <screen> + make check + </screen> + + <para>If the <filename>Makefile</filename> has a + <emphasis>check</emphasis> target, it saves some effort. For + instance, it can set up any auxiliary programs or other files + needed by the tests. The most common file the + <emphasis>check</emphasis> target creates is + <filename>site.exp</filename>. The <filename>site.exp</filename> + file contains various variables that DejaGnu uses to determine + the configuration of the program being tested. This is mostly + used to support remote testing.</para> + + <para>The <emphasis>check</emphasis> target is supported by GNU + <productname>Automake</productname>. To have DejaGnu support added to your + generated <filename>Makefile.in</filename>, just add the keyword + dejagnu to the AUTOMAKE_OPTIONS variable in your + <filename>Makefile.am</filename> file.</para> + + <para>Once you have run <command>make check</command> to build + any auxiliary files, you can invoke the test driver + <command>runtest</command> directly to repeat the tests. You + will also have to execute <command>runtest</command> directly + for test collections with no <emphasis>check</emphasis> target + in the <filename>Makefile</filename>.</para> + + </sect1> + + <sect1 id=runtest xreflabel="Runtest"> + <title>Runtest</title> + + <para><command>runtest</command> is the executable test driver + for DejaGnu. You can specify two kinds of things on the + <command>runtest</command> command line: options and Tcl + variable assignments for the test scripts. The options are + listed alphabetically below.</para> + + <para><command>runtest</command> returns an exit code of + <emphasis>1</emphasis> if any test has an unexpected result; otherwise + (if all tests pass or fail as expected) it returns <emphasis>0</emphasis> + as the exit code.</para> + + <sect2 id=outputs xreflabel="Output States"> + <title>Test Result States</title> + + <para><filename>runtest</filename> flags the outcome of each + test as one of the following cases. See <xref linkend=posix> + for a discussion of how POSIX specifies the meanings of these + cases.</para> + + <variablelist> + <varlistentry> + <term>PASS</term> + <listitem><para>The most desirable outcome: the test was + expected to succeed and did succeed.</para></listitem> + </varlistentry> + + <varlistentry> + <term>XPASS</term> + <listitem><para>A pleasant kind of failure: a test was + expected to fail, but succeeded. This may indicate progress; + inspect the test case to determine whether you should amend + it to no longer expect failure.</para></listitem> + </varlistentry> + + <varlistentry> + <term>FAIL</term> + <listitem><para>A test was expected to succeed, but failed. + This may indicate a regression; inspect the test case and + the failing software to locate the bug.</para></listitem> + </varlistentry> + + <varlistentry> + <term>XFAIL</term> + <listitem><para>A test that was expected to fail did fail. + This result indicates no change in a known bug. If a test + fails because the environment running the test lacks some + facility required by the test, the outcome is + <emphasis>UNSUPPORTED</emphasis> instead.</para></listitem> + </varlistentry> + + <varlistentry> + <term>UNRESOLVED</term> + <listitem><para>Output from an unresolved test requires + manual inspection, as the testsuite could not automatically + determine the outcome. A test can report this outcome, for + instance, when a test is not completed as expected.</para></listitem> + </varlistentry> + + <varlistentry> + <term>UNTESTED</term> + <listitem><para>A test case is not yet complete, and in + particular cannot yet produce a <emphasis>PASS</emphasis> or + <emphasis>FAIL</emphasis>. You can also use this outcome for + placeholder tests that note explicitly the absence of a real + test case for a particular property.</para></listitem> + </varlistentry> + + <varlistentry> + <term>UNSUPPORTED</term> + <listitem><para>A test depends on a conditionally available feature + that does not exist (in the configured testing environment). For + example, you can use this outcome to report on a test case that does + not work on a particular target because its operating system support + does not include a required subroutine.</para></listitem> + </varlistentry> + </variablelist> + + <para><command>runtest</command> may also display the following + messages:</para> + + <variablelist> + <varlistentry> + <term>ERROR</term> + <listitem><para>Indicates a major problem (detected by the test case + itself) in running the test. This is usually an unrecoverable error, + such as a missing file or loss of communication to the target. (POSIX + testsuites should not emit this message; use + <emphasis>UNSUPPORTED</emphasis>, <emphasis>UNTESTED</emphasis>, or + <emphasis>UNRESOLVED</emphasis> instead, as + appropriate.)</para></listitem> + </varlistentry> + + <varlistentry> + <term>WARNING</term> + <listitem><para>Indicates a possible problem in running the + test. Usually warnings correspond to recoverable errors or + display an important message.</para></listitem> + </varlistentry> + + <varlistentry> + <term>NOTE</term> + <listitem><para>A message about the test case.</para></listitem> + </varlistentry> + </variablelist> + + </sect2> + + <sect2 id=invoking xreflabel="Invoking Runtest"> + <title>Invoking runtest</title> + + <para>This is the full set of command line options that + <filename>runtest</filename> recognizes. Arguments may be + abbreviated to the shortest unique string.</para> + + <variablelist> + <varlistentry> + <term><option>--all</option>, <option>-a</option></term> + <listitem><para>Display all test output. By default, + <command>runtest</command> shows only the output of tests + that produce unexpected results. That is, tests with result + states of <emphasis>FAIL</emphasis>, + <emphasis>XPASS</emphasis>, or + <emphasis>ERROR</emphasis>. Specifying + <command>--all</command> will include output for tests with + result states <emphasis>PASS</emphasis>, + <emphasis>XFAIL</emphasis> and + <emphasis>WARNING</emphasis>.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--build [string]</option></term> + <listitem><para><emphasis>string</emphasis> is a system + triplet as used by <command>configure</command>. This is the + type of system the program to be tested is built on. For a + normal cross-compiler this is the same as the host triplet, + but for a Canadian cross-compiler, they are + distinct.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--host [string]</option></term> + + <listitem><para><emphasis>string</emphasis> is a system + triplet as used by <command>configure</command>. Use this + option to override the default string recorded by your + configuration's choice of host. This choice does not change + how anything is actually configured unless + <option>--build</option> is also specified; it affects + <emphasis>only</emphasis> DejaGnu procedures that compare + the host string with particular values. The procedures + <emphasis>ishost</emphasis>, <emphasis>istarget</emphasis>, + <emphasis>isnative</emphasis>, and + <emphasis>setup</emphasis>xfail} are affected by + <emphasis>--host</emphasis>. In this usage, + <emphasis>host</emphasis> refers to the machine that the + tests are to be run on, which may not be the same as the + <emphasis>build</emphasis> machine. If + <emphasis>--build</emphasis> is also specified, then + <emphasis>--host</emphasis> refers to the machine that the + tests wil, be run on, not the machine DejaGnu is run + on.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--host_board [name]</option></term> + <listitem><para>The host board to use.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--target [string]</option></term> + <listitem><para>Use this option to override the default + target setting. <emphasis>string</emphasis> is a system + triplet as used by <command>configure</command>. This option + changes the configuration <command>runtest</command> uses + for the default tool names, and other setup + information.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--debug</option>, <option>-de</option></term> + <listitem><para>Enables internal + <productname>Expect</productname> debug output. Debug + output is displayed as part of the + <command>runtest</command> output and is additionally logged + to a file called <filename>dbg.log</filename>. The extra + debugging output does <emphasis>not</emphasis> appear on + standard output, unless the verbose level is greater than 2. + For instance, to see debug output immediately, specify + <option>--debug</option> <option>-v</option> + <option>-v</option>. The debugging output shows all + attempts at matching the output of the program under test + with the scripted patterns describing expected + output.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--help</option>, <option>-he</option></term> + <listitem><para>Prints a summary of + <command>runtest</command> options and then exits. This + option overrides all other options in this + regard.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--ignore [test(s)] </option></term> + <listitem><para>The names of specific tests to + ignore.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--objdir [path]</option></term> + <listitem><para>Use <emphasis>path</emphasis> as the top + directory containing any auxiliary pre-compiled test + code. This defaults to <filename>.</filename> and a + <filename>Makefile</filename> can be used to prepare any + auxiliary files that are needed.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--outdir [path]</option></term> + <listitem><para>Write output logs in directory + <filename>path</filename>. The default is + <filename>.</filename> and is the directory where you invoke + <command>runtest</command>. This option affects only the + summary (<filename>.sum</filename>) and the detailed log + (<filename>.log</filename>) files. The debug log + <filename>dbg.log</filename> is always written to the + current working directory.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--reboot [name]</option></term> + <listitem><para>Reboot the target board when + <command>runtest</command> initializes. When running tests + on a separate target board, it is generally safer to reboot + the target to be certain of its state. However, when + developing test scripts, rebooting takes a lot of time and + can reduce the life of prototype boards.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--srcdir [path]</option></term> + + <listitem><para>Use <emphasis>path</emphasis> as the top + directory for test scripts to + run. <command>runtest</command> looks in this directory for + any subdirectory whose name begins with the tool name + (specified with <emphasis>--tool</emphasis>). For instance, + with <option>--tool gdb</option>, <command>runtest</command> + searches subdirectories matching <filename>gdb.*</filename> + for test scripts. If you do not use + <option>--srcdir</option>, <command>runtest</command> looks + for test directories under the current working + directory.</para></listitem> + + </varlistentry> + + <varlistentry> + <term><option>--strace [number]</option></term> + <listitem><para>Turn on internal tracing for + <productname>Expect</productname>, to <emphasis>n</emphasis> + levels deep. By adjusting the level, you can control the + extent to which your output expands multi-level Tcl + statements. This allows you to ignore some levels of + <emphasis>case</emphasis> or <emphasis>if</emphasis> + statements. Each procedure call or control structure counts + as one level. The output is recorded in the debug log file + <filename>dbg.log</filename>.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--connect [program]</option></term> + <listitem><para>Connect to a target system using by + <emphasis>program</emphasis>, if the target system is + distinct from the computer running + <command>runtest</command>. The possible values for + <emphasis>program</emphasis> in the DejaGnu 1.4.4 + distribution are <emphasis>rlogin</emphasis>, + <emphasis>telnet</emphasis>, <emphasis>rsh</emphasis>, + <emphasis>tip</emphasis>, <emphasis>kermit</emphasis> and + <emphasis>mondfe</emphasis>.</para> + </varlistentry> + + <varlistentry> + <term><option>--baud [rate]</option></term> + <listitem><para>Set the baud rate to + <emphasis>rate</emphasis> bits per second. Some serial + interface programs, such as <command>tip</command>, use a + separate initialization file and will ignore this + option.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--target_board [board(s)]</option></term> + <listitem><para>The list of target boards to run tests + on.</para></listitem> + </varlistentry> + + <varlistentry id=tool-opt> + <term><option>--tool [tool(s)]</option></term> + + <listitem><para>Specifies which testsuite to run and what + initialization module to use. <option>--tool</option> is + used only for these two purposes. It is + <emphasis>not</emphasis> used to name the executable program + to test. Executable tool names and pathsare recorded in + <filename>site.exp</filename> and you can override them by + specifying Tcl variable values on the command line.</para> + + <para>For example, including <option>--tool gcc</option> on + the <command>runtest</command> command line will run tests + from all subdirectories whose names match + <filename>gcc.*</filename> and will use one of the + initialization modules named + <filename>config/*-gcc.exp</filename>. To specify the path + to the compiler, use <option>GCC=/path/to/gcc</option> on + the <command>runtest</command> command + line.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--tool_exec [name]</option></term> + <listitem><para>The path to the tool executable to + test.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--tool_opts [options]</option></term> + <listitem><para>A list of additional options to pass to the + tool.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--verbose</option>, <option>-v</option></term> + <listitem><para>Raises the level of output from + <command>runtest</command>. Repeating this option increases + the amount of output displayed. Level one + (<emphasis>-v</emphasis>) is simply test output. Level two + (<emphasis>-v</emphasis>-v}) shows messages on options, + configuration, and process control. Verbose messages appear + in the detailed log file (<filename>*.log</filename>), but + not in the summary log file + (<filename>*.sum</filename>).</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--version</option>, <option>-V</option></term> + <listitem><para>Prints the version numbers of &dj;, + <productname>Expect</productname> and + <productname>Tcl</productname> and then terminates without + running any tests.</para></listitem> + </varlistentry> + + <varlistentry> + <term><option>--D[0-1]</option></term> + <listitem><para>Start the internal Tcl debugger. The Tcl + debugger supports breakpoints, single stepping, and other + common debugging activities. If you specify + <option>-D1</option>, the <command>expect</command> shell + stops at a breakpoint as soon as DejaGnu invokes it. If you + specify <emphasis>-D0</emphasis>, DejaGnu starts as usual, + but you can enter the debugger by sending an interrupt with + <keycombo><keycap>C</keycap><keycap>c</keycap></keycombo>. + </para></listitem> + </varlistentry> + + <varlistentry> + <term><filename>testfile.exp</filename>[=arg(s)]</term> + <listitem><para>Specify the names of testsuites to run. By + default, <command>runtest</command> runs all tests for the + tool, but you can restrict it to particular testsuites by + giving the names of the <filename>.exp</filename> + <productname>Expect</productname> scripts that control + them. <emphasis>testfile.exp</emphasis> may not include + directory names; use base filenames only.</para> + + <para>By listing filenames in <emphasis>arg(s)</emphasis>, + it is possible to specify a subset of tests in a suite to + run. For compiler or assembler tests, which often use a + single <productname>Expect</productname> script covering + many different input files, this option allows you to + further restrict the tests by listing particular input files + to test. Some tools additionally support wildcards. The + wildcards supported depend upon the tool, but typically they + are <emphasis>?</emphasis>, <emphasis>*</emphasis>, and + <emphasis>[chars]</emphasis>.</para></listitem> + </varlistentry> + + <varlistentry> + <term><symbol>tclvar</symbol>=value</term> + <listitem><para>You can define Tcl variables for use by your + test scripts in the same style used by + <command>make</command> for environment variables. For + example, <command>runtest GDB=gdb.old</command> defines a + Tcl variable called <symbol>GDB</symbol>. When test scripts + refer to <symbol>$GDB</symbol>, they will receive the value + <emphasis>gdb.old</emphasis>.</para> + + <para>The default Tcl variables used for most tools are defined in + the main DejaGnu <emphasis>Makefile</emphasis>; their values are + captured in the <filename>site.exp</filename> file.</para></listitem> + </varlistentry> + </variablelist> + </sect2> + + <sect2 id=common xreflabel="Common Options"> + <title>Common Options</title> + + <para>Typically, no command line options are required. The + <option>--tool</option> option is only required when there is + more than one testsuite in the same directory. The default + options are in the local site.exp file, created by + <command>make site.exp</command>.</para> + + <para>For example, if the directory <filename>gdb/testsuite</filename> + contains a collection of DejaGnu tests for GDB, you can run them like + this:</para> + + <screen> + $ cd gdb/testsuite + $ runtest --tool gdb + </screen> + + <para>Test output will follow, ending with:</para> + + <screen> + === gdb Summary === + + # of expected passes 508 + # of expected failures 103 + /usr/latest/bin/gdb version 4.14.4 -nx + </screen> + + <para>You can use the option <emphasis>--srcdir</emphasis> to point to + some other directory containing a collection of tests:</para> + + <screen> + $ runtest --srcdir /devo/gdb/testsuite + </screen> + + <para>By default, <command>runtest</command> prints only the + names of the tests it runs, output from any tests that have unexpected + results, and a summary showing how many tests passed and how many + failed. To display output from all tests (whether or not they behave + as expected), use the <emphasis>--all</emphasis> option. For more + verbose output about processes being run, communication, and so on, use + <emphasis>--verbose</emphasis>. To see even more output, use multiple + <emphasis>--verbose</emphasis> options.</para> + + <para>Test output goes into two files in your current + directory: summary output in <filename>tool.sum</filename>, + and detailed output in <filename> tool.log</filename>. Here, + <emphasis>tool</emphasis> refers to the collection of tests. + After a run with <emphasis>--tool</emphasis> gdb, the output + files will be named <filename>gdb.sum</filename> and + <filename>gdb.log</filename>.</para> </sect2> </sect1> + + <sect1 id=outputfiles xreflabel="Output Files"> + <title>DejaGnu output files</title> + + <para>When <command>runtest</command> is invoked, DejaGnu + generates two output files: a summary log and a detailed log. The + contents of these are determined by the test scripts.</para> + + <para>For troubleshooting, a third kind of output file can be + requested with the <option>--debug</option> option. This file, + called <filename>dbg.log</filename> shows what + <productname>Expect</productname> is doing internally.</para> + + <sect2 id=sum xreflabel="Summary File"> + <title>Summary File</title> + + <para>DejaGnu always produces a summary output file called + <filename>tool.sum</filename>, where <emphasis>tool</emphasis> + is the name of the tool under test. This summary shows the names + of all test scripts run and, for each test script, one line of + output for each test result, trailing summary statistics that + tally the number of passing and failing tests (both expected and + unexpected); and the full pathname and version number of the + tool tested. All possible test outcomes and errors are included + in the summary output file, regardless of whether or not you + specify the <option>--all</option> option.</para> + + <para>If any of your tests use the procedures + <command>unresolved</command>, <command>unsupported</command>, + or <command>runtested</command>, the summary output also + tabulates the corresponding outcomes.</para> + + <para>For example, after <command>runtest --tool + binutils</command>, &dj; will produce a summary log file called + <filename>binutils.sum</filename>. Normally, DejaGnu writes this + file in your current working directory; use the + <option>--outdir</option> option to select a different + directory.</para> + + <example> + <title>Sample summary log</title> + + <screen> + Test Run By rob on Tue Feb 3 23:14:04 2004 + + === gdb tests === + Running ./gdb.t00/echo.exp ... + PASS: Echo test + Running ./gdb.all/help.exp ... + PASS: help add-symbol-file + PASS: help aliases + PASS: help breakpoint "bre" abbreviation + FAIL: help run "r" abbreviation + Running ./gdb.t10/crossload.exp ... + PASS: m68k-elf (elf-big) explicit format; loaded + XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed C types + === gdb Summary === + # of expected passes 5 + # of expected failures 1 + # of unexpected failures 1 + /usr/latest/bin/gdb version 4.6.5 -q + </screen> + </example> + + </sect2> + + <sect2 id=log xreflabel="Log File"> + <title>Log File</title> + + <para>DejaGnu also produces a detailed log file called + <filename>tool.log</filename>, showing any output generated by + tests as well as the summary output. For example, after + <command>runtest --tool binutils</command>, &dj; will produce a + detailed log file named + <filename>binutils.log</filename>. Normally, DejaGnu writes this + file in your current working directory; use the + <option>--outdir</option> option to select a different + directory.</para> + + + <example> + <title>Detailed log for <productname>G++</productname> tests</title> + + <screen> + Test Run By rob on Tue Feb 3 23:16:23 2004 + + === g++ tests === + + --- Running ./g++.other/t01-1.exp --- + PASS: operate delete + + --- Running ./g++.other/t01-2.exp --- + FAIL: i960 bug EOF + p0000646.C: In function `int warn_return_1 ()': + p0000646.C:109: warning: control reaches end of non-void function + p0000646.C: In function `int warn_return_arg (int)': + p0000646.C:117: warning: control reaches end of non-void function + p0000646.C: In function `int warn_return_sum (int, int)': + p0000646.C:125: warning: control reaches end of non-void function + p0000646.C: In function `struct foo warn_return_foo ()': + p0000646.C:132: warning: control reaches end of non-void function + + --- Running ./g++.other/t01-4.exp --- + FAIL: abort + 900403_04.C:8: zero width for bit-field `foo' + --- Running ./g++.other/t01-3.exp --- + FAIL: segment violation + 900519_12.C:9: parse error before `;' + 900519_12.C:12: Segmentation violation + /usr/latest/bin/gcc: Internal compiler error: program cc1plus got fatal signal + + === g++ Summary === + + # of expected passes 1 + # of expected failures 3 + /usr/latest/bin/g++ version cygnus-2.0.1 + </screen> + </example> + + </sect2> + + <sect2 id=debugfile xreflabel="Debug Log File"> + <title>Debug Log File</title> + + <para>The <option>--debug</option> option will generate a debug + log file showing the internal output from + <productname>Expect</productname> running in debugging + mode. This file, called <filename>dbg.log</filename>, is created + in the directory where <command>runtest</command>) is invoked + and shows each pattern <productname>Expect</productname> + considers in analyzing program output.</para> + + <para>This file reflects each <command>send</command> command, + showing the string sent as input to the program under test; and + each <productname>Expect</productname> command, showing each + pattern it compares with the program output. The log messages + begin with a message of the form:</para> + + <screen> + expect: does {<symbol>tool output</symbol>} (spawn_id <symbol>n</symbol>) + match pattern {<emphasis>expected pattern</emphasis>}? + </screen> + + <para>For every unsuccessful match, + <productname>Expect</productname> issues a + <emphasis>no</emphasis> after this message; if other patterns + are specified for the same <productname>Expect</productname> + command, they are reflected also, but without the first part of + the message (<emphasis>expect... match pattern</emphasis>).</para> + + <para>When <productname>Expect</productname> finds a match, the + log for the successful match ends with <emphasis>yes</emphasis>, + followed by a record of the <productname>Expect</productname> + variables set to describe a successful match.</para> + + <example> + <title>Debug log for a <productname>GDB</productname> test:</title> + + <screen> + send: sent {break gdbme.c:34\n} to spawn id 6 + expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file + gdbme.c, line 34.*\(gdb\) $}? no + {.*\(gdb\) $}? no + expect: does {} (spawn_id 0) match pattern {return} ? no + {\(y or n\) }? no + {buffer_full}? no + {virtual}? no + {memory}? no + {exhausted}? no + {Undefined}? no + {command}? no + break gdbme.c:34 + Breakpoint 8 at 0x23d8: file gdbme.c, line 34. + (gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: + file gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern + {Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? yes + expect: set expect_out(0,start) {18} + expect: set expect_out(0,end) {71} + expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file + gdbme.c, line 34.\r\n(gdb) } + epect: set expect_out(spawn_id) {6} + expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8 + at 0x23d8: file gdbme.c, line 34.\r\n(gdb) } + PASS: 70 0 breakpoint line number in file + </screen> + </example> + + <para>This example exhibits three properties of + <productname>Expect</productname> and + <productname>DejaGnu</productname> that might be surprising at + first glance:</para> + + <itemizedlist mark="bullet"> + <listitem><para>Empty output for the first attempted match. The + first set of attempted matches shown ran against the output + <emphasis>{}</emphasis> --- that is, no + output. <productname>Expect</productname> begins + attempting to match the patterns supplied immediately; often, + the first pass is against incomplete output (or completely + before all output, as in this case).</para></listitem> + + <listitem><para>Interspersed tool output. The beginning of + the log entry for the second attempted match may be hard to + spot: this is because the prompt <emphasis>{(gdb) }</emphasis> + appears on the same line, just before the + <emphasis>expect:</emphasis> that marks the beginning of the + log entry.</para></listitem> + + <listitem><para>Fail-safe patterns. Many of the patterns + tested are fail-safe patterns provided by + <productname>GDB</productname> testing utilities, to reduce + possible indeterminacy. It is useful to anticipate potential + variations caused by extreme system conditions + (<productname>GDB</productname> might issue the message + <emphasis>virtual memory exhausted</emphasis> in rare + circumstances), or by changes in the tested program + (<emphasis>Undefined command</emphasis> is the likeliest + outcome if the name of a tested command changes).</para> + + <para>The pattern <emphasis>{return}</emphasis> is a + particularly interesting fail-safe to notice; it checks for an + unexpected <keycap>RET</keycap> prompt. This may happen, + for example, if the tested tool can filter output through a + pager.</para> + + <para>These fail-safe patterns (like the debugging log itself) + are primarily useful while developing test scripts. Use the + <command>error</command> procedure to make the actions for + fail-safe patterns produce messages starting with + <emphasis>ERROR</emphasis> on standard output, and in the + detailed log file.</para></listitem> + </itemizedlist> + </sect2> + </sect1> + </chapter> + +<chapter id=tutorial> + <title>Tutorial</title> <para>This chapter was originally written by Niklaus Giger (ngiger@mus.ch) because he lost a week to figure out how DejaGnu works and how to write a first test.</para> <para>Follow these instructions as closely a possible in order get a good insight into how DejaGnu works, otherwise you might run into a -lot of subtle problems.</para> - -<para>It should be no problem to install DejaGnu using your package -manager or from the source code. Under a Debian/GNU/Linux systems just -type (as root) <programlisting>apt-get dejagnu</programlisting>. The -examples given in this chapter were run on an AMD K6 machine with a -AMD K6 and a Mac Powerbook G3 acting as a remote target.</para> - -<para>The tests for Windows were run under Windows NT using +lot of subtle problems. The examples given in this chapter were run +on an AMD K6 machine with a AMD K6 and a Mac Powerbook G3 acting as a +remote target. The tests for Windows were run under Windows NT using Cygwin. Its target system was a PowerPC embedded system running vxWorks.</para> @@ -760,751 +1500,546 @@ powerpc-linux-gcc -g -O2 -o calc calc.o </sect2> </sect1> </chapter> - <chapter id=runningtests> - <title>Running Tests</title> - - <para>There are two ways to execute a testsuite. The most common - way is to rely upon support in a <filename>Makefile</filename> for - a <emphasis>check</emphasis> target. The other way is to execute - the <command>runtest</command> program directly. To run - <command>runtest</command> directcly from the command line - requires either all the correct options, or the <xref - linkend=local> must be setup correctly. Automake can generate a - <filename>Makefile</filename> that does all of the right things - when the user invokes <command>make check</command> and is the - preferred approach. Both ways of executing a testsuite will be - covered in more detail below.</para> - - <sect1 id=makecheck xreflabel="Make Check"> - <title>make check</title> - <para>To run tests from an existing collection, use - <command>configure</command> to configure a build directory and - then type:</para> - - <screen> - make check - </screen> - - <para>If the <filename>Makefile</filename> has a - <emphasis>check</emphasis> target, it saves some effort. For - instance, it can set up any auxiliary programs or other files - needed by the tests. The most common file the - <emphasis>check</emphasis> target creates is - <filename>site.exp</filename>. The <filename>site.exp</filename> - file contains various variables that DejaGnu uses to determine - the configuration of the program being tested. This is mostly - used to support remote testing.</para> +<chapter id=Writing xreflabel="Writing Testsuites"> + <title>Writing Testsuites</title> - <para>The <emphasis>check</emphasis> target is supported by GNU - <productname>Automake</productname>. To have DejaGnu support added to your - generated <filename>Makefile.in</filename>, just add the keyword - dejagnu to the AUTOMAKE_OPTIONS variable in your - <filename>Makefile.am</filename> file.</para> + <sect1 id=addsuite xreflabel="Adding a new Testsuite"> + <title>Adding A New Testsuite</title> - <para>Once you have run <command>make check</command> to build - any auxiliary files, you can invoke the test driver - <command>runtest</command> directly to repeat the tests. You - will also have to execute <command>runtest</command> directly - for test collections with no <emphasis>check</emphasis> target - in the <filename>Makefile</filename>.</para> + <para>The testsuite for a new tool should always be located in that tools + source directory. DejaGnu require the directory be named + <filename>testsuite</filename>. Under this directory, the test cases go + in a subdirectory whose name begins with the tool name. For example, for + a tool named <emphasis>flubber</emphasis>, each subdirectory containing + testsuites must start with <emphasis>"flubber."</emphasis>.</para> </sect1> - <sect1 id=runtest xreflabel="Runtest"> - <title>Runtest</title> - - <para><command>runtest</command> is the executable test driver - for DejaGnu. You can specify two kinds of things on the - <command>runtest</command> command line: options and Tcl - variable assignments for the test scripts. The options are - listed alphabetically below.</para> - - <para><command>runtest</command> returns an exit code of - <emphasis>1</emphasis> if any test has an unexpected result; otherwise - (if all tests pass or fail as expected) it returns <emphasis>0</emphasis> - as the exit code.</para> - - <sect2 id=outputs xreflabel="Output States"> - <title>Test Result States</title> - - <para><filename>runtest</filename> flags the outcome of each - test as one of the following cases. See <xref linkend=posix> - for a discussion of how POSIX specifies the meanings of these - cases.</para> + <sect1 id=addtool xreflabel="Adding A New Tool"> + <title>Adding A New Tool</title> - <variablelist> - <varlistentry> - <term>PASS</term> - <listitem><para>The most desirable outcome: the test was - expected to succeed and did succeed.</para></listitem> - </varlistentry> + <para>In general, the best way to learn how to write (code or even prose) + is to read something similar. This principle applies to test cases and + to testsuites. Unfortunately, well-established testsuites have a way + of developing their own conventions: as test writers become more + experienced with DejaGnu and with Tcl, they accumulate more utilities, + and take advantage of more and more features of + <productname>Expect</productname> and <productname>Tcl</productname> in + general.</para> - <varlistentry> - <term>XPASS</term> - <listitem><para>A pleasant kind of failure: a test was - expected to fail, but succeeded. This may indicate progress; - inspect the test case to determine whether you should amend - it to no longer expect failure.</para></listitem> - </varlistentry> + <para>Inspecting such established testsuites may make the prospect of + creating an entirely new testsuite appear overwhelming. Nevertheless, + it is quite straightforward to get a new testsuite going.</para> - <varlistentry> - <term>FAIL</term> - <listitem><para>A test was expected to succeed, but failed. - This may indicate a regression; inspect the test case and - the failing software to locate the bug.</para></listitem> - </varlistentry> + <para>There is one testsuite that is guaranteed not to grow more + elaborate over time: both it and the tool it tests were created expressly + to illustrate what it takes to get started with DejaGnu. The + <filename>example/</filename> directory of the DejaGnu distribution + contains both an interactive tool called <command>calc</command>, and a + testsuite for it. Reading this testsuite, and experimenting with it, + is a good way to supplement the information in this section. (Thanks to + Robert Lupton for creating calc and its testsuite---and also the first + version of this section of the manual!)</para> - <varlistentry> - <term>XFAIL</term> - <listitem><para>A test that was expected to fail did fail. - This result indicates no change in a known bug. If a test - fails because the environment running the test lacks some - facility required by the test, the outcome is - <emphasis>UNSUPPORTED</emphasis> instead.</para></listitem> - </varlistentry> + <para>To help orient you further in this task, here is an outline of the + steps to begin building a testsuite for a program example.</para> - <varlistentry> - <term>UNRESOLVED</term> - <listitem><para>Output from an unresolved test requires - manual inspection, as the testsuite could not automatically - determine the outcome. A test can report this outcome, for - instance, when a test is not completed as expected.</para></listitem> - </varlistentry> + <itemizedlist mark=bullet> - <varlistentry> - <term>UNTESTED</term> - <listitem><para>A test case is not yet complete, and in - particular cannot yet produce a <emphasis>PASS</emphasis> or - <emphasis>FAIL</emphasis>. You can also use this outcome for - placeholder tests that note explicitly the absence of a real - test case for a particular property.</para></listitem> - </varlistentry> + <listitem><para>Create or select a directory to contain your new + collection of tests. Change into that directory (shown here as + <filename>testsuite</filename>):</para> - <varlistentry> - <term>UNSUPPORTED</term> - <listitem><para>A test depends on a conditionally available feature - that does not exist (in the configured testing environment). For - example, you can use this outcome to report on a test case that does - not work on a particular target because its operating system support - does not include a required subroutine.</para></listitem> - </varlistentry> - </variablelist> + <para>Create a <filename>configure.in</filename> file in this directory, + to control configuration-dependent choices for your tests. So far as + DejaGnu is concerned, the important thing is to set a value for the + variable <symbol>target_abbrev</symbol>; this value is the link to the + init file you will write soon. (For simplicity, we assume the + environment is Unix, and use <emphasis>unix</emphasis> as the + value.)</para> - <para><command>runtest</command> may also display the following - messages:</para> + <para>What else is needed in <filename>configure.in</filename> depends on + the requirements of your tool, your intended test environments, and which + configure system you use. This example is a minimal configure.in for use + with <productname>GNU Autoconf</productname>. </para></listitem> - <variablelist> - <varlistentry> - <term>ERROR</term> - <listitem><para>Indicates a major problem (detected by the test case - itself) in running the test. This is usually an unrecoverable error, - such as a missing file or loss of communication to the target. (POSIX - testsuites should not emit this message; use - <emphasis>UNSUPPORTED</emphasis>, <emphasis>UNTESTED</emphasis>, or - <emphasis>UNRESOLVED</emphasis> instead, as - appropriate.)</para></listitem> - </varlistentry> + <listitem><para>Create <filename>Makefile.in</filename> (if you are using + Autoconf), or <filename>Makefile.am</filename>(if you are using + Automake), the source file used by configure to build your + <filename>Makefile</filename>. If you are using GNU Automake.just add the + keyword <emphasis>dejagnu</emphasis> to the + <emphasis>AUTOMAKE_OPTIONS</emphasis> variable in your + <filename>Makefile.am</filename> file. This will add all the Makefile + support needed to run DejaGnu, and support the <xref linkend=makecheck> + target.</para> - <varlistentry> - <term>WARNING</term> - <listitem><para>Indicates a possible problem in running the - test. Usually warnings correspond to recoverable errors or - display an important message.</para></listitem> - </varlistentry> + <para>You also need to include two targets important to DejaGnu: + <emphasis>check</emphasis>, to run the tests, and + <emphasis>site.exp</emphasis>, to set up the Tcl copies of + configuration-dependent values. This is called the <xref linkend=local> + The check target must run the <command>runtest</command> program to + execute the tests.</para> - <varlistentry> - <term>NOTE</term> - <listitem><para>A message about the test case.</para></listitem> - </varlistentry> - </variablelist> + <para>The <filename>site.exp</filename> target should usually set up + (among other things) the $tool variable for the name of your program. If + the local site.exp file is setup correctly, it is possible to execute the + tests by merely typing <command>runtest</command> on the command + line.</para> - </sect2> + <example> + <title>Sample Makefile.in Fragment</title> - <sect2 id=invoking xreflabel="Invoking Runtest"> - <title>Invoking runtest</title> + <programlisting> + # Look for a local version of DejaGnu, otherwise use one in the path + RUNTEST = `if test -f $(top_srcdir)/../dejagnu/runtest; then \ + echo $(top_srcdir) ../dejagnu/runtest; \ + else \ + echo runtest; \ + fi` - <para>This is the full set of command line options that - <filename>runtest</filename> recognizes. Arguments may be - abbreviated to the shortest unique string.</para> + # The flags to pass to runtest + RUNTESTFLAGS = - <variablelist> - <varlistentry> - <term><option>--all</option>, <option>-a</option></term> - <listitem><para>Display all test output. By default, - <command>runtest</command> shows only the output of tests - that produce unexpected results. That is, tests with result - states of <emphasis>FAIL</emphasis>, - <emphasis>XPASS</emphasis>, or - <emphasis>ERROR</emphasis>. Specifying - <command>--all</command> will include output for tests with - result states <emphasis>PASS</emphasis>, - <emphasis>XFAIL</emphasis> and - <emphasis>WARNING</emphasis>.</para></listitem> - </varlistentry> + # Execute the tests + check: site.exp all + $(RUNTEST) $(RUNTESTFLAGS) \ + --tool <symbol>${example}</symbol> --srcdir $(srcdir) - <varlistentry> - <term><option>--build [string]</option></term> - <listitem><para><emphasis>string</emphasis> is a system - triplet as used by <command>configure</command>. This is the - type of system the program to be tested is built on. For a - normal cross-compiler this is the same as the host triplet, - but for a Canadian cross-compiler, they are - distinct.</para></listitem> - </varlistentry> + # Make the local config file + site.exp: ./config.status Makefile + @echo "Making a new config file..." + -@rm -f ./tmp? + @touch site.exp - <varlistentry> - <term><option>--host [string]</option></term> + -@mv site.exp site.bak + @echo "## these variables are automatically\ + generated by make ##" > ./tmp0 + @echo "# Do not edit here. If you wish to\ + override these values" >> ./tmp0 + @echo "# add them to the last section" >> ./tmp0 + @echo "set host_os ${host_os}" >> ./tmp0 + @echo "set host_alias ${host_alias}" >> ./tmp0 + @echo "set host_cpu ${host_cpu}" >> ./tmp0 + @echo "set host_vendor ${host_vendor}" >> ./tmp0 + @echo "set target_os ${target_os}" >> ./tmp0 + @echo "set target_alias ${target_alias}" >> ./tmp0 + @echo "set target_cpu ${target_cpu}" >> ./tmp0 + @echo "set target_vendor ${target_vendor}" >> ./tmp0 + @echo "set host_triplet ${host_canonical}" >> ./tmp0 + @echo "set target_triplet ${target_canonical}">>./tmp0 + @echo "set tool binutils" >> ./tmp0 + @echo "set srcdir ${srcdir}" >> ./tmp0 + @echo "set objdir `pwd`" >> ./tmp0 + @echo "set <symbol>${examplename}</symbol> <symbol>${example}</symbol>" >> ./tmp0 + @echo "## All variables above are generated by\ + configure. Do Not Edit ##" >> ./tmp0 + @cat ./tmp0 > site.exp + @sed < site.bak \ + -e '1,/^## All variables above are.*##/ d' \ + >> site.exp + -@rm -f ./tmp? - <listitem><para><emphasis>string</emphasis> is a system - triplet as used by <command>configure</command>. Use this - option to override the default string recorded by your - configuration's choice of host. This choice does not change - how anything is actually configured unless - <option>--build</option> is also specified; it affects - <emphasis>only</emphasis> DejaGnu procedures that compare - the host string with particular values. The procedures - <emphasis>ishost</emphasis>, <emphasis>istarget</emphasis>, - <emphasis>isnative</emphasis>, and - <emphasis>setup</emphasis>xfail} are affected by - <emphasis>--host</emphasis>. In this usage, - <emphasis>host</emphasis> refers to the machine that the - tests are to be run on, which may not be the same as the - <emphasis>build</emphasis> machine. If - <emphasis>--build</emphasis> is also specified, then - <emphasis>--host</emphasis> refers to the machine that the - tests wil, be run on, not the machine DejaGnu is run - on.</para></listitem> - </varlistentry> + </programlisting> + </example> + </listitem> - <varlistentry> - <term><option>--host_board [name]</option></term> - <listitem><para>The host board to use.</para></listitem> - </varlistentry> + <listitem><para>Create a directory (in <filename>testsuite</filename>) + called <filename>config</filename>. Make a <emphasis>Tool Init + File</emphasis> in this directory. Its name must start with the + <symbol>target_abbrev</symbol> value, or be named + <filename>default.exp</filename> so call it + <filename>config/unix.exp</filename> for our Unix based example. This + is the file that contains the target-dependent procedures. + Fortunately, on Unix, most of them do not have to do very much in + order for <command>runtest</command> to run.</para> - <varlistentry> - <term><option>--target [string]</option></term> - <listitem><para>Use this option to override the default - target setting. <emphasis>string</emphasis> is a system - triplet as used by <command>configure</command>. This option - changes the configuration <command>runtest</command> uses - for the default tool names, and other setup - information.</para></listitem> - </varlistentry> + <para>If the program being tested is not interactive, you can get + away with this minimal <filename>unix.exp</filename> to begin + with:</para> - <varlistentry> - <term><option>--debug</option>, <option>-de</option></term> - <listitem><para>Enables internal - <productname>Expect</productname> debug output. Debug - output is displayed as part of the - <command>runtest</command> output and is additionally logged - to a file called <filename>dbg.log</filename>. The extra - debugging output does <emphasis>not</emphasis> appear on - standard output, unless the verbose level is greater than 2. - For instance, to see debug output immediately, specify - <option>--debug</option> <option>-v</option> - <option>-v</option>. The debugging output shows all - attempts at matching the output of the program under test - with the scripted patterns describing expected - output.</para></listitem> - </varlistentry> + <example> + <title>Simple Batch Program Tool Init File</title> - <varlistentry> - <term><option>--help</option>, <option>-he</option></term> - <listitem><para>Prints a summary of - <command>runtest</command> options and then exits. This - option overrides all other options in this - regard.</para></listitem> - </varlistentry> + <programlisting> - <varlistentry> - <term><option>--ignore [test(s)] </option></term> - <listitem><para>The names of specific tests to - ignore.</para></listitem> - </varlistentry> + proc foo_exit {} {} + proc foo_version {} {} - <varlistentry> - <term><option>--objdir [path]</option></term> - <listitem><para>Use <emphasis>path</emphasis> as the top - directory containing any auxiliary pre-compiled test - code. This defaults to <filename>.</filename> and a - <filename>Makefile</filename> can be used to prepare any - auxiliary files that are needed.</para></listitem> - </varlistentry> + </programlisting> + </example> - <varlistentry> - <term><option>--outdir [path]</option></term> - <listitem><para>Write output logs in directory - <filename>path</filename>. The default is - <filename>.</filename> and is the directory where you invoke - <command>runtest</command>. This option affects only the - summary (<filename>.sum</filename>) and the detailed log - (<filename>.log</filename>) files. The debug log - <filename>dbg.log</filename> is always written to the - current working directory.</para></listitem> - </varlistentry> + <para>If the program being tested is interactive, however, you might + as well define a <emphasis>start</emphasis> routine and invoke it by + using an init file like this:</para> - <varlistentry> - <term><option>--reboot [name]</option></term> - <listitem><para>Reboot the target board when - <command>runtest</command> initializes. When running tests - on a separate target board, it is generally safer to reboot - the target to be certain of its state. However, when - developing test scripts, rebooting takes a lot of time and - can reduce the life of prototype boards.</para></listitem> - </varlistentry> + <example> + <title>Simple Interactive Program Tool Init File</title> - <varlistentry> - <term><option>--srcdir [path]</option></term> + <programlisting> + + proc foo_exit {} {} + proc foo_version {} {} - <listitem><para>Use <emphasis>path</emphasis> as the top - directory for test scripts to - run. <command>runtest</command> looks in this directory for - any subdirectory whose name begins with the tool name - (specified with <emphasis>--tool</emphasis>). For instance, - with <option>--tool gdb</option>, <command>runtest</command> - searches subdirectories matching <filename>gdb.*</filename> - for test scripts. If you do not use - <option>--srcdir</option>, <command>runtest</command> looks - for test directories under the current working - directory.</para></listitem> + proc foo_start {} { + global ${examplename} + spawn ${examplename} + expect { + -re "" {} + } + } - </varlistentry> + # Start the program running we want to test + foo_start - <varlistentry> - <term><option>--strace [number]</option></term> - <listitem><para>Turn on internal tracing for - <productname>Expect</productname>, to <emphasis>n</emphasis> - levels deep. By adjusting the level, you can control the - extent to which your output expands multi-level Tcl - statements. This allows you to ignore some levels of - <emphasis>case</emphasis> or <emphasis>if</emphasis> - statements. Each procedure call or control structure counts - as one level. The output is recorded in the debug log file - <filename>dbg.log</filename>.</para></listitem> - </varlistentry> + </programlisting> + </example> + </listitem> - <varlistentry> - <term><option>--connect [program]</option></term> - <listitem><para>Connect to a target system using by - <emphasis>program</emphasis>, if the target system is - distinct from the computer running - <command>runtest</command>. The possible values for - <emphasis>program</emphasis> in the DejaGnu 1.4.4 - distribution are <emphasis>rlogin</emphasis>, - <emphasis>telnet</emphasis>, <emphasis>rsh</emphasis>, - <emphasis>tip</emphasis>, <emphasis>kermit</emphasis> and - <emphasis>mondfe</emphasis>.</para> - </varlistentry> + <listitem><para>Create a directory whose name begins with your tool's + name, to contain tests. For example, if your tool's name is + <emphasis>gcc</emphasis>, then the directories all need to start with + <emphasis>"gcc."</emphasis>.</para></listitem> - <varlistentry> - <term><option>--baud [rate]</option></term> - <listitem><para>Set the baud rate to - <emphasis>rate</emphasis> bits per second. Some serial - interface programs, such as <command>tip</command>, use a - separate initialization file and will ignore this - option.</para></listitem> - </varlistentry> + <listitem><para>Create a sample test file. Its name must end with + <filename>.exp</filename>. You can use + <filename>first-try.exp</filename>. To begin with, just write there a + line of Tcl code to issue a message.</para> - <varlistentry> - <term><option>--target_board [board(s)]</option></term> - <listitem><para>The list of target boards to run tests - on.</para></listitem> - </varlistentry> - - <varlistentry id=tool-opt> - <term><option>--tool [tool(s)]</option></term> + <example> + <title>Testing A New Tool Config</title> - <listitem><para>Specifies which testsuite to run and what - initialization module to use. <option>--tool</option> is - used only for these two purposes. It is - <emphasis>not</emphasis> used to name the executable program - to test. Executable tool names and pathsare recorded in - <filename>site.exp</filename> and you can override them by - specifying Tcl variable values on the command line.</para> + <programlisting> - <para>For example, including <option>--tool gcc</option> on - the <command>runtest</command> command line will run tests - from all subdirectories whose names match - <filename>gcc.*</filename> and will use one of the - initialization modules named - <filename>config/*-gcc.exp</filename>. To specify the path - to the compiler, use <option>GCC=/path/to/gcc</option> on - the <command>runtest</command> command - line.</para></listitem> - </varlistentry> + send_user "Testing: one, two...\n" - <varlistentry> - <term><option>--tool_exec [name]</option></term> - <listitem><para>The path to the tool executable to - test.</para></listitem> - </varlistentry> + </programlisting> + </example> + </listitem> - <varlistentry> - <term><option>--tool_opts [options]</option></term> - <listitem><para>A list of additional options to pass to the - tool.</para></listitem> - </varlistentry> + <listitem><para>Back in the <filename>testsuite</filename> (top + level) directory, run <command>configure</command>. Typically you do + this while in the build directory. You may have to specify more of a + path, if a suitable configure is not available in your execution + path.</para></listitem> - <varlistentry> - <term><option>--verbose</option>, <option>-v</option></term> - <listitem><para>Raises the level of output from - <command>runtest</command>. Repeating this option increases - the amount of output displayed. Level one - (<emphasis>-v</emphasis>) is simply test output. Level two - (<emphasis>-v</emphasis>-v}) shows messages on options, - configuration, and process control. Verbose messages appear - in the detailed log file (<filename>*.log</filename>), but - not in the summary log file - (<filename>*.sum</filename>).</para></listitem> - </varlistentry> + <listitem><para>e now ready to triumphantly type <command>make + check</command> or <command>runtest</command>. You should see + something like this:</para> - <varlistentry> - <term><option>--version</option>, <option>-V</option></term> - <listitem><para>Prints the version numbers of &dj;, - <productname>Expect</productname> and - <productname>Tcl</productname> and then terminates without - running any tests.</para></listitem> - </varlistentry> + <example> + <title>Example Test Case Run</title> - <varlistentry> - <term><option>--D[0-1]</option></term> - <listitem><para>Start the internal Tcl debugger. The Tcl - debugger supports breakpoints, single stepping, and other - common debugging activities. If you specify - <option>-D1</option>, the <command>expect</command> shell - stops at a breakpoint as soon as DejaGnu invokes it. If you - specify <emphasis>-D0</emphasis>, DejaGnu starts as usual, - but you can enter the debugger by sending an interrupt with - <keycombo><keycap>C</keycap><keycap>c</keycap></keycombo>. - </para></listitem> - </varlistentry> + <screen> + Test Run By rhl on Fri Jan 29 16:25:44 EST 1993 - <varlistentry> - <term><filename>testfile.exp</filename>[=arg(s)]</term> - <listitem><para>Specify the names of testsuites to run. By - default, <command>runtest</command> runs all tests for the - tool, but you can restrict it to particular testsuites by - giving the names of the <filename>.exp</filename> - <productname>Expect</productname> scripts that control - them. <emphasis>testfile.exp</emphasis> may not include - directory names; use base filenames only.</para> + === example tests === - <para>By listing filenames in <emphasis>arg(s)</emphasis>, - it is possible to specify a subset of tests in a suite to - run. For compiler or assembler tests, which often use a - single <productname>Expect</productname> script covering - many different input files, this option allows you to - further restrict the tests by listing particular input files - to test. Some tools additionally support wildcards. The - wildcards supported depend upon the tool, but typically they - are <emphasis>?</emphasis>, <emphasis>*</emphasis>, and - <emphasis>[chars]</emphasis>.</para></listitem> - </varlistentry> + Running ./example.0/first-try.exp ... + Testing: one, two... - <varlistentry> - <term><symbol>tclvar</symbol>=value</term> - <listitem><para>You can define Tcl variables for use by your - test scripts in the same style used by - <command>make</command> for environment variables. For - example, <command>runtest GDB=gdb.old</command> defines a - Tcl variable called <symbol>GDB</symbol>. When test scripts - refer to <symbol>$GDB</symbol>, they will receive the value - <emphasis>gdb.old</emphasis>.</para> + === example Summary === - <para>The default Tcl variables used for most tools are defined in - the main DejaGnu <emphasis>Makefile</emphasis>; their values are - captured in the <filename>site.exp</filename> file.</para></listitem> - </varlistentry> - </variablelist> - </sect2> + </screen> + </example> - <sect2 id=common xreflabel="Common Options"> - <title>Common Options</title> - - <para>Typically, no command line options are required. The - <option>--tool</option> option is only required when there is - more than one testsuite in the same directory. The default - options are in the local site.exp file, created by - <command>make site.exp</command>.</para> + <para>There is no output in the summary, because so far the example + does not call any of the procedures that establish a test + outcome.</para></listitem> - <para>For example, if the directory <filename>gdb/testsuite</filename> - contains a collection of DejaGnu tests for GDB, you can run them like - this:</para> + <listitem><para>Write some real tests. For an interactive tool, you + should probably write a real exit routine in fairly short order. In + any case, you should also write a real version routine + soon. </para></listitem> + </itemizedlist> + </sect1> - <screen> - $ cd gdb/testsuite - $ runtest --tool gdb - </screen> + <sect1 id=writingcase xreflabel="Writing A Test Case"> + <title>Writing A Test Case</title> - <para>Test output will follow, ending with:</para> + <para>The easiest way to prepare a new test case is to base it + on an existing one for a similar situation. There are two major + categories of tests: batch or interactive. Batch oriented tests + are usually easier to write.</para> - <screen> - === gdb Summary === + <para>The GCC tests are a good example of batch oriented tests. + All GCC tests consist primarily of a call to a single common + procedure, Since all the tests either have no output, or only + have a few warning messages when successfully compiled. Any + non-warning output is a test failure. All the C code needed is + kept in the test directory. The test driver, written in Tcl, + need only get a listing of all the C files in the directory, and + compile them all using a generic procedure. This procedure and a + few others supporting for these tests are kept in the library + module <filename>lib/c-torture.exp</filename> in the GCC test + suite. Most tests of this kind use very few + <productname>expect</productname> features, and are coded almost + purely in Tcl.</para> - # of expected passes 508 - # of expected failures 103 - /usr/latest/bin/gdb version 4.14.4 -nx - </screen> + <para>Writing the complete suite of C tests, then, consisted of + these steps:</para> - <para>You can use the option <emphasis>--srcdir</emphasis> to point to - some other directory containing a collection of tests:</para> + <itemizedlist mark=bullet> + <listitem><para>Copying all the C code into the test directory. + These tests were based on the C-torture test created by Torbjorn + Granlund (on behalf of the Free Software Foundation) for GCC + development.</para></listitem> - <screen> - $ runtest --srcdir /devo/gdb/testsuite - </screen> + <listitem><para>Writing (and debugging) the generic Tcl procedures for + compilation.</para></listitem> - <para>By default, <command>runtest</command> prints only the - names of the tests it runs, output from any tests that have unexpected - results, and a summary showing how many tests passed and how many - failed. To display output from all tests (whether or not they behave - as expected), use the <emphasis>--all</emphasis> option. For more - verbose output about processes being run, communication, and so on, use - <emphasis>--verbose</emphasis>. To see even more output, use multiple - <emphasis>--verbose</emphasis> options.</para> + <listitem><para>Writing the simple test driver: its main task is to + search the directory (using the Tcl procedure + <emphasis>glob</emphasis> for filename expansion with wildcards) + and call a Tcl procedure with each filename. It also checks for + a few errors from the testing procedure.</para></listitem> + </itemizedlist> - <para>Test output goes into two files in your current - directory: summary output in <filename>tool.sum</filename>, - and detailed output in <filename> tool.log</filename>. Here, - <emphasis>tool</emphasis> refers to the collection of tests. - After a run with <emphasis>--tool</emphasis> gdb, the output - files will be named <filename>gdb.sum</filename> and - <filename>gdb.log</filename>.</para> </sect2> </sect1> + <para>Testing interactive programs is intrinsically more + complex. Tests for most interactive programs require some trial + and error before they are complete.</para> - <sect1 id=outputfiles xreflabel="Output Files"> - <title>DejaGnu output files</title> + <para>However, some interactive programs can be tested in a + simple fashion reminiscent of batch tests. For example, prior + to the creation of DejaGnu, the GDB distribution already + included a wide-ranging testing procedure. This procedure was + very robust, and had already undergone much more debugging and + error checking than many recent DejaGnu test cases. + Accordingly, the best approach was simply to encapsulate the + existing GDB tests, for reporting purposes. Thereafter, new GDB + tests built up a family of Tcl procedures specialized for GDB + testing.</para> - <para>When <command>runtest</command> is invoked, DejaGnu - generates two output files: a summary log and a detailed log. The - contents of these are determined by the test scripts.</para> + </sect1> - <para>For troubleshooting, a third kind of output file can be - requested with the <option>--debug</option> option. This file, - called <filename>dbg.log</filename> shows what - <productname>Expect</productname> is doing internally.</para> + <sect1 id=debugging xreflabel="Debugging A Test Case"> + <title>Debugging A Test Case</title> - <sect2 id=sum xreflabel="Summary File"> - <title>Summary File</title> + <para>These are the kinds of debugging information available + from DejaGnu:</para> - <para>DejaGnu always produces a summary output file called - <filename>tool.sum</filename>, where <emphasis>tool</emphasis> - is the name of the tool under test. This summary shows the names - of all test scripts run and, for each test script, one line of - output for each test result, trailing summary statistics that - tally the number of passing and failing tests (both expected and - unexpected); and the full pathname and version number of the - tool tested. All possible test outcomes and errors are included - in the summary output file, regardless of whether or not you - specify the <option>--all</option> option.</para> + <itemizedlist mark=bullet> - <para>If any of your tests use the procedures - <command>unresolved</command>, <command>unsupported</command>, - or <command>runtested</command>, the summary output also - tabulates the corresponding outcomes.</para> + <listitem><para>Output controlled by test scripts themselves, + explicitly allowed for by the test author. This kind of + debugging output appears in the detailed output recorded in the + DejaGnu log file. To do the same for new tests, use the + <command>verbose</command> procedure (which in turn uses the + variable also called <emphasis>verbose</emphasis>) to control + how much output to generate. This will make it easier for other + people running the test to debug it if necessary. Whenever + possible, if <emphasis>$verbose</emphasis> is + <emphasis>0</emphasis>, there should be no output other than the + output from <emphasis>pass</emphasis>, + <emphasis>fail</emphasis>, <emphasis>error</emphasis>, and + <emphasis>warning</emphasis>. Then, to whatever extent is + appropriate for the particular test, allow successively higher + values of <emphasis>$verbose</emphasis> to generate more + information. Be kind to other programmers who use your tests: + provide for a lot of debugging information.</para></listitem> - <para>For example, after <command>runtest --tool - binutils</command>, &dj; will produce a summary log file called - <filename>binutils.sum</filename>. Normally, DejaGnu writes this - file in your current working directory; use the - <option>--outdir</option> option to select a different + <listitem><para>Output from the internal debugging functions of + Tcl and <productname>Expect</productname>. There is a command + line options for each; both forms of debugging output are + recorded in the file <filename>dbg.log</filename> in the current directory.</para> - <example> - <title>Sample summary log</title> + <para>Use <option>--debug</option> for information from the + expect level; it generates displays of the expect attempts to + match the tool output with the patterns specified. This output + can be very helpful while developing test scripts, since it + shows precisely the characters received. Iterating between the + latest attempt at a new test script and the corresponding + <filename>dbg.log</filename> can allow you to create the final + patterns by ``cut and paste''. This is sometimes the best way + to write a test case.</para></listitem> - <screen> - Test Run By rob on Tue Feb 3 23:14:04 2004 + <listitem><para>Use <option>--strace</option> to see more + detail at the Tcl level; this shows how Tcl procedure + definitions expand, as they execute. The associated number + controls the depth of definitions expanded.</para></listitem> - === gdb tests === - Running ./gdb.t00/echo.exp ... - PASS: Echo test - Running ./gdb.all/help.exp ... - PASS: help add-symbol-file - PASS: help aliases - PASS: help breakpoint "bre" abbreviation - FAIL: help run "r" abbreviation - Running ./gdb.t10/crossload.exp ... - PASS: m68k-elf (elf-big) explicit format; loaded - XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed C types - === gdb Summary === - # of expected passes 5 - # of expected failures 1 - # of unexpected failures 1 - /usr/latest/bin/gdb version 4.6.5 -q - </screen> - </example> + <listitem><para>Finally, if the value of + <emphasis>verbose</emphasis> is 3 or greater,DejaGnu turns on + the expect command <command>log_user</command>. This command + prints all expect actions to the expect standard output, to the + detailed log file, and (if <option>--debug</option> is on) to + <filename>dbg.log</filename>.</para></listitem> + </itemizedlist> - </sect2> + </sect1> - <sect2 id=log xreflabel="Log File"> - <title>Log File</title> + <sect1 id=adding xreflabel="Adding A Test Case To A Testsuite"> + <title>Adding A Test Case To A Testsuite.</title> - <para>DejaGnu also produces a detailed log file called - <filename>tool.log</filename>, showing any output generated by - tests as well as the summary output. For example, after - <command>runtest --tool binutils</command>, &dj; will produce a - detailed log file named - <filename>binutils.log</filename>. Normally, DejaGnu writes this - file in your current working directory; use the - <option>--outdir</option> option to select a different - directory.</para> + <para>There are two slightly different ways to add a test + case. One is to add the test case to an existing directory. The + other is to create a new directory to hold your test. The + existing test directories represent several styles of testing, + all of which are slightly different; examine the directories for + the tool of interest to see which (if any) is most suitable.</para> + <para>Adding a GCC test can be very simple: just add the C code + to any directory beginning with <filename>gcc</filename>. and it + runs on the next <programlisting>runtest --tool + gcc</programlisting>.</para> - <example> - <title>Detailed log for <productname>G++</productname> tests</title> + <para>To add a test to GDB, first add any source code you will + need to the test directory. Then you can either create a new + expect file, or add your test to an existing one (any + file with a <emphasis>.exp</emphasis> suffix). Creating a new + .exp file is probably a better idea if the test is significantly + different from existing tests. Adding it as a separate file also + makes upgrading easier. If the C code has to be already compiled + before the test will run, then you'll have to add it to the + <filename>Makefile.in</filename> file for that test directory, + then run <command>configure</command> and + <command>make</command>.</para> - <screen> - Test Run By rob on Tue Feb 3 23:16:23 2004 + <para>Adding a test by creating a new directory is very + similar:</para> - === g++ tests === + <itemizedlist mark=bullet> - --- Running ./g++.other/t01-1.exp --- - PASS: operate delete + <listitem><para>Create the new directory. All subdirectory names + begin with the name of the tool to test; e.g. G++ tests might be + in a directory called <filename>g++.other</filename>. There can + be multiple test directories that start with the same tool name + (such as <emphasis>g++</emphasis>).</para></listitem> - --- Running ./g++.other/t01-2.exp --- - FAIL: i960 bug EOF - p0000646.C: In function `int warn_return_1 ()': - p0000646.C:109: warning: control reaches end of non-void function - p0000646.C: In function `int warn_return_arg (int)': - p0000646.C:117: warning: control reaches end of non-void function - p0000646.C: In function `int warn_return_sum (int, int)': - p0000646.C:125: warning: control reaches end of non-void function - p0000646.C: In function `struct foo warn_return_foo ()': - p0000646.C:132: warning: control reaches end of non-void function + <listitem><para>Add the new directory name to the + <symbol>configdirs</symbol> definition in the + <filename>configure.in</filename> file for the testsuite + directory. This way when <command>make</command> and + <command>configure</command> next run, they include the new + directory.</para></listitem> - --- Running ./g++.other/t01-4.exp --- - FAIL: abort - 900403_04.C:8: zero width for bit-field `foo' - --- Running ./g++.other/t01-3.exp --- - FAIL: segment violation - 900519_12.C:9: parse error before `;' - 900519_12.C:12: Segmentation violation - /usr/latest/bin/gcc: Internal compiler error: program cc1plus got fatal signal + <listitem><para>Add the new test case to the directory, as + above. </para></listitem> - === g++ Summary === + <listitem><para>To add support in the new directory for + configure and make, you must also create a + <filename>Makefile.in</filename> and a + <filename>configure.in</filename>.</para></listitem> + </itemizedlist> - # of expected passes 1 - # of expected failures 3 - /usr/latest/bin/g++ version cygnus-2.0.1 - </screen> - </example> + </sect1> - </sect2> + <sect1 id=hints xreflabel="Hints On Writing A Test Case"> + <title>Hints On Writing A Test Case</title> - <sect2 id=debugfile xreflabel="Debug Log File"> - <title>Debug Log File</title> + <para>It is safest to write patterns that match all the output + generated by the tested program; this is called closure. + If a pattern does not match the entire output, any output that + remains will be examined by the next <command>expect</command> + command. In this situation, the precise boundary that determines + which <command>expect</command> command sees what is very + sensitive to timing between the Expect task and the task running + the tested tool. As a result, the test may sometimes appear to + work, but is likely to have unpredictable results. (This problem + is particularly likely for interactive tools, but can also + affect batch tools---especially for tests that take a long time + to finish.) The best way to ensure closure is to use the + <option>-re</option> option for the <command>expect</command> + command to write the pattern as a full regular expressions; then + you can match the end of output using a <emphasis>$</emphasis>. + It is also a good idea to write patterns that match all + available output by using <emphasis>.*\</emphasis> after the + text of interest; this will also match any intervening blank + lines. Sometimes an alternative is to match end of line using + <emphasis>\r</emphasis> or <emphasis>\n</emphasis>, but this is + usually too dependent on terminal settings.</para> - <para>The <option>--debug</option> option will generate a debug - log file showing the internal output from - <productname>Expect</productname> running in debugging - mode. This file, called <filename>dbg.log</filename>, is created - in the directory where <command>runtest</command>) is invoked - and shows each pattern <productname>Expect</productname> - considers in analyzing program output.</para> + <para>Always escape punctuation, such as <emphasis>(</emphasis> + or <emphasis>"</emphasis>, in your patterns; for example, write + <emphasis>\(</emphasis>. If you forget to escape punctuation, + you will usually see an error message like <programlisting>extra + characters after close-quote.</programlisting></para> - <para>This file reflects each <command>send</command> command, - showing the string sent as input to the program under test; and - each <productname>Expect</productname> command, showing each - pattern it compares with the program output. The log messages - begin with a message of the form:</para> + <para>If you have trouble understanding why a pattern does not + match the program output, try using the <option>--debug</option> + option to <command>runtest</command>, and examine the debug log + carefully.</para> - <screen> - expect: does {<symbol>tool output</symbol>} (spawn_id <symbol>n</symbol>) - match pattern {<emphasis>expected pattern</emphasis>}? - </screen> + <para>Be careful not to neglect output generated by setup rather + than by the interesting parts of a test case. For example, + while testing GDB, I issue a send <emphasis>set height + 0\n</emphasis> command. The purpose is simply to make sure GDB + never calls a paging program. The <emphasis>set + height</emphasis> command in GDB does not generate any + output; but running any command makes GDB issue a new + <emphasis>(gdb) </emphasis> prompt. If there were no + <command>expect</command> command to match this prompt, the + output <emphasis>(gdb) </emphasis> begins the text seen by the + next <command>expect</command> command---which might make that + pattern fail to match.</para> - <para>For every unsuccessful match, - <productname>Expect</productname> issues a - <emphasis>no</emphasis> after this message; if other patterns - are specified for the same <productname>Expect</productname> - command, they are reflected also, but without the first part of - the message (<emphasis>expect... match pattern</emphasis>).</para> + <para>To preserve basic sanity, I also recommended that no test + ever pass if there was any kind of problem in the test case. To + take an extreme case, tests that pass even when the tool will + not spawn are misleading. Ideally, a test in this sort of + situation should not fail either. Instead, print an error + message by calling one of the DejaGnu procedures + <command>error</command> or <command>warning</command>.</para> - <para>When <productname>Expect</productname> finds a match, the - log for the successful match ends with <emphasis>yes</emphasis>, - followed by a record of the <productname>Expect</productname> - variables set to describe a successful match.</para> + </sect1> - <example> - <title>Debug log for a <productname>GDB</productname> test:</title> + <sect1 id=tvariables xreflabel="Test Case Variables"> + <title>Special variables used by test cases.</title> - <screen> - send: sent {break gdbme.c:34\n} to spawn id 6 - expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file - gdbme.c, line 34.*\(gdb\) $}? no - {.*\(gdb\) $}? no - expect: does {} (spawn_id 0) match pattern {return} ? no - {\(y or n\) }? no - {buffer_full}? no - {virtual}? no - {memory}? no - {exhausted}? no - {Undefined}? no - {command}? no - break gdbme.c:34 - Breakpoint 8 at 0x23d8: file gdbme.c, line 34. - (gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: - file gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern - {Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? yes - expect: set expect_out(0,start) {18} - expect: set expect_out(0,end) {71} - expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file - gdbme.c, line 34.\r\n(gdb) } - epect: set expect_out(spawn_id) {6} - expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8 - at 0x23d8: file gdbme.c, line 34.\r\n(gdb) } - PASS: 70 0 breakpoint line number in file - </screen> - </example> + <para>There are special variables used by test cases. These contain + other information from DejaGnu. Your test cases can use these variables, + with conventional meanings (as well as the variables saved in + <filename>site.exp</filename>. You can use the value of these variables, + but they should never be changed.</para> - <para>This example exhibits three properties of - <productname>Expect</productname> and - <productname>DejaGnu</productname> that might be surprising at - first glance:</para> + <variablelist> + <varlistentry> + <term>$prms_id</term> + <listitem><para>The tracking system (e.g. GNATS) number identifying + a corresponding bugreport. (<emphasis>0</emphasis>} if you do not + specify it in the test script.)</para></listitem> + </varlistentry> - <itemizedlist mark="bullet"> - <listitem><para>Empty output for the first attempted match. The - first set of attempted matches shown ran against the output - <emphasis>{}</emphasis> --- that is, no - output. <productname>Expect</productname> begins - attempting to match the patterns supplied immediately; often, - the first pass is against incomplete output (or completely - before all output, as in this case).</para></listitem> + <varlistentry> + <term>$item bug_id</term> + <listitem><para>An optional bug id; may reflect a bug + identification from another organization. (<emphasis>0</emphasis> + if you do not specify it.)</para></listitem> + </varlistentry> - <listitem><para>Interspersed tool output. The beginning of - the log entry for the second attempted match may be hard to - spot: this is because the prompt <emphasis>{(gdb) }</emphasis> - appears on the same line, just before the - <emphasis>expect:</emphasis> that marks the beginning of the - log entry.</para></listitem> + <varlistentry> + <term>$subdir</term> + <listitem><para>The subdirectory for the current test + case.</para></listitem> + </varlistentry> - <listitem><para>Fail-safe patterns. Many of the patterns - tested are fail-safe patterns provided by - <productname>GDB</productname> testing utilities, to reduce - possible indeterminacy. It is useful to anticipate potential - variations caused by extreme system conditions - (<productname>GDB</productname> might issue the message - <emphasis>virtual memory exhausted</emphasis> in rare - circumstances), or by changes in the tested program - (<emphasis>Undefined command</emphasis> is the likeliest - outcome if the name of a tested command changes).</para> + <varlistentry> + <term>$expect_out(buffer)</term> + <listitem><para>The output from the last command. This is an + internal variable set by Expect. More information can be found in + the Expect manual.</para></listitem> + </varlistentry> - <para>The pattern <emphasis>{return}</emphasis> is a - particularly interesting fail-safe to notice; it checks for an - unexpected <keycap>RET</keycap> prompt. This may happen, - for example, if the tested tool can filter output through a - pager.</para> + <varlistentry> + <term>$exec_output</term> + <listitem><para>This is the output from a + <function>${tool}_load</function> command. This only applies to + tools like GCC and GAS which produce an object file that must in + turn be executed to complete a test.</para></listitem> + </varlistentry> - <para>These fail-safe patterns (like the debugging log itself) - are primarily useful while developing test scripts. Use the - <command>error</command> procedure to make the actions for - fail-safe patterns produce messages starting with - <emphasis>ERROR</emphasis> on standard output, and in the - detailed log file.</para></listitem> - </itemizedlist> - </sect2> - </sect1> - </chapter> + <varlistentry> + <term>$comp_output</term> + <listitem><para>This is the output from a + <function>${tool}_start</function> command. This is conventionally + used for batch oriented programs, like GCC and GAS, that may + produce interesting output (warnings, errors) without further + interaction.</para></listitem> + </varlistentry> + </variablelist> + </sect1> +</chapter> <chapter id=Customizing xreflabel="Customizing DejaGnu"> <title>Customizing DejaGnu</title> @@ -2113,260 +2648,194 @@ powerpc-linux-gcc -g -O2 -o calc calc.o </chapter> - <chapter id=Extending xreflabel="Extending DejaGnu"> - <title>Extending DejaGnu</title> - - <sect1 id=addsuite xreflabel="Adding a new Testsuite"> - <title>Adding A New Testsuite</title> - - <para>The testsuite for a new tool should always be located in that tools - source directory. DejaGnu require the directory be named - <filename>testsuite</filename>. Under this directory, the test cases go - in a subdirectory whose name begins with the tool name. For example, for - a tool named <emphasis>flubber</emphasis>, each subdirectory containing - testsuites must start with <emphasis>"flubber."</emphasis>.</para> - - </sect1> - - <sect1 id=addtool xreflabel="Adding A New Tool"> - <title>Adding A New Tool</title> - - <para>In general, the best way to learn how to write (code or even prose) - is to read something similar. This principle applies to test cases and - to testsuites. Unfortunately, well-established testsuites have a way - of developing their own conventions: as test writers become more - experienced with DejaGnu and with Tcl, they accumulate more utilities, - and take advantage of more and more features of - <productname>Expect</productname> and <productname>Tcl</productname> in - general.</para> - - <para>Inspecting such established testsuites may make the prospect of - creating an entirely new testsuite appear overwhelming. Nevertheless, - it is quite straightforward to get a new testsuite going.</para> - - <para>There is one testsuite that is guaranteed not to grow more - elaborate over time: both it and the tool it tests were created expressly - to illustrate what it takes to get started with DejaGnu. The - <filename>example/</filename> directory of the DejaGnu distribution - contains both an interactive tool called <command>calc</command>, and a - testsuite for it. Reading this testsuite, and experimenting with it, - is a good way to supplement the information in this section. (Thanks to - Robert Lupton for creating calc and its testsuite---and also the first - version of this section of the manual!)</para> - - <para>To help orient you further in this task, here is an outline of the - steps to begin building a testsuite for a program example.</para> - - <itemizedlist mark=bullet> - - <listitem><para>Create or select a directory to contain your new - collection of tests. Change into that directory (shown here as - <filename>testsuite</filename>):</para> - - <para>Create a <filename>configure.in</filename> file in this directory, - to control configuration-dependent choices for your tests. So far as - DejaGnu is concerned, the important thing is to set a value for the - variable <symbol>target_abbrev</symbol>; this value is the link to the - init file you will write soon. (For simplicity, we assume the - environment is Unix, and use <emphasis>unix</emphasis> as the - value.)</para> - - <para>What else is needed in <filename>configure.in</filename> depends on - the requirements of your tool, your intended test environments, and which - configure system you use. This example is a minimal configure.in for use - with <productname>GNU Autoconf</productname>. </para></listitem> - - <listitem><para>Create <filename>Makefile.in</filename> (if you are using - Autoconf), or <filename>Makefile.am</filename>(if you are using - Automake), the source file used by configure to build your - <filename>Makefile</filename>. If you are using GNU Automake.just add the - keyword <emphasis>dejagnu</emphasis> to the - <emphasis>AUTOMAKE_OPTIONS</emphasis> variable in your - <filename>Makefile.am</filename> file. This will add all the Makefile - support needed to run DejaGnu, and support the <xref linkend=makecheck> - target.</para> - - <para>You also need to include two targets important to DejaGnu: - <emphasis>check</emphasis>, to run the tests, and - <emphasis>site.exp</emphasis>, to set up the Tcl copies of - configuration-dependent values. This is called the <xref linkend=local> - The check target must run the <command>runtest</command> program to - execute the tests.</para> - - <para>The <filename>site.exp</filename> target should usually set up - (among other things) the $tool variable for the name of your program. If - the local site.exp file is setup correctly, it is possible to execute the - tests by merely typing <command>runtest</command> on the command - line.</para> - - <example> - <title>Sample Makefile.in Fragment</title> - - <programlisting> - # Look for a local version of DejaGnu, otherwise use one in the path - RUNTEST = `if test -f $(top_srcdir)/../dejagnu/runtest; then \ - echo $(top_srcdir) ../dejagnu/runtest; \ - else \ - echo runtest; \ - fi` - - # The flags to pass to runtest - RUNTESTFLAGS = - - # Execute the tests - check: site.exp all - $(RUNTEST) $(RUNTESTFLAGS) \ - --tool <symbol>${example}</symbol> --srcdir $(srcdir) - - # Make the local config file - site.exp: ./config.status Makefile - @echo "Making a new config file..." - -@rm -f ./tmp? - @touch site.exp - - -@mv site.exp site.bak - @echo "## these variables are automatically\ - generated by make ##" > ./tmp0 - @echo "# Do not edit here. If you wish to\ - override these values" >> ./tmp0 - @echo "# add them to the last section" >> ./tmp0 - @echo "set host_os ${host_os}" >> ./tmp0 - @echo "set host_alias ${host_alias}" >> ./tmp0 - @echo "set host_cpu ${host_cpu}" >> ./tmp0 - @echo "set host_vendor ${host_vendor}" >> ./tmp0 - @echo "set target_os ${target_os}" >> ./tmp0 - @echo "set target_alias ${target_alias}" >> ./tmp0 - @echo "set target_cpu ${target_cpu}" >> ./tmp0 - @echo "set target_vendor ${target_vendor}" >> ./tmp0 - @echo "set host_triplet ${host_canonical}" >> ./tmp0 - @echo "set target_triplet ${target_canonical}">>./tmp0 - @echo "set tool binutils" >> ./tmp0 - @echo "set srcdir ${srcdir}" >> ./tmp0 - @echo "set objdir `pwd`" >> ./tmp0 - @echo "set <symbol>${examplename}</symbol> <symbol>${example}</symbol>" >> ./tmp0 - @echo "## All variables above are generated by\ - configure. Do Not Edit ##" >> ./tmp0 - @cat ./tmp0 > site.exp - @sed < site.bak \ - -e '1,/^## All variables above are.*##/ d' \ - >> site.exp - -@rm -f ./tmp? - - </programlisting> - </example> - </listitem> - - <listitem><para>Create a directory (in <filename>testsuite</filename>) - called <filename>config</filename>. Make a <emphasis>Tool Init - File</emphasis> in this directory. Its name must start with the - <symbol>target_abbrev</symbol> value, or be named - <filename>default.exp</filename> so call it - <filename>config/unix.exp</filename> for our Unix based example. This - is the file that contains the target-dependent procedures. - Fortunately, on Unix, most of them do not have to do very much in - order for <command>runtest</command> to run.</para> - - <para>If the program being tested is not interactive, you can get - away with this minimal <filename>unix.exp</filename> to begin - with:</para> - - <example> - <title>Simple Batch Program Tool Init File</title> + <chapter id=unit xreflabel="Unit Testing"> + <title>Unit Testing</title> - <programlisting> + <sect1 id=unittest xreflabel="What Is Unit Testing?"> + <title>What Is Unit Testing?</title> - proc foo_exit {} {} - proc foo_version {} {} + <para>Most regression testing as done by DejaGnu is system + testing. This is the complete application is tested all at + once. Unit testing is for testing single files, or small + libraries. In this case, each file is linked with a test case in + C or C++, and each function or class and method is tested in + series, with the test case having to check private data or + global variables to see if the function or method worked.</para> - </programlisting> - </example> + <para>This works particularly well for testing APIs and at level + where it is easier to debug them, than by needing to trace through + the entire appication. Also if there is a specification for the + API to be tested, the testcase can also function as a compliance + test.</para> - <para>If the program being tested is interactive, however, you might - as well define a <emphasis>start</emphasis> routine and invoke it by - using an init file like this:</para> + </sect1> - <example> - <title>Simple Interactive Program Tool Init File</title> + <sect1 id=djh xreflabel="The dejagnu.h Header File"> + <title>The dejagnu.h Header File</title> - <programlisting> - - proc foo_exit {} {} - proc foo_version {} {} + <para>DejaGnu uses a single header file to assist in unit + testing. As this file also produces it's one test state output, + it can be run standalone, which is very useful for testing on + embedded systems. This header file has a C and C++ API for the + test states, with simple totals, and standardized + output. Because the output has been standardized, DejaGnu can be + made to work with this test case, without writing almost any + Tcl. The library module, dejagnu.exp, will look for the output + messages, and then merge them into DejaGnu's.</para> - proc foo_start {} { - global ${examplename} - spawn ${examplename} - expect { - -re "" {} - } - } + </sect1> - # Start the program running we want to test - foo_start + <sect1 id=cunit xreflabel="C Unit Testing API"> + <title>C Unit Testing API</title> - </programlisting> - </example> - </listitem> + <para>All of the functions that take a + <parameter>msg</parameter> parameter use a C char * that is + the message to be dislayed. There currently is no support for + variable length arguments.</para> - <listitem><para>Create a directory whose name begins with your tool's - name, to contain tests. For example, if your tool's name is - <emphasis>gcc</emphasis>, then the directories all need to start with - <emphasis>"gcc."</emphasis>.</para></listitem> + <variablelist> + <varlistentry> + <term>Pass Function</term> - <listitem><para>Create a sample test file. Its name must end with - <filename>.exp</filename>. You can use - <filename>first-try.exp</filename>. To begin with, just write there a - line of Tcl code to issue a message.</para> + <listitem><para>This prints a message for a successful test + completion.</para> - <example> - <title>Testing A New Tool Config</title> + <funcsynopsis role="C"> + <funcdef><function>pass</function></funcdef> + <paramdef><parameter>msg</parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> - <programlisting> + <varlistentry> + <term>Fail Function</term> - send_user "Testing: one, two...\n" + <listitem><para>This prints a message for an unsuccessful + test completion.</para> - </programlisting> - </example> - </listitem> + <funcsynopsis role="C"> + <funcdef><function>fail</function></funcdef> + <paramdef><parameter>msg</parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> - <listitem><para>Back in the <filename>testsuite</filename> (top - level) directory, run <command>configure</command>. Typically you do - this while in the build directory. You may have to specify more of a - path, if a suitable configure is not available in your execution - path.</para></listitem> + <varlistentry> + <term>Untested Function</term> - <listitem><para>e now ready to triumphantly type <command>make - check</command> or <command>runtest</command>. You should see - something like this:</para> + <listitem><para>This prints a message for an test case that + isn't run for some technical reason.</para> - <example> - <title>Example Test Case Run</title> + <funcsynopsis role="C"> + <funcdef><function>untested</function></funcdef> + <paramdef><parameter>msg</parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> - <screen> - Test Run By rhl on Fri Jan 29 16:25:44 EST 1993 + <varlistentry> + <term>Unresolved Function</term> - === example tests === + <listitem><para>This prints a message for an test case that is + run, but there is no clear result. These output states + require a human to look over the results to determine what + happened.</para> - Running ./example.0/first-try.exp ... - Testing: one, two... + <funcsynopsis role="C"> + <funcdef><function>unresolved</function></funcdef> + <paramdef><parameter>msg</parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> - === example Summary === + <varlistentry> + <term>Totals Function</term> - </screen> - </example> + <listitem><para>This prints out the total numbers of all the + test state outputs.</para> - <para>There is no output in the summary, because so far the example - does not call any of the procedures that establish a test - outcome.</para></listitem> + <funcsynopsis role="C"> + <funcdef><function>totals</function></funcdef> + <paramdef><parameter></parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> + </variablelist> - <listitem><para>Write some real tests. For an interactive tool, you - should probably write a real exit routine in fairly short order. In - any case, you should also write a real version routine - soon. </para></listitem> + </sect1> - </itemizedlist> + <sect1 id=cppunit xreflabel="C++ Unit Testing API"> + <title>C++ Unit Testing API</title> + + <para>All of the methods that take a + <parameter>msg</parameter> parameter use a C char * + or STL string, that is the message to be + dislayed. There currently is no support for variable + length arguments.</para> + + <variablelist> + <varlistentry> + <term>Pass Method</term> + + <listitem><para>This prints a message for a successful test + completion.</para> + + <funcsynopsis role="C++"> + <funcdef><function>TestState::pass</function></funcdef> + <paramdef><parameter>msg</parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> + + <varlistentry> + <term>Fail Method</term> + + <listitem><para>This prints a message for an unsuccessful + test completion.</para> + + <funcsynopsis role="C++"> + <funcdef><function>TestState::fail</function></funcdef> + <paramdef><parameter>msg</parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> + + <varlistentry> + <term>Untested Method</term> + + <listitem><para>This prints a message for an test case that + isn't run for some technical reason.</para> + + <funcsynopsis role="C++"> + <funcdef><function>TestState::untested</function></funcdef> + <paramdef><parameter>msg</parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> + + <varlistentry> + <term>Unresolved Method</term> + + <listitem><para>This prints a message for an test case that is run, + but there is no clear result. These output states require a + human to look over the results to determine what happened. + </para> + + <funcsynopsis role="C++"> + <funcdef><function>TestState::unresolved</function></funcdef> + <paramdef><parameter>msg</parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> + + <varlistentry> + <term>Totals Method</term> + + <listitem><para>This prints out the total numbers of all the test + state outputs.</para> + + <funcsynopsis role="C++"> + <funcdef><function>TestState::totals</function></funcdef> + <paramdef><parameter></parameter></paramdef> + </funcsynopsis></listitem> + </varlistentry> + </variablelist> + </sect1> +</chapter> - </sect1> + <chapter id=Extending xreflabel="Extending DejaGnu"> + <title>Extending DejaGnu</title> <sect1 id=addtarget xreflabel="Adding A New Target"> <title>Adding A New Target</title> @@ -2828,336 +3297,9 @@ powerpc-linux-gcc -g -O2 -o calc calc.o </tgroup> </table> </para> - - </sect1> - - <sect1 id=writing xreflabel="Writing A Test Case"> - <title>Writing A Test Case</title> - - <para>The easiest way to prepare a new test case is to base it - on an existing one for a similar situation. There are two major - categories of tests: batch or interactive. Batch oriented tests - are usually easier to write.</para> - - <para>The GCC tests are a good example of batch oriented tests. - All GCC tests consist primarily of a call to a single common - procedure, Since all the tests either have no output, or only - have a few warning messages when successfully compiled. Any - non-warning output is a test failure. All the C code needed is - kept in the test directory. The test driver, written in Tcl, - need only get a listing of all the C files in the directory, and - compile them all using a generic procedure. This procedure and a - few others supporting for these tests are kept in the library - module <filename>lib/c-torture.exp</filename> in the GCC test - suite. Most tests of this kind use very few - <productname>expect</productname> features, and are coded almost - purely in Tcl.</para> - - <para>Writing the complete suite of C tests, then, consisted of - these steps:</para> - - <itemizedlist mark=bullet> - <listitem><para>Copying all the C code into the test directory. - These tests were based on the C-torture test created by Torbjorn - Granlund (on behalf of the Free Software Foundation) for GCC - development.</para></listitem> - - <listitem><para>Writing (and debugging) the generic Tcl procedures for - compilation.</para></listitem> - - <listitem><para>Writing the simple test driver: its main task is to - search the directory (using the Tcl procedure - <emphasis>glob</emphasis> for filename expansion with wildcards) - and call a Tcl procedure with each filename. It also checks for - a few errors from the testing procedure.</para></listitem> - </itemizedlist> - - <para>Testing interactive programs is intrinsically more - complex. Tests for most interactive programs require some trial - and error before they are complete.</para> - - <para>However, some interactive programs can be tested in a - simple fashion reminiscent of batch tests. For example, prior - to the creation of DejaGnu, the GDB distribution already - included a wide-ranging testing procedure. This procedure was - very robust, and had already undergone much more debugging and - error checking than many recent DejaGnu test cases. - Accordingly, the best approach was simply to encapsulate the - existing GDB tests, for reporting purposes. Thereafter, new GDB - tests built up a family of Tcl procedures specialized for GDB - testing.</para> - - </sect1> - - <sect1 id=debugging xreflabel="Debugging A Test Case"> - <title>Debugging A Test Case</title> - - <para>These are the kinds of debugging information available - from DejaGnu:</para> - - <itemizedlist mark=bullet> - - <listitem><para>Output controlled by test scripts themselves, - explicitly allowed for by the test author. This kind of - debugging output appears in the detailed output recorded in the - DejaGnu log file. To do the same for new tests, use the - <command>verbose</command> procedure (which in turn uses the - variable also called <emphasis>verbose</emphasis>) to control - how much output to generate. This will make it easier for other - people running the test to debug it if necessary. Whenever - possible, if <emphasis>$verbose</emphasis> is - <emphasis>0</emphasis>, there should be no output other than the - output from <emphasis>pass</emphasis>, - <emphasis>fail</emphasis>, <emphasis>error</emphasis>, and - <emphasis>warning</emphasis>. Then, to whatever extent is - appropriate for the particular test, allow successively higher - values of <emphasis>$verbose</emphasis> to generate more - information. Be kind to other programmers who use your tests: - provide for a lot of debugging information.</para></listitem> - - <listitem><para>Output from the internal debugging functions of - Tcl and <productname>Expect</productname>. There is a command - line options for each; both forms of debugging output are - recorded in the file <filename>dbg.log</filename> in the current - directory.</para> - - <para>Use <option>--debug</option> for information from the - expect level; it generates displays of the expect attempts to - match the tool output with the patterns specified. This output - can be very helpful while developing test scripts, since it - shows precisely the characters received. Iterating between the - latest attempt at a new test script and the corresponding - <filename>dbg.log</filename> can allow you to create the final - patterns by ``cut and paste''. This is sometimes the best way - to write a test case.</para></listitem> - - <listitem><para>Use <option>--strace</option> to see more - detail at the Tcl level; this shows how Tcl procedure - definitions expand, as they execute. The associated number - controls the depth of definitions expanded.</para></listitem> - - <listitem><para>Finally, if the value of - <emphasis>verbose</emphasis> is 3 or greater,DejaGnu turns on - the expect command <command>log_user</command>. This command - prints all expect actions to the expect standard output, to the - detailed log file, and (if <option>--debug</option> is on) to - <filename>dbg.log</filename>.</para></listitem> - </itemizedlist> - - </sect1> - - <sect1 id=adding xreflabel="Adding A Test Case To A Testsuite"> - <title>Adding A Test Case To A Testsuite.</title> - - <para>There are two slightly different ways to add a test - case. One is to add the test case to an existing directory. The - other is to create a new directory to hold your test. The - existing test directories represent several styles of testing, - all of which are slightly different; examine the directories for - the tool of interest to see which (if any) is most suitable.</para> - - <para>Adding a GCC test can be very simple: just add the C code - to any directory beginning with <filename>gcc</filename>. and it - runs on the next <programlisting>runtest --tool - gcc</programlisting>.</para> - - <para>To add a test to GDB, first add any source code you will - need to the test directory. Then you can either create a new - expect file, or add your test to an existing one (any - file with a <emphasis>.exp</emphasis> suffix). Creating a new - .exp file is probably a better idea if the test is significantly - different from existing tests. Adding it as a separate file also - makes upgrading easier. If the C code has to be already compiled - before the test will run, then you'll have to add it to the - <filename>Makefile.in</filename> file for that test directory, - then run <command>configure</command> and - <command>make</command>.</para> - - <para>Adding a test by creating a new directory is very - similar:</para> - - <itemizedlist mark=bullet> - - <listitem><para>Create the new directory. All subdirectory names - begin with the name of the tool to test; e.g. G++ tests might be - in a directory called <filename>g++.other</filename>. There can - be multiple test directories that start with the same tool name - (such as <emphasis>g++</emphasis>).</para></listitem> - - <listitem><para>Add the new directory name to the - <symbol>configdirs</symbol> definition in the - <filename>configure.in</filename> file for the testsuite - directory. This way when <command>make</command> and - <command>configure</command> next run, they include the new - directory.</para></listitem> - - <listitem><para>Add the new test case to the directory, as - above. </para></listitem> - - <listitem><para>To add support in the new directory for - configure and make, you must also create a - <filename>Makefile.in</filename> and a - <filename>configure.in</filename>.</para></listitem> - </itemizedlist> - - </sect1> - - <sect1 id=hints xreflabel="Hints On Writing A Test Case"> - <title>Hints On Writing A Test Case</title> - - <para>It is safest to write patterns that match all the output - generated by the tested program; this is called closure. - If a pattern does not match the entire output, any output that - remains will be examined by the next <command>expect</command> - command. In this situation, the precise boundary that determines - which <command>expect</command> command sees what is very - sensitive to timing between the Expect task and the task running - the tested tool. As a result, the test may sometimes appear to - work, but is likely to have unpredictable results. (This problem - is particularly likely for interactive tools, but can also - affect batch tools---especially for tests that take a long time - to finish.) The best way to ensure closure is to use the - <option>-re</option> option for the <command>expect</command> - command to write the pattern as a full regular expressions; then - you can match the end of output using a <emphasis>$</emphasis>. - It is also a good idea to write patterns that match all - available output by using <emphasis>.*\</emphasis> after the - text of interest; this will also match any intervening blank - lines. Sometimes an alternative is to match end of line using - <emphasis>\r</emphasis> or <emphasis>\n</emphasis>, but this is - usually too dependent on terminal settings.</para> - - <para>Always escape punctuation, such as <emphasis>(</emphasis> - or <emphasis>"</emphasis>, in your patterns; for example, write - <emphasis>\(</emphasis>. If you forget to escape punctuation, - you will usually see an error message like <programlisting>extra - characters after close-quote.</programlisting></para> - - <para>If you have trouble understanding why a pattern does not - match the program output, try using the <option>--debug</option> - option to <command>runtest</command>, and examine the debug log - carefully.</para> - - <para>Be careful not to neglect output generated by setup rather - than by the interesting parts of a test case. For example, - while testing GDB, I issue a send <emphasis>set height - 0\n</emphasis> command. The purpose is simply to make sure GDB - never calls a paging program. The <emphasis>set - height</emphasis> command in GDB does not generate any - output; but running any command makes GDB issue a new - <emphasis>(gdb) </emphasis> prompt. If there were no - <command>expect</command> command to match this prompt, the - output <emphasis>(gdb) </emphasis> begins the text seen by the - next <command>expect</command> command---which might make that - pattern fail to match.</para> - - <para>To preserve basic sanity, I also recommended that no test - ever pass if there was any kind of problem in the test case. To - take an extreme case, tests that pass even when the tool will - not spawn are misleading. Ideally, a test in this sort of - situation should not fail either. Instead, print an error - message by calling one of the DejaGnu procedures - <command>error</command> or <command>warning</command>.</para> - </sect1> - - <sect1 id=tvariables xreflabel="Test Case Variables"> - <title>Special variables used by test cases.</title> - - <para>There are special variables used by test cases. These contain - other information from DejaGnu. Your test cases can use these variables, - with conventional meanings (as well as the variables saved in - <filename>site.exp</filename>. You can use the value of these variables, - but they should never be changed.</para> - - <variablelist> - <varlistentry> - <term>$prms_id</term> - <listitem><para>The tracking system (e.g. GNATS) number identifying - a corresponding bugreport. (<emphasis>0</emphasis>} if you do not - specify it in the test script.)</para></listitem> - </varlistentry> - - <varlistentry> - <term>$item bug_id</term> - <listitem><para>An optional bug id; may reflect a bug - identification from another organization. (<emphasis>0</emphasis> - if you do not specify it.)</para></listitem> - </varlistentry> - - <varlistentry> - <term>$subdir</term> - <listitem><para>The subdirectory for the current test - case.</para></listitem> - </varlistentry> - - <varlistentry> - <term>$expect_out(buffer)</term> - <listitem><para>The output from the last command. This is an - internal variable set by Expect. More information can be found in - the Expect manual.</para></listitem> - </varlistentry> - - <varlistentry> - <term>$exec_output</term> - <listitem><para>This is the output from a - <function>${tool}_load</function> command. This only applies to - tools like GCC and GAS which produce an object file that must in - turn be executed to complete a test.</para></listitem> - </varlistentry> - - <varlistentry> - <term>$comp_output</term> - <listitem><para>This is the output from a - <function>${tool}_start</function> command. This is conventionally - used for batch oriented programs, like GCC and GAS, that may - produce interesting output (warnings, errors) without further - interaction.</para></listitem> - </varlistentry> - </variablelist> - - </sect1> - </chapter> - <chapter id=unit xreflabel="Unit Testing"> - <title>Unit Testing</title> - - <sect1 id=unittest xreflabel="What Is Unit Testing ?"> - <title>What Is Unit Testing ?</title> - - <para>Most regression testing as done by DejaGnu is system - testing. This is the complete application is tested all at - once. Unit testing is for testing single files, or small - libraries. In this case, each file is linked with a test case in - C or C++, and each function or class and method is tested in - series, with the test case having to check private data or - global variables to see if the function or method worked.</para> - - <para>This works particularly well for testing APIs and at level - where it is easier to debug them, than by needing to trace through - the entire appication. Also if there is a specification for the - API to be tested, the testcase can also function as a compliance - test.</para> - - </sect1> - - <sect1 id=djh xreflabel="The dejagnu.h Header File"> - <title>The dejagnu.h Header File</title> - - <para>DejaGnu uses a single header file to assist in unit - testing. As this file also produces it's one test state output, - it can be run standalone, which is very useful for testing on - embedded systems. This header file has a C and C++ API for the - test states, with simple totals, and standardized - output. Because the output has been standardized, DejaGnu can be - made to work with this test case, without writing almost any - Tcl. The library module, dejagnu.exp, will look for the output - messages, and then merge them into DejaGnu's.</para> - - </sect1> - -</chapter> <!-- Keep this comment at the end of the file Local variables: |