diff options
Diffstat (limited to 'doc/overview.sgml')
| -rw-r--r-- | doc/overview.sgml | 439 |
1 files changed, 439 insertions, 0 deletions
diff --git a/doc/overview.sgml b/doc/overview.sgml new file mode 100644 index 0000000..58bce0c --- /dev/null +++ b/doc/overview.sgml @@ -0,0 +1,439 @@ +<!DOCTYPE book PUBLIC "-//Davenport//DTD DocBook V3.0//EN" [ + +<!-- Begin Document Specific Declarations --> + +<?Fm: Validation Off> + +<!ENTITY version "0.5"> +<!ENTITY dj "DejaGnu"> + +<!ENTITY dejagnu-copyright " + <YEAR>1998</YEAR> + <HOLDER>Free Software Foundation, Inc.</HOLDER>"> + +<!ENTITY dejagnu-code-copyright " +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +This file documents the GNU Testing Framework ``DejaGnu'' + +Copyright (C) 92, 93, 94, 95, 96, 97, 98, 1999 Free Software Foundation, Inc. + +This text may be freely distributed under the terms of the GNU +General Public License. +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +"> + +<!ENTITY dejagnu-copyright " +Copyright 92, 93, 94, 95, 96, 97, 98, 1999 Free Software Foundation, Inc. + +Permission is granted to make and distribute verbatim copies of +this manual provided the copyright notice and this permission notice +are preserved on all copies. + +Permission is granted to copy and distribute modified versions of this +manual under the conditions for verbatim copying, provided also that +the entire resulting derived work is distributed under the terms of a +permission notice identical to this one. + +Permission is granted to copy and distribute translations of this manual +into another language, under the above conditions for modified versions. +"> + +<!-- The reference material --> +<!entity ref SYSTEM "ref.sgml"> + +<!-- The user manual --> +<!entity user SYSTEM "user.sgml"> + +<!-- End Document Specific Declarations --> +]> + +<book> + <bookinfo> + <title>&dj;</title> + <subtitle>The GNU Testing Framework</subtitle> + <date>1998 Nov 24</date> + <edition> &version</edition> + <releaseinfo> for circulation within Cygnus</releaseinfo> + <authorgroup> + <author> + <firstname>Rob Savoye</firstname> + <affiliation> + <orgname>Free Software Foundation</orgname></affiliation> + <!-- <authorblurb> + <title>Rob Savoye</title> + <para> + His home page is at <ulink> + URL="http://www.welcomehome.org/rob.html">this + location</ulink> + </para> + </authorblurb> + --> + </author> + </authorgroup> + <address> + <email>rob@welcomehome.org</email> + </address> + <!-- &cygnus-street-address; --> + <copyright>&dejagnu-copyright;</copyright> + <!-- <legalnotice> + <para> --> + <!-- [FIXME: must put legal notice here] --> + <!-- </para> --> + <!-- &cygnus-legal-notice; --> + <!-- </legalnotice> --> + <revhistory> + <revision> + <revnumber> 0.1</revnumber> + <date>1998-11</date> + <authorinitials>rob@welcomehome.org</authorinitials> + <revremark>Initial version after conversion to DocBook.</revremark> + </revision> + </revhistory> + + </bookinfo> + + <toc></toc> + + <preface id=preface> + <title>Abstract</title> + + <para>This document attempts to describe the functionality of + DejaGnu, the GNU Testing Framework. DejaGnu is entirely written in + <productname>Expect</productname>, which uses + <productname>Tcl</productname> as a command + language. <productname>Expect</productname> serves as a very + programmable shell; you can run any program, as with the usual + Unix command shells---but once the program is started, your + test script has fully programmable control of + its input and output. This does not just apply to the programs + under test; <command>expect</command> can also run any auxiliary + program, such as <command>diff</command> or <command>sh</command>, + with full control over its input and output.</para> + + <para>DejaGnu itself is merely a framework for creation of a test + suites. Test suites are distributed separately for each GNU + tool.</para> + + </preface> + + <chapter id=overview xreflabel=Overview> + <title>Overview</title> + + <sect1 id=whatis xreflabel="What is &dj; ?"> + <title>What is &dj; ?</title> + + <para><productname>DejaGnu</productname> is a framework for + testing other programs. Its purpose is to provide a single + front end for all tests. Think of it as a custom library of + Tcl procedures crafted to support writing a test harness. A + <emphasis>Test Harness</emphasis> is the testing + infrastructure that is created to support a specific program + or tool. Each program can have multiple test suites, all + supported by a single test harness. DejaGnu is written in + <productname>Expect</productname>, which in turn uses + <productname>Tcl</productname> -- Tool command + language. There is more information on Tcl at the <ulink + URL="http://www.scriptics.com">Scriptics</ulink> web site, and the + Expect web site is at <ulink + URL="http://expect.nist.gov">NIST</ulink>.</para> + + <para>DejaGnu offers several advantages for testing:</para> + + <itemizedlist mark="bullet" spacing="compact"> + + <listitem><para>The flexibility and consistency of the DejaGnu + framework make it easy to write tests for any program, with + either batch oriented, or interactive programs.</para> + </listitem> + + <listitem><para>DejaGnu provides a layer of abstraction which + allows you to write tests that are portable to any host or + target where a program must be tested. For instance, a test + for <command>GDB</command> can run (from any Unix + based host) on any target architecture that DejaGnu + supports. Currently DejaGnu runs tests on many single board + computers, whose operating software ranges from just a boot + monitor to a full-fledged, Unix-like realtime OS.</para> + </listitem> + + <listitem><para>All tests have the same output format. This + makes it easy to integrate testing into other software + development processes. DejaGnu's output is designed to be + parsed by other filtering script, and it is also human + readable.</para> + </listitem> + + <listitem><para>Using Tcl and expect, it's easy to create wrappers + for existing test suites. By incorporating existing tests under + DejaGnu, it's easier to have a single set of report analyse + programs..</para> + + </listitem> + </itemizedlist> + + <para>Running tests requires two things: the testing framework, and + the test suites themselves. Tests are usually written in + <productname>Expect</productname> using Tcl, but you can also use a + Tcl script to run a test suite that is not based on + <productname>Expect</productname>. + (<productname>expect</productname> script filenames conventionally + use <emphasis>.exp</emphasis> as a suffix; for example, the main + implementation of the DejaGnu test driver is in the file + <productname>runtest.exp</productname>.)</para> + + <para>Julia Menapace first coined the term ``Deja Gnu'' to describe an + earlier testing framework at Cygnus Support she had written for + <command>GDB</command>. When we replaced it with the Expect-based + framework, it was like DejaGnu all over again... But more importantly, it + was also named after my daughter,<ulink + URL="mailto:deja@welcomehome.org">Deja Snow Savoye</ulink> (now 9 + years old in Dec of 1998), who was a toddler during DejaGnu's + creation.</para> + + </sect1> + + <sect1 id=new xreflabel="Release Notes"> + <title>What's New In This Release</title> + + <para>This release has a number of substantial changes over version + 1.3. The most visible change is that the version of Expect and Tcl + included in the release are up-to-date with the current stable net + releases. The biggest change is years of modifications to the + target configuration system, used for cross testing. While this + greatly improved cross testing, is has made that subsystem very + complicated. The goal is to have this entirely rewritten using + <productname>iTcl</productname> by the next release. Other changes + are:</para> + + <itemizedlist> + <listitem><para>More builtin support for building target binaries + with the correct linker flags. Currently this only works with + <productname>GCC</productname> as the cross compiler, + preferably with a target supported by + <xref linkend=libgloss>.</para></listitem> + + <listitem><para>Lots of little bug fixes from years of heavy + use at Cygnus Solutions.</para></listitem> + + <listitem><para>DejaGnu now uses + <productname>Automake</productname> for Makefile + configuration.</para></listitem> + + <listitem><para>Updated documentation, now in SGML + (using the <ulink + URL="http://nis-www.lanl.gov/~rosalia/mydocs/docbook-intro.html">free + GNU DocBook tools</ulink>) format.</para></listitem> + + <listitem><para>NT support. There is beta level support for NT + that is still a work in progress. This requires the <ulink + URL=httpd://sourceware.cygnus.com>Cygwin</ulink> POSIX system + for NT.</para></listitem> + + </itemizedlist> + + <sect2 id=cygwin xreflabel="NT Support"> + <title>NT Support</title> + + <para>To use DejaGnu on NT, you need to first install the + <ulink URL="http://sourceware.cygnus.com">Cygwin</ulink> + release. This works as of the B20.1 release. Cygwin is a POSIX + system for NT. This covers both utility programs, and a libray + that adds POSIX system calls to NT. Among them is pseudo tty + support for NT that emulates the POSIX pty standard. The + latest Cygwin is always available from <ulink + URL="http://sourceware.cygnus.com">this location</ulink>. This + works well enough to run <emphasis>"make check"</emphasis> of + the GNU development tree on NT after a native build. But the + nature of pty's on NT is still evolving. Your mileage may + vary...</para> + + </sect2> + + </sect1> + + <sect1 id=designgoals xreflabel="Design Goals"> + <title>Design Goals</title> + + <para>DejaGnu grew out of the internal needs of Cygnus Solutions. (then + Cygnus Support). Cygnus maintains and enhances a variety of free programs + in many different environments, and we needed a testing tool that:</para> + + <itemizedlist mark="bullet"> + <listitem><para>is useful to developers while fixing bugs.</para> + <listitem><para>automates running many tests during a software + release process.</para> + <listitem><para>is portable among a variety of host + computers.</para> + <listitem><para>supports cross-development testing.</para> + <listitem><para>permits testing interactive programs, like + <command>GDB</command>; and </para> + <listitem><para>permits testing batch oriented programs, like + <command>GCC</command>.</para> + </itemizedlist> + + <para>Some of the requirements proved challenging. For example, + interactive programs do not lend themselves very well to automated testing. + But all the requirements are important: for instance, it is imperative to + make sure that <command>GDB</command> works as well when cross-debugging + as it does in a native configuration. </para> + + <para>Probably the greatest challenge was testing in a cross-development + environment (which can be a real nightmare). Most cross-development + environments are customized by each developer. Even when buying packaged + boards from vendors there are many differences. The communication + interfaces vary from a serial line to ethernet. DejaGnu was designed with + a modular communication setup, so that each kind of communication can be + added as required, and supported thereafter. Once a communication + procedure is coded, any test can use it. Currently DejaGnu can use + <command>rsh</command>, <command>rlogin</command>, + <command>telnet</command>, <command>tip</command>, + <command>kermit</command>, and <command>mondfe</command> for remote + communications.</para> + + </sect1> + + <sect1 id=posix xreflabel="A POSIX Conforming Test Framework"> + <title>A POSIX conforming test framework</title> + + <para>DejaGnu conforms to the POSIX 1003.3 standard for test + frameworks. I was also a member of that committe.</para> + + <para>The POSIX standard 1003.3 defines what a testing framework needs to + provide, in order to permit the creation of POSIX conformance test + suites. This standard is primarily oriented to running POSIX conformance + tests, but its requirements also support testing of features not related + to POSIX conformance. POSIX 1003.3 does not specify a particular testing + framework, but at this time there is only one other POSIX conforming test + framework: TET. TET was created by Unisoft for a consortium comprised of + X/Open, Unix International, and the Open Software Foundation.</para> + + <para>The POSIX documentation refers to <firstterm>assertions</firstterm>. + An assertion is a description of behavior. For example, if a standard + says ``The sun shall shine'', a corresponding assertion might be ``The + sun is shining.'' A test based on this assertion would pass or fail + depending on whether it is daytime or nighttime. It is important to note + that the standard being tested is never 1003.3; the standard being tested + is some other standard, for which the assertions were written.</para> + + <para>As there is no test suite to test testing frameworks for POSIX + 1003.3 conformance, verifying conformance to this standard is done by + repeatedly reading the standard and experimenting. One of the main + things 1003.3 does specify is the set of allowed output messages, and + their definitions. Four messages are supported for a required feature of + POSIX conforming systems, and a fifth for a conditional feature. DejaGnu + supports the use of all five output messages; in this sense a test suite + that uses exactly these messages can be considered POSIX conforming. + These definitions specify the output of a test case:</para> + + <variablelist> + <varlistentry> + <term>PASS</term> + <listitem><para>A test has succeeded. That is, it demonstrated that + the assertion is true.</para></listitem> + </varlistentry> + + <varlistentry> + <term>XFAIL</term> + <listitem><para>POSIX 1003.3 does not incorporate the notion of + expected failures, so <emphasis>PASS</emphasis>, instead of + <emphasis>XPASS</emphasis>, must also be returned for test cases + which were expected to fail and did not. This means that + <emphasis>PASS</emphasis> is in some sense more ambiguous than if + <emphasis>XPASS</emphasis> is also used.</para></listitem> + </varlistentry> + + <varlistentry> + <term>FAIL</term> + <listitem><para>A test has produced the bug it was intended to + capture. That is, it has demonstrated that the assertion is false. + The <emphasis>FAIL</emphasis> message is based on the test case only. + Other messages are used to indicate a failure of the framework. As + with <emphasis>PASS</emphasis>, POSIX tests must return + <emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even + if a failure was expected.</para></listitem> + </varlistentry> + + <varlistentry> + <term>UNRESOLVED</term> + <listitem><para>A test produced indeterminate results. Usually, this + means the test executed in an unexpected fashion; this outcome + requires that a human being go over results, to determine if the test + should have passed or failed. This message is also used for any test + that requires human intervention because it is beyond the abilities + of the testing framework. Any unresolved test should resolved to + <emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test + run can be considered finished.</para> + + <para>Note that for POSIX, each assertion must produce a test result + code. If the test isn't actually run, it must produce + <emphasis>UNRESOLVED</emphasis> rather than just leaving that test + out of the output. This means that you have to be careful when + writing tests, to not carelessly use tcl statements like + <emphasis>return</emphasis>---if you alter the flow of control of the + tcl code you must insure that every test still produces some result + code.</para> + + <para>Here are some of the ways a test may wind up + <emphasis>UNRESOLVED</emphasis>:</para> + + <itemizedlist mark=bullet> + <listitem><para>A test's execution is interrupted.</listitem> + + <listitem><para>A test does not produce a clear result. This is + usually because there was an <emphasis>ERROR</emphasis> from + DejaGnu while processing the test, or because there were three or + more <emphasis>WARNING</emphasis> messages. Any + <emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis> messages + can invalidate the output of the test. This usually requires a + human being to examine the output to determine what really + happened---and to improve the test case.</para></listitem> + + <listitem><para>A test depends on a previous test, which + fails.</para></listitem> + + <listitem><para>The test was set up incorrectly.</para></listitem> + </itemizedlist> + + <varlistentry> + <term>UNTESTED</term> + <listitem><para>A test was not run. This is a placeholder, used + when there is no real test case yet.</para> + </varlistentry> + </variablelist> + + <para>The only remaining output message left is intended to test + features that are specified by the applicable POSIX standard as + conditional:</para> + + <variablelist> + <varlistentry> + <term>UNSUPPORTED</term> + <listitem><para>There is no support for the tested case. This may + mean that a conditional feature of an operating system, or of a + compiler, is not implemented. DejaGnu also uses this message when + a testing environment (often a ``bare board'' target) lacks basic + support for compiling or running the test case. For example, a + test for the system subroutine <emphasis>gethostname</emphasis> + would never work on a target board running only a boot monitor. + </varlistentry> + </variablelist> + + <para>DejaGnu uses the same output procedures to produce these messages + for all test suites, and these procedures are already known to conform + to POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3, + you must avoid the <emphasis>setup</emphasis>xfail} procedure as + described in the <emphasis>PASS</emphasis> section above, and you must + be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate, + as described in the <emphasis>UNRESOLVED</emphasis> section + above.</para> + </sect1> + + </chapter> + + <!-- include the user manual --> + &user; + + <!-- include the reference manual --> + &ref; + +</book> |