diff options
Diffstat (limited to 'doc/dejagnu.xml')
| -rw-r--r-- | doc/dejagnu.xml | 370 |
1 files changed, 0 insertions, 370 deletions
diff --git a/doc/dejagnu.xml b/doc/dejagnu.xml deleted file mode 100644 index 488ab3a..0000000 --- a/doc/dejagnu.xml +++ /dev/null @@ -1,370 +0,0 @@ -<?xml version="1.0"?> -<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN" - "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" [ - <!ENTITY legal SYSTEM "legal.xml"> - <!ENTITY appversion "1.5.4"> - <!ENTITY version "1.5.4"> - <!ENTITY manrevision "1.0"> - <!ENTITY date "November 2015"> - <!ENTITY app "<application>DejaGnu</application>"> - <!ENTITY appname "DejaGnu"> - <!ENTITY dj "DejaGnu"> - <!-- The reference material --> - <!ENTITY ref SYSTEM "ref.xml"> - <!-- The user manual --> - <!ENTITY user SYSTEM "user.xml"> -]> -<!-- Begin Document Specific Declarations --> -<article> - <articleinfo> - <title>&dj;</title> - <subtitle>The GNU Testing Framework</subtitle> - <date>November 2015</date> - <authorgroup> - <author> - <firstname>Rob</firstname> - <surname>Savoye</surname> - </author> - <author> - <firstname>Ben</firstname> - <surname>Elliston</surname> - </author> - </authorgroup> - - <copyright> - <year>2015</year> - <holder>Free Software Foundation, Inc.</holder> - </copyright> - <!-- <legalnotice> - <para> --> - <!-- [FIXME: must put legal notice here] --> - <!-- </para> --> - <!-- </legalnotice> --> - <revhistory> - <revision> - <revnumber>1.5.3</revnumber> - <date>2015-11-12</date> - <authorinitials>bje</authorinitials> - <revremark>Overhaul the manual.</revremark> - </revision> - </revhistory> - - </articleinfo> - - <sect1 id="intro" xreflabel="Introduction"> - <title>Introduction</title> - <sect2 id="whatis" xreflabel="What is &dj;?"> - <title>What is &dj;?</title> - - <para><productname>&dj;</productname> is a framework for testing - other programs, providing a single front-end for all - tests. You can think of it as a custom library of Tcl - procedures crafted to help with writing a test harness. A - <emphasis>test harness</emphasis> is the testing - infrastructure that is created to support a specific program - or tool. Each program can have multiple testsuites, all - supported by a single test harness. &dj; is written in - <productname>Expect</productname>, which in turn uses - <productname>Tcl</productname>, the Tool command - language. There is more information on Tcl at - the <ulink url="http://www.tcl.tk">Tcl/Tk</ulink> web site and - the Expect web site is - at <ulink url="http://expect.nist.gov">NIST</ulink>.</para> - - <para>Julia Menapace first coined the term ``&dj;'' to describe an - earlier testing framework she wrote at Cygnus Support for - testing GDB. When we replaced it with the Expect-based - framework, it was like &dj; all over again. More importantly, - it was also named after my daughter, Deja Snow Savoye, who was - a toddler during &dj;'s beginnings.</para> - - <para>&dj; offers several advantages for testing:</para> - - <itemizedlist mark="bullet" spacing="compact"> - - <listitem><para>The flexibility and consistency of the &dj; - framework make it easy to write tests for any program, with - either batch oriented, or interactive programs.</para> - </listitem> - - <listitem><para>&dj; provides a layer of abstraction which - allows you to write tests that are portable to any host or - target where a program must be tested. For instance, a test - for <command>GDB</command> can run from any supported host - system on any supported target system. &dj; runs tests on - many single board computers, whose operating software ranges - from a simple boot monitor to a real-time OS.</para> - </listitem> - - <listitem><para>All tests have the same output format. This - makes it easy to integrate testing into other software - development processes. &dj;'s output is designed to be - parsed by other filtering script and it is also human - readable.</para> - </listitem> - - <listitem><para>Using Tcl and Expect, it's easy to create wrappers - for existing testsuites. By incorporating existing tests under - &dj;, it's easier to have a single set of report analyse - programs..</para> - - </listitem> - </itemizedlist> - - <para>Running tests requires two things: the testing framework and - the testsuites themselves. Tests are usually written in - <productname>Expect</productname> using Tcl, but you can also use - a Tcl script to run a testsuite that is not based on - <productname>Expect</productname>. <productname>Expect</productname> - script filenames conventionally use <emphasis>.exp</emphasis> as a - suffix; for example, the main implementation of the &dj; test - driver is in the file - <productname>runtest.exp</productname>.)</para> - - </sect2> - - <sect2 id="new" xreflabel="Release Notes"> - <title>New In This Release</title> - - <itemizedlist> - <listitem> - <para>A completely new manual.</para> - </listitem> - - </itemizedlist> - - </sect2> - - <sect2 id="designgoals" xreflabel="Design Goals"> - <title>Design Goals</title> - - <para>&dj; grew out of the internal needs of Cygnus Solutions - (formerly Cygnus Support). Cygnus maintained and enhanced a - variety of free programs in many different environments and needed - a testing tool that:</para> - - <itemizedlist mark="bullet"> - <listitem><para>was useful to developers while fixing - bugs;</para></listitem> - <listitem><para>automated running many tests during a software - release process;</para></listitem> - <listitem><para>was portable among a variety of host - computers;</para></listitem> - <listitem><para>supported a cross-development environment; - </para></listitem> - <listitem><para>permitted testing of interactive programs - like <command>GDB</command>; and </para></listitem> - <listitem><para>permitted testing of batch-oriented programs - like <command>GCC</command>.</para></listitem> - </itemizedlist> - - <para>Some of the requirements proved challenging. For example, - interactive programs do not lend themselves very well to automated testing. - But all the requirements are important. For instance, it is imperative to - make sure that <command>GDB</command> works as well when cross-debugging - as it does in a native configuration. </para> - - <para>Probably the greatest challenge was testing in a - cross-development environment. Most cross-development - environments are customized by each developer. Even when buying - packaged boards from vendors there are many differences. The - communication interfaces vary from a serial line to Ethernet. - &dj; was designed with a modular communication setup, so that - each kind of communication can be added as required and supported - thereafter. Once a communication procedure is written, any test can - use it. Currently &dj; can use <command>rsh</command>, - <command>rlogin</command>, <command>telnet</command>, - <command>tip</command>, and <command>kermit</command> for remote - communications.</para> - - </sect2> - - <sect2 id="posix" xreflabel="A POSIX Conforming Test Framework"> - <title>A POSIX compliant test framework</title> - - <para>&dj; conforms to the POSIX 1003.3 standard for test - frameworks. Rob Savoye was a member of that committee.</para> - - <para>POSIX standard 1003.3 defines what a testing framework - needs to provide to create a POSIX compliant testsuite. This - standard is primarily oriented to checking POSIX conformance, - but its requirements also support testing of features not - related to POSIX conformance. POSIX 1003.3 does not specify a - particular testing framework, but at this time there is only one - other POSIX conforming test framework. TET was created by - Unisoft for a consortium comprised of X/Open, Unix International - and the Open Software Foundation.</para> - - <para>The POSIX documentation refers to <firstterm>assertions</firstterm>. - An assertion is a description of behavior. For example, if a standard - says ``The sun shall shine'', a corresponding assertion might be ``The - sun is shining.'' A test based on this assertion would pass or fail - depending on whether it is day or night. It is important to note - that the standard being tested is never 1003.3; the standard being tested - is some other standard, for which the assertions were written.</para> - - <para>As there is no testsuite to verify that testing frameworks - are POSIX 1003.3 compliant, this is done by repeatedly reading - the standard and experimenting. One of the main things POSIX - 1003.3 does specify is the set of allowed output messages and - their definitions. Four messages are supported for a required - feature of POSIX conforming systems and a fifth for a - conditional feature. &dj; supports all five output messages. In - this sense a testsuite that uses exactly these messages can be - considered POSIX compliant. These definitions specify the - output of a test case:</para> - - <variablelist> - <varlistentry> - <term>PASS</term> - <listitem><para>A test has succeeded. That is, it demonstrated that - the assertion is true.</para></listitem> - </varlistentry> - - <varlistentry> - <term>FAIL</term> - <listitem><para>A test has not succeeded -- the assertion is - false. The <emphasis>FAIL</emphasis> message is based on - this test case only. Other messages are used to indicate a - failure of the framework. As with <emphasis>PASS</emphasis>, - POSIX tests must return - <emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even - if a failure was expected.</para></listitem> - </varlistentry> - - <varlistentry> - <term>XFAIL</term> - <listitem><para>POSIX 1003.3 does not incorporate the notion of - expected failures, so <emphasis>PASS</emphasis>, instead of - <emphasis>XPASS</emphasis>, must also be returned for test cases - which were expected to fail and did not. This means that - <emphasis>PASS</emphasis> is in some sense more ambiguous than if - <emphasis>XPASS</emphasis> is also used.</para></listitem> - </varlistentry> - - <varlistentry> - <term>UNRESOLVED</term> - <listitem><para>A test produced indeterminate results. Usually, this - means the test executed in an unexpected fashion. This outcome - requires a human to go over results to determine if the test - should have passed or failed. This message is also used for any test - that requires human intervention because it is beyond the abilities - of the testing framework. Any unresolved test should resolved to - <emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test - run can be considered finished.</para> - - <para>Note that for POSIX, each assertion must produce a test result - code. If the test isn't actually run, it must produce - <emphasis>UNRESOLVED</emphasis> rather than just leaving that test - out of the output. This means that you have to be careful when - writing tests to not carelessly use Tcl commands like - <emphasis>return</emphasis>---if you alter the flow of control of the - Tcl code you must insure that every test still produces some result - code.</para> - - <para>Here are some of the ways a test may wind up - <emphasis>UNRESOLVED</emphasis>:</para></listitem> - - </varlistentry> - </variablelist> - - <itemizedlist> - <listitem><para>Execution of a test is - interrupted.</para></listitem> - - <listitem><para>A test does not produce a clear - result. This is usually because there was an - <emphasis>ERROR</emphasis> from &dj; while processing - the test, or because there were three or more - <emphasis>WARNING</emphasis> messages. Any - <emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis> - messages can invalidate the output of the test. This - usually requires a human to examine the output to - determine what really happened -- and to improve the test - case.</para></listitem> - - <listitem><para>A test depends on a previous test, which - has failed.</para></listitem> - - <listitem><para>The test was set up - incorrectly.</para></listitem> - </itemizedlist> - - <variablelist> - <varlistentry> - <term>UNTESTED</term> - <listitem><para>A test was not run. This is a placeholder - used when there is no real test case - yet.</para></listitem> - </varlistentry> - </variablelist> - - <variablelist> - <varlistentry> - <term>UNSUPPORTED</term> - <listitem><para>There is no support for the tested case. This may - mean that a conditional feature of an operating system, or of a - compiler, is not implemented. &dj; also uses this message when - a testing environment (often a ``bare board'' target) lacks basic - support for compiling or running the test case. For example, a - test for the system subroutine <emphasis>gethostname</emphasis> - would never work on a target board running only a boot - monitor.</para></listitem> - </varlistentry> - </variablelist> - - <para>&dj; uses the same output procedures to produce these messages - for all testsuites and these procedures are already known to conform - to POSIX 1003.3. For a &dj; testsuite to conform to POSIX 1003.3, - you must avoid the <emphasis>setup_xfail</emphasis> procedure as - described in the <emphasis>PASS</emphasis> section above and you must - be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate, - as described in the <emphasis>UNRESOLVED</emphasis> section - above.</para> - </sect2> - - <sect2 id="installation" xreflabel="Installation"> - <title>Installation</title> - - <para>Refer to the <filename>INSTALL</filename> in the source - distribution for detailed installation instructions. Note that - there is no compilation step as with many other GNU packages, as - &dj; consists of interpreted code only.</para> - - <para>Save for its own small testsuite, the &dj; distribution does - not include any testsuites. Testsuites for the various GNU - development tools are included with those packages. After - configuring the top-level &dj; directory, unpack and configure - the test directories for the tools you want to test; then, in - each test directory, run <emphasis>make check</emphasis> to - build auxiliary programs required by some of the tests, and run - the test suites.</para> - </sect2> - - </sect1> - - <!-- include the user manual --> - &user; - - <!-- include the reference manual --> - &ref; - -</article> - -<!-- Keep this comment at the end of the file -Local variables: -mode: sgml -sgml-omittag:t -sgml-shorttag:t -sgml-namecase-general:t -sgml-general-insert-case:lower -sgml-minimize-attributes:nil -sgml-always-quote-attributes:t -sgml-indent-step:1 -sgml-indent-data:nil -sgml-parent-document:nil -sgml-exposed-tags:nil -sgml-local-catalogs:nil -sgml-local-ecat-files:nil -End: ---> |