path: root/doc/dejagnu.xml

<?xml version="1.0"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
    "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" [
  <!ENTITY legal SYSTEM "legal.xml">
  <!ENTITY appversion "1.5.4">
  <!ENTITY version "1.5.4">
  <!ENTITY manrevision "1.0">
  <!ENTITY date "November 2015">
  <!ENTITY app "<application>DejaGnu</application>">
  <!ENTITY appname "DejaGnu">
  <!ENTITY dj "DejaGnu">
  <!-- The reference material -->
  <!ENTITY ref SYSTEM "ref.xml">
  <!-- The user manual -->
  <!ENTITY user SYSTEM "user.xml">
]>
<!-- Begin Document Specific Declarations -->
<article>
  <articleinfo>
    <title>&dj;</title>
    <subtitle>The GNU Testing Framework</subtitle>
    <date>November 2015</date>
    <authorgroup>
      <author>
        <firstname>Rob</firstname>
        <surname>Savoye</surname>
      </author>
      <author>
        <firstname>Ben</firstname>
        <surname>Elliston</surname>
      </author>
    </authorgroup>

    <copyright>
       <year>2015</year>
       <holder>Free Software Foundation, Inc.</holder>
    </copyright>
    <!-- <legalnotice>
      <para> -->
        <!-- [FIXME: must put legal notice here] -->
      <!-- </para> -->
    <!-- </legalnotice> -->
    <revhistory>
      <revision>
        <revnumber>1.5.3</revnumber>
        <date>2015-11-12</date>
        <authorinitials>bje</authorinitials>
        <revremark>Overhaul the manual.</revremark>
      </revision>
    </revhistory>

  </articleinfo>

  <sect1 id="intro" xreflabel="Introduction">
    <title>Introduction</title>
    <sect2 id="whatis" xreflabel="What is &dj;?">
      <title>What is &dj;?</title>

    <para><productname>&dj;</productname> is a framework for testing
	other programs, providing a single front-end for all
	tests. You can think of it as a custom library of Tcl
	procedures crafted to help with writing a test harness. A
	<emphasis>test harness</emphasis> is the testing
	infrastructure that is created to support a specific program
	or tool. Each program can have multiple testsuites, all
	supported by a single test harness. &dj; is written in
	<productname>Expect</productname>, which in turn uses
	<productname>Tcl</productname>, the Tool command
	language. There is more information on Tcl at
	the <ulink url="http://www.tcl.tk">Tcl/Tk</ulink> web site and
	the Expect web site is
	at <ulink url="http://expect.nist.gov">NIST</ulink>.</para>
	
   <para>Julia Menapace first coined the term ``&dj;'' to describe an
        earlier testing framework she wrote at Cygnus Support for
        testing GDB. When we replaced it with the Expect-based
        framework, it was like &dj; all over again.  More importantly,
        it was also named after my daughter, Deja Snow Savoye, who was
        a toddler during &dj;'s beginnings.</para>

    <para>&dj; offers several advantages for testing:</para>

    <itemizedlist mark="bullet" spacing="compact">

      <listitem><para>The flexibility and consistency of the &dj;
	framework make it easy to write tests for any program, with
	either batch oriented, or interactive programs.</para>
      </listitem>

      <listitem><para>&dj; provides a layer of abstraction which
	allows you to write tests that are portable to any host or
	target where a program must be tested. For instance, a test
	for <command>GDB</command> can run from any supported host
	system on any supported target system. &dj; runs tests on
	many single board computers, whose operating software ranges
	from a simple boot monitor to a real-time OS.</para>
	</listitem>

      <listitem><para>All tests have the same output format. This
	makes it easy to integrate testing into other software
	development processes. &dj;'s output is designed to be
	parsed by other filtering script and it is also human
	readable.</para>
      </listitem>

      <listitem><para>Using Tcl and Expect, it's easy to create wrappers
      for existing testsuites. By incorporating existing tests under
      &dj;, it's easier to have a single set of report analyse
      programs..</para>

      </listitem>
    </itemizedlist>

    <para>Running tests requires two things: the testing framework and
    the testsuites themselves. Tests are usually written in
    <productname>Expect</productname> using Tcl, but you can also use
    a Tcl script to run a testsuite that is not based on
    <productname>Expect</productname>. <productname>Expect</productname>
    script filenames conventionally use <emphasis>.exp</emphasis> as a
    suffix; for example, the main implementation of the &dj; test
    driver is in the file
    <productname>runtest.exp</productname>.)</para>
	
  </sect2>

  <sect2 id="new" xreflabel="Release Notes">
    <title>New In This Release</title>

    <itemizedlist>
      <listitem>
        <para>A completely new manual.</para>
      </listitem>

    </itemizedlist>

  </sect2>

  <sect2 id="designgoals" xreflabel="Design Goals">
    <title>Design Goals</title>

    <para>&dj; grew out of the internal needs of Cygnus Solutions
    (formerly Cygnus Support). Cygnus maintained and enhanced a
    variety of free programs in many different environments and needed
    a testing tool that:</para>

    <itemizedlist mark="bullet">
      <listitem><para>was useful to developers while fixing
      bugs;</para></listitem>
      <listitem><para>automated running many tests during a software
      release process;</para></listitem>
      <listitem><para>was portable among a variety of host
      computers;</para></listitem>
      <listitem><para>supported a cross-development environment;
      </para></listitem>
      <listitem><para>permitted testing of interactive programs
      like <command>GDB</command>; and </para></listitem>
      <listitem><para>permitted testing of batch-oriented programs
      like <command>GCC</command>.</para></listitem>
    </itemizedlist>

    <para>Some of the requirements proved challenging.  For example,
    interactive programs do not lend themselves very well to automated testing.
    But all the requirements are important. For instance, it is imperative to
    make sure that <command>GDB</command> works as well when cross-debugging
    as it does in a native configuration. </para>

    <para>Probably the greatest challenge was testing in a
    cross-development environment.  Most cross-development
    environments are customized by each developer.  Even when buying
    packaged boards from vendors there are many differences.  The
    communication interfaces vary from a serial line to Ethernet.
    &dj; was designed with a modular communication setup, so that
    each kind of communication can be added as required and supported
    thereafter.  Once a communication procedure is written, any test can
    use it.  Currently &dj; can use <command>rsh</command>,
    <command>rlogin</command>, <command>telnet</command>,
    <command>tip</command>, and <command>kermit</command> for remote
    communications.</para>

    </sect2>

    <sect2 id="posix" xreflabel="A POSIX Conforming Test Framework">
      <title>A POSIX compliant test framework</title>

      <para>&dj; conforms to the POSIX 1003.3 standard for test
      frameworks. Rob Savoye was a member of that committee.</para>

      <para>POSIX standard 1003.3 defines what a testing framework
      needs to provide to create a POSIX compliant testsuite. This
      standard is primarily oriented to checking POSIX conformance,
      but its requirements also support testing of features not
      related to POSIX conformance.  POSIX 1003.3 does not specify a
      particular testing framework, but at this time there is only one
      other POSIX conforming test framework. TET was created by
      Unisoft for a consortium comprised of X/Open, Unix International
      and the Open Software Foundation.</para>

      <para>The POSIX documentation refers to <firstterm>assertions</firstterm>.
      An assertion is a description of behavior.  For example, if a standard
      says ``The sun shall shine'', a corresponding assertion might be ``The
      sun is shining.''  A test based on this assertion would pass or fail
      depending on whether it is day or night.  It is important to note
      that the standard being tested is never 1003.3; the standard being tested
      is some other standard, for which the assertions were written.</para>

      <para>As there is no testsuite to verify that testing frameworks
      are POSIX 1003.3 compliant, this is done by repeatedly reading
      the standard and experimenting.  One of the main things POSIX
      1003.3 does specify is the set of allowed output messages and
      their definitions.  Four messages are supported for a required
      feature of POSIX conforming systems and a fifth for a
      conditional feature. &dj; supports all five output messages.  In
      this sense a testsuite that uses exactly these messages can be
      considered POSIX compliant.  These definitions specify the
      output of a test case:</para>

      <variablelist>
	<varlistentry>
	  <term>PASS</term>
	  <listitem><para>A test has succeeded.  That is, it demonstrated that
	  the assertion is true.</para></listitem>
	</varlistentry>

        <varlistentry>
	  <term>FAIL</term>
	  <listitem><para>A test has not succeeded -- the assertion is
	  false.  The <emphasis>FAIL</emphasis> message is based on
	  this test case only.  Other messages are used to indicate a
	  failure of the framework. As with <emphasis>PASS</emphasis>,
	  POSIX tests must return
	  <emphasis>FAIL</emphasis> rather than <emphasis>XFAIL</emphasis> even
	  if a failure was expected.</para></listitem>
	</varlistentry>

        <varlistentry>
	  <term>XFAIL</term>
	  <listitem><para>POSIX 1003.3 does not incorporate the notion of
	  expected failures, so <emphasis>PASS</emphasis>, instead of
	  <emphasis>XPASS</emphasis>, must also be returned for test cases
	  which were expected to fail and did not.  This means that
	  <emphasis>PASS</emphasis> is in some sense more ambiguous than if
	  <emphasis>XPASS</emphasis> is also used.</para></listitem>
	</varlistentry>

        <varlistentry>
	  <term>UNRESOLVED</term>
	  <listitem><para>A test produced indeterminate results.  Usually, this
	  means the test executed in an unexpected fashion. This outcome
	  requires a human to go over results to determine if the test
	  should have passed or failed.  This message is also used for any test
	  that requires human intervention because it is beyond the abilities
	  of the testing framework.  Any unresolved test should resolved to
	  <emphasis>PASS</emphasis> or <emphasis>FAIL</emphasis> before a test
	  run can be considered finished.</para>

	  <para>Note that for POSIX, each assertion must produce a test result
	  code.  If the test isn't actually run, it must produce
	  <emphasis>UNRESOLVED</emphasis> rather than just leaving that test
	  out of the output.  This means that you have to be careful when
	  writing tests to not carelessly use Tcl commands like
	  <emphasis>return</emphasis>---if you alter the flow of control of the
	  Tcl code you must insure that every test still produces some result
	  code.</para>

	  <para>Here are some of the ways a test may wind up
	  <emphasis>UNRESOLVED</emphasis>:</para></listitem>

	</varlistentry>
	</variablelist>

          <itemizedlist>
	    <listitem><para>Execution of a test is
	    interrupted.</para></listitem>

	    <listitem><para>A test does not produce a clear
	    result. This is usually because there was an
	    <emphasis>ERROR</emphasis> from &dj; while processing
	    the test, or because there were three or more
	    <emphasis>WARNING</emphasis> messages. Any
	    <emphasis>WARNING</emphasis> or <emphasis>ERROR</emphasis>
	    messages can invalidate the output of the test.  This
	    usually requires a human to examine the output to
	    determine what really happened -- and to improve the test
	    case.</para></listitem>

	    <listitem><para>A test depends on a previous test, which
	    has failed.</para></listitem>

	    <listitem><para>The test was set up
	    incorrectly.</para></listitem>
	  </itemizedlist>

	<variablelist>
	  <varlistentry>
	    <term>UNTESTED</term>
	    <listitem><para>A test was not run.  This is a placeholder
	    used when there is no real test case
	    yet.</para></listitem>
	  </varlistentry>
	</variablelist>

	<variablelist>
	  <varlistentry>
  	    <term>UNSUPPORTED</term>
	    <listitem><para>There is no support for the tested case.  This may
  	    mean that a conditional feature of an operating system, or of a
  	    compiler, is not implemented.  &dj; also uses this message when
  	    a testing environment (often a ``bare board'' target) lacks basic
  	    support for compiling or running the test case.  For example, a
  	    test for the system subroutine <emphasis>gethostname</emphasis>
  	    would never work on a target board running only a boot
  	    monitor.</para></listitem>
	  </varlistentry>
	</variablelist>

        <para>&dj; uses the same output procedures to produce these messages
	for all testsuites and these procedures are already known to conform
	to POSIX 1003.3.  For a &dj; testsuite to conform to POSIX 1003.3,
	you must avoid the <emphasis>setup_xfail</emphasis> procedure as
	described in the <emphasis>PASS</emphasis> section above and you must
	be careful to return <emphasis>UNRESOLVED</emphasis> where appropriate,
	as described in the <emphasis>UNRESOLVED</emphasis> section
	above.</para>
     </sect2>

  <sect2 id="installation" xreflabel="Installation">
    <title>Installation</title>

    <para>Refer to the <filename>INSTALL</filename> in the source
    distribution for detailed installation instructions.  Note that
    there is no compilation step as with many other GNU packages, as
    &dj; consists of interpreted code only.</para>

    <para>Save for its own small testsuite, the &dj; distribution does
      not include any testsuites. Testsuites for the various GNU
      development tools are included with those packages. After
      configuring the top-level &dj; directory, unpack and configure
      the test directories for the tools you want to test; then, in
      each test directory, run <emphasis>make check</emphasis> to
      build auxiliary programs required by some of the tests, and run
      the test suites.</para>
  </sect2>

  </sect1>

  <!-- include the user manual -->
  &user;

  <!-- include the reference manual -->
  &ref;

</article>

<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
sgml-omittag:t
sgml-shorttag:t
sgml-namecase-general:t
sgml-general-insert-case:lower
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:nil
sgml-parent-document:nil
sgml-exposed-tags:nil
sgml-local-catalogs:nil
sgml-local-ecat-files:nil
End:
-->