path: root/doc/test-writing-guidelines.txt

LTP Test Writing Guidelines
===========================

This document describes LTP guidelines and LTP test interface and is intended
for anybody who want to write or modify a LTP testcase. It's not a definitive
guide and it's not, by any means, a substitute for common sense.


1. General Rules
----------------

1.1 Simplicity
~~~~~~~~~~~~~~

For all it's worth keep the testcases simple or better as simple as possible.
The kernel and libc are tricky beasts and the complexity imposed by their
interfaces is quite high. Concentrate on the interface you want to test and
follow the UNIX philosophy. It's a good idea to make the test as
self-contained as possible too (it should not depend on tools or libraries
that are not widely available).

Do not reinvent the wheel

* Use LTP standard interface
* Do not add custom PASS/FAIL reporting functions
* Do not write Makefiles from scratch,
  use LTP build system instead, etc.
* ...

1.2 Code duplication
~~~~~~~~~~~~~~~~~~~~

Copy & paste is a good servant but very poor master. If you are about to copy a
large part of the code from one testcase to another, think what would happen if
you find bug in the code that has been copied all around the tree. What about
moving it to a library instead?

The same goes for short but complicated parts, whenever you are about to copy &
paste a syscall wrapper that packs arguments accordingly to machine
architecture or similarly complicated code, put it into a header instead.

1.3 Coding style
~~~~~~~~~~~~~~~~

1.3.1 C coding style
^^^^^^^^^^^^^^^^^^^^

LTP adopted Linux kernel coding style. If you aren't familiar with its rules
locate 'linux/Documentation/CodingStyle' in the kernel sources and read it,
it's a well written introduction.

There is also a checkpatch (see 'linux/scripts/checkpatch.pl') script that can
be used to check your patches before the submission.

NOTE: If checkpatch does not report any problems, the code still may be wrong
      as the tool only looks for common mistakes.

1.3.2 Shell coding style
^^^^^^^^^^^^^^^^^^^^^^^^

When writing testcases in shell write in portable shell only, it's a good idea
to try to run the test using alternative shell (alternative to bash, for
example dash) too.

Here are some common sense style rules for shell

* Keep lines under 80 chars

* Use tabs for indentation

* Keep things simple, avoid unnecessary subshells

* Don't do confusing things (i.e. don't name your functions like common shell
  commands, etc.)

* Quote variables

* Be consistent

1.4 Commenting code
~~~~~~~~~~~~~~~~~~~

Comments can sometimes save you day but they can easily do more harm than
good. There has been several cases where comments and actual implementation
were drifting slowly apart which yielded into API misuses and hard to find
bugs. Remember there is only one thing worse than no documentation, wrong
documentation.

Generally everybody should write code that is obvious (which unfortunately
isn't always possible). If there is a code that needs to be commented keep it
short and to the point. Never ever comment the obvious.

In case of LTP testcases it's customary to add a paragraph with highlevel test
description somewhere at the beginning of the file (usually right under the GPL
header). This helps other people to understand the overall goal of the test
before they dive into the technical details.

1.5 Backwards compatibility
~~~~~~~~~~~~~~~~~~~~~~~~~~~

LTP test should be as backward compatible as possible. Think of an enterprise
distributions with long term support (more than five years since the initial
release) or of an embedded platform that needs to use several years old
toolchain supplied by the manufacturer.

Therefore LTP test for more current features should be able to cope with older
systems. It should at least compile fine and if it's not appropriate for the
configuration it should return 'TCONF' (see test interface description below).

There are several types of checks we use:

The *configure script* is usually used to detect availability of a function
declarations in system headers. It's used to disable tests at compile time.

The *tst_kvercmp()* which is runtime kernel version detection and comparison
and is used to disable tests at runtime.

Checking the *errno* value is another type of runtime check. Most of the
syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented
or was disabled upon kernel compilation.

Sometimes it also makes sense to define a few macros instead of creating
configure test. One example are Linux specific POSIX clock ids in
'include/lapi/posix_clocks.h'.

1.6 Dealing with messed up legacy code
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

LTP contains a lot of old and messy code and we are cleaning it up as fast as
we can but despite the efforts there is still a lot. If you start modifying
old or a messed up testcase and your changes are more complicated than simple
typo fixes you should do a cleanup first (in a separate patch). It's easier to
review the changes if you separate the formatting fixes from the changes that
affects the test behavior.

The same goes for moving files. If you need a rename or move file do it in a
separate patch.

1.7 License
~~~~~~~~~~~

Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or
any later version).

2. Writing a testcase
---------------------

2.1 LTP Structure
~~~~~~~~~~~~~~~~~

The structure of LTP is quite simple. Each test is a binary written either in
portable shell (no bashism) or C. The test gets a configuration via
environment variables and/or command line parameters, it prints additional
information into the stdout and reports overall success/failure via the exit
value.

Tests are generally placed under the 'testcases/' directory. Everything that
is a syscall or (slightly confusingly) libc syscall wrapper goes under
'testcases/kernel/syscalls/'. Then there is 'testcases/open_posix_testsuite'
which is a well maintained fork of the upstream project that has been dead
since 2005 and also a number of directories with tests for more specific
features.

2.1.1 Runtest Files
^^^^^^^^^^^^^^^^^^^

The list of tests to be executed is stored in runtest files under the
'runtest/' directory. The default set of runtest files to be executed is
stored in 'scenario_groups/default'. When you add a test you should add
corresponding entries into some runtest file(s) as well.

For syscall tests (these placed under 'testcases/kernel/syscalls/') use
'runtest/syscalls' file, for kernel related tests for memory management we
have 'runtest/mm', etc.

IMPORTANT: The runtest files should have one entry per a test. Creating a
           wrapper that runs all your tests and adding it as a single test
           into runtest file is strongly discouraged.

2.1.2 Datafiles
^^^^^^^^^^^^^^^

If your test needs datafiles to work, these should be put into a subdirectory
named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to
do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the
'datafiles/Makefile').

You can obtain path to datafiles via $LTP_DATAROOT provided by test.sh
'$LTP_DATAROOT/...'
or via C function 'tst_dataroot()' provided by libltp:

[source,c]
-------------------------------------------------------------------------------
const char *dataroot = tst_dataroot();
-------------------------------------------------------------------------------

Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...',
but '$LTP_DATAROOT' and 'tst_dataroot()' are preffered as these can be used
when running testcases directly in git tree as well as from install
location.

The path is constructed according to these rules:

1. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID'
2. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles'
   (where '$STARTWD' is initial working directory as recorded by 'tst_tmdir()')
3. else return '$CWD/datafiles'

See 'testcases/commands/ade/ldd/ldd01' for example.

2.1.3 Subexecutables
^^^^^^^^^^^^^^^^^^^^

If you test needs to execute a binary, place it in the same directory as the
testcase and name the file starting with testname_ ('TCID_' see below).
Once the test is executed by the framework, the path to the directory with all
LTP binaries is added to the '$PATH' and you can execute it just by its name.

TIP: If you need to execute such test from the LTP tree, you can add path to
     current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'.

2.2 Writing a test in C
~~~~~~~~~~~~~~~~~~~~~~~

2.2.1 Basic test structure
^^^^^^^^^^^^^^^^^^^^^^^^^^

Let's start with an example, following code is a simple test for a getenv().

[source,c]
-------------------------------------------------------------------------------
/*
 * This is test for basic functionality of getenv().
 *
 *  - create an env variable and verify that getenv() can get get it
 *
 *  - call getenv() with nonexisting variable name, check that it returns NULL
 */

#include "test.h"

char *TCID = "getenv01";
int TST_TOTAL = 2;

#define TEST_ENV "LTP_TEST_ENV"
#define TEST_NE_ENV "LTP_TEST_THIS_DOES_NOT_EXIST"
#define TEST_ENV_VAL "val"

static void setup(void)
{
	if (setenv(TEST_ENV, TEST_ENV_VAL, 1))
		tst_brkm(TBROK | TERRNO, NULL, "setenv() failed");
}

static void test(void)
{
	char *ret;

	ret = getenv(TEST_ENV);

	if (ret) {
		if (!strcmp(ret, TEST_ENV_VAL))
			tst_resm(TPASS, "getenv(" TEST_ENV ") = '"
			         TEST_ENV_VAL "'");
		else
			tst_resm(TFAIL, "getenv(" TEST_ENV ") = '%s', "
			         "expected '" TEST_ENV_VAL "'", ret);
	} else {
		tst_resm(TFAIL, "getenv(" TEST_ENV ") = NULL");
	}

	ret = getenv(TEST_NE_ENV);

	if (ret)
		tst_resm(TFAIL, "getenv(" TEST_NE_ENV ") = '%s'", ret);
	else
		tst_resm(TPASS, "getenv(" TEST_NE_ENV ") = NULL");
}

int main(int argc, char *argv[])
{
	const char *msg;
	int lc;

	if ((msg = parse_opts(argc, argv, NULL, NULL)))
		tst_brkm(TBROK, NULL, "OPTION PARSING ERROR - %s", msg);

	setup();

	for (lc = 0; TEST_LOOPING(lc); lc++)
		test();

	tst_exit();
}
-------------------------------------------------------------------------------

Each test must define 'TCID' and 'TST_TOTAL'.

'TCID' defines test name (usually syscall/libcall name + number) and is used
as a base for temporary directory name (if 'tst_tmpdir()' is used). In most of
the cases the 'TCID' is the same as test filename (without the extension).

'TST_TOTAL' defines total number of tests.

NOTE: The test should report 'TST_TOTAL' PASSES/FAILS on each iteration.

The overall test initialization is usually done in a 'setup()' function and
the overall cleanup is done in a 'cleanup()' function. Here 'cleanup()' is
omitted as the test don't have to clean anything up. Usually it's called right
before the 'tst_exit()' to do the cleanup and passed to library functions that
can exit the test execution, for example 'tst_brkm()', so that the test can do
a cleanup before it exits.

WARNING: Never pass a cleanup function to functions called from cleanup.
         If you don't see why this is a problem you are not ready to write
	 a testcase yet.

WARNING: Don't use 'tst_brkm()' in 'cleanup()' unless you really want to exit
         the cleanup prematurely. See discussion below for further
	 information.

The 'parse_opts()' parses the test command line arguments, it's important to
use it even if the test has no (other than default) parameters.

The last important thing is the 'TEST_LOOPING()' macro, each test has standard
options so that it can be executed N times or for M seconds.

NOTE: The 'test()' function must work correctly even if executed several times
      from test main loop.

A word about the cleanup() callback
+++++++++++++++++++++++++++++++++++

There are a few rules that needs to be followed in order to write correct
cleanup() callback.

1. Don't call callback() from within the callback().
(This includes passing callback pointer to library 'tst_' functions.)

2. Make sure to free resources in the reverse order they were
   initialized. (Some of the steps may not depend on others and everything
   will work if there were swapped but let's keep it in order.)

3. Free only resources that were initialized.

The third rule needs a bit more detailed discussion. Consider, for example,
test setup below which is example of a setup that prepares temporary
directory, two file descriptors and allocates a buffer.

[source,c]
-------------------------------------------------------------------------------
static int fd0, fd1;
static void *buf;
#define BUFSIZE (1024 * 1024)

static void setup(void)
{
	tst_require_root();
	tst_tmpdir();

	fd0 = SAFE_OPEN(cleanup, "test_file1", O_CREAT | O_RDWR, 0666);
	SAFE_UNLINK(cleanup, "test_file1");
	fd1 = SAFE_OPEN(cleanup, "test_file2", O_CREAT | O_RDWR, 0666);
	SAFE_UNLINK(cleanup, "test_file2");

	buf = SAFE_MALLOC(cleanup, BUFSIZE);
}
-------------------------------------------------------------------------------

In this case the 'cleanup()' function may be entered in five different states:

* The first 'SAFE_OPEN()' has failed, temporary directory is created
  no files are open and +fd0 == -1+, +fd1 == 0+ and +buf == NULL+.

* The first 'SAFE_UNLINK()' has failed, +fd0+ holds file descriptor and
  +fd1 == 0+.

* The second 'SAFE_OPEN()' has failed, +fd0+ holds file descriptor and
  +fd1 == -1+.

* The second 'SAFE_UNLINK()' or 'SAFE_MALLOC()' has failed and both 'fd0' and
  'fd1' holds file descriptors, +buf+ is still +NULL+.

* The 'cleanup()' was called at the end of the test, all +fd0+, +fd1+ and
  +buf+ are initialized.

The 'cleanup()' functions should be able to cope with all scenarios. In this
case following code will do:

[source,c]
-------------------------------------------------------------------------------
static void cleanup(void)
{
	free(buf);

	if (fd1 > 0)
		close(fd1);

	if (fd0 > 0)
		close(fd0);

	tst_rmdir();
}
-------------------------------------------------------------------------------

2.2.2 Basic test interface
^^^^^^^^^^^^^^^^^^^^^^^^^^

[source,c]
-------------------------------------------------------------------------------
void tst_resm(int ttype, char *arg_fmt, ...);
-------------------------------------------------------------------------------

Printf-like function to report test result, it's mostly used with ttype:

|==============================
| 'TPASS' | Test has passed.
| 'TFAIL' | Test has failed.
| 'TINFO' | General message.
|==============================

The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print
'errno', 'TEST_ERRNO' respectively.

[source,c]
-------------------------------------------------------------------------------
void tst_brkm(int ttype, void (*cleanup)(void), char *arg_fmt, ...);
-------------------------------------------------------------------------------

Printf-like function to report error and exit the test, it can be used with ttype:

|============================================================
| 'TBROK' | Something has failed in test preparation phase.
| 'TCONF' | test is not appropriate for current configuration
            (syscall not implemented, unsupported arch, ...)
| 'TFAIL' | test has failed
|============================================================

The 'ttype' can be combined bitwise with 'TERRNO' or 'TTERRNO' to print
'errno', 'TEST_ERRNO' respectively.

[source,c]
-------------------------------------------------------------------------------
void tst_require_root(void (*cleanup)(void));
-------------------------------------------------------------------------------

Abort the test if it's not executed with root privileges. If needed this should
be one of the first checks in the test 'setup()'.

WARNING: The cleanup parameter is deprecated and should always be 'NULL'.


[source,c]
-------------------------------------------------------------------------------
void tst_exit(void);
-------------------------------------------------------------------------------

Exits the tests, note that this function has no parameters, the PASSES/FAILS
reported by the 'tst_resm()' interfaces were stored and are used for the exit
value.

[source,c]
-------------------------------------------------------------------------------
const char *tst_strsig(int sig);
-------------------------------------------------------------------------------

Return the given signal number's corresponding string.


[source,c]
-------------------------------------------------------------------------------
const char *tst_strerrno(int err);
-------------------------------------------------------------------------------

Return the given errno number's corresponding string.


2.2.3 Test temporary directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If the test needs to create files (which is common case) it must create a test
temporary directory in LTP temp directory and work with files only inside of
this directory. Happily there are easy to use library functions exactly for
this purpose.

[source,c]
-------------------------------------------------------------------------------
void tst_tmpdir(void);
-------------------------------------------------------------------------------

Creates a directory with a unique name (based on the test 'TCID') under the
LTP temporary directory (defaults to '/tmp/') and changes the test working
directory to it. It's usually called from the test 'setup()'.

[source,c]
-------------------------------------------------------------------------------
void tst_rmdir(void);
-------------------------------------------------------------------------------

Removes the directory recursively. It's usually called from test 'cleanup()'.

It's important to close all file descriptors (that point to files in test
temporary directory, even the unlinked ones) before the test calls
'tst_rmdir()' otherwise the test may break on NFS mounted temp dir (look for
"NFS silly rename").

[[2.2.4]]
2.2.4 Safe macros
^^^^^^^^^^^^^^^^^

Safe macros aim to simplify error checking in test preparation. Instead of
calling system API functions, checking for their return value and aborting the
test if the operation has failed, you just use corresponding safe macros.

Use them whenever it's possible.

NOTE: Have a look at the the xref:2.2.7[doing real test in the child process]
      if you want to use safe macros from within a child process.

Instead of writing:

[source,c]
-------------------------------------------------------------------------------
	fd = open("/dev/null", O_RDONLY);
	if (fd < 0)
		tst_brkm(TBROK | TERRNO, cleanup, "opening /dev/null failed");
-------------------------------------------------------------------------------

You write just:

[source,c]
-------------------------------------------------------------------------------
	fd = SAFE_OPEN(cleanup, "/dev/null", O_RDONLY);
-------------------------------------------------------------------------------

They can also simplify reading and writing of sysfs files, you can, for
example, do:

[source,c]
-------------------------------------------------------------------------------
	SAFE_FILE_SCANF(cleanup, "/proc/sys/kernel/pid_max", "%lu", &pid_max);
-------------------------------------------------------------------------------

See 'include/safe_macros.h', 'include/safe_stdio.h' and
'include/safe_file_ops.h' for a complete list.

NOTE: It's wise NOT to use safe macros in test cleanup(). This is because
      all safe macros call tst_brkm(), which exits the test immediately, making
      the cleanup() exit prematurely. (Actually, this is hacked around in
      the test library at the moment so that the cleanup() will finish, but
      the hack will be removed in the future).

2.2.5 Runtime kernel version detection
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[source,c]
-------------------------------------------------------------------------------
int tst_kvercmp(int r1, int r2, int r3);

struct tst_kern_exv {
        char *dist_name;
        char *extra_ver;
};

int tst_kvercmp2(int r1, int r2, int r3, struct tst_kern_exv *vers);
-------------------------------------------------------------------------------

These two functions are intended for runtime kernel version detection. They
parse the output from 'uname()' and compare it to the passed values.

The return value is similar to the 'strcmp()' function, i.e. zero means equal,
negative value means that the kernel is older than than the expected value and
positive means that it's newer.

The second function 'tst_kvercmp2()' allows for specifying per-vendor table of
kernel versions as vendors typically backport fixes to their kernels and the
test may be relevant even if the kernel version does not suggests so. See
'testcases/kernel/syscalls/inotify/inotify04.c' for example usage.

2.2.6 Fork()-ing
^^^^^^^^^^^^^^^^

Be wary that if the test forks and there were messages printed by the tst_*
interfaces, the data may still be in kernel buffers and these are NOT flushed
automatically.

This happens when 'stdout' gets redirected to a file. In this case, the
'stdout' is not line buffered, but block buffered, and buffered messages will be
printed by the parent and each of the children.

To avoid that, you should either call 'tst_flush()' right before the 'fork()',
or even better - use 'tst_fork()' instead.

[[2.2.7]]
2.2.7 Doing real test in the child process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you have to do the test in a child process you have two possibilities:

* You can use test interface ('tst_resm()', 'tst_brkm()', 'tst_exit()', ...)
  but there are some rules to obey:

** If you use 'tst_resm()' to record test the results in child process, you
   *MUST* use either 'tst_exit()' or 'tst_brkm()' to exit the child process.

** If you use 'tst_brkm()' with a non-NULL cleanup argument in the child
   process, you're strongly recommended to create a separate cleanup function
   for child process that will only clean up what has been allocated in child
   process. Otherwise the 'cleanup()' will be called both from the parent and
   child and the test will likely fail (for example in 'tst_rmdir()' because
   both parent and child will attempt to remove the directory and one of them
   will fail).

** The same applies to xref:2.2.4[safe macros] because they call 'tst_brkm()'
   when anything goes wrong.

* Or you can stick to plain old 'exit()' with 'TPASS', 'TBROK', 'TFAIL' and
  'TCONF' constants.

Then you call 'tst_record_childstatus()' to records the result of the test
(done in child process) which propagates the child result (child exit status)
to the parent process correctly.

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

void tst_record_childstatus(void (*cleanup)(void), pid_t child);
-------------------------------------------------------------------------------

This function does a 'waitpid()' on child process and record child process's
return value into the overall test result.

2.2.8 Fork() and Parent-child synchronization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

As LTP tests are written for Linux, most of the test involves fork()-ing and
parent-child process synchronization. We have a checkpoint library code
that works even for two different processes, all they need to is to run with
the same working directory (they use FIFO for synchronization). The checkpoint
interface provides two pairs of signal and wait functions. One pair to be used
to signal child from parent and second to signal parent from child.

Checkpoint is represented by 'struct tst_checkpoint', which has to be
initialized before first usage and FIFO has to be created. This is
usually done in parent process in single step by calling
'tst_checkpoint_create()'.

Child processes created via fork() are ready to use tst_checkpoint_*
synchronization functions, as they inherited already initialized
'struct tst_checkpoint'.

Child processes started via exec*, or any other process can use already
created FIFO, provided they initialize 'struct tst_checkpoint' first by
call to 'tst_checkpoint_init()'. This function does not create any FIFO,
it relies on fact, that it was already created by some other process.
Note, that if you use multiple FIFOs in this scenario, these should be
initialized in same order as in process you are sychronizing against.

IMPORTANT: Be aware, that following scenario is _NOT_ safe when using
           only single checkpoint. You are advised to use two checkpoints
           in this case.

* tst_checkpoint_signal_child() followed by tst_checkpoint_parent_wait()

             parent            |            child
-------------------------------+-------------------------------
                               | tst_checkpoint_child_wait()
                               |   blocking read, waits until
                               |   parent opens write side
tst_checkpoint_signal_child()  |
  NONBLOCK write               |
                               |   child is now able to read
                               |   from FIFO
tst_checkpoint_parent_wait()   |
  NONBLOCK read parent able to |
  read from FIFO and can race  |
  with read in child           |
                               | tst_checkpoint_signal_parent()
-------------------------------+--------------------------------

For the details of the interface, look into the 'include/tst_checkpoint.h' and
'lib/tests/tst_checkpoint_*'.

There is also an interface that allows parent to wait until child is blocked
in kernel (for example waits in 'pause()'), see 'include/tst_process_state.h'
for more.

2.2.9 Signal handlers
^^^^^^^^^^^^^^^^^^^^^

If you need to use signal handlers, keep the code short and simple. Don't
forget that the signal handler is called asynchronously and can interrupt the
code execution at any place.

This means that problems arise when global state is changed both from the test
code and signal handler, which will occasionally lead to:

* Data corruption (data gets into inconsistent state), this may happen, for
  example, for any operations on 'FILE' objects.

* Deadlock, this happens, for example, if you call 'malloc(2)', 'free(2)',
  etc. from both the test code and the signal handler at the same time because
  'malloc' has global lock for it's internal data structures. (Be wary that
  'malloc(2)' is used by the libc functions internally too.)

* Any other unreproducible and unexpected behavior.

Quite common mistake is to call 'exit(3)' from a signal handler. Note that this
function is not signal-async-safe as it flushes buffers, etc. If you need to
exit a test immediately from a signal handler use '_exit(2)' instead.

TIP: See 'man 7 signal' for the list of signal-async-safe functions.

If a signal handler sets a variable, its declaration must be 'volatile',
otherwise compiler may misoptimize the code, because the variable may not be
changed in the code flow analysis. There is 'sig_atomic_t' type defined in C99
but this one DOES NOT imply volatile (it's just a typedef to int). So the
correct type for a flag that is changed from a signal handler is either
'volatile int' or 'volatile sig_atomic_t'.

2.2.10 Kernel Modules
^^^^^^^^^^^^^^^^^^^^^

There are certain cases where the test needs a kernel part and userspace part,
happily, LTP can build a kernel module and then insert it to the kernel on test
start for you. See 'testcases/kernel/device-drivers/block' for details.

2.2.11 Useful macros
^^^^^^^^^^^^^^^^^^^^^

[source,c]
-------------------------------------------------------------------------------
ARRAY_SIZE(arr)
-------------------------------------------------------------------------------

Returns the size of statically defined array, i.e.
'(sizeof(arr) / sizeof(*arr))'

[source,c]
-------------------------------------------------------------------------------
LTP_ALIGN(x, a)
-------------------------------------------------------------------------------

Aligns the x to be next multiple of a. The a must be power of 2.

2.2.12 Filesystem type detection
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some tests are known to fail on certain filesytems (you cannot swap on TMPFS,
there are unimplemented fcntl() etc.).

If your test needs to be skipped on certain filesystems, use the interface
below:

[source,c]
-------------------------------------------------------------------------------
#include "tst_fs_type.h"

	/*
	 * Unsupported only on NFS.
	 */
	if (tst_fs_type(cleanup, ".") == TST_NFS_MAGIC) {
		tst_brkm(TCONF, cleanup,
		         "Test not supported on NFS filesystem");
	}

	/*
	 * Unsupported on NFS, TMPFS and RAMFS
	 */
	long type;

	switch ((type = tst_fs_type(cleanup, "."))) {
	case TST_NFS_MAGIC:
	case TST_TMPFS_MAGIC:
	case TST_RAMFS_MAGIC:
		tst_brkm(TCONF, cleanup, "Test not supported on %s filesystem",
		         tst_fs_type_name(type));
	break;
	}
-------------------------------------------------------------------------------

2.2.13 Thread-safety in the LTP library
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It is safe to use library 'tst_' functions in multi-threaded tests.
Synchronization mechanism is enabled only if the test is linked with pthread
library, otherwise no-op stubs (defined in libc) are used.

Also, LTP has a helper "TST_DECLARE_ONCE_FN" macro to declare a function which
must be run only once (e.g. init or cleanup function), but might be called by
multiple threads at the same time. See example below:

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

...

void do_cleanup(void)
{
	/* cleanup code */
	...
}
TST_DECLARE_ONCE_FN(cleanup, do_cleanup);

...

void test01(void)
{
	sfd = socket(AF_INET, SOCK_STREAM, 0);
	if (sfd == -1)
		tst_brkm(TBROK, cleanup, "Failed to create a socket");
}
-------------------------------------------------------------------------------

2.2.14 Acquiring a block device
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some tests needs a block device (inotify tests, syscall 'EROFS' failures,
etc.). For your convenience LTP library includes a function to acquire and
release a testing device.

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

const char *tst_acquire_device(void (cleanup_fn)(void));
-------------------------------------------------------------------------------

This function returns a path to small block device suitable for formatting and
mounting.

WARNING: You can acquire only one device, calling it for second time without
         releasing the device first will abort the test.

In case that 'LTP_DEV' is passed to the test in environment, the function just
checks that the file exists and that it's a block device.

Otherwise a temporary file is created and attached to a free loop device.

In case that that there are no unused loop devices, 'NULL' is returned. The
test should skip the particular part of the test that needs the device with
'TCONF'.

WARNING: Because 'tst_acquire_device()' may create temporary file, you must
         call 'tst_tmpdir()' before you call 'tst_acquire_device()'

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

void tst_release_device(void (cleanup_fn)(void), const char *dev);
-------------------------------------------------------------------------------

Releases the device acquired by 'tst_acquire_device()'.

2.2.15 Formatting a device with a filesystem
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

const char *tst_dev_fs_type(void);
-------------------------------------------------------------------------------

Returns name of filesystem that should be used for tests. Unless is your test
designed to test specific filesystem you should use this function to select
filesystem for the 'tst_mkfs()'.

NOTE: The default filesytem is hardcoded to 'ext2' in the sources and can be
      overridden by setting 'LTP_DEV_FS_TYPE' environment variable.

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

void tst_mkfs(void (cleanup_fn)(void), const char *dev,
              const char *fs_type, const char *const fs_opts[]);
-------------------------------------------------------------------------------

This function takes a path to a device, filesystem type and an array of extra
options passed to mkfs.

The extra options 'fs_opts' should either be 'NULL' if there are none, or a
'NULL' terminated array of strings such as '{"-b", "1024", NULL}'.

2.2.16 Verifying a filesystem's free space
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some tests have size requirements for the filesystem's free space. If these
requirements are not satisfied, the tests are not appropriate to run.
Especially when you run runltp with "-z" option to specify a big block device,
different tests have different requirements for the free space of this block
device.

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

int tst_fs_has_free(void (*cleanup)(void), const char *path,
		    unsigned int size, unsigned int mult);
-------------------------------------------------------------------------------

The 'tst_fs_has_free()' function returns 1 if there is enough space and 0 if
there is not.

The 'path' is the pathname of any directory/file within a filesystem you want to
verify its free space of.

The 'mult' is a multiplier, one of 'TST_BYTES', 'TST_KB', 'TST_MB' or 'TST_GB'.

The required free space is calculated by 'size * mult', e.g.
'tst_fs_has_free(cleanup, "/tmp/testfile", 64, TST_MB)' will return 1 if the
filesystem, which '"/tmp/testfile"' is in, has 64MB free space at least, and 0
if not.

2.2.17 Getting the maximum number of links to a regular file or directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some tests need to know the maximum count of links to a regular file or
directory, such as 'rename(2)' or 'linkat(2)' to test 'EMLINK' error.

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

int tst_fs_fill_hardlinks(void (*cleanup)(void), const char *dir);
-------------------------------------------------------------------------------

Try to get maximum count of hard links to a regular file inside the 'dir'.

NOTE: This number depends on the filesystem 'dir' is on.

This function uses 'link(2)' to create hard links to a single file until it
gets 'EMLINK' or creates 65535 links. If the limit is hit, the maximum number of
hardlinks is returned and the 'dir' is filled with hardlinks in format
"testfile%i", where i belongs to [0, limit) interval. If no limit is hit or if
'link(2)' failed with 'ENOSPC' or 'EDQUOT', zero is returned and previously
created files are removed.

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

int tst_fs_fill_subdirs(void (*cleanup)(void), const char *dir);
-------------------------------------------------------------------------------

Try to get maximum number of subdirectories in directory.

NOTE: This number depends on the filesystem 'dir' is on. For current kernel,
subdir limit is not available for all filesystems (available for ext2, ext3,
minix, sysv and more). If the test runs on some other filesystems, like ramfs,
tmpfs, it will not even try to reach the limit and return 0.

This function uses 'mkdir(2)' to create directories in 'dir' until it gets
'EMLINK' or creates 65535 directories. If the limit is hit, the maximum number
of subdirectories is returned and the 'dir' is filled with subdirectories in
format "testdir%i", where i belongs to [0, limit - 2) interval (because each
newly created dir has two links already - the '.' and the link from parent
dir). If no limit is hit or if 'mkdir(2)' failed with 'ENOSPC' or 'EDQUOT',
zero is returned and previously created directories are removed.

2.2.18 Getting an unused PID number
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some tests require a 'PID', which is not used by the OS (does not belong to
any process within it). For example, kill(2) should set errno to 'ESRCH' if
it's passed such 'PID'.

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

pid_t tst_get_unused_pid(void (*cleanup)(void));
-------------------------------------------------------------------------------

Return a 'PID' value not used by the OS or any process within it.

2.2.19 Umounting devices
^^^^^^^^^^^^^^^^^^^^^^^^

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

int tst_umount(const char *path);
-------------------------------------------------------------------------------

Since various desktop deamons (gvfsd-trash is known for that) may be stupid
enough to probe all newly mounted filesystem which results in 'umount(2)'
failing with 'EBUSY' LTP library includes 'tst_umount()' function that behaves
exactly as 'umount(2)' but retries several times on a failure.

IMPORTANT: All testcases should use 'tst_umount()' instead of 'umount(2)' to
           umount filesystems.

2.2.20 Running executables
^^^^^^^^^^^^^^^^^^^^^^^^^^

[source,c]
-------------------------------------------------------------------------------
#include "test.h"

int tst_run_cmd(void (cleanup_fn)(void),
               const char *const argv[],
	       const char *stdout_path,
	       const char *stderr_path,
	       int pass_exit_val);
-------------------------------------------------------------------------------

'tst_run_cmd' is a wrapper for 'vfork() + execvp()' which provides a way
to execute an external program.

'argv[]' is a NULL-terminated array of strings starting with the program name
which is followed by optional arguments.

A non-zero 'pass_exit_val' makes 'tst_run_cmd' return the program exit code to
the caller. A zero for 'pass_exit_val' makes 'tst_run_cmd' exit the tests
on failure and call 'cleanup_fn' (if not NULL) beforehand.

'stdout_path' and 'stderr_path' determine where to redirect the program
stdout and stderr I/O streams.

.Example
[source,c]
-------------------------------------------------------------------------------
#include "test.h"

const char *const cmd[] = { "ls", "-l", NULL };

...
	/* Store output of 'ls -l' into log.txt */
	tst_run_cmd(cleanup, cmd, "log.txt", NULL, 0);
...
-------------------------------------------------------------------------------

2.3 Writing a testcase in shell
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

LTP supports testcases to be written in a portable shell too.

There is a shell library modeled closely to the C interface (the source is
located at 'testcases/lib/test.sh') and is installed to the same directory as
the rest of the LTP test binaries.

WARNING: All identifiers starting with TST_ or tst_ are reserved for the
         'test.sh' library.

2.3.1 Basic shell test structure
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# This is a basic test for true shell buildin
#

TCID=true01
TST_TOTAL=1
. test.sh

true
ret=$?

if [ $ret -eq 0 ]; then
	tst_resm TPASS "true returned 0"
else
	tst_resm TFAIL "true returned $ret"
fi

tst_exit
-------------------------------------------------------------------------------

TIP: To execute this test the 'test.sh' library must be in '$PATH'. If you are
     executing the test from a git checkout you can run it as
     'PATH="$PATH:../../lib" ./foo01.sh'

WARNING: Do not forget to add the 'tst_exit' at the end of the test,
         otherwise the test return value would be the return value of last
	 executed command.

2.3.2 Basic test interface
^^^^^^^^^^^^^^^^^^^^^^^^^^

Following functions similar to the LTP C interface are available.

* tst_resm()
* tst_brkm()
* tst_exit()
* tst_require_root()
* tst_tmpdir()
* tst_rmdir()
* tst_fs_has_free()

There is one more function called 'tst_check_cmds()' that gets unspecified
number of parameters and asserts that each parameter is a name of an
executable in '$PATH' and exits the test with 'TCONF' on first missing.

.tst_fs_has_free
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh

...

# whether current directory has 100MB free space at least.
if ! tst_fs_has_free . 100MB; then
	tst_brkm TCONF "Not enough free space"
fi

...
-------------------------------------------------------------------------------

The tst_fs_has_free shell interface returns 0 if the specified free space is
satisfied, 1 if not, and 2 on error.

The second argument supports suffixes kB, MB and GB, the default unit is Byte.

2.3.3 Cleanup
^^^^^^^^^^^^^

Due to differences between C and shell, the cleanup callback is done using a
'TST_CLEANUP' shell variable that, if not empty, is evaluated before the test
exits (either after calling 'tst_exit()' or 'tst_brkm()'). See example below.

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# Test cleanup example
#

TCID=true01
TST_TOTAL=1
. test.sh

cleanup()
{
	tst_rmdir
}

tst_tmpdir
TST_CLEANUP=cleanup

# Do the test here

tst_exit
-------------------------------------------------------------------------------

2.3.4 Restarting daemons
^^^^^^^^^^^^^^^^^^^^^^^^

Restarting system daemons is a complicated task for two reasons.

* There are different init systems
  (SysV init, systemd, etc...)

* Daemon names are not unified between distributions
  (apache vs httpd, cron vs crond, various syslog variations)

To solve these problems LTP has 'testcases/lib/daemonlib.sh' library that
provides functions to start/stop/query daemons as well as variables that store
correct daemon name.

.Supported operations
|==============================================================================
| start_daemon()   | Starts daemon, name is passed as first parameter.
| stop_daemon()    | Stops daemon, name is passed as first parameter.
| restart_daemon() | Restarts daemon, name is passed as first parameter.
| status_daemon()  | Returns daemon status, TODO: what is return value?
|==============================================================================

.Variables with detected names
|==============================================================================
| CROND_DAEMON | Cron daemon name (cron, crond).
| SYSLOG_DAEMON | Syslog daemon name (syslog, syslog-ng, rsyslog).
|==============================================================================

.Cron daemon restart example
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
#
# Cron daemon restart example
#
TCID=cron01
TST_COUNT=1
. test.sh
. daemonlib.sh

...

restart_daemon $CROND_DAEMON

...

tst_exit
-------------------------------------------------------------------------------


3. Common problems
------------------

This chapter describes common problems/misuses and less obvious design patters
(quirks) in UNIX interfaces. Read it carefully :)

3.1 umask()
~~~~~~~~~~~

I've been hit by this one several times already... When you create files
with 'open()' or 'creat()' etc, the mode specified as the last parameter *is
not* the mode the file is created with. The mode depends on current 'umask()'
settings which may clear some of the bits. If your test depends on specific
file permissions you need either to change umask to 0 or 'chmod()' the file
afterwards or use SAFE_TOUCH() that does the 'chmod()' for you.

3.2 access()
~~~~~~~~~~~

If 'access(some_file, W_OK)' is executed by root, it will return success even
if the file doesn't have write permission bits set (the same holds for R_OK
too). For sysfs files you can use 'open()' as a workaround to check file
read/write permissions. It might not work for other filesystems, for these you
have to use 'stat()', 'lstat()' or 'fstat()'.

3.3 umount() EBUSY
~~~~~~~~~~~~~~~~~~

Various desktop deamons (gvfsd-trash is known for that) may be stupid enough
to probe all newly mounted filesystem which results in 'umount(2)' failing
with 'EBUSY'; use 'tst_umount()' described in 2.2.19 that retries in this case
instead of plain 'umount(2)'.


4. Test Contribution Checklist
------------------------------

1. Test compiles and runs fine (check with -i 10 too)
2. Checkpatch does not report any errors
3. The runtest entires are in place
4. Test files are added into corresponding .gitignore files
5. Patches apply over the latest git


4.1 About .gitignore files
~~~~~~~~~~~~~~~~~~~~~~~~~~

There are numerous '.gitignore' files in the LTP tree. Usually there is a
'.gitignore' file per a group of tests. The reason for this setup is simple.
It's easier to maintain a '.gitignore' file per directory with tests, rather
than having single file in the project root directory. This way, we don't have
to update all the gitignore files when moving directories, and they get deleted
automatically when a directory with tests is removed.