============================================================================ | OpenMP Validation Suite V 3.0 | | High Performance Computing Center, Stuttgart | | High Performance Computing and Tools, University of Houston | | Jan. 2012 | ============================================================================ TABLE OF CONTENTS I INTRODUCTION I.1. Aims and general function I.2. Files and directories II USAGE II.1. First run with make II.2 Where to search for results II.3. Using the runtest script III Adding and modifying tests III.1. The template structure IV Known Issues and Workaround V Contact and Support ------------------------------------------------------------------------------ I. INTRODUCTION ============================================================================== I.1. Aims and general function -------------------------------- The OpenMP validation suite is designed to verify the correct implementation of OpenMP in compilers. It is capable of checking Fortran as well as c compiler. Testing the implementation is based on statistics. Each directive is tested by computing and verifying the result with the already known value. In most cases a wrong implementation can result in the right values. So the tests are run several times. Additionally, the validation suite creates so called crosstests for each directive. These are tests in which either the directive is missing or used with different arguments. If this so called crosstest fails, this indicates strongly that the previous test is capable of testing the directive. Lastly, an orphaned test is also run to determine if the directive being tested is able to correctly run when 'orphaned' from the main function. Essentially, the directive's code is placed into its own function which is called during execution of the main function and often inside a parallel region. I.2. Files and directories ---------------------------- d c directory containing the templates for the c tests d fortran directory containing the templates for the Fortran tests Makefile Makefile containing options for compilation common_utility.f omp_my_sleep.h thread save sleep function omp_testsuite.f Fortran header file omp_testsuite.h autogenerated c-header file ompts-c.conf configuration file for the c tests about how often the tests shall be executed or how large the loop size is ompts_makeHeader.pl perl module for automatically generation of an up to date header file ompts_parserFunctions.pm perl module containing general functions for the parser.pl script ompts_parser.pl script for generating the source code out of the templates ompts_standaloneProc.c framework for the c tests ompts_standaloneProc.f framework for the Fortran tests README the README file you've already found ;-) LICENSE contains license information runtest.pl the frame program of the test suite testlist-f.txt test list containing the available tests for Fortran testlist-c.txt test list containing the available tests for c ------------------------------------------------------------------------------ II. USAGE ============================================================================== II.1. First run with make -------------------------- You can do a first simple run of the testsuite only after one step of configuration: 1) Modify the ompts.conf and ompts-c.conf file, change the number of threads and number of repetitions of each test. 2) Modify the Makefile, uncommenting the CC/FC and CFLAGS/FFLAGS variables for the compiler of your choice. And now you can run the testsuite either for a C compiler with: > make ctest or for a Fortran compiler with: > make ftest II.2. Running custom tests -------------------------- In order to run single tests or custom groups of tests, two make commmands are defined: make stest and make fstest. These two command reference the file customtest.txt when looking for a testlist to use. Simply edit customtest.txt to include the desired test or tests. If customtest.txt contains c tests, > make stest or for fortran tests > make fstest In order to change the number of threads used in the tests, simply edit the Makefile variables MINTHREADS and MAXTHREADS. By default, they are configured to use 2 threads. To change the number of times each test is run, for c tests edit the REPETITIONS variable in the file ompts-c.conf. The LOOPCOUNT and SLEEPTIME variables can also be changed here. For fortran tests, edit the file omp_testsuite.f to change both the LOOPCOUNT and the number of times each test is run. II.3. Understanding the results --------------------------------- When running the testsuite the results will be shown on the screen. If you need the results for further purpose you can use the results.txt, which is a simple list containing the results for each directive in a single line. Each line starts with the name of the directive. Then follows the result of the test given in the percentage of the passed tests. If 100% of the tests passed successfully, the second number gives the result of the corresponding crosstest. Crosstests will only be run if the normal test passes with 100% accuracy. If a crosstest was not run or a test does not exist, it is denotated by a "-". After the results of the normal tests, there follow a series of tests in the orphaned mode. If there were no orphaned tests available this is shown by a "-". If you run the testsuite with different numbers of threads (e.g. using the runtest.pl script) the results are shown in blocks of 4 columns for each number of threads. If a test fails you can find more detailed information in the ompts.log, bin/c/*.out and *.log files. While the ompts.log file contains all compiler error messages for all tests, the *.out and *.log files contain detailed inforamtion on the execution process of the single tests. In the *.out files there are listed all the results of the single executions of the tests. In the *.log files there are error messages of the tests itself. II.4. Cleaning Up ----------------- Because many files are generated for each tested directive, it is often necessary to clean the main directory after a battery of tests. To clean all generated files in the main directory including the results and log files, > make clean To clean only the logs and out files, > make cleanlogs To clean only the results, > make cleanresults II.4. Using the runtest script ------------------------------- So for special purpose you can use the runtest.pl script, which allows a lot more options for the execution process than the execution with make. Using the runtest.pl script is rather easy. You can use the the test suite only after two steps of modifications: 1.) Modify the Makefile to your wishes choosing your compiler and the necessary compiler flags. 2.) If necessary edit one of the test lists (testlist-c.txt) and comment out the tests you do not want to run using # at the beginning of a line. Testlists for Fortran end with -f.txt while test lists for c with -c.txt. And now you can run the test suite either for Fortran using # > ./runtest.pl -lang=fortran -d=fortran testlist-f.txt or for c # > ./runtest.pl -lang=c -d=c testlist-c.txt With the --help option you can show the complete list of options and their explanations. The test results are summarized in cresults or fresults.txt while *.log keep details for individual tests. There is also a file (ompts.log) keeping compilation information. (see section II.2 ) If you don't want to test the directives in orphaned mode you can use the -norphan option. You also can use the runtest.pl script either to compile all tests or run compiled tests e.g. for cross compilation on other platforms. For this there are the options -norun and -nocompile. Happy testing! ------------------------------------------------------------------------------ III. How to add new tests / The structure of test templates ============================================================================== III.1 The template structure ------------------------------ The test suite is based on templates so that you only have one file for test, crosstest and the orphaned versions of them. A) Description of the template structure The syntax of the templates is much like xml. So each test begins with '' and ends with ''. In between there are several other blocks holding information: - In between this tag you can give a description on what the test checks and how it works. - This tag is used to specify the OpenMP-version which includes the tested directive. - Used to specify the directive how it is called in the programming language. - With this tag you can specify other omp directives which are necessary for the correct execution of your test. The directives have to be listed by their directive names as it is called in the programming language. Multiple directives are separated by ','. - In this tag stands the whole source code for the test / crosstest. Each test has to be written as a function. The syntax of the functions differs between C and Fortran: In C it has to take a file pointer 'FILE * logFile' and return an int. If the test has been passed successful it has to return a value unequal to 0. The file pointer can be used to write information into a log file. In Fortran the function takes no argument and the function name must not exceed XX characters. The return value has to be specified using the '' tags. It has to be 1 if the test succeeded and 0 if the test failed. For details see the example. To tell the test suite the name of your test function you have to enclose it into the ' ' tag. If there are differences between test and crosstest you can use the and tag. When generating the test the parser will use the code enclosed in tags and cut out the code written in tags. So you have two possibilities to write your template for test and crosstest: The first way you can write the complete code is to write the test in one tag and later the crosstest in one tag. The second way is to write both tests only by enclosing differing parts in corresponding tags. The first method should be preferred if test and crosstest differ much from each other. The second e.g. if you only want to change a few options like replacing an omp singleprivate clause by an omp private clause or to cut out a directive like omp flush. When you use the first way you have to take care of the function name! You have to declare it twice with ! - This tag can be used if you want to enable your test to check the directive in orphan regions, too. The code enclosed in this part will be written to a separate function which will be called instead. If you have variables which are used outside this region you have to declare them as global variables enclosed in an tag. For further information see the description of the tag. - This tag is used to specify global variables for an orphan region which allow the exchange of values between the main program and the orphaned functions. The usage differs between C and Fortran. In C you have to use a single declaration for each variable. You can either declare all variables in one single or in several different regions. You must not initialize the variables inside! In Fortran you have to put all declarations in one single tag. Because there exist no global variables as in C you have to use common blocks. For further information see the examples. III.2. Adding tests to the test lists -------------------------------------- After you have created a new test you have to add them to a testlist. Simply add the function name in a new Line into a file. ------------------------------------------------------------------------------ IV. Known Issues and Workaround ============================================================================== The Sun OS has a problem with the -maxdepth option on the 'make cleanall' command. This prevents the tests from being removed from the working directory and can cause problems with future tests. To remedy, edit the makefile line under the clean command: -rm [cf]test*.[cf] [cf]crosstest*.[cf] [cf]ctest*.[cf] [cf]orphan*.[cf] Change to: -rm [cf]test* [cf]crosstest* [cf]ctest* [cf]orphan* ------------------------------------------------------------------------------ V. Contact and Support ============================================================================== Contact: http://www.cs.uh.edu/~hpctools