diff options
author | Daniel Sanders <daniel.sanders@imgtec.com> | 2016-08-09 11:50:53 +0000 |
---|---|---|
committer | Daniel Sanders <daniel.sanders@imgtec.com> | 2016-08-09 11:50:53 +0000 |
commit | 055eba88d72e693a75bae2535c304d3428fb9c93 (patch) | |
tree | e2871e12717559d8b5526c2e5c557011ad1e0561 /docs | |
parent | 662d5497c3a4fbcdbf125d109f4abe5a9385cc20 (diff) |
[sanitizers] Make it possible to XFAIL on the effective target, not just the default.
Summary:
The triple is not the right thing to XFAIL on since LIT only sees the default
triple and not the effective triple chosen by any -target option in the RUN
directives. This discrepancy is shown in the table below:
Default Triple | Options | XFAIL | LIT's expected result | Desired expectation
=================+===================================+========+=======================+====================
mips-linux-gnu | -target mips-linux-gnu | | Pass | Pass
mips-linux-gnu | -target mips64-linux-gnu -mabi=64 | | Pass | Pass
mips-linux-gnu | -target mips-linux-gnu | mips | Fail | Fail
mips-linux-gnu | -target mips64-linux-gnu -mabi=64 | mips | Fail | Fail/Pass* (debatable**)
mips-linux-gnu | -target mips-linux-gnu | mips- | Fail | Fail
mips-linux-gnu | -target mips64-linux-gnu -mabi=64 | mips- | Fail | Pass*
mips-linux-gnu | -target mips-linux-gnu | mips64 | Pass | Pass
mips-linux-gnu | -target mips64-linux-gnu -mabi=64 | mips64 | Pass | Fail*
mips64-linux-gnu | -target mips-linux-gnu | | Pass | Pass
mips64-linux-gnu | -target mips64-linux-gnu -mabi=64 | | Pass | Pass
mips64-linux-gnu | -target mips-linux-gnu | mips | Fail | Fail*
mips64-linux-gnu | -target mips64-linux-gnu -mabi=64 | mips | Fail | Fail/Pass (debatable**)
mips64-linux-gnu | -target mips-linux-gnu | mips- | Pass | Fail*
mips64-linux-gnu | -target mips64-linux-gnu -mabi=64 | mips- | Pass | Pass
mips64-linux-gnu | -target mips-linux-gnu | mips64 | Fail | Pass*
mips64-linux-gnu | -target mips64-linux-gnu -mabi=64 | mips64 | Fail | Fail
x64_64-linux-gnu | -target i386-linux-gnu | | Pass | Pass
x64_64-linux-gnu | -target x86_64-linux-gnu | | Pass | Pass
x64_64-linux-gnu | -target i386-linux-gnu | i386 | Pass | Fail*
x64_64-linux-gnu | -target x86_64-linux-gnu | i386 | Pass | Pass
x64_64-linux-gnu | -target i386-linux-gnu | x86_64 | Fail | Pass
x64_64-linux-gnu | -target x86_64-linux-gnu | x86_64 | Fail | Fail*
* These all differ from LIT's current behaviour.
** People's expectations vary depending on whether they know that LIT does a
substring match on the default triple or think it's an exact match on an
architecture.
This patch adds "target-is-${target_arch}" to the available features list and
updates the mips XFAIL's to use them. XFAIL'ing on these features will
correctly account for the target being tested. Making the table:
Options | XFAIL | LIT's expected result
==================================+==================+======================
-target mips-linux-gnu | | Pass
-target mips64-linux-gnu -mabi=64 | | Pass
-target mips-linux-gnu | target-is-mips | Fail
-target mips64-linux-gnu -mabi=64 | target-is-mips | Pass
-target mips-linux-gnu | target-is-mips64 | Pass
-target mips64-linux-gnu -mabi=64 | target-is-mips64 | Fail
-target i386-linux-gnu | | Pass
-target x86_64-linux-gnu | | Pass
-target i386-linux-gnu | target-is-i386 | Fail
-target x86_64-linux-gnu | target-is-i386 | Pass
-target i386-linux-gnu | target-is-x86_64 | Pass
-target x86_64-linux-gnu | target-is-x86_64 | Fail
Reviewers: probinson
Subscribers: probinson, kubabrecka, llvm-commits, samsonov
Differential Revision: https://reviews.llvm.org/D22802
git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@278116 91177308-0d34-0410-b5e6-96231b3b80d8
Diffstat (limited to 'docs')
-rw-r--r-- | docs/TestingGuide.rst | 66 |
1 files changed, 66 insertions, 0 deletions
diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst new file mode 100644 index 000000000..4edda6738 --- /dev/null +++ b/docs/TestingGuide.rst @@ -0,0 +1,66 @@ +======================================== +Compiler-rt Testing Infrastructure Guide +======================================== + +.. contents:: + :local: + +Overview +======== + +This document is the reference manual for the compiler-rt modifications to the +testing infrastructure. Documentation for the infrastructure itself can be found at +:ref:`llvm_testing_guide`. + +LLVM testing infrastructure organization +======================================== + +The compiler-rt testing infrastructure contains regression tests which are run +as part of the usual ``make check-all`` and are expected to always pass -- they +should be run before every commit. + +Quick start +=========== + +The regressions tests are in the "compiler-rt" module and are normally checked +out in the directory ``llvm/projects/compiler-rt/test``. Use ``make check-all`` +to run the regression tests after building compiler-rt. + +REQUIRES, XFAIL, etc. +--------------------- + +Sometimes it is necessary to restrict a test to a specific target or mark it as +an "expected fail" or XFAIL. This is normally achieved using ``REQUIRES:`` or +``XFAIL:`` with a substring of LLVM's default target triple. Unfortunately, the +behaviour of this is somewhat quirky in compiler-rt. There are two main +pitfalls to avoid. + +The first pitfall is that these directives perform a substring match on the +triple and as such ``XFAIL: mips`` affects more triples than expected. For +example, ``mips-linux-gnu``, ``mipsel-linux-gnu``, ``mips64-linux-gnu``, and +``mips64el-linux-gnu`` will all match a ``XFAIL: mips`` directive. Including a +trailing ``-`` such as in ``XFAIL: mips-`` can help to mitigate this quirk but +even that has issues as described below. + +The second pitfall is that the default target triple is often inappropriate for +compiler-rt tests since compiler-rt tests may be compiled for multiple targets. +For example, a typical build on an ``x86_64-linux-gnu`` host will often run the +tests for both x86_64 and i386. In this situation ``XFAIL: x86_64`` will mark +both the x86_64 and i386 tests as an expected failure while ``XFAIL: i386`` +will have no effect at all. + +To remedy both pitfalls, compiler-rt tests provide a feature string which can +be used to specify a single target. This string is of the form +``target-is-${arch}`` where ``${arch}}`` is one of the values from the +following lines of the CMake output:: + + -- Compiler-RT supported architectures: x86_64;i386 + -- Builtin supported architectures: i386;x86_64 + +So for example ``XFAIL: target-is-x86_64`` will mark a test as expected to fail +on x86_64 without also affecting the i386 test and ``XFAIL: target-is-i386`` +will mark a test as expected to fail on i386 even if the default target triple +is ``x86_64-linux-gnu``. Directives that use these ``target-is-${arch}`` string +require exact matches so ``XFAIL: target-is-mips``, +``XFAIL: target-is-mipsel``, ``XFAIL: target-is-mips64``, and +``XFAIL: target-is-mips64el`` all refer to different MIPS targets. |