LKFT: kselftest: Introduce yaml skipfile

This skipfile is consumable using skipgen
(https://github.com/Linaro/skipgen), included in this repo in
/automated/bin/<arch>/skipgen.

The skips in this file accurately reflect the skips in skipfiles/, with
the following considerations. When there were inconsistencies, new skips
were added to resolve, or multiple skip blocks were used. In no case
were skips removed from a board/branch combination.

The inconsistencies, missing data, etc in this file are due to
inconsistencies in skipfiles/, and will be resolved in a second pass
once migration to the yaml skipfile is complete and all the skipfiles/
are removed. That said, this file was generated manually (and validated
computationally), so there may also be errors introduced.

Verified with the following two scripts:

    # Compare complete lists of skips to find any that are entirely missing. Note
    # that this catches the 'rpk' skips, which should be ignored.
    skipgen skipfile-lkft.yaml | sort > some
    cat skipfiles/* | grep -v ^$ | grep -v ^# | awk '{print $1}' | sort | uniq > all
    diff some all

    # Check each board/branch combination for matching skiplists
    for board in dragonboard-410c hi6220-hikey juno-r2 x15 x86 qemu_arm64 qemu_x86_64; do
        for branch in 4.15 4.14 4.9 4.4; do
            echo $board $branch
            skipgen --board $board --branch $branch --environment production skipfile-lkft.yaml | sort > 1
            cat skipfiles/skiplist_${board}_${branch} | grep -v ^$ | grep -v ^# | awk '{print $1}' | sort | uniq > 2
            if ! diff 1 2; then
                read
            fi
        done
    done

Change-Id: Id06bf6bfa16c9f103bed254bbbf6a0242e91c65b
Signed-off-by: Dan Rue <dan.rue@linaro.org>

2 files changed, 482 insertions, 0 deletions

diff --git a/automated/linux/kselftest/skipfile-lkft.yaml b/automated/linux/kselftest/skipfile-lkft.yaml
new file mode 100644
index 0000000..bef6239
--- /dev/null
+++ b/automated/linux/kselftest/skipfile-lkft.yaml
@@ -0,0 +1,481 @@
+# This skipfile is consumable using skipgen
+# (https://github.com/Linaro/skipgen), included in this repo in
+# /automated/bin/<arch>/skipgen.
+#
+# reason:
+#   Please provide a human readable explanation of the reason for the skip.
+# url:
+#   A url referencing the issue. Often a link to the bug. Only one url may be
+#   specified.
+# environments:
+#   Should be 'production', 'staging', or 'all' for both. Our
+#   staging environment contains a very minimal skiplist. Only tests which make
+#   testing unstable should include staging.
+# boards:
+#   Should match a list of board names, or 'all' to include all boards.
+# branches:
+#   Should match a list of branch names, or 'all' to include all branches.
+# tests:
+#   List of tests to skip
+
+skiplist:
+
+  - reason: >
+      Adding skiplist according to the below ticket
+      mainline kernel tests baselining
+    url: https://projects.linaro.org/projects/CTT/queues/issue/CTT-585
+    environments: production
+    boards: all
+    branches: all
+    tests:
+      - test_maps
+
+  - reason: >
+      LKFT: kselftest: test_progs: libbpf: failed to open ./test_pkt_access.o:
+      No such file or directory
+    url: https://bugs.linaro.org/show_bug.cgi?id=3120
+    environments: production
+    boards: all
+    branches: all
+    tests:
+      - test_progs
+
+  - reason: "LKFT: linux-next: x86: kselftest: test_kmod.sh test failed"
+    url: https://bugs.linaro.org/show_bug.cgi?id=3219
+    environments: production
+    boards: all
+    branches: all
+    tests:
+      - test_kmod.sh
+
+  - reason: >
+      LKFT: linux-next: x15: kselftest: ftracetest hangs forever
+      LKFT: linux-mainline: kernel panic on X15 when running ftrace kseltest
+      https://bugs.linaro.org/show_bug.cgi?id=3304
+    url: https://bugs.linaro.org/show_bug.cgi?id=3297
+    environments: all
+    boards: all
+    branches: all
+    tests:
+      - ftracetest
+
+  - reason: "LKFT: linux-next: x86: kselftest: pstore_tests failed"
+    url: https://bugs.linaro.org/show_bug.cgi?id=3222
+    environments: production
+    boards: all
+    branches: all
+    tests:
+      - pstore_tests
+
+  - reason: "LKFT: selftests: seccomp TRACE_syscall.skip_after_RET_TRACE"
+    url: https://bugs.linaro.org/show_bug.cgi?id=2980
+    environments: production
+    boards: all
+    branches: all
+    tests:
+      - seccomp_bpf
+
+  - reason: "LKFT: linux-next: kselftest: breakpoint_test_arm64 build failed"
+    url: https://bugs.linaro.org/show_bug.cgi?id=3208
+    environments: all
+    boards:
+      - dragonboard-410c
+      - hi6220-hikey
+      - juno-r2
+      - qemu_arm64
+      - x15
+    branches: all
+    tests:
+      - breakpoint_test_arm64
+
+  - reason: "LKFT: linux-mainline: kselftest sync_test failed"
+    url: https://bugs.linaro.org/show_bug.cgi?id=3504
+    environments: production
+    boards: all
+    branches: all
+    tests:
+      - sync_test
+
+  - reason: >
+      LKFT: linux-mainline: kselftest BPF test_dev_cgroup failed (new test
+      case) on all devices
+    url: https://bugs.linaro.org/show_bug.cgi?id=3500
+    environments: production
+    boards: all
+    branches: all
+    tests:
+      - test_dev_cgroup
+
+  - reason: >
+      Adding skiplist according to the below ticket
+      mainline kernel tests baselining
+    url: https://projects.linaro.org/projects/CTT/queues/issue/CTT-585
+    environments: production
+    boards: all
+    branches: all
+    tests:
+      - test_tag
+      - test_lru_map
+      - test_lpm_map
+      - run.sh
+      - run_fuse_test.sh
+      - run_vmtests
+
+  - reason: >
+      LKFT: 4.4-rc 4.9-rc 4.13-rc 4.14-rc: x86: kselftest mpx-mini-test_64 - no
+      MPX support - failed - 3869 Aborted (core dumped)
+    url: https://bugs.linaro.org/show_bug.cgi?id=3497
+    environments: production
+    boards: x86
+    branches: all
+    tests:
+      - mpx-mini-test_64
+
+  - reason: >
+      LKFT: linux-mainline: x86, x15, juno-r2: kselftest fw_filesystem.sh
+      failed
+    url: https://bugs.linaro.org/show_bug.cgi?id=3503
+    environments: production
+    boards: all
+    branches: all
+    tests:
+      - fw_filesystem.sh
+
+  - reason: >
+      LKFT: linux-mainline: x86: kselftest NET reuseport_bpf failed (new test)
+    url: https://bugs.linaro.org/show_bug.cgi?id=3502
+    environments: production
+    boards:
+      - x86
+      - x15
+    branches: all
+    tests:
+      - reuseport_bpf
+  - reason: >
+      LKFT: linux-mainline: x86: kselftest NET reuseport_bpf failed (new test)
+    url: https://bugs.linaro.org/show_bug.cgi?id=3502
+    environments: production
+    boards: all
+    branches:
+      - 4.4
+      - 4.9
+    tests:
+      - reuseport_bpf
+
+  - reason: >
+      LKFT: mainline: x86: kselftests fsgsbase_64 failed - GS/BASE changed from
+      0x1/0x0 to 0x0/0x0
+      Fails intermittently
+    url: https://bugs.linaro.org/show_bug.cgi?id=3596
+    environments: production
+    boards:
+      - x86
+      - qemu_x86_64
+    branches: all
+    tests:
+      - fsgsbase_64
+
+  - reason: >
+      LKFT: linux-mainline: x15: kselfteest NET reuseport_bpf_numa failed (new
+      test)
+    url: https://bugs.linaro.org/show_bug.cgi?id=3501
+    environments: production
+    boards: all
+    branches:
+      - 4.4
+      - 4.9
+    tests:
+      - reuseport_bpf_numa
+  - reason: >
+      LKFT: linux-mainline: x15: kselfteest NET reuseport_bpf_numa failed (new
+      test)
+    url: https://bugs.linaro.org/show_bug.cgi?id=3501
+    environments: production
+    boards: x15
+    branches: all
+    tests:
+      - reuseport_bpf_numa
+
+  - reason: >
+      LKFT: linux-next, 4.9 and 4.4: bpf: test_align and test_verifier: No such
+      file or directory
+    url: https://bugs.linaro.org/show_bug.cgi?id=3170
+    environments: production
+    boards: all
+    branches:
+      - 4.4
+      - 4.9
+      - 4.14
+      - 4.15
+    tests:
+      - test_align
+      - test_verifier
+
+  - reason:
+    url:
+    environments: all
+    boards:
+      - x15
+      - qemu_x86_64
+      - qemu_arm64
+      - dragonboard-410c
+    branches: all
+    tests:
+      - step_after_suspend_test
+
+  - reason: >
+      LKFT: linux-mainline: x15: printf.sh bitmap.sh netns_netlink - section 4
+      reloc 2 sym 'memset': relocation 28 out of range (0xbf046044 ->
+      0xc109f720)
+    url: https://bugs.linaro.org/show_bug.cgi?id=3484
+    environments: production
+    boards: all
+    branches: 4.4
+    tests:
+      - bitmap.sh
+  - reason: >
+      LKFT: linux-mainline: x15: printf.sh bitmap.sh netns_netlink - section 4
+      reloc 2 sym 'memset': relocation 28 out of range (0xbf046044 ->
+      0xc109f720)
+    url: https://bugs.linaro.org/show_bug.cgi?id=3484
+    environments: production
+    boards: x15
+    branches: 4.15
+    tests:
+      - bitmap.sh
+
+  - reason: "skip all tests: memory hotplug is not supported"
+    url:
+    environments: production
+    boards: all
+    branches:
+      - 4.4
+      - 4.9
+    tests:
+      - mem-on-off-test.sh
+
+  - reason: "skip all tests: efivarfs is not mounted on /sys/firmware/efi/efivars"
+    url:
+    environments: production
+    boards: all
+    branches:
+      - 4.9
+    tests:
+      - efivarfs.sh
+  - reason: "skip all tests: efivarfs is not mounted on /sys/firmware/efi/efivars"
+    url:
+    environments: production
+    boards:
+      - dragonboard-410c
+      - hi6220-hikey
+    branches:
+      - 4.4
+    tests:
+      - efivarfs.sh
+
+  - reason: >
+      LKFT: linux-next: gpio: gpio-mockup-chardev: No such file or directory -
+      Build failed.
+    url: https://bugs.linaro.org/show_bug.cgi?id=3122
+    environments: production
+    boards: all
+    branches:
+      - 4.4
+      - 4.9
+    tests:
+      - gpio-mockup.sh
+
+  - reason: >
+      LKFT: linux-mainline: x86, x15, juno-r2: kselftest fw_filesystem.sh
+      failed
+    url: https://bugs.linaro.org/show_bug.cgi?id=3503
+    environments: production
+    boards: all
+    branches:
+      - 4.4
+      - 4.9
+    tests:
+      - fw_fallback.sh
+
+  - reason: "LKFT: 4.9-rc: Hikey: sysctl.sh need CONFIG_TEST_SYSCTL=y"
+    url: https://bugs.linaro.org/show_bug.cgi?id=3251
+    environments: production
+    boards: all
+    branches:
+      - 4.4
+      - 4.9
+    tests:
+      - sysctl.sh
+
+  - reason: "LKFT: timer: inconsistency-check failed on Hikey"
+    url: https://bugs.linaro.org/show_bug.cgi?id=2950
+    environments: production
+    boards:
+      - hi6220-hikey
+      - dragonboard-410c
+    branches:
+      - 4.4
+      - 4.9
+    tests:
+      - inconsistency-check
+
+  - reason: "kselftests: ldt_gdt_64 fails on x86"
+    url: https://bugs.linaro.org/show_bug.cgi?id=3564
+    environments: production
+    boards:
+      - x86
+      - qemu_x86_64
+    branches:
+      - 4.4
+      - 4.9
+    tests:
+      - ldt_gdt_64
+
+  - reason: >
+      LKFT: linux-stable-rc-4.4: Juno: kselftest cpufreq test failed - No cpu
+      is managed by cpufreq core, exiting
+    url: https://bugs.linaro.org/show_bug.cgi?id=3489
+    environments: production
+    boards: juno-r2
+    branches: 4.4
+    tests:
+      - main.sh
+  - reason: >
+      LKFT: linux-stable-rc-4.4: Juno: kselftest cpufreq test failed - No cpu
+      is managed by cpufreq core, exiting
+    url: https://bugs.linaro.org/show_bug.cgi?id=3489
+    environments: production
+    boards:
+      - dragonboard-410c
+      - qemu_arm64
+      - qemu_x86_64
+    branches: all
+    tests:
+      - main.sh
+
+  - reason:
+    url:
+    environments: production
+    boards: all
+    branches: 4.4
+    tests:
+      - owner
+      - pidns
+      - pstore_post_reboot_tests
+      - reuseport_bpf_cpu
+      - run_afpackettests
+      - sas
+
+  - reason: >
+      LKFT: linux-mainline, 4.9 and 4.4: x15: rtctest - PIE delta error:
+      0.018197 should be close to 0.015625
+    url: https://bugs.linaro.org/show_bug.cgi?id=3402
+    environments: production
+    boards:
+      - hi6220-hikey
+    branches: 4.4
+    tests:
+      - rtctest
+  - reason: >
+      LKFT: linux-mainline, 4.9 and 4.4: x15: rtctest - PIE delta error:
+      0.018197 should be close to 0.015625
+    url: https://bugs.linaro.org/show_bug.cgi?id=3402
+    environments: production
+    boards:
+      - dragonboard-410c
+      - x15
+    branches: all
+    tests:
+      - rtctest
+
+  - reason:
+    url:
+    environments: production
+    boards:
+      - x86
+      - qemu_x86_64
+    branches: 4.4
+    tests:
+      - sigreturn_64
+
+  - reason:
+    url:
+    environments: production
+    boards:
+      - dragonboard-410c
+      - hi6220-hikey
+    branches: 4.4
+    tests:
+      - test_bpf.sh
+
+  - reason:
+    url:
+    environments: all
+    boards: qemu_x86_64
+    branches: all
+    tests:
+      - breakpoint_test
+  - reason:
+    url:
+    environments: production
+    boards: qemu_arm64
+    branches:
+      - 4.9
+      - mainline
+    tests:
+      - breakpoint_test
+
+  - reason:
+    url:
+    environments: production
+    boards: qemu_arm64
+    branches: all
+    tests:
+      - mq_open_tests
+      - mq_perf_tests
+
+  - reason: >
+      LKFT: BPF: test_tcpbpf_user test_libbpf.sh failed - test_tcpbpf_kern.o
+      open test_l4lb.o : No such file or directory
+    url: https://bugs.linaro.org/show_bug.cgi?id=3636
+    environments: production
+    boards:
+      - qemu_x86_64
+      - qemu_arm64
+    branches:
+      - 4.14
+      - 4.15
+      - mainline
+    tests:
+      - test_libbpf.sh
+      - test_offload.py
+      - test_tcpbpf_user
+
+  - reason:
+    url:
+    environments: production
+    boards:
+      - qemu_x86_64
+      - qemu_arm64
+    branches:
+      - mainline
+    tests:
+      - test_static_keys.sh
+      - test_user_copy.sh
+      - printf.sh
+
+  - reason: >
+      LKFT: linux-mainline: x15: printf.sh bitmap.sh zram.sh netns_netlink -
+      section 4 reloc 2 sym 'memset': relocation 28 out of range (0xbf046044 ->
+      0xc109f720)
+    url: https://bugs.linaro.org/show_bug.cgi?id=3484
+    environments: production
+    boards:
+      - x15
+      - qemu_arm64
+      - qemu_x86_64
+    branches:
+      - 4.15
+      - mainline
+    tests:
+      - printf.sh
diff --git a/automated/linux/kselftest/skipfiles/skiplist_dragonboard-410c_mainline b/automated/linux/kselftest/skipfiles/skiplist_dragonboard-410c_mainline
new file mode 120000
index 0000000..bf3c260
--- /dev/null
+++ b/automated/linux/kselftest/skipfiles/skiplist_dragonboard-410c_mainline
@@ -0,0 +1 @@
+skiplist_hikey_mainline
\ No newline at end of file