More tolerance for flaky tests in libc++ on NetBSD

Summary:
Tests marked with the flaky attribute ("FLAKY_TEST.")
can still report false positives in local tests and on the
NetBSD buildbot.

Additionally a number of tests (probably all threaded
ones) unmarked with the flaky attribute is flaky on
NetBSD.

An ideal solution on the libcxx side would be to raise
max retries for NetBSD and mark failing tests with
the flaky flag, however this adds more maintenance
burden and constant monitoring of flaky tests.

Reduce the work and handle flaky tests as more flaky
on NetBSD and allow flakiness  of other tests on
NetBSD.

Reviewers: mgorny, EricWF

Reviewed By: mgorny

Subscribers: christof, llvm-commits, libcxx-commits

Differential Revision: https://reviews.llvm.org/D56064

1 files changed, 7 insertions, 0 deletions

diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index 6a334ac31cf..46b2e46ac8c 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -12,6 +12,7 @@ import errno
 import os
 import time
 import random
+import platform
 
 import lit.Test        # pylint: disable=import-error
 import lit.TestRunner  # pylint: disable=import-error
@@ -202,6 +203,12 @@ class LibcxxTestFormat(object):
                           for f in os.listdir(local_cwd) if f.endswith('.dat')]
             is_flaky = self._get_parser('FLAKY_TEST.', parsers).getValue()
             max_retry = 3 if is_flaky else 1
+
+            # LIBC++ tests tend to be more flaky on NetBSD, so add more retries.
+            # We don't do this on other platforms because it's slower.
+            if platform.system() in ['NetBSD']:
+                max_retry = max_retry * 3
+
             for retry_count in range(max_retry):
                 cmd, out, err, rc = self.executor.run(exec_path, [exec_path],
                                                       local_cwd, data_files,