summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2017-06-06test-runner: add support for parse patternChase Qi
LAVA V2 is able to catch and save result lines that send to stdout instantly. For tests that run for a long time and generates a plenty of results as it runs, like kselftest and ltp tests, LAVA's live result collection feature could be useful. In Linaro Test Suite, to support completely local run and reduce the complexity of log parser debugging, we parse test log in test script and save results in LAVA's format to result.txt file. Then use 'send-to-lava.sh [path]/result.txt' to send result lines to stdout after test finished. This should work for most test cases, but maybe not good enough for tests mentioned above as the tests may crash or hang or cause Kernel panic or system halt during test run, in those cases, the result sending line has no chance to get executed. Using parse pattern in test definition file is one way to enable LAVA's live result collection feature. This patch is used to add parse pattern support in test-runner so that we can run the test cases using parse pattern on the local. It also could be useful for pattern debugging on the local. Change-Id: I5756af84f653ab00fc52c62ff888b71fe8817944 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-06-06automated: linux: kselftest: fix duplicate resultsNaresh Kamboju
Change-Id: I344e60f52a48cb14cf20101db3525ced0daaa911 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-06-02linux: sysbench: cope with shellcheckChase Qi
Change-Id: I43f893dafb9f0d7c245d2a6a9d76b20c2df0d394 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-06-01linux: sysbench: cope with LAVA V2Chase Qi
Split sysbench's result line avg/stddev into two metrics to avoid using slash in test-case-id and measurement. Using slash in test-case-id or measurement is valid in LAVA V1, but invalid in V2. Refer to lava-dispatcher's source code 'lava_dispatcher/pipeline/actions/test/shell.py' line #389 and #420. Change-Id: Ic3c411745c88827ebff78421e9ceebaec8f94cbd Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-31automated: linux: kselftest: Adding SKIPLIST and SKIPFILE parametersNaresh Kamboju
Known unsupported test cases list can be provided by using, SKIPLIST or SKIPFILE parameters which will be skipped from the run. Change-Id: I844dd6a9cded5a18bd1a21bc63320f4a3a3e7d83 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-30test-runner: Skip duplicate testsDan Rue
This commit adds checking to ensure that duplicate tests are not added to the test plan. Change-Id: I045c16467cc17812a4559a14ac66e55a9c33a189 Signed-off-by: Dan Rue <dan.rue@linaro.org>
2017-05-26automated/linux/ltp: renamed TIMEOUT variableMilosz Wasilewski
Change-Id: I021774e1f9581df05b69d026972ba50c201855fd Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-05-26automated: utils: use lava-test-case for result saving in LAVAChase Qi
In LAVA, use lava-test-case for result saving makes the upload helper script more reliable. For local run, saving file upload 'fail/skip' result to 'result.txt' isn't necessary as it is always skipped. And some test cases may don't use 'result.txt' at all, in these cases, we will need to handle file creating in the upload helper script which increases complicity. Change-Id: Ie9ed79b66bab6abc861c2a3da7a3c6fd21b19c88 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-26automated: linux: ltp: increase timeout for slow machinesNaresh Kamboju
The default timeout is 5min per test case. Increase timeout for slow machines with the help of variable ${MINUTES_TO_TIMEOUT} from yaml file. Change-Id: Ibd4c022d15f203cc8eedca76ff754ab030f73391 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-25automated: linux: optee-xtest: improve result parserNaresh Kamboju
Results print format changed on upstream optee_test tree https://github.com/OP-TEE/optee_test/commit/213ca8aaf4eafe9d26af76db7b31d9595ce57e6f This patch simplify result parser to make it more reliable. Change-Id: Iacc3282b5460c0d1d8bfbb237917c087e9657108 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-25automated: android: xtest: improve result parserChase Qi
Change-Id: I08cc2a8200c35e9551612fe9d1c25362f90f544d Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-25test-runner: add support for Python3Chase Qi
* Use os.chmod('file', 0o755) to change file permission. It is supported in Python3 and higher version Python2(tested with v2.7.12 on Ubuntu 16.04). To support lower version Python2, we will need to detect Python version at runtime and use different syntax respectively. That probably isn't necessary. * Use print("string") format for printing which is supported by both Python2 and Python3. * Cope with Python3's strict separation of strings and bytes. Change-Id: I11ab8b956a750e7d6bfafe6ead37686248572854 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-25android: optee-xtest: improve test results parserNaresh Kamboju
Results print format changed on upstream optee_test tree https://github.com/OP-TEE/optee_test/commit/213ca8aaf4eafe9d26af76db7b31d9595ce57e6f Use ${TEST_SUITE} as prefix to detect testcase names. Change-Id: I5f5dc30201f61049fa0882903ee830b01acac445 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-24automated: android: add bootchart testChase Qi
* Migrate bootchart to v2. * Generate bootchart graphic. Change-Id: Iea19523d30d036a8197ee6cfc1a39a58512725f7 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-24automated: add support for workload automation test suiteChase Qi
Change-Id: Ie07d58c902e4c905c938b1aad8a7c126cdcf7ebf Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-23test-runner: fix test kind selectionChase Qi
Change-Id: I89c8760d20b6d5fb9705adb4fe237916487b3a30 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-22automated: linux: lapack: fetch upstream sourceNaresh Kamboju
Fetch upstream source tree from github git clone https://github.com/Reference-LAPACK/lapack.git Change-Id: Ib2a508b62b2e5ddce36f0872fd06e5ffdbef37b6 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-19automated: android: tjbench: add support for Android O and masterChase Qi
Change-Id: I909ec3a6285dd2a31bd60f3034c5875c3de06bc3 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-18tjbench: add support for O preview and masterYongqin Liu
in O preview, the name of tjbench command is changed to tj Change-Id: I47dac9bca6225181a2bdd5203d9577ec3115f19e Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>
2017-05-17automated: linux: libhugetlbfs: cleanup results summaryNaresh Kamboju
Print results status in lowercase pass/fail/skip testcase name containing /dev/full coming twice in results summary. once it is printing as pass as expected. next time the default lava results parser printing as "unknown". This issue is been fixed by changing the "/" to "-" Change-Id: Ib3f5be575e0b97c9c73b9e9e408218973d92a319 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-17automated: linux: ltp-realtime: Bump LTP version to 20170516Naresh Kamboju
Change-Id: Idafb588b44a0da4d10c29ebdad3e8986b5733b72 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-17automated: linux: ltp: Bump LTP version to 20170516Naresh Kamboju
Change-Id: I67b1d174fd1a679fd90dee85710c10a8975e2ea0 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-16automated: sysbench: run the number of processors threads by defaultChase Qi
Change-Id: I757fa6baf40d8bab674d1d931c17e9c8fab63fa3 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-15automated: sysbench: add percpu benchmarkNicolas Dechesne
On multi core platforms, running 'sysbench --test=cpu' is not deterministic, since we cannot guarantee on which CPU the test is being run. It is even more problematic on big little platforms. In order for sysbench to be useful, we need to run sysbench 'cpu' benchmarks in the following conditions: 1. with --num-thread=<number of cpu> 2. with --num-thread=1 , for each cpu , using task affinity So that we can benchmark the platform as a whole, and each CPU individually. This patch adds a specific 'percpu' mode that takes care of #2 above. For #1, I would be in favor of removing NUMBER_THREADS as a params, or at least set it to <number of CPU> by default, but this is not done in this patch. A tweak in general_parser() was needed to add the cpu ID to the test name, running the test with 'percpu cpu' as PARAMS gives the following output: sysbench,percpu0-total-time,pass,9.2697,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu0-total-number-of-events,pass,10000,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu0-total-time-taken-by-event-execution,pass,9.2674,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu0-response-time-min,pass,0.84,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu0-response-time-avg,pass,0.93,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu0-response-time-max,pass,6.10,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu0-response-time-approx,pass,1.12,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu0-events-avg/stddev,pass,10000.0000/0.00,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu0-execution-time-avg/stddev,pass,9.2674/0.00,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu1-total-time,pass,9.9461,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu1-total-number-of-events,pass,10000,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu1-total-time-taken-by-event-execution,pass,9.9425,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu1-response-time-min,pass,0.84,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu1-response-time-avg,pass,0.99,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu1-response-time-max,pass,4.46,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu1-response-time-approx,pass,1.23,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu1-events-avg/stddev,pass,10000.0000/0.00,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu1-execution-time-avg/stddev,pass,9.9425/0.00,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu2-total-time,pass,9.4700,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu2-total-number-of-events,pass,10000,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu2-total-time-taken-by-event-execution,pass,9.4663,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu2-response-time-min,pass,0.84,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu2-response-time-avg,pass,0.95,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu2-response-time-max,pass,1.78,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu2-response-time-approx,pass,1.12,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu2-events-avg/stddev,pass,10000.0000/0.00,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu2-execution-time-avg/stddev,pass,9.4663/0.00,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu3-total-time,pass,9.4429,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu3-total-number-of-events,pass,10000,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu3-total-time-taken-by-event-execution,pass,9.4406,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu3-response-time-min,pass,0.84,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu3-response-time-avg,pass,0.94,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu3-response-time-max,pass,3.67,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu3-response-time-approx,pass,1.09,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu3-events-avg/stddev,pass,10000.0000/0.00,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,percpu3-execution-time-avg/stddev,pass,9.4406/0.00,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,cpu-total-time,pass,9.8187,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,cpu-total-number-of-events,pass,10000,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,cpu-total-time-taken-by-event-execution,pass,9.8148,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,cpu-response-time-min,pass,0.84,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,cpu-response-time-avg,pass,0.98,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,cpu-response-time-max,pass,8.63,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,cpu-response-time-approx,pass,1.16,ms,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,cpu-events-avg/stddev,pass,10000.0000/0.00,times,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false sysbench,cpu-execution-time-avg/stddev,pass,9.8148/0.00,s,NUM_THREADS=1;TESTS=percpu cpu;SKIP_INSTALL=false Change-Id: I25b10f0b5f1f5083e82b813b10e9ddd0ae184033 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-05-15automated: kselftests download right archRiku Voipio
Unless speficically requested, download aarch64 copy when running on aarch64. Change-Id: I816b27ee9d4e501c3c37964d6ce1b64a5a719772 Signed-off-by: Riku Voipio <riku.voipio@linaro.org>
2017-05-15plans: Added lapack tests to rpb_ee_perfomance test planNaresh Kamboju
The LAPACK project supports the package in the sense that reports of errors or poor performance will gain immediate attention from the developers. Change-Id: I8cdfb072da3ce285fc1a02b4b2c53253af614d1b Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-13boottime2.sh: ignore case for the grep commandYongqin Liu
"Starting service 'logd'" changed to "starting service 'logd'" in O preview build, so we specify -i for the grep command to search with ignoring case Change-Id: I3bc5d49d219e0646dea7be33f10d7e2d7c65caa8 Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>
2017-05-12kvm-cloud: de-race shutdownRiku Voipio
- move to automated dir - start qemu with pidfile - call shutdown with 1m timeout in guest for clean poweroff - wait for qemu process recorded in pidfile to exit to avoid primary lava connection to poweroff the machine before guest has finished Change-Id: I105ec3bbe3246a05552b3d13123ef4a3c7ea4c82 Signed-off-by: Riku Voipio <riku.voipio@linaro.org>
2017-05-12Merge "automated: android: add lapack test"Milosz Wasilewski
2017-05-12Add a script to populate local DNS in NFSNeil Williams
When used with a Debian initramfs (jessie, stretch or later) when the kernel reports dns0 : 10.15.0.1 when mounting the NFS, use this data to populate /etc/resolv.conf Avoids overwriting an existing /etc/resolv.conf Usage: ./automated/utils/nfs-dns.sh eth0 Change-Id: Ie61fde7081f71110d8a4bc6ebe154ca1cfc5beac
2017-05-12utils: skip file uploading on empty tokenChase Qi
Change-Id: Ib6a4dc1be433b161d665ddcf07d1eb8c3382ca9f Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-11test-runner.py: use python2, not pythonNicolas Dechesne
test-runner.py does not work with python3. The first error is the following: File "/work/linaro/test-definitions/automated/bin/test-runner", line 201 os.chmod('%s/run.sh' % self.test['test_path'], 0755) ^ SyntaxError: invalid token Until we have support for python3, let's make sure that we use python2, this is especially needed when trying to use distro where python3 is the default (e.g. archlinux). Change-Id: Id387bf0d6092c6555081eb891de12ff0a4df3bb7 Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
2017-05-11automated: android: add lapack testNaresh Kamboju
Change-Id: I636b0c0f5aeee5cbb600a1ece81328cddb0ba5a9 Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-05-10automated: android: add pm-qa testChase Qi
Change-Id: Ibd4368cdc5ec7e0f2e0098125e9cc50f33f555c2 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-09tradefed: install vts dependenciesVishal Bhoj
Change-Id: Icfd1dcc82cfe605750d51e052930d9a69d8f985d Signed-off-by: Vishal Bhoj <vishal.bhoj@linaro.org>
2017-05-08android: tradefed: fix package installationChase Qi
* Added support for Ubuntu * Removed support for Debian Change-Id: I4cc4074e455b8007f8f68e33815312846c344435 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-08change disable suspendVishal Bhoj
Change-Id: Ic06f1a674a43e47847f6c82082c5b5210d79836d Signed-off-by: Vishal Bhoj <vishal.bhoj@linaro.org>
2017-05-08android: tradefed: Wait for 5 minutes before terminatingVishal Bhoj
tradefed itself will finish execution if device is not available anymore so lets give 5 minutes for execution to finish Change-Id: I2423ce7706ac0518792183aa4c890dfe381eed82 Signed-off-by: Vishal Bhoj <vishal.bhoj@linaro.org>
2017-05-05kselftest: moved to LAVA parsing in YAML fileMilosz Wasilewski
LAVA v2 is able to parse results as they come. This way timeouts and crashes don't prevent all results to be registered in LAVA. Change-Id: I75e1b9af8a5159844ae20ab97e3e1f5e89e3473d Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-05-04Automated: Android: Added support for running VTSMilosz Wasilewski
CTS test is almost compatible with VTS as they use the same test shell - tradefed. The CTS test was updated to also support executing and reporting VTS results. Change-Id: Ida3ceda733c5e74f87256cf2ebefbaf93a6fce74 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-05-04Exit on install_deps errorChase Qi
If test script runs with '-e' argument, test should exit on errors. If test script runs without '-e' argument, 'if [ $? -ne 0 ]' will throw an error message and exit on installation error. Change-Id: I6365801582deaf7f5f105fe7014eadc4d0d3cdf7 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-04automated: linux: add rcutorture testChase Qi
Change-Id: Idc1506e2025e26a6c6f361f78037a55c0672c974 Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-04automated: linux: add RT pmqtestChase Qi
Change-Id: I1386f165df1c446f94b22be4253b919cd8540e0b Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-05-04automated: linux: add preempt-rt tag for hackbench and cyclictestChase Qi
Other tests from rt-tests already have this tag. With this patch, we will be able to filter them easily. Change-Id: Ic1e62ffe4680a09da73f5e62d136e82a79b5d37e Signed-off-by: Chase Qi <chase.qi@linaro.org>
2017-04-28automated: android: add refactored apk-automationMilosz Wasilewski
Change-Id: I5a81b3c4317dc56f90df37c5a8f83d2547542fab Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-04-27automated: android: add monkey test caseerin.liu
Change-Id: I3eeabe9bdeb74cc90d9f49d18359a4d7a304388e Signed-off-by: erin.liu <erin.liu@linaro.org>
2017-04-25android bootchart: add support for bootchart_parseYongqin Liu
also changed to use lava command for test result collection in script Change-Id: I0fdb8c71bd6de69e06de1bbbaae8ac5924842aee Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>
2017-04-21kselftest: fix results parserNaresh Kamboju
Change-Id: I591c4c3e7a6c1850d6160654418092f1cd96286a Signed-off-by: Naresh Kamboju <naresh.kamboju@linaro.org>
2017-04-20automated: doc: Added section on using test-runnerMilosz Wasilewski
The paragraph on test-runner was contributed by Nicolas Dechesne. The section was mostly copied from: https://github.com/96boards/documentation/blob/master/ConsumerEdition/DragonBoard-410c/Guides/Testing.md Change-Id: I217d33f2259ae16b924c6a0a8fec8c7690bcb65b Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
2017-04-19automated: android: added disable_suspend functionMilosz Wasilewski
The function unlocks the device and sets it to be awake all the time Change-Id: I53e39a047e8983e2d08c53b650df9251a793e7ab Signed-off-by: Milosz Wasilewski <milosz.wasilewski@linaro.org>
generated by cgit v1.2.3 (git 2.25.1) at 2024-07-18 10:19:17 +0000