aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
3 daysgen_interesting_symbols.py: Interesting symbols only list among the ↵HEADmasterLaurent Alfonsi
succesful benchmarks Change-Id: Ic5d3942f2a7d38dc42ef940ca1de600dc6e3b7e1
2024-05-06Revert "gather-metrics.sh: Failing if no successful benchmark (perf.csv is ↵Laurent Alfonsi
empty)" I initially proposed this commit in the following case : if none of the benchmark is successful, we could consider it as a failure. This case was finally considered as a EXTERNAL_FAILURE - probably an infrastructure problem - no need to bisect. This sounded reasonnable, however, if the tcwg_bmk-build.sh is launched on a single benchmark only, which is the case during the bisection, when the single asked benchmark is failing, it ends with a EXTERNAL_FAILURE. Which is a problem for bisection. This is why I am now reverting this commit. This reverts commit 5a30cd3d010c25c351ae551d64420f80f5bdd1b1. Change-Id: I50d30de74e9040da9fdd979e7ff25f0e00788a81
2024-04-29spec2xxx-result: Add spec run id in the extension of the results filesLaurent Alfonsi
The spec builds are splitted in 8 pieces if 8 machines are online. If some of the machines are busy, one machine can run successively 2 of that 8 slices. In such case, the results of the second run will overrides the results of the first ones. Some benchmarks are losts. Deciding to upload to dest dir the files with spec run id. This should not affect gather-metrics.sh, and the results should be well found and used. Change-Id: Icb541ec6378c01f24fdea05e2b753f979bbb48e4
2024-04-26break-up-bmks.sh: Disabling 548.exchange2_rLaurent Alfonsi
When using llvm -Os, bechmark 548.exchange2_r never ends running Disabling for the moment Change-Id: Ia6f70fd1ef8a72349ccadb32b8da0b0ec5819611
2024-04-18cpu2006.cfg: Add option for recent compilers to build these benchmarksLaurent Alfonsi
Change-Id: I64dfbff754f0a91ae73bbb4d26dabc9f284fe080
2024-04-18prepare-board.sh: keep udev service activeLaurent Alfonsi
udev service can be reactivated silently. and there's an (unclear) interaction between udev and set-ntp commands. when udev service is stopped, the "timedatectl set-ntp true" can hang and finally timeout. Change-Id: I99c2517977aafdf76436b9556c075aeebe472d46
2024-04-15cpu20**.cfg: Remove --std=legacy flag as unsupported now by flangLaurent Alfonsi
cpu2017 : removed std=legacy flang flag : error: Only -std=f2018 is allowed currently. cpu2006 : Removed std=legacy from cam4_s no such FPORTABILITY option described in https://www.spec.org/cpu2017/Docs/benchmarks/527.cam4_r.html Change-Id: I49bf82ed9b30691c3dab9d1e62919d4bc60d2f85
2024-04-10merge-metric-csvs.py: arrange result columns orderLaurent Alfonsi
Change-Id: If702a822a2c4762a3ab863f38d6907592c1a83ad
2024-04-03run.sh: Limit the number of parallel SPEC installationLaurent Alfonsi
Change-Id: I34469842f8374d4d11f078755079f480db7906ad
2024-03-27compute-variability.py: Enhance peak detection for 2-peaks-linear algorithmLaurent Alfonsi
Change-Id: I6099890fb71730d7f48782f7ead27a0ca08ec77e
2024-03-21gather-metrics.sh: Relevant changes from other patches in the series.Prathamesh Kulkarni
Change-Id: Ib78282453ba2479079893a0a4aa85359a052ed8f
2024-03-21Pass status.csv to metric files.Prathamesh Kulkarni
This patch introduces a new option --status_csv for passing status.csv to metric computing scripts (*-data-to-csv.py), and merge status field in merge-metric-csvs.py. Change-Id: If4fa4f7f99c1cdc574c326422b03b87d7af3dc71
2024-03-21merge-metric-csvs.py: Add --interesting_symbols and --metric_csvs options.Prathamesh Kulkarni
The patch adds option --metric_csvs=<metric csv files> which is mandatory, and an optional --interesting_symbols to pass interesting-symbols.csv. The rationale of this patch is to emit only interesting symbols in results.csv. Change-Id: I9d0dd778ad3a1c2b1848ac41c85211d82696ec03
2024-03-21Remove merging of status metric in fixup-perf-csv.py.Prathamesh Kulkarni
This patch partially reverts commit to merge status.csv and perf.csv, and filter out failed benchmarks, thus avoiding special casing of status metric. Instead the current workflow is to have metric computing scripts filter out successful bmks, and merge-metric-csvs.py merge the status field. Change-Id: If21201dfbed70ab3a064c25771618331e368050f
2024-03-14compute-variability.py: new weight method 2-peaks-linearLaurent Alfonsi
This is a matter of weights we are applying to each variability data in the history. The 2-peaks-linear algorithm gives a weight of - A linear weights affectation between --> most recent : 100% --> and second most recent noise variation : 50% - No weight for older noise variations : 0% Change-Id: Iab9dc783786560e257543e3cdff45e83e0dd9c90 Change-Id: Ifd4b56f40047477e7bd60316d5ec0430ce35ffb8
2024-03-14compute-variability.py: compute weight list for each inputLaurent Alfonsi
Change-Id: I74b802a5ffa0f6533c52443fe42487be7e319032
2024-03-04output-bmk-results.py: Fix typoLaurent Alfonsi
..related to compute-variability.py changes Change-Id: Idb0ee8a95b628f9a67942128fcf4a92fb7472697
2024-02-29Fix order of columns after merging and drop duplicate entries.Prathamesh Kulkarni
Change-Id: I867fd8f509542cee9e5e94638bdfabd305af60ca
2024-02-28output-bmk-results.py: Add parameter to specify the kind of data in ↵Laurent Alfonsi
specific_variability file ... and reworked script parameter parsing to use argparse The variability_file_data can be either "avg" or "max". When using "avg", it means the values in the specific_variability file is the sigma std deviation. The used threshold will be - 2 sigmas in build project (which cover 95% of random values) - 3 sigmas in bisect project (which cover 98% of random values) When using "max", the values stored in specific_variability file is the maximum noise for previous builds. In that case, we directly use this value as threshold for each symbol Change-Id: I7a0303b3c88668692b4599327113d87f33f5c8a0
2024-02-28compute-variability.py: extend to other method than averageLaurent Alfonsi
Change-Id: I947cbd887f7f5b13503c53dfd0d46f17b41203cb
2024-02-27compute-variability.py: Remove handling of current csvLaurent Alfonsi
Change-Id: Ie98b3063a9aae4304551881f40e2bd6e5f31e690
2024-02-27compute-variability.py: Fixed weight average computationLaurent Alfonsi
Change-Id: Idf791494742965fe89672dc3ffedff9b4e74b16f
2024-02-21Temporarily store perf-temp.csv to debug issue with field ordering in perf.csv.Prathamesh Kulkarni
Change-Id: I089397196345198ba915a6af2692167571478f86
2024-02-21csvs2table.py: Add a workaround to make tcwg_aosp-build.sh workMaxim Kuvyrkov
Change-Id: I26400fdbe040c4ff576e479770ddd4c14405c030
2024-02-20compute-variability.py: Can now use current compare-results.csvLaurent Alfonsi
Change-Id: Ie4abe9f9cdf03d0388bb8ca5779407fd00db5cfc
2024-02-15output-bmk-results.py: Add debug printsLaurent Alfonsi
Change-Id: Ic8cb88e88ef36b157bc8ddd7458e5da28da6b723
2024-01-31Update position of status field.Prathamesh Kulkarni
After 146fbc595f5ae1f904abfee37af7731f29f05941, we merge status field in fixup-perf-csv.py, rather than relying on merge-metric-csvs.py. As a result, the position of status filed in results.py changes from last to one following the symbol. For example, in: https://ci.linaro.org/view/tcwg_bmk/job/tcwg_bmk-code_size-coremark--gnu_eabi-arm_eabi-master-O2-build/106/artifact/artifacts/09-check_regression/results-vs-prev/csv-results-1/results.csv/*view*/ It can be seen that status is now at index 3, rather than being last field. Change-Id: I779e128b0a29f51763786a18098c0db0699b184d
2024-01-31[RFC] Extend gen_interesting_symbols.py to handle multiple perf csvs.Prathamesh Kulkarni
The patch extends gen_interesting_symbols.py, to pick set of interesting symbols from last N perf csvs, instead of just using from the current run. The patch tries to keep logic similar to csvs2table.py by recording the position of the symbol from each perf csv, and accumulates that, and sorts the combined result in descending order w.r.t accumulated position. For example, 1.csv: benchmark,symbol,sample,dso bmk1,exe.default,5000,na bmk1,sym1,3000,exe.default bmk1,sym2,900,libm.so bmk1,sym3,100,na bmk1,sym5,40,na 2.csv: benchmark,symbol,sample,dso bmk1,exe.default,4000,na bmk1,sym2,1500,libm.so bmk1,sym4,900,na bmk1,sym1,200,exe.default The set of interesting symbols along with it's position ("num_interesting" in csvs2table.py) for 1.csv: idf1: benchmark,symbol,dso,index bmk1,exe.default,na,100 bmk1,sym1,exe.default,99 bmk1,sym2,libm.so,98 Note that it doesn't contain samples anymore, and a new "index" column, that represents the position of the symbol from perf.csv. Likewise the set of interesting symbols along with it's position for 2.csv: idf2: benchmark,symbol,dso,index bmk1,exe.default,na,100 bmk1,sym2,libm.so,99 bmk1,sym4,na,98 After merging idf1 and idf2, we get the final result: benchmark,symbol,index,dso bmk1,exe.default,200,na bmk1,sym2,197,libm.so bmk1,sym1,99,exe.default bmk1,sym4,98,na For symbols that are present in both profiles, we add the "index" column, rationale being that a symbol appearing more frequently in profiles is more "interesting" than a symbol appearing less frequently. In the final out_csv, we drop the "index" column (since it's no longer needed), and keep benchmark,symbol,dso which is consistent with the current format. Change-Id: Iafe136f1262eacb3bd34f1cfb835e970ce5cb888
2024-01-10fixup-perf-csv.py: Merge status info in perf.csv and remove symbol entries ↵Prathamesh Kulkarni
for failed bmks. The rationale for this patch is to avoid issues like LLVM-1070 in which metric and comparison scripts still process symbols for failed benchmarks resulting in spurious regressions. The patch simply removes any symbol entries for failed bmks, since IMHO we should only care about bmk profile if it's run successfully. Change-Id: I198dc15333d3bcd13a1c3e25e8d75009619978ed
2023-12-29spec2xxx-config: Remove old code to autodetect mcpu / mfpuLaurent Alfonsi
Change-Id: Ic3ca023800666556b04c7c570b564508723a559e
2023-12-21compute-variability.py: Cast values to float to avoid internal python error ↵Laurent Alfonsi
in np.isnan() Also specify the order of the columns in the outputed csv file. This is useful for parsing this file in lnt-utils.sh Change-Id: I6e6339725958389820f2cfe7c7ac961f70094704
2023-12-13spec2xxx-config,cpu2017_llvm.cfg: Os and Oz unsupported in flangLaurent Alfonsi
Change-Id: I24a4658613b523b36947be6e04d7d6ee89de7a17
2023-12-06break-up-bmks.sh: Add spec2017_fast_nofortran benchmark setLaurent Alfonsi
Change-Id: I73ca4741b64d4b6b0473edc802d29e3446fb669d
2023-12-05csvs2table.py: Fix header for non relative mode.Prathamesh Kulkarni
We pass [options.base] + args to gen_output_fieldnames, even if options.base is empty, which is the case for non relative mode, and end up with args = ['', <arg1>.csv, <arg2>.csv, ...] and up with spurious header in output csv. Contrived example: benchmark,symbol,:sample,1:sample,2:sample bmk1,exe.default,5000,4000 bmk1,sym2,900,1500 Which has a spurious ":sample" in the header. The patch fixes this by adding options.base to args only if it is non empty. Change-Id: I65884d204c4e4136d0856c8f7b23c473c1a67097
2023-12-05compute-variability.py: Use np.average for computing weighted average.Prathamesh Kulkarni
Change-Id: I1f3d2d52b54fb3ded9968aebd2b4216560500074
2023-11-30gather-metrics.sh: Failing if no successful benchmark (perf.csv is empty)Laurent Alfonsi
Change-Id: Id65cad230b6a6bc5a55cf1c5edccb13e176f0bc9
2023-11-28gather-metrics.sh,coremark.sh: Fix status.csvLaurent Alfonsi
- Fix status for coremark - Correctly regenerate one for old results - fixed missing passed benchs for spec results Change-Id: Ibf6ce684abdc711994019c9404d91188a150f649
2023-11-24spec2xxx-config: remove flag flang-experimental-execAntoine Moynault
This flag has been removed from llvm several months ago (0867d4157328) Change-Id: Ic9ee674f13c6ded4bcec55dae2fee04536c18278
2023-11-17gather-metrics.sh: regenerate status.csv in case we have only results.csvLaurent Alfonsi
Change-Id: I3149c9f89c8f549ab7cb2782ca4bb5f881ddba61
2023-11-14compare-results.py, csvs2table.py: Include failed benchmarks in output reportLaurent Alfonsi
Also treate "status" field as non numeric values. Change-Id: I8fb9fdb62bef658b8c034ceac422a550abf0a7ff
2023-11-14compute-variability.py: no need anymore to handle 888888888 and 999999999 ↵Laurent Alfonsi
magic numbers Change-Id: Ie7f1d2f4c0390f5679d9c65ac13288e66a208be2
2023-11-14merge-metric-csvs.py, output-bmk-results.py: Adapt to use "status" field ↵Laurent Alfonsi
instead of magic numbers For status changes, it will create 1 or 2 new files status.{regression,improvement} Change-Id: Icc9e74298e399b720724a4b064cde25d7bda65ae
2023-11-14gather-metrics.sh, perfdatadir2csv.sh: Use status.csvLaurent Alfonsi
... or regenerate it for old results This is needed to process all existing bare spec results That will always be necessary because stored bare results will stay untouched Change-Id: I1bc972b7aaa5c08b9ed5ddbb9740bde37ed46782
2023-11-14coremark.sh, spec2xxx-result: Create status.csvLaurent Alfonsi
... containing eihter success, failed-to-build, or failed-to-run keyword Change-Id: I45fd0ce834c4844206990f3f27b19934c12d49a0
2023-10-19metric_utils.py: Add entries for status.Prathamesh Kulkarni
Change-Id: Ic0428c0406fc70a7ddd388f493c3f151024f1eb3
2023-10-02compute-variability.py: Compute variability for all symbolsLaurent Alfonsi
.. including omp_outlined functions that were skipped up to now. Change-Id: Id1c2e933d7801a2d638990fbcf5784255e86ec49
2023-10-02prepare-board.sh: retry when failing to set a system valueLaurent Alfonsi
Initial reason for that change is that, sometimes Qualcomm board refuse to set the scaling_min_freq. First supposition was that was due to the board was too hot and refuse to setup a higher frequency. But I can now observe that it sometimes refuse to set lower frequency. Change-Id: I766eff99b5214ec7d2a56883fdf796590183269f
2023-09-29output-bmk-results.py (get_short_long_diag): Change output formatMaxim Kuvyrkov
... from "BMK slowed down by X%" to "slowed down by X% - BMK". The new format makes for better summary fields in jira cards. Change-Id: I4e950323777848843e4210d73412e94998aa9ecd
2023-09-15perfdatadir2csv.sh: report failed benchmarks again in generated csv filesLaurent Alfonsi
Change-Id: I740a5d1e9c8d4d0124214694c506447823a3986c
2023-09-15prepare-board/freq-qc: decrease Qualcomm cpu frequencyLaurent Alfonsi
As sometimes the board doesn't accept to set the 2.5GHz on cpu 7. This might be due to a safety protection when the board becomes too hot. Using 2GHz for cpu7. Using 0.38GHz for cpu0-3 Change-Id: I28134edba77465f32f9f9667e867aadc1783f4ab
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-29 09:34:32 +0000