blob: 6114540f877865fef49020593cddbef2044047ce (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
|
= EzBench =
This repo contains a collection of tools to benchmark graphics-related
patch-series.
== Ezbench.sh ==
This tool is responsible for collecting the data and generating logs that will
be used by another tool to generate a visual report.
By default, the logs will be outputed in logs/<date of the run>/ and are stored
mostly as csv files. The main report is found under the name results and needs
to read with "less -r" to get the colours out! The list of commits tested is
found under the name commit_list. A comprehensive documentation of the file
structure will be written really soon.
You may specify whatever name you want by adding -N <name> to the command line.
This is very useful when testing kernel-related stuff as we need to reboot on
a new kernel to test a new commit. Support for this is not perfect yet as some
duplicates in commit_list may happen.
=== Dependencies ===
- A recent-enough version of bash
- make
- awk
- all the other typical binutils binaries
=== Configuration ===
The tests configuration file is named test_options.sh. A sample file called
test_options.sh.sample comes with the repo and is a good basis for your first
configuration file.
You will need to adjust this file to give the location of the base directory of
all the benchmark folders.
Another important note about ezbench.sh is that it is highly modular and
hook-based. Have a look at conf.d/README for the documentation about the
different hooks. The hooks can be defined in test_options.sh or in whatever file
whose extension would be .conf and located in the conf.d directory.
=== Examples ===
==== Testing every patchset of a series ====
The following command will test all the GLB27:Egypt cases but the ones
containing the word cpu in them. It will run all the benchmarks 5 times on
the 10 commits before HEAD~~.
./ezbench.sh -p ~/repos/mesa -B cpu -b GLB27:Egypt -r 5 -n 10 -H HEAD~~
The following command run the synmark:Gl21Batch2 benchmark (note the $ at the
end that indicates that we do not want the :cpu variant). It will run all the
benchmarks 3 times on 3 commits (in this order), HEAD~5 HEAD~2 HEAD~10.
./ezbench.sh -p ~/repos/mesa -b synmark:Gl21Batch2$ -r 3 HEAD~5 HEAD~2 HEAD~10
==== Retrospectives ====
Here is an example of how to generate a retrospective. The interesting part is
the call to utils/get_commit_list.py which generates a list of commits
./ezbench.sh -p ~/repos/mesa -B cpu -b GLB27:Egypt:offscreen \
-b GLB27:Trex:offscreen -b GLB30:Manhattan:offscreen \
-b GLB30:Trex:offscreen -b unigine:heaven -b unigine:valley \
-r 3 -m "./recompile-release.sh" `utils/get_commit_list.py \
-p ~/repos/mesa -s 2014-12-01 -i "1 week"`
== stats/gen_report.py ==
The goal of this tool is to read the reports from ezbench and make them
presentable to engineers and managers.
Commits can be renamed by having a file named 'commit_labels' in the logs
folder. The format is to have one label per line. The short SHA1 first, a space
and then the label. Here is an example:
bb19f2c 2014-12-01
If you want to generate date labels for commits, you can use the tool
utils/gen_date_labels.py to generates the 'commit_labels' file. Example:
utils/gen_date_labels.py -p ~/repos/mesa logs/seekreet_stuff/
It is also possible to add notes to the HTML report by adding a file called
'notes' in the report folder. Every line of the note file will be added in
an unordered list. It is possible to use HTML inside the file.
=== Dependencies ===
- python3
- matplotlib
- scipy
- mako
- an internet connection to read the report
=== Example ===
This command will create an HTML report named
logs/public_benchmarks_trend_broadwell/index.html. Nothing more, nothing less.
./stats/gen_report.py logs/public_benchmarks_trend_broadwell/
== utils/perf_bisect.py ==
The perf_bisect.py tool allows bisecting performance issues. It is quite trivial
to use, so just check out the example.
=== Examples ===
The following command will bisect a performance difference between commit
HEAD~100 and HEAD. The -p, -b, -r and -m arguments are the same as ezbench.sh.
utils/perf_bisect.py -p ~/repos/mesa -b glxgears:window -r 1 HEAD~100 HEAD
|
