partial update of the TODO list and README

2 files changed, 204 insertions, 57 deletions

diff --git a/README b/README
index 0bef94b..db37f34 100644
--- a/README
+++ b/README
@@ -3,11 +3,22 @@
 This repo contains a collection of tools to benchmark graphics-related
 patch-series.
 
-== Ezbench.sh ==
+== Core.sh ==
+
+WARNING: This tool can be used directly from the CLI but it is recommenced that
+you use ezbench in conjunction with ezbenchd which together support testing the
+kernel and which are much more robust to errors.
 
 This tool is responsible for collecting the data and generating logs that will
 be used by another tool to generate a visual report.
 
+To operate, this script requires a git repo, the ability to compile and deploy
+a commit and then benchmarks to be run. To simplify the usage, a profile should
+be written for each repo you want to test that will allow ezbench to check the
+current version that is deployed, to compile and install a new version and set
+some default parameters to avoid typing very-long command lines every time a
+set of benchmarks needs to be run.
+
 By default, the logs will be outputed in logs/<date of the run>/ and are stored
 mostly as csv files. The main report is found under the name results and needs
 to read with "less -r" to get the colours out! The list of commits tested is
@@ -16,8 +27,7 @@ structure will be written really soon.
 
 You may specify whatever name you want by adding -N <name> to the command line.
 This is very useful when testing kernel-related stuff as we need to reboot on
-a new kernel to test a new commit. Support for this is not perfect yet as some
-duplicates in commit_list may happen.
+a new kernel to test a new commit.
 
 === Dependencies ===
 
@@ -28,17 +38,16 @@ duplicates in commit_list may happen.
 
 === Configuration ===
 
-The tests configuration file is named test_options.sh. A sample file called
-test_options.sh.sample comes with the repo and is a good basis for your first
+The tests configuration file is named user_parameters.sh. A sample file called
+user_parameters.sh.sample comes with the repo and is a good basis for your first
 configuration file.
 
 You will need to adjust this file to give the location of the base directory of
-all the benchmark folders.
+all the benchmark folders and repositories for the provided profiles.
 
 Another important note about core.sh is that it is highly modular and
-hook-based. Have a look at conf.d/README for the documentation about the
-different hooks. The hooks can be defined in test_options.sh or in whatever file
-whose extension would be .conf and located in the conf.d directory.
+hook-based. Have a look at profiles.d/$profile/conf.d/README for the
+documentation about the different hooks.
 
 === Examples ===
 
@@ -56,6 +65,11 @@ benchmarks 3 times on 3 commits (in this order), HEAD~5 HEAD~2 HEAD~10.
 
     ./core.sh -p ~/repos/mesa -b synmark:Gl21Batch2$ -r 3 HEAD~5 HEAD~2 HEAD~10
 
+To use the mesa profile, which has the advantage of checking that the deployment
+was successful, you may achieve the same result by running:
+
+    ./core.sh -P mesa -b synmark:Gl21Batch2$ -r 3 HEAD~5 HEAD~2 HEAD~10
+
 ==== Retrospectives ====
 
 Here is an example of how to generate a retrospective. The interesting part is
@@ -63,13 +77,93 @@ the call to utils/get_commit_list.py which generates a list of commits
 
     ./core.sh -p ~/repos/mesa -B cpu -b GLB27:Egypt:offscreen \
                  -b GLB27:Trex:offscreen -b GLB30:Manhattan:offscreen \
-                 -b GLB30:Trex:offscreen -b unigine:heaven -b unigine:valley \
-                 -r 3 -m "./recompile-release.sh" `utils/get_commit_list.py \
-                          -p ~/repos/mesa -s 2014-12-01 -i "1 week"`
+                 -b GLB30:Trex:offscreen -b unigine:heaven:1080p:fullscreen \
+                 -b unigine:valley:1080p:fullscreen -r 3 \
+                 -m "./recompile-release.sh" \
+                 `utils/get_commit_list.py -p ~/repos/mesa -s 2014-12-01 -i "1 week"`
+
+== ezbench ==
+
+This tool is meant to make the usage of core.sh easy and support testing
+performance across reboots.
+
+It allows creating a new performance report, scheduling benchmark
+runs, changing the execution rounds on the fly and then start, pause or halt
+the execution of this report.
+
+This tool uses core.sh as a backend for checking that the commits SHA1 and tests
+do exist so you are sure that the work can be executed when the time comes.
+
+=== Dependencies ===
+
+ - python3
+ - numpy
+
+=== Examples ===
+
+==== Creating a report ====
+
+The ezbench command allows you to create a new performance report. To create
+a performance report named 'mesa-tracking-pub-benchmarks', using the core.sh
+profile 'mesa', you need to run the following command:
+
+    ./ezbench -p mesa mesa-tracking-pub-benchmarks
+
+==== Adding benchmarks runs ====
+
+Adding the 2 rounds of benchmark GLB27:Egypt:offscreen to the report
+mesa-tracking-pub-benchmarks for the commit HEAD can be done using the following
+command:
+
+    ./ezbench -r 2 -b GLB27:Egypt:offscreen -c HEAD mesa-tracking-pub-benchmarks
+
+A retrospective can be made in the same fashion as with core.sh at the exception
+made that it would also work across reboots which is good when testing kernels:
+
+    ./ezbench -r 3 -b GLB27:Egypt:offscreen -b GLB27:Trex:offscreen \
+              -b GLB30:Manhattan:offscreen -b GLB30:Trex:offscreen \
+              -b unigine:heaven:1080p:fullscreen -b unigine:valley:1080p:fullscreen \
+              -c "`utils/get_commit_list.py -p ~/repos/mesa -s 2014-12-01 -i "1 week"`"
+              mesa-tracking-pub-benchmarks
+
+==== Checking the status of a report ====
+
+You can check the status of the 'mesa-tracking-pub-benchmarks' report by calling
+the following command:
+
+    ./ezbench mesa-tracking-pub-benchmarks status
+
+==== Changing the execution status of the report ====
+
+When creating a report, the default state of the report is "initial" which means
+that nothing will happen until the state is changed. To change the state, you
+need to run the following command:
+
+    ./ezbench mesa-tracking-pub-benchmarks (run|pause|abort)
+
+ - The "run" state says that the report is ready to be run by ezbenchd.py.
+
+ - The "pause" and "abort" states indicate that ezbenchd.py should not be
+ executing any benchmarks from this report. The difference between the "pause"
+ and "abort" states is mostly for humans, to convey the actual intent.
+
+==== Starting collecting data without ezbenchd.py ====
+
+If you are not using ezbenchd.py, you may simply run the following command to
+start collecting data:
+
+    ./ezbench mesa-tracking-pub-benchmarks start
+
+This command will automatically change the state of the report to "run".
+
+== utils/ezbenchd.py ==
 
 
 == stats/gen_report.py ==
 
+WARNING: This tool is deprecated, compare_reports.py is the prefered way now even
+if the single-report mode is not as advanced as the gen_report.py.
+
 The goal of this tool is to read the reports from ezbench and make them
 presentable to engineers and managers.
 
@@ -104,6 +198,8 @@ logs/public_benchmarks_trend_broadwell/index.html. Nothing more, nothing less.
 
 == utils/perf_bisect.py ==
 
+WARNING: The introduction of smart ezbench made this tool absolutely useless
+
 The perf_bisect.py tool allows bisecting performance issues. It is quite trivial
 to use, so just check out the example.
 
diff --git a/TODO b/TODO
index 27ee59b..83e236e 100644
--- a/TODO
+++ b/TODO
@@ -1,92 +1,143 @@
 = TODO =
 
 == Priority list ==
- - ezbench profiles : Blocks smart ezbench for checking which version of the repo is currently installed
- -
+ - watchdogs
+ - auto deploying of the component being tested
+ - experiment mode
 
-== Ezbench.sh ==
+== core.sh ==
+
+=== Watchdog support ===
+
+Add watchdog support to reboot the machine if the test is taking too long to
+execute and the machine does not want to work anymore.
+
+Systemd has some nice features which could be built-in smart-ezbench and ezbenchd
+but this watchdog cannot be stopped or re-programmed during compilation which may
+take any amount of time.
+
+More investigation needs to be made.
 
 === Profiles ===
 
-Profiles allow users to link hooks, tests list and repo path under one name to
-reduce any kind of error and ease reproducibility.
+ - Add auto-deployment support which would download from the git repo of
+ interest and setup ezbench to make use of it.
 
- - Have a folder called profiles.d/ that will store profiles
- - Profiles may be stored in sub-directories (to allow sharing between machines)
- - A profile's ID is its path in the
- - A git repo's config may contain the ezbench profile to be used (overridable by the user, but displaying a warning)
+ - Create a new repo every time we are about to compile, this should be very
+ cheap and would guarantee that we do not destroy the current state of the repo.
 
-=== Tests ===
+ - Add information about how to create working URLs for every commit SHA1, this
+ is useful when creating reports.
 
-Let tests tell what kind of units are returned and if bigger is better or worse.
+==== Kernel ====
 
-=== Compilation ===
+Make the default kernel profile work on ubuntu and not just archlinux by
+supporting the initramfs creation tool from ubuntu.
 
-Do not recompile if the version that is currently being used is the right one!
+=== Tests ===
 
-== Reports ==
+ - Auto deploy benchmarks
 
-=== Move commits to a different folder ===
+ - Do dependency checking a bit better so as we can report what is missing for
+ one benchmark
 
-The current report folder is kind of a mess because it potentially contains thousands of files.
+ - Add a benchmark-testing mode that will run the benchmark a hundred time and
+ deduce the variance of it. This will be used by smart ezbench to figure out how
+ many runs are necessary.
 
-We could create one folder per commit and store the data there. The name could be suffixed with the commit date.
+ - Output multiple metrics, possibly under the name
+ ${commit}_bench_${benchmark}.${metric}_metric
 
-=== Store the environment used when benchmarking ===
+=== Reset the environment to a previously-used one ===
 
-We need this to allow devs to reproduce the entire environment the benchmark used.
+When wanting to add data to a certain commit, we need to check the environment
+did not change, or, if it did, make at least that the results do not have a
+different average. At the very least, we should prevent adding data if anything
+in the environment changed except volatile information like the throttling count.
 
-The environment contains:
+== Reports ==
 
-- All the versions and commit IDs of graphics stack:
-    - kernel (git, sha1 of the file and /proc/kconfig.gz)
-    - libdrm
-    - mesa
-    - xserver
-    - xf86-video-xxxx
-    - etc
-- The hardware used
-- The BIOS version
-- The screens layout and resolution
+ - Store the execution runid along with the value in the result file to avoid
+ mis-labeling run IDs and to detect execution errors!
 
-This could be made easier if the graphics stack was entirely compiled using a set of scripts hosted by ezbench.
+=== Move commits to a different folder ===
 
-=== Potentially share commit results between reports ===
+The current report folder is kind of a mess because it potentially contains
+thousands of files.
 
-Benchmarks take forever to run, so it really is infuriating to have to re-run them over and over again when nothing changed!
+We could create one folder per commit and store the data there. The name could
+be suffixed with the commit date.
 
-This issue will be mitigated when Smart_Ezbench lands as one could copy the benchmark results of a previous run in the new report folder to avoid re-executing them. This would be made easy if using folders per commits.
+=== Potentially share commit results between reports ===
 
-The actual solution will be to store results in a separate folder with the state that got used to generate them. This would allow re-using results when nothing changed :)
+Benchmarks take forever to run, so it really is infuriating to have to re-run
+them over and over again when nothing changed!
 
-In the mean time, we can store the results of each commits in a separate folder
+This issue will be mitigated when Smart_Ezbench lands as one could copy the
+benchmark results of a previous run in the new report folder to avoid
+re-executing them. This would be made easy if using folders per commits.
 
-== Smart Ezbench ==
+The actual solution will be to store results in a separate folder with the state
+that got used to generate them. This would allow re-using results when nothing
+changed :)
 
-Smart Ezbench is trying to store all the necessary information about benchmarking report to the disk so as if the computer crashes or reboots, the benchmarking can keep on going.
+In the mean time, we can store the results of each commits in a separate folder
 
 === Experiment mode ===
 
 There is currently only one mode to ezbench, it is making a report.
 
-This is not very convenient during the development phase as we often want to compare different approches to a baseline.
+This is not very convenient during the development phase as we often want to
+compare different approches to a baseline.
 
-The goal would be to remember what were the settings set for the baseline and automatically run the experiment when asked to by the user.
+The goal would be to remember what were the settings set for the baseline and
+automatically run the experiment when asked to by the user.
 
-At a user's wish (amd probably at the end of every run), a report should be created to show the differences.
+At a user's wish (amd probably at the end of every run), a report should be
+created to show the differences.
 
-Upon changes to the baseline parameters (set of benchmarks, # of runs, ...), the previous experiments should also be updated to contain all the needed data. This allows the developer to add data over-night for a broader set of benchmarks or reducing the variance by adding runs.
+Upon changes to the baseline parameters (set of benchmarks, # of runs, ...), the
+previous experiments should also be updated to contain all the needed data. This
+allows the developer to add data over-night for a broader set of benchmarks or
+reducing the variance by adding runs.
+
+EDIT: Profiles already get us close to this, but not entirely. I will need to
+think more about this
 
 == Utils ==
 
 === Easy way to build the graphics stack from git ===
 
-To ease up the set-up of the build environment which is also error-prone and prone to having differences from machines to machines, we propose adding a bunch of scripts that would set up the graphics stack from git, have sane defaults values and work together.
+To ease up the set-up of the build environment which is also error-prone and
+prone to having differences from machines to machines, we propose adding a bunch
+of scripts that would set up the graphics stack from git, have sane defaults
+values and work together.
+
+Having such a script set could allow us to store the git SHA1 IDs and build-ids
+in the generated binaries.
 
-Having such a script set could allow us to store the git SHA1 IDs and build-ids in the generated binaries.
+EDIT: Chad's script would be good for mesa already!
+    http://sarah.thesharps.us/2015/12/17/building-a-custom-intel-graphics-stack/
 
 == gen_report.py ==
 
 === Allow filtering the results ===
 
-There is a ton of data in a performance report. It would be nice if we could filter the rest of the data when we unselect a benchmark in the trend view.
+There is a ton of data in a performance report. It would be nice if we could
+filter the rest of the data when we unselect a benchmark in the trend view.
+
+== env-dump ==
+
+ - Save the SHA1 of the config (zcat /proc/config.gz | sha1sum -) and compare
+ that to the SHA1_DB to get where it comes from. If it is not found in the DB,
+ I guess we will have to keep it as unknown and hope the kernel version is
+ enough information for us to check the config.
+
+ - Save the kernel parameters
+
+ - Save the list of modules loaded
+
+ - Save the configure command line in the SHA1-DB
+
+ - Add a way to save storage by storing a diff with a previous report. This
+ means we need to add support for applying a patch