| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
So that random keys show up somewhat nicely.
|
|
Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
|
|
This was initially generated via the following sed command:
sed -i \
-e 's@in \(.*\)\.iter\(values,keys,items\)()@in six.iter\2(\1)@g' \
-e 's@in \(.*\..*\)\.iter\(values,keys,items\)()@in six.iter\2(\1)@g'
Then cleaned up by hand, including changes for view*.
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
Acked-by: Jose Fonseca <jfonseca@vmware.com>
|
|
This is not feature complete for python 3.x in and of itself, but it
gets started by using six functions rather than str and unicode.
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
Acked-by: Jose Fonseca <jfonseca@vmware.com>
|
|
Exceptions were not reaching it.
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
This adds a new "summary feature" command to piglit which creates a HTML table
with the feature x DUT status (useful for multiple features, multiple DUTs).
Another use case is the feature status for subsequent test results (depending
on the meaning of that test result - DUT or build)
A feature readiness is defined by the piglit regexp which selects the tests
relevant for that feature and the acceptance percentage threshold (pass rate).
It requires an input json file containing the list of features, in the
following format (this is just an example):
{
"glsl" : {
"include_tests" : "glsl",
"exclude_tests" : "",
"target_rate" : 90
},
"arb" : {
"include_tests" : "arb_gpu",
"exclude_tests" : "",
"target_rate" : 10
}
}
v3:
Changed json rate to int instead of string
Applied other review comments
v2:
Apply review comments (Dylan, Thomas)
Fixed 2nd round of review comments
Signed-off-by: Gabriel Feceoru <gabriel.feceoru@intel.com>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
This makes use of the time TimeAttribute for handling times in results,
both for tests and for runs.
This change requires bumping the json version, and a large chunk of the
changes in this patch are for that change.
This could be split into two patches, but that would require making two
sequential bumps to the json results version, and I felt that the larger
patch was better than making two version bumps.
Reviewed-by: Mark Janes <mark.a.janes@intel.com>
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
|
|
It's like glxinfo or wglinfo, but for OpenCL.
Reviewed-by: Serge Martin <edb+piglit@sigluy.net>
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
|
|
This patch fixes the handling of subtests when more than one result is
presented. Currently this will raise a KeyError and die horribly.
cc: Ilia Mirkin <imirkin@alum.mit.edu>
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
|
|
This reduces the amount of code considerably, and doesn't add too much
complexity to the template. The result is easier to maintain, and that
the code used is better broken up.
This version is also slightly faster (as measured with two results),
moving from ~1.8 seconds to ~1.2
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
|
|
This is a very invasive patch, because it replaces one of our core
data-structures with a completely different kind of object. This new
object is not a dict-like object (it doesn't use obj[key] = value
syntax), instead it's a standard object with the standard object
attributes. So result['time'] becomes result.time.
This approach has a couple of advantages. First, it allows us to use
properties, which allows us to encapsulate a lot of distributed logic
from the summary module in the results module, and in a way that is
easier to test for correctness. The second advantage of that
encapsulation is that correct behavior is used everywhere, instead of
just in most places. Finally, it allows us to use the flyweight pattern
on the results objects, limiting the amount of memory consumed.
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
|
|
This will be used later in this series
This status has a background color of purple in the html summary, and is
considered the worst status.
My logic for making it the worst status is as follows: basically we
always want this to show up as a regression, because it's a very unusual
status (only available when piglit is aborted or when the whole system
goes down).
v2: - use white text instead of black, it makes things infinitely more
readable
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
Reviewed-by: Mark Janes <mark.a.janes@intel.com>
|
|
The commit above changed the template to take a single value rather than
nearly a dozen, but the template arguments changed a few of the names to
shorter versions, including environment to env. The template was updated
incorrectly, and environment has not been printed since then.
This patch fixes this simple bug.
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
|
|
Add the totals to the testrun_info template, including the total number
of tests evaluated.
Signed-off-by: Thomas Wood <thomas.wood@intel.com>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
With this patch tests with subtests are treated as groups, their status
is thrown away, and they inherit the 'worst' status from their children
just like a group.
This reverts commit de4b13de226e140313c3571e59a57438626da183.
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
Signed-off-by: Thomas Wood <thomas.wood@intel.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
This adds support for running XTS from piglit.
This patch was original written by Eric Anholt, but I've cleaned it up
and reworked parts of it for upstreaming, and as a result I've attached
my name as the author so ``git bisect'' and ``git blame'' will send
angry mail my way instead of Eric's.
v2: - Don't break dmesg reporting
- Use str.format consistently
- populate profile after check for xtest symlink
v3: - rebase against master
- fix spelling errors
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
|
|
Signed-off-by: Damien Lespiau <damien.lespiau@intel.com>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
Previously each value of a dictionary was pulled out and passed one by
one to mako, that results in a lot of values being passed. With this
patch the dictionary is passed and the keys are pulled out in the mako.
This is much cleaner and easier to read.
v2: - fix dmesg is list problem by altering the dictionary
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
|
|
Recent changes to the JSON file spilt stderr, stdout, and returncode
values out of a single 'info' string into three distinct values, but
didn't update the summary to accept those values.
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
|
|
This reverts commit ab2eb660925e3680e868c5f469a4ef32e6dd9a59.
This patch has a bad interaction with the locking in the json writing
library, running concurrently with this patch enabled can cause json
corruption that cannot easily be untangled, and even when it is is the
result is of low trustworthiness.
v2: - Add message explaining why this patch is to be reverted
- fix framework/status.py docstring that was mangled after the
revert
Acked-by: Daniel Vetter <daniel@ffwll.ch>
Reviewed-by: Kenneth Gruanke <kenneth@whitecape.org>
Signed-off-by: Dylan Baker <baker.dylan.c@mgmail.com>
|
|
This reverts commit e0d4bdbaa38a7441b0f38e115bd2daf3c6a05da3.
|
|
This patch causes tests with subtests to be treated as a group, rather
than as a test. This means that the status the test itself stores will
be overwritten by those in the subtest.
There is one oddity about this to be aware of; a test with subtests that
crashes or fails before any of the subtests run will report a fraction
of 0/1 with the appropriate color, even though all of the subtests will
report Not Run.
v2: - Add subtests to the results file as full tests (the internal view
of the json), without this they will not appear in changes, fixes,
etc
- Render the background color of Not Run tests correctly in HTML
- Apply subtest fractions down the stack
v3: - Don't generate a link for Not Run tests in html page
Tested-by: Ben Widawsky <ben@bwidawsk.net>
Tested-by: Tom Stellard <thomas.stellard@amd.com>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
i-g-t tests can take a long time, and for a bunch of reasons (bad
tuning on disparate set of platforms, stuck in the kernel which often
can be recovered by interrupting with a signal, ...) that sometimes
extends to a good approximation of forever.
Hence this adds a per-test timeout value and a bit of infrastructure
to adjust the results. Test results adjusting is done after calling
interpretResult so that we don't have to replicate this all over the
place. This might need to be adjusted for the piglit-native subtest
stuff, but otoh igt is a bit special with it's crazy long-running
tests. So I think we can fix this once it's actually needed.
The default timeout is None, so this is purely opt-in.
Note on the implementation: SIG_ALARM doesn't exists on Windows and
stackoverflow overwhelmingly recommended to go with this thread-based
approach here. But only tested on my Linux box here.
I've also timed quick.tests run a bit and the overhead due to the
additional thread we launch seems to be in the wash. So I didn't opt
to make the thread launching optional if there's no timeout limit.
v2: Also add all the boilerplate needed to handle the new test status
in summaries. For the color I've opted for two shades of light blue,
they nicely stick out from the current selection.
v3: Fix GLSLParserTest and ShaderTest. They both derive from
PlainExecTest but only call the __init__ function of the Test
baseclass. I haven't really figured why this is and also not really
what command I should pass to PlainExecTest.__init__, so just
replicated the init code for now.
v4: Initialize timeout earlier for tests where we use the ENOENT
handling to skip them.
Fix up the out, err passing from the thread. Apparently my idea of how
python closures work was completely misguided - like with a function
call a closure captures a copy of of a reference pointing at the
original object. So assinging anything to them won't have any effect
outside of the lambda. Hence we need to jump through hoops and pass an
array around. Nicer fix would be to create a class or something, but
that seems like overkill.
v5: Fixup the fraction setting in status.py
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
Cc: Dylan Baker <baker.dylan.c@gmail.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
|
|
Specifically for glean tests setting certain environment variables are
needed to run a certain subtest in glean. This patch prints that in the
HTML results.
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
If a test generates output containing '<', '>', or '&', we need to
HTML escape it so that the web browser doesn't interpret it as
containing HTML tags.
While we're at it, go ahead and HTML escape the traceback and dmesg
outputs too.
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
The Radeon driver writes GPU page faults to dmesg and we need to know which
tests caused them.
If there is any change in dmesg during a test run, the test result is changed
as follows:
* pass -> dmesg-warn
* warn -> dmesg-warn
* fail -> dmesg-fail
Dmesg is captured before and after each test and the difference between the two
is stored in the test summary.
The piglit-run.py parameter which enables this behavior is --dmesg. It's also
recommended to use -c0.
v2: - address some of Dylan's remarks, mainly bug fixes
- fix get_dmesg_diff
Reviewed-by: Dylan Baken <baker.dylan.c@gmail.com>
|
|
If the Python framework fails to run a program for some reason, it'll
record a 'fail' status. It also records a 'traceback' key in the
dictionary.
The new summary code missed this when writing the test detail page,
resulting in tests marked as 'fail' with no indication as to why.
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
Signed-off-by: Kenneth Graunke <kenneth@whitecape.org>
|
|
This new page is able to convey the same information, but takes less
time to generate
v2: - Change message to say "No ${page}".
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
It is absolutely silly to generate pages for fixes, changes, and
regressions when there is only one test result is provided. This still
generates empty pages when two results are provided, because some
developers have expressed a work flow that uses these empty pages.
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
This provides a switch and method for excluding the generation of the
test result HTML files (those that live under <testrun name>/). This
allows the user to trade some verbosity in the results for a significant
increase in generation speed. A run of quick.tests with all options
enabled takes ~5.5 seconds, with just pass and skip disabled it only
takes ~2.5 seconds. This only becomes more significant as more testsruns
are added the HTML page.
V3: - Remove some bad comments
- adds 'skip' to exclude list
V4: - Renames --exclude-generation to --exclude-details
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-By: Aaron Watry <awatry@gmail.com>
Reviewed-By: Kenneth Graunke <kenneth@whitecape.org>
|
|
In the old summary code one was set equal to the other, and then
written. In order to produce the same output they were both brought over
to the mako code. But they're rather silly and worthless and only one
should be kept.
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-By: Aaron Watry <awatry@gmail.com>
Reviewed-By: Kenneth Graunke <kenneth@whitecape.org>
|
|
With 10,000+ tests all living in the same folder it can be hard to
identify a single file, especially when trying to work with the HTML
generation itself. This patch changes the behavior so that each test
has a directory tree for the group results with tests under it (ie
spec/GLSL 1.0/foo/bar/test.html)
V3: - Correct some spelling errors/remove excess comments
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-By: Aaron Watry <awatry@gmail.com>
Reviewed-By: Kenneth Graunke <kenneth@whitecape.org>
|
|
Change the div:nth-child to operating on everything except head to only
working on group. This is lays groundwork for later changes.
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-By: Aaron Watry <awatry@gmail.com>
Reviewed-By: Kenneth Graunke <kenneth@whitecape.org>
|
|
These template files are no longer used, and so they should be removed
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-By: Aaron Watry <awatry@gmail.com>
Reviewed-By: Kenneth Graunke <kenneth@whitecape.org>
|
|
Adds the actual templates used to by the new mako based summary code.
V3: - Replace tabs with spaces like python code
- Rename test_index.mako to testrun_info.mako
- Refactor testrun_info.mako links bar generation
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
Reviewed-By: Aaron Watry <awatry@gmail.com>
Reviewed-By: Kenneth Graunke <kenneth@whitecape.org>
|
|
We weren't specifying a DTD, which made validator.w3.org angry. There's
no reason not to be strict or use SGML-based HTML, so go with XHTML.
Signed-off-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
The usual "auto" table layout algorithm makes the browser look at the
contents of each row in the table in order to determine the layout. We
have 10,000 rows. This is insanely expensive.
The "fixed" algorithm just uses the <col/> info, and possibly the first
row. This works out just fine.
Signed-off-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
When a report contains many columns, the table may need to be larger
than the width of the page. Currently that happens due to the browser
looking at the table content in order to determine the size, but when we
switch to table-layout: fixed, it would limit it to the page size.
Doing this now avoids the problem before it first occurs.
Signed-off-by: Kenneth Graunke <kenneth@whitecape.org>
|
|
This accommodates the increase in the total number of tests, so that the
column default width is great enough for the 'all' category, which is
now over 5 digits, a total of 11 characters.
Signed-off-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
There's no need to specify width="50pt" on every <col/>; we can easily
specify a width for all but the first column with CSS3.
At that point, using a template file that contains only <col/> seems
absurd; just repeat that the right number of times in the python code.
Signed-off-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
It was entirely empty.
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
Most browsers support the :nth-child selector by now, which allows us to
zebra-stripe the tables in CSS rather than making the python code
annotate every row with an 'a' or 'b' class.
While we're at it, stripe the pass/fail/crash results too. Based on
some CSS fragments, this was the intent all along, but didn't work.
This also changes the striping slightly: it now stripes purely based on
whether the row is even/odd, rather than striping on a per-group basis.
Signed-off-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
Previously, we indented groups of tests by adding 10 columns
of whitespace to the table and adding colspan. This is pretty
ugly; we really should use CSS for things like this.
The new CSS-based approach should work with deeper nesting levels,
and simplifies the tables.
Signed-off-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Dylan Baker <baker.dylan.c@gmail.com>
|
|
While distinguishing between 'fail' and 'crash' in extremely useful,
I've never encountered a situation where 'abort' or 'trap' offers more
insight than simply calling it a 'crash'.
We may as well simplify things a bit.
Signed-off-by: Kenneth Graunke <kenneth@whitecape.org>
Acked-by: Marek Olšák <maraeo@gmail.com>
|
|
Previously piglit-summary-html didn't explicitly encode its output at
all--it relied on the default behavior of the Python file.write()
method (which, in Python 2.x, is to write 8-bit strings verbatim and
to encode unicode strings using ASCII). This meant that if a
non-ASCII character appeared in any of the test data,
piglit-summary-html would fail with an exception.
This patch changes piglit-summary-html to explicitly encode its output
using UTF-8. It also updates all of the HTML templates to specify
UTF-8 as the encoding, so that when results are viewed in a web
browser, non-ASCII characters will display correctly.
|
|
To distiguish between 'fail' and unexpected termination.
Acked-by: Ian Romanick <ian.d.romanick@intel.com>
Acked-by: Eric Anholt <eric@anholt.net>
Acked-by: Chad Versace <chad.versace@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
|
|
|
|
|
|
|
