summaryrefslogtreecommitdiff
path: root/framework
AgeCommit message (Collapse)AuthorFilesLines
2017-07-21framework: remove duplicate import of framework.exceptionsHEADmasterDylan Baker1-1/+1
Reported-by: Matt Turner <mattst88@gmail.com> Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Reviewed-by: Martin Peres <martin.peres@linux.intel.com>
2017-07-18framework: Don't stop resume because no tests are left to be runDylan Baker1-5/+10
It's completely possible that if piglit resume is run with the -n flag (don't retry incomplete tests), that piglit resume could be invoked with no tests left to run. In this case it makes sense to let piglit resume just finalize the backend and produce a unified result. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> cc: Martin Peres <martin.peres@linux.intel.com> Tested-by: Hector Edmundo Ramirez Gomez
2017-07-11wgl: initial check-in of wgl utility code and wgl sanity check programBrian Paul1-1/+1
Like the glx tests/utility code, but for wgl. Note, one must set the PIGLIT_PLATFORM env var to "wgl" before running Piglit. It looks like there's some Waffle work to look at before this can be made automatic.
2017-05-25test/gleantest: fix missing --quick property from quick profileTimothy Arceri1-1/+1
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
2017-05-18framework: Add command.setter method to TestDylan Baker4-0/+17
This allows the command to be overwritten or modified after instantiation, which is useful for adding additional arguments in a profile. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2017-04-27framework/test/shader_test: use -fbo option for shader testsBrian Paul1-2/+2
Running shader tests with a FBO instead of a window is faster in some environments. For example, for a Windows7 VM (debug build of everything) the runtime of a large piglit run drops from to 3.7 hours to 3.2 hours. In a Linux guest, piglit-run.py -t built-in-functions drops from 19 minutes to 15 minutes. No regressions found with VMware driver in Windows VM and Linux VM. Reviewed-by: Eric Anholt <eric@anholt.net> Reviewed-by: Dylan Baker <dylan@pnwbakers.com>
2017-04-13framework: Exit if a filter removes all tests from a profileDylan Baker1-0/+4
It doesn't makes sense to run if a user has removed all tests from selected profiles, and currently if all tests are removed, then an assertion will be hit in the backend that isn't extremely clear about what went wrong. This should be much easier to understand. v2: - simplify logic somewhat. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Reviewed-by: Brian Paul <brianp@vmware.com> (v1) Tested-by: Brian Paul <brianp@vmware.com> (v1)
2017-04-13framework/exceptions: Add an exception class for user errorsDylan Baker1-0/+11
It's nice to separate "something internally went wrong" and "you mis-configured or passed bad flags". Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2017-02-10framework/backends/json.py: preserve 'tests' orderTomi Sarvela1-12/+18
Use OrderedDict when reading tests data in. Reviewed-by: Dylan Baker <dylanx.c.baker@intel.com>
2017-02-10framework/results.py: Copy 'tests' internally to OrderedDictTomi Sarvela1-3/+4
To preserve the original test running order, use OrderedDict when copying 'tests' values over. Reviewed-by: Dylan Baker <dylanx.c.baker@intel.com>
2017-02-06framework/test/glsl_parser_test.py: fix python script to run glslparsertest ↵Neha Bhende1-2/+7
on Windows glslparsertest was not running on windows because python script is looking for glslparsertest. On windows, piglit creates glslparsertest.exe. With this fix, all glslparsertest will run on Windows. Reviewed-by: Charmaine Lee <charmainel@vmware.com> Reviewed-by: Brian Paul <brianp@vmware.com>
2017-02-02framework/programs/run.py: Allow comments in test-list files.Petri Latvala1-2/+3
IGT testing in Intel's CI is using static lists of tests to run. Commenting out tests and annotating them will help human-readability. v2: Use comprehensions (Dylan) CC: Dylan Baker <dylan@pnwbakers.com> Signed-off-by: Petri Latvala <petri.latvala@intel.com> Reviewed-by: Dylan Baker <dylan@pnwbakers.com>
2017-01-30framework/backends/json: always flush the metadata to the diskMartin Peres1-0/+4
This guarantees that whatever the state we are in, we can always continue the piglit run. Do not use FileBackend.__fsync as it would not do the fsync call in the non-sync mode. This change makes it easier to integrate into a CI system that is hard-reboot-resistant. Reviewed-by: Dylan Baker <dylan@pnwbakers.com> Signed-off-by: Martin Peres <martin.peres@linux.intel.com>
2017-01-12resume: use the previously-set test-list when resumingMartin Peres1-11/+20
Previously, the test-list was not saved when running and resuming a run would ignore the test-list and execute the entire test set. This patch tries to follow Dylan's proposal[0]. [0] https://patchwork.freedesktop.org/patch/122189/ Cc: Rami Ben Hassine <ramix.ben.hassine@intel.com> Cc: Olivier Berthier <olivierx.berthier@intel.com> Cc: Julian Dumez <julian.dumez@intel.com> Cc: Dylan Baker <dylan@pnwbakers.com> Reported-by: Rami Ben Hassine <ramix.ben.hassine@intel.com> Reviewed-by: Dylan Baker <dylan@pnwbakers.com> Signed-off-by: Martin Peres <martin.peres@linux.intel.com>
2017-01-09deqp: Search stdout and stderr for X connection failureMark Janes1-2/+4
Recent versions of the GL CTS report X connection errors on standard out. Support all variants by checking both stdout and stderr for the error string. Reviewed-by: Dylan Baker <dylan@pnwbakers.com>
2016-11-21framework/profile: Use re.IGNORECASE for RegexFilterDylan Baker1-1/+1
This flag makes the regular expressions match case-insensitive, and is useful since piglit treats test names as case insensitive (lowering them all by default). This was inadvertently changed to not use the flag in e92555a647, which replaced the part that added this flag with a simpler implementation, so this patch reverts to the previous behavior. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Reviewed-by: Kenneth Graunke <kenneth@whitecape.org> Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
2016-11-15framework/profile: don't copy.deepcopy when copying profilesDylan Baker1-3/+2
This reduces memory consumption by a lot (about 60% on my system), which fixes some tests that fail sporadically on low memory systems. bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=98725 Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Tested-by: Mark Janes <mark.a.janes@intel.com> Reviewed-by: Mark Janes <mark.a.janes@intel.com>
2016-11-15framework/backends/json: support non-piglit junit filesMark Janes1-8/+22
The junit loader is unnecessarily strict with the input that it accepts. It expects input generated by piglit, but can be made to handle junit from other test suites like crucible. v2: Remove unnecessary initializers Conditionally parse command for piglit tests
2016-11-11framework/programs/run: Fix bug when using --test-profileDylan Baker1-1/+1
3d887afe4762 has a rebase error in it that causes --test-profile to raise an exception due to a typo. In a previous version of the patch args.test_profile had been changed to args.test_profiles since more than one can be specified, but was removed since it was more than one logical change in a single patch, and was dropped from the series. This trivial patch corrects the issue. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/test/glsl_parser_test.py: split the parser out of the classDylan Baker1-39/+46
This pulls the code for parsing the test file out of GLSLParserTest and puts them in a new Parser class. This is going be useful later for allowing the parsing to be done during the compile stage, and not require searching the file system at runtime. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/test/base: Remove timeout parameterDylan Baker1-4/+1
This doesn't work and never has, setting it would cause an exception, and I'm not exactly sure how to fix it. It's also unused, so we can bring it back later if we need it. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Move group_manager from TestProfile to TestDictDylan Baker1-56/+48
This move is going to allow us to supplement the TestDict with a different class that can be used instead that loads xml. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: remove unused argument from TestProfile.group_managerDylan Baker1-1/+1
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: replace Testprofile.{dmesg,monitoring} with dictDylan Baker4-70/+34
This allows a significant amount of cleanup to happen. It allows removing attributes from the global OPTIONS, removing tests that no longer apply, and because of the split run method it allows more values to simply be passed. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Split try/except blockDylan Baker1-5/+8
This will avoid catching AttributeErrors when importing, and only catch them if mod.profile doesn't exist. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Update docstringsDylan Baker1-53/+48
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Update __all__Dylan Baker1-0/+2
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Don't alter the TestProfile before runningDylan Baker3-81/+45
This patch changes the way that the tests from TestProfile are processed. Rather than having a process_test_list method that modifies TestProfile.test_list, a new itertests() method creates an iterator that yields tests that are not excluded by filters. This saves a bit of code, increases the memory usage, and reduces surprise. It would be nice if there wasn't a need to create a concrete test list, but there wouldn't be any way to get the number of tests otherwise. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Factor out check_all closureDylan Baker1-11/+3
This does away with the check_all closure, by simplifying it down to an equivalent lambda expression. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework: Pull {include,exclude}_filter out of OptionsDylan Baker5-172/+79
Since these are also just special cases of filters for the standard TestProfile filtering mechanism, and they have a lot of unique classes. This is just a waste, the same can be achieved with a much simpler class structure. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework: Remove exclude_tests from options.OPTIONSDylan Baker3-3/+4
Rather than putting this in a global variable, just add a filter for this in the runner. Far simpler, and removes more globals. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/programs/run: import framework.profile as profileDylan Baker1-7/+7
Since there isn't a conflict in the name anymore, this just makes the code cleaner and easier to read. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Drop TestProfile.filter_testsDylan Baker1-13/+3
This was just a thin wrapper around the already public TestProfile.filters list. It seems silly to have such a method when it does nothing but call self.filters.append. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/programs/run: remove redundant list() around comprehensionDylan Baker1-1/+1
This is a list comprehension, it already results in a list. There is no need to wrap it in the list() constructor call. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Don't merge profilesDylan Baker3-76/+58
Because we can copy profiles, we don't need to merge them to run more than one of them. Instead we can simply have a list of profiles, and run them one by one. One side effect of this is that tests will be run one profile at a time, so if running with out the -1/--no-concurrency or -c/--all-concurrent options tests will run in a sort of zipper pattern: <p1 concurrent>, <p1 non-concurrent>, <p2 concurrent>, etc. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/programs/run: Only allow --test-list if one profileDylan Baker1-0/+4
This limitation has basically always existed, but for the next patch to function this is a requirement. This limitation will be properly fixed in a follow up series (that was originally part of this series, but the size and scope of the series got too big so this was split.) Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: add a copy method to profileDylan Baker2-1/+17
This will allow a profile to be copied and "subclassed" without affecting the original profile. This will allow a long-standing bug that made it impossible to run two subclasses of all.py (say shader.py and glslparser.py) at the same time, since they would both try to modify the all.py profile in incompatible ways. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/programs/run: Remove use of TestrunResultDylan Baker1-13/+6
We basically only used it to set the name and the time elapsed. That's silly, just do those things directly. It needs less code and doesn't require creating a big object. This is leftover from the days before atomic writes, when results were all stored in memory until the end of the run. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework: remove concurrent from OPTIONS, pass directly to profileDylan Baker3-9/+12
This removes a value out of the global OPTIONS, which is nice. It's very trivial to pass this instead of putting it in options. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework: Split the run method out of profile.Dylan Baker3-70/+71
There are a couple of reasons for doing this. First, profile is a big complex mess that does entirely too much, and this helps with that. Second, there are bugs in the way two profiles run at the same time work, and this is going to fix that. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Drop unneeded exceptDylan Baker1-7/+1
A try/finally works better here anyway. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: make run hooks publicDylan Baker1-20/+8
These are used in a couple of places, and since they're meant to be changed by subclasses they need to be public. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/result: TestrunResult.options should be a dictDylan Baker1-1/+1
This changes the default type from None to a dict, which is what it is when setup in framework/programs/run. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-09framework/test/shader_test: Fix MultiShaderRunner handling of directoriesDylan Baker1-1/+15
Currently MultiShaderRunner (used with --process-isolation false) expects each directory will contain either GL, GLES2, or GLES3 shaders. The barrier between GLES2 and GLES3 is somewhat artificial, since they can be promoted to GLES3. This patch allows GLES2 and GLES3 shaders to be run together but still enforces a separation between GLES and GL shaders. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Tested-by: Michel Dänzer <michel.daenzer@amd.com> Tested-by: Marek Olšák <marek.olsak@amd.com>
2016-11-04framework/backends/json: Handle updating results without pid fieldDylan Baker1-1/+4
If it's not present make it an empty list. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Tested-by: Brian Paul <brianp@vmware.com>
2016-10-26framework: fix loading JUnit resultsDylan Baker1-1/+5
The JUnit loader was never updated to handle PID's as lists rather than a single int. This patch corrects that and allows JUnit results to be loaded again. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Tested-by: Mark Janes <mark.a.janes@intel.com>
2016-10-26framework: Properly load JSON into internal representationDylan Baker2-6/+19
Currently piglit doesn't consistently deserialize JSON into it's internal representation, leaving some bits as dictionaries or lists, which breaks the summary code. This patch corrects that. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Tested-by: Mark Janes <mark.a.janes@intel.com> Tested-by: Michel Dänzer <michel.daenzer@amd.com>
2016-10-24framework/backends/json: Don't convert to TestrunResult while updatingDylan Baker2-47/+33
This changes the way updates are done in the backend, instead of converting to a TestrunResult immediately, all of the transformations are done to the JSON data in it's rawest form, ie, as dicts and lists, and then transform to a TestrunResult (and children) after that. This makes the loading code more robust and simpler, since it's decoupled from the representation, making the transformations easier to test and manage. Part of this change is fixing the .to_json and .from_dict methods, many of which "worked" because their shortcomings were papered over by using json.load with a custom decoder. This patch fixes them to actually work correctly. Despite my best attempts I couldn't decouple this work for this patch because of the close coupling of the JSON loading code and the class representations. There are a number of fixups to the tests in this patch, since a number of issues were being covered by the TestrunResult.to_json() method filling it missing values. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-10-24framework: Drop support for JSON formats < 7Dylan Baker1-256/+10
Most of these are pretty old at this point (over a year), and it seems unlikely anyone still has results in those formats. The biggest problem with them is that some of them (especially the early ones) are complex transformations with a lot of validation needed, and some of the middle ones rely on the class representations to do the transformations, which is what this series is trying to change, since the tight coupling makes changes to the schema difficult, or nearly impossible in other cases. If there is compelling reason I could push support back a bit more, but I really want to drop some of the versions we support since it simplifies the code considerably. This also drops support for the older "main" file, and only supports results.json and results.json.<compression> Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-10-24framework/backends/json: replace str() with six.text_type()Dylan Baker1-1/+1
since str() is incorrect in python2.x. This also fixes an indentation error. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-23 08:05:34 +0000