summaryrefslogtreecommitdiff
path: root/unittests
AgeCommit message (Collapse)AuthorFilesLines
2018-09-20framework/test: Convert tests to new fast-skip interfaceDylan Baker3-116/+127
This converts the two test types that use the fast-skipping mechanism (glsl parser and shader) to use the new interfaces. This has been verified by running with PIGLIT_NO_FAST_SKIP=1 and without, and sans 1 test (that is fixed later in the series) the results are the same.
2018-09-20unittests/wflinfo: Update tests for new APIDylan Baker1-83/+35
2018-09-20wflinfo: Reimplement wflinfo separating the various API'sDylan Baker1-3/+20
Currently fast skipping is implemented such that it assumes there's a single version of ES, a single version of desktop, and all extensions are shared between them. This has basically worked because 1) there are basically no gles1 tests, and 2) piglit didn't have compat profile. But worked and correct are two different things. With the addition of compat profiles it's time to re-evaluate how fast skipping works. Namely we need to have different attributes for ES1, ES1+, core, compat, and I've added on for "legacy" (pre-profile), since waffle supports that. This maintains legacy interfaces so that existing code continues to work. v2: - Fix versions < 3.1 on implementations without core profile
2018-07-24JSON: move info utilities out of rootDylan Baker4-39/+234
And into a generic sub element in the output. This makes information like glxinfo and clinfo not a hard requirement and will allow any sort of system information to be encoded.
2018-07-20framework: deepcopy TestProfile's filters when copying itAndres Gomez2-2/+2
This avoids modifications in the original filters object. Detected through Travis CI. Fixed also a couple of typos. Fixes: 1f0f2ad25 ("framework: Add filters class") Cc: Dylan Baker <dylanx.c.baker@intel.com> Cc: Michel Dänzer <michel.daenzer@amd.com> Signed-off-by: Andres Gomez <agomez@igalia.com> Reviewed-by: Dylan Baker <dylan@pnwbakers.com> Reviewed-by: Rhys Kidd <rhyskidd@gmail.com>
2018-05-01tests/glsl_parser_test.py: fix is_skip for serialized profilesDylan Baker1-7/+6
Currently is_skip() relies on runtime detection of which glslparsertest binaries are built, but we can't assume that at build time. Instead always assign the appropriate binary, and then check for the existence of that binary at run time. Tested-by: Rafael Antognolli <rafael.antognolli@intel.com>
2018-05-01profile: Add support for loading xml based profilesDylan Baker1-5/+0
Which includes de-serialization on demand. This will vastly reduce the amount of resident memory that is necessary to hold the tests since test instances are made on demand, and can be garbage collected as soon as the run in complete. Tested-by: Rafael Antognolli <rafael.antognolli@intel.com>
2018-05-01framework/test: Split multishader tooDylan Baker1-2/+15
Tested-by: Rafael Antognolli <rafael.antognolli@intel.com>
2018-05-01framework: do the same for shader testDylan Baker1-10/+10
Tested-by: Rafael Antognolli <rafael.antognolli@intel.com>
2018-05-01framework: use a class method for building test via parsingDylan Baker1-24/+24
Since the serialized tests wont need this path, build a generic constructor that this special constructor can leverage. Tested-by: Rafael Antognolli <rafael.antognolli@intel.com>
2018-04-20unittests: mustpass is a text listJuan A. Suarez Romero1-19/+9
Commit 7cf9e743a updated deqp mustpass lists to use text based mustpass. Hence unit tests should use text based mustpass too. Fixes: 7cf9e743a ("framework: update deqp mustpass list for text based mustpass") Reviewed-by: Andres Gomez <agomez@igalia.com> Reviewed-by: Dylan Baker <dylan@pnwbakers.com>
2018-02-05framework: if a test with subtests crashes mark the offending subtestDylan Baker1-0/+36
This relies on the fact that subtests are guaranteed to be ordered to mark the crashing subtest as such. This ensures that the correct status will be propagated up the totals tree. Signed-off-by: Fabian Bieler <fabianbieler@fastmail.fm> Reviewed-by: Dylan Baker <dylan@pnwbakers.com>
2017-12-19unittests: update python generatorAndres Gomez1-1/+6
Cc: Dylan Baker <dylanx.c.baker@intel.com> Signed-off-by: Andres Gomez <agomez@igalia.com> Reviewed-by: Dylan Baker <dylan@pnwbakers.com> Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
2017-12-19unittests: correct tessellation typoAndres Gomez1-8/+8
Cc: Dylan Baker <dylanx.c.baker@intel.com> Signed-off-by: Andres Gomez <agomez@igalia.com> Reviewed-by: Dylan Baker <dylan@pnwbakers.com> Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
2017-12-16Remove GleanTest framework.Fabian Bieler2-134/+0
2017-11-20unittests: test WflInfo in its own module and correct depending onesAndres Gomez3-359/+395
Fixes: edc41a1db ("framework: move WflInfo class into new wflinfo.py module") Cc: Brian Paul <brianp@vmware.com> Cc: Dylan Baker <dylanx.c.baker@intel.com> Signed-off-by: Andres Gomez <agomez@igalia.com> Reviewed-by: Brian Paul <brianp@vmware.com>
2017-11-13unittests: include new elapsed time from the console summary outputAndres Gomez1-5/+9
Fixes: e76ecd048 ("framework: also print elapsed time with piglit-summary.py") Cc: Brian Paul <brianp@vmware.com> Cc: Dylan Baker <dylanx.c.baker@intel.com> Signed-off-by: Andres Gomez <agomez@igalia.com> Reviewed-by: Brian Paul <brianp@vmware.com>
2017-07-21framework: Fix the same bug for resume in the previous commitDylan Baker1-8/+7
The resume path duplicates the 2 lines of code, so the same fix is required there. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2017-07-21framework: Only sort files for json backend that end in .jsonDylan Baker1-1/+0
This fixes trying to load .tmp files that don't get moved during an interrupted run. bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=101503 Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Tested-by: Hector Edmundo Ramirez Gomez <hector.edmundox.ramirez.gomez@intel.com>
2017-07-21unittests: Add tests for file in tests dir not ending in .jsonDylan Baker1-0/+11
This is currently expected to fail. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2017-07-21unittests/framework: remove 'autouse' from fixtureDylan Baker1-13/+16
This reworks the fixture to not use autouse, so that other tests can be added to the class without getting the setup it doesn't use. This also uses a class scope fixture instead of an instance scope one. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2017-05-25test_gleantest: make sure --quick is returned when set in GLOBAL_PARAMSTimothy Arceri1-0/+1
2017-05-18framework: Add command.setter method to TestDylan Baker4-12/+51
This allows the command to be overwritten or modified after instantiation, which is useful for adding additional arguments in a profile. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2017-01-19unittests: Add new python generator to unittestsDylan Baker1-0/+1
2016-11-10framework/profile: Move group_manager from TestProfile to TestDictDylan Baker1-79/+74
This move is going to allow us to supplement the TestDict with a different class that can be used instead that loads xml. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: replace Testprofile.{dmesg,monitoring} with dictDylan Baker3-32/+8
This allows a significant amount of cleanup to happen. It allows removing attributes from the global OPTIONS, removing tests that no longer apply, and because of the split run method it allows more values to simply be passed. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Don't alter the TestProfile before runningDylan Baker1-6/+0
This patch changes the way that the tests from TestProfile are processed. Rather than having a process_test_list method that modifies TestProfile.test_list, a new itertests() method creates an iterator that yields tests that are not excluded by filters. This saves a bit of code, increases the memory usage, and reduces surprise. It would be nice if there wasn't a need to create a concrete test list, but there wouldn't be any way to get the number of tests otherwise. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework: Pull {include,exclude}_filter out of OptionsDylan Baker2-264/+40
Since these are also just special cases of filters for the standard TestProfile filtering mechanism, and they have a lot of unique classes. This is just a waste, the same can be achieved with a much simpler class structure. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework: Remove exclude_tests from options.OPTIONSDylan Baker1-17/+0
Rather than putting this in a global variable, just add a filter for this in the runner. Far simpler, and removes more globals. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: Don't merge profilesDylan Baker2-27/+9
Because we can copy profiles, we don't need to merge them to run more than one of them. Instead we can simply have a list of profiles, and run them one by one. One side effect of this is that tests will be run one profile at a time, so if running with out the -1/--no-concurrency or -c/--all-concurrent options tests will run in a sort of zipper pattern: <p1 concurrent>, <p1 non-concurrent>, <p2 concurrent>, etc. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework/profile: add a copy method to profileDylan Baker1-0/+52
This will allow a profile to be copied and "subclassed" without affecting the original profile. This will allow a long-standing bug that made it impossible to run two subclasses of all.py (say shader.py and glslparser.py) at the same time, since they would both try to modify the all.py profile in incompatible ways. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10framework: Split the run method out of profile.Dylan Baker1-5/+5
There are a couple of reasons for doing this. First, profile is a big complex mess that does entirely too much, and this helps with that. Second, there are bugs in the way two profiles run at the same time work, and this is going to fix that. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-10unittests: Add tests for the feature completeness summaryDylan Baker1-0/+116
This adds a few simple tests for the feature completeness summary object. Nothing to strenuous but much better than nothing. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-08unittests/generators: Add gen_shader_framebuffer_fetch_testsDylan Baker1-0/+1
Adds this new generator to the list of generators to test. Trivial. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-11-04framework/backends/json: Handle updating results without pid fieldDylan Baker1-0/+23
If it's not present make it an empty list. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Tested-by: Brian Paul <brianp@vmware.com>
2016-10-26framework: fix loading JUnit resultsDylan Baker1-3/+3
The JUnit loader was never updated to handle PID's as lists rather than a single int. This patch corrects that and allows JUnit results to be loaded again. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Tested-by: Mark Janes <mark.a.janes@intel.com>
2016-10-26framework: Properly load JSON into internal representationDylan Baker3-52/+32
Currently piglit doesn't consistently deserialize JSON into it's internal representation, leaving some bits as dictionaries or lists, which breaks the summary code. This patch corrects that. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Tested-by: Mark Janes <mark.a.janes@intel.com> Tested-by: Michel Dänzer <michel.daenzer@amd.com>
2016-10-24framework/backends/json: Don't convert to TestrunResult while updatingDylan Baker3-64/+36
This changes the way updates are done in the backend, instead of converting to a TestrunResult immediately, all of the transformations are done to the JSON data in it's rawest form, ie, as dicts and lists, and then transform to a TestrunResult (and children) after that. This makes the loading code more robust and simpler, since it's decoupled from the representation, making the transformations easier to test and manage. Part of this change is fixing the .to_json and .from_dict methods, many of which "worked" because their shortcomings were papered over by using json.load with a custom decoder. This patch fixes them to actually work correctly. Despite my best attempts I couldn't decouple this work for this patch because of the close coupling of the JSON loading code and the class representations. There are a number of fixups to the tests in this patch, since a number of issues were being covered by the TestrunResult.to_json() method filling it missing values. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-10-24framework: Drop support for JSON formats < 7Dylan Baker2-747/+0
Most of these are pretty old at this point (over a year), and it seems unlikely anyone still has results in those formats. The biggest problem with them is that some of them (especially the early ones) are complex transformations with a lot of validation needed, and some of the middle ones rely on the class representations to do the transformations, which is what this series is trying to change, since the tight coupling makes changes to the schema difficult, or nearly impossible in other cases. If there is compelling reason I could push support back a bit more, but I really want to drop some of the versions we support since it simplifies the code considerably. This also drops support for the older "main" file, and only supports results.json and results.json.<compression> Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-10-07framework: Add class for running multiple shader_tests in a single processDylan Baker1-1/+62
This adds a class based on the ReducedProcessMixin that allows the python layer to understand the output of shader_runner when it runs more than one test per process. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-10-07framework: Add a Mixin class for running multiple tests in a single processDylan Baker1-2/+136
This Mixin makes writing classes for handling tests that run multiple tests in a single process simpler. It does this through the use of the subtest feature. It makes it possible to implement two new methods, and an aware interpret_result method and have support for this feature, including a cherry-like resume feature that starts again after a test crashes, without rerunning the crashed test. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-10-07framework: Bump JSON on disk format to version 9Dylan Baker3-2/+209
This change makes the TestResult.pid field into a list of ints instead of a list. This will be used by the multiple test per process tests to ensure that all PIDs are recorded when the cherry features is invoked. This is groundwork for later patches in the series. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-10-07framework: Split FastSkipMixin.Dylan Baker1-84/+108
This splits the FastSkipMixin into two classes. One that actually does the fast skipping, and one that provides a mixin for Test. This split will allow the class to be used in the traditional way, but also in shader_test.MultiShaderTest. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-10-05framework: Fix verbose loggerDylan Baker1-0/+6
Which was broken by the last commit touching the logger, as were several unittests that continued passing because of a dirty environment. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-09-16unittests: Add test for DriverClassifier.get_glxinfoDylan Baker1-0/+25
Ensure that it converts bytes to str and also collects correctly. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-09-16framework: Add a driver classifier based on the renderer string.Eric Anholt1-0/+60
For the apitrace test runner, I need to be able to store known-good images for various drivers, which means I need some way to group drivers. Not all Mesa drivers we care about are covered here, but this is a sample to get things started. v2: Apply feedback from Dylan, add classification for my Skylake. Reviewed-by: Dylan Baker <dylan@pnwbakers.com>
2016-09-09unittests: Fix JSON schema test to pass time_elapsedDylan Baker1-1/+2
Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-09-09unittests: Fix shared data for backends.Dylan Baker1-2/+5
It turns out that the backends coincidently put certain fields in, but the new jsonstreams backend does not automatically add these fields, it must be passed them. (This is the behavior of piglit when not testing). Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-09-09unittests: don't require totals section in schemaDylan Baker1-1/+1
We generate this on the fly if it's not there and add it. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com>
2016-08-31framework/backends/junit.py: Add a writer class that handles subtestsDylan Baker1-0/+165
This class handles subtests by making a test with subtests into a testsuite element, and then makes each subtest a testcase element. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Acked-by: Jose Fonseca <jfonseca@vmware.com> (v1) Reviewed-by: Mark Janes <mark.a.janes@intel.com> Tested-by: Mark Janes <mark.a.janes@intel.com>
generated by cgit v1.2.3 (git 2.39.1) at 2024-05-28 09:32:21 +0000