framework: Add a vulkan tests profile

This searches for files named *.vk_shader_test in the tests/vulkan
directory and runs them with VkRunner. VkRunner is executed as an
external dependency. It is found either with the vkrunner:bin config
option, by setting the PIGLIT_VKRUNNER_BINARY environment variable, or
just in the search path.

v2: Move VkShaderTest to piglit_test.py and rename to VkRunnerTest.
    Add future imports. Remove unused import.
v3: Support the PIGLIT_VKRUNNER_BINARY variable to specify the
    location of VkRunner.
v4: Add documentation to the README. Add an option in piglit.conf to
    set the binary location. (Suggested by Samuel Iglesias)

Reviewed-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Reviewed-by: Dylan Baker <dylan@pnwbakers.com>

4 files changed, 76 insertions, 1 deletions

diff --git a/README.md b/README.md
index 2ae0febc6..c75c5f6c3 100644
--- a/README.md
+++ b/README.md
@@ -47,6 +47,8 @@ Optionally, you can install the following:
     (https://simplejson.readthedocs.org/en/latest/)
   - jsonstreams. A JSON stream writer for python.
     (https://jsonstreams.readthedocs.io/en/stable/)
+  - VkRunner. A shader script testing tool for Vulkan.
+    (https://github.com/igalia/vkrunner)
 
 For Python 2.x you can install the following to add features, these are
 unnecessary for python3:
@@ -285,6 +287,13 @@ behaves.
     When this variable is true in python then any timeouts given by tests
     will be ignored, and they will run until completion or they are killed.
 
+  - `PIGLIT_VKRUNNER_BINARY`
+
+    Can be used to override the path to the vkrunner executable for
+    running Vulkan shader tests. Alternatively the config option
+    vkrunner:bin can be used instead. If neither are set then vkrunner
+    will be searched for in the search path.
+
 
 ### 3.2 Note
 
@@ -338,7 +347,13 @@ The following test sets are currently available:
     opencv and oclconform.
 
 
-### 4.3 External Integration
+### 4.3 Vulkan tests
+
+  - **vulkan.py** This suite contains all Vulkan tests. Note that
+    currently all of the Vulkan tests require VkRunner. If it is not
+    installed then all of the tests will be skipped.
+
+### 4.4 External Integration
 
   - **xts.py** Support for running the X Test Suite using piglit.
   - **igt.py** Support for running Intel-gpu-tools test suite using piglit.
diff --git a/framework/test/piglit_test.py b/framework/test/piglit_test.py
index f52915d18..c80e355d1 100644
--- a/framework/test/piglit_test.py
+++ b/framework/test/piglit_test.py
@@ -44,6 +44,7 @@ __all__ = [
     'PiglitCLTest',
     'PiglitGLTest',
     'PiglitBaseTest',
+    'VkRunnerTest',
     'CL_CONCURRENT',
     'ROOT_DIR',
     'TEST_BIN_DIR',
@@ -229,3 +230,25 @@ class CLProgramTester(PiglitCLTest):
         command = super(CLProgramTester, self).command
         command.insert(1, os.path.join(ROOT_DIR, self.filename))
         return command
+
+
+class VkRunnerTest(PiglitBaseTest):
+    """ Make a PiglitTest instance for a VkRunner shader test file """
+
+    def __init__(self, filename):
+        vkrunner_bin = os.environ.get('PIGLIT_VKRUNNER_BINARY')
+
+        if vkrunner_bin is None:
+            vkrunner_bin = core.PIGLIT_CONFIG.safe_get(
+                'vkrunner', 'bin', fallback='vkrunner')
+
+        super(VkRunnerTest, self).__init__(
+            [vkrunner_bin, filename],
+            run_concurrent=True)
+
+    @PiglitBaseTest.command.getter
+    def command(self):
+        # This is overriden because we don’t want PiglitBaseTest to
+        # prepend TEST_BIN_DIR so that it will look for vkrunner in
+        # the search path.
+        return self._command
diff --git a/piglit.conf.example b/piglit.conf.example
index 5d1ff5474..1877187df 100644
--- a/piglit.conf.example
+++ b/piglit.conf.example
@@ -187,6 +187,10 @@ run_test=./%(test_name)s
 ; Default: True
 ;process isolation=True
 
+[vkrunner]
+; Path to the VkRunner executable
+; bin=/home/neil/local/bin/vkrunner
+
 [expected-failures]
 ; Provide a list of test names that are expected to fail.  These tests
 ; will be listed as passing in JUnit output when they fail.  Any
diff --git a/tests/vulkan.py b/tests/vulkan.py
new file mode 100644
index 000000000..7058f3108
--- /dev/null
+++ b/tests/vulkan.py
@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""All Vulkan tests that come with piglit, using default settings."""
+
+from __future__ import (
+    absolute_import, division, print_function, unicode_literals
+)
+
+import os
+
+from framework.profile import TestProfile
+from framework import grouptools
+from framework.test.piglit_test import VkRunnerTest
+from .py_modules.constants import TESTS_DIR, GENERATED_TESTS_DIR
+
+__all__ = ['profile']
+
+profile = TestProfile()
+
+# Find and add all shader tests.
+for basedir in [TESTS_DIR, GENERATED_TESTS_DIR]:
+    _basedir = os.path.join(basedir, 'vulkan')
+    for dirpath, _, filenames in os.walk(_basedir):
+        groupname = grouptools.from_path(os.path.relpath(dirpath, _basedir))
+        for filename in filenames:
+            testname, ext = os.path.splitext(filename)
+            if ext != '.vk_shader_test':
+                continue
+            test = VkRunnerTest(os.path.join(dirpath, filename))
+            group = grouptools.join(groupname, testname)
+            assert group not in profile.test_list, group
+
+            profile.test_list[group] = test