drm/gem: Check for valid formats

Currently, drm_gem_fb_create() doesn't check if the pixel format is
supported, which can lead to the acceptance of invalid pixel formats
e.g. the acceptance of invalid modifiers. Therefore, add a check for
valid formats on drm_gem_fb_create().

Note that this check is only valid for atomic drivers, because, for
non-atomic drivers, checking drm_any_plane_has_format() is not
possible since the format list for the primary plane is fake, and we'd
therefore reject valid formats.

Adding this check to drm_gem_fb_create() will guarantee that the
igt@kms_addfb_basic@addfb25-bad-modifier IGT test passes for drivers
using this callback.

This commit is a recapture of a series sent a while ago. Initially,
I sent a patch [1] similar to this one in which I introduced the
format check to drm_gem_fb_create().

Based on the feedback on the patch, I placed the check inside
framebuffer_check() [2] so that it wouldn't be needed to hit any
driver-specific code path when the check fails. Therefore, we could
remove the check from the specific drivers (i915, amdgpu, and vmwgfx).

But, with some new feedback, it was shown that introducing this check
inside framebuffer_check() is problematic for the i915 driver [3].
For the i915 driver, in the legacy case, in which we don't get the
modifier from the userspace, i915's fb_create hook computes the right
modifier, which isn't necessarily linear.  Therefore, if we check the
modifier before that point, we might get wrong answers.

So, I kept the check inside the i915 driver and removed the check from
amdgpu and vmwgfx [4]. But, this yet hasn't solved the i915 problem [5].

As we cannot add the check inside framebuffer_check() without
affecting the i915 behavior, this commit went back to the original
patch. This way we can guarantee a more uniform behavior from the
drivers that use the drm_gem_fb_create() callback.

[1] https://lore.kernel.org/dri-devel/20230103125322.855089-1-mcanal@igalia.com/T/
[2] https://lore.kernel.org/dri-devel/20230109105807.18172-1-mcanal@igalia.com/T/
[3] https://lore.kernel.org/dri-devel/Y8AAdW2y7zN7DCUZ@intel.com/
[4] https://lore.kernel.org/dri-devel/20230113112743.188486-1-mcanal@igalia.com/T/
[5] https://lore.kernel.org/dri-devel/Y8FXWvEhO7GCRKVJ@intel.com/

Signed-off-by: Maíra Canal <mcanal@igalia.com>
Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Maíra Canal <mairacanal@riseup.net>
Link: https://patchwork.freedesktop.org/patch/msgid/20230412142923.136707-1-mcanal@igalia.com

1 files changed, 9 insertions, 0 deletions

diff --git a/drivers/gpu/drm/drm_gem_framebuffer_helper.c b/drivers/gpu/drm/drm_gem_framebuffer_helper.c
index e93533b86037..b8a615a138cd 100644
--- a/drivers/gpu/drm/drm_gem_framebuffer_helper.c
+++ b/drivers/gpu/drm/drm_gem_framebuffer_helper.c
@@ -9,6 +9,7 @@
 #include <linux/module.h>
 
 #include <drm/drm_damage_helper.h>
+#include <drm/drm_drv.h>
 #include <drm/drm_fourcc.h>
 #include <drm/drm_framebuffer.h>
 #include <drm/drm_gem.h>
@@ -164,6 +165,14 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev,
 		return -EINVAL;
 	}
 
+	if (drm_drv_uses_atomic_modeset(dev) &&
+	    !drm_any_plane_has_format(dev, mode_cmd->pixel_format,
+				      mode_cmd->modifier[0])) {
+		drm_dbg(dev, "Unsupported pixel format %p4cc / modifier 0x%llx\n",
+			&mode_cmd->pixel_format, mode_cmd->modifier[0]);
+		return -EINVAL;
+	}
+
 	for (i = 0; i < info->num_planes; i++) {
 		unsigned int width = mode_cmd->width / (i ? info->hsub : 1);
 		unsigned int height = mode_cmd->height / (i ? info->vsub : 1);