xwayland: Apply output rotation for screen size

Previously, we would swap the width/height of the Xwayland output based
on the output rotation, so that the overall screen size would match the
actual rotation of each output.

Problem is the RandR's ConstrainCursorHarder() handler will also apply
the output rotation, meaning that when the output is rotated, the
pointer will be constrained within the wrong dimension.

Moreover, XRandR assumes the original output width/height are unchanged
when the output is rotated, so by changing the Xwayland output width and
height based on rotation, Xwayland causes XRandr to report the wrong
output sizes (an output of size 1024x768 rotated left or right should
remain 1024x768, not 768x1024).

So to avoid this issue and keep things consistent between Wayland and
Xwayland outputs, leave the actual width/height unchanged but apply the
rotation when computing the screen size. This fixes both the output size
being wrong in "xrandr -q" and the pointer being constrained in the
wrong dimension with rotated with weston.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=99663
Signed-off-by: Olivier Fourdan <ofourdan@redhat.com>
Reviewed-by: Adam Jackson <ajax@redhat.com>

1 files changed, 16 insertions, 12 deletions

diff --git a/hw/xwayland/xwayland-output.c b/hw/xwayland/xwayland-output.c
index fb46448e9..d9047c596 100644
--- a/hw/xwayland/xwayland-output.c
+++ b/hw/xwayland/xwayland-output.c
@@ -108,14 +108,8 @@ output_handle_mode(void *data, struct wl_output *wl_output, uint32_t flags,
     if (!(flags & WL_OUTPUT_MODE_CURRENT))
         return;
 
-    if (xwl_output->rotation & (RR_Rotate_0 | RR_Rotate_180)) {
-        xwl_output->width = width;
-        xwl_output->height = height;
-    } else {
-        xwl_output->width = height;
-        xwl_output->height = width;
-    }
-
+    xwl_output->width = width;
+    xwl_output->height = height;
     xwl_output->refresh = refresh;
 }
 
@@ -123,11 +117,21 @@ static inline void
 output_get_new_size(struct xwl_output *xwl_output,
                     int *height, int *width)
 {
-    if (*width < xwl_output->x + xwl_output->width)
-        *width = xwl_output->x + xwl_output->width;
+    int output_width, output_height;
+
+    if (xwl_output->rotation & (RR_Rotate_0 | RR_Rotate_180)) {
+        output_width = xwl_output->width;
+        output_height = xwl_output->height;
+    } else {
+        output_width = xwl_output->height;
+        output_height = xwl_output->width;
+    }
+
+    if (*width < xwl_output->x + output_width)
+        *width = xwl_output->x + output_width;
 
-    if (*height < xwl_output->y + xwl_output->height)
-        *height = xwl_output->y + xwl_output->height;
+    if (*height < xwl_output->y + output_height)
+        *height = xwl_output->y + output_height;
 }
 
 /* Approximate some kind of mmpd (m.m. per dot) of the screen given the outputs