randr: check rotated virtual size limits correctly

Commit d1107918d4626268803b54033a07405122278e7f introduced checks to
the RandR path that cause RRSetScreenConfig requests to fail if the
size is too large.  Unfortunately, when RandR 1.1 rotation is enabled
it compares the rotated screen dimensions to the unrotated limits,
which causes 90- and 270-degree rotation to fail unless your screen
happens to be square:

  X Error of failed request:  BadValue (integer parameter out of range for operation)
    Major opcode of failed request:  153 (RANDR)
    Minor opcode of failed request:  2 (RRSetScreenConfig)
    Value in failed request:  0x780
    Serial number of failed request:  14
    Current serial number in output stream:  14

Fix this by moving the check above the code that swaps the dimensions
based on the rotation.

Signed-off-by: Aaron Plattner <aplattner@nvidia.com>
Reviewed-by: Jeremy Huddleston <jeremyhu@apple.com>
Tested-by: Robert Hooker <robert.hooker@canonical.com>
Tested-by: Kent Baxley <kent.baxley@canonical.com>
Signed-off-by: Keith Packard <keithp@keithp.com>

1 files changed, 6 insertions, 6 deletions

diff --git a/randr/rrscreen.c b/randr/rrscreen.c
index 1bc1a9ea1..da6d48d6e 100644
--- a/randr/rrscreen.c
+++ b/randr/rrscreen.c
@@ -910,12 +910,6 @@ ProcRRSetScreenConfig (ClientPtr client)
      */
     width = mode->mode.width;
     height = mode->mode.height;
-    if (rotation & (RR_Rotate_90|RR_Rotate_270))
-    {
-	width = mode->mode.height;
-	height = mode->mode.width;
-    }
-
     if (width < pScrPriv->minWidth || pScrPriv->maxWidth < width) {
 	client->errorValue = width;
 	free(pData);
@@ -927,6 +921,12 @@ ProcRRSetScreenConfig (ClientPtr client)
 	return BadValue;
     }
 
+    if (rotation & (RR_Rotate_90|RR_Rotate_270))
+    {
+	width = mode->mode.height;
+	height = mode->mode.width;
+    }
+
     if (width != pScreen->width || height != pScreen->height)
     {
 	int	c;