From 61d3f56877b23241757490efcdb759b905caca1d Mon Sep 17 00:00:00 2001 From: Alex Goins Date: Mon, 8 Jan 2018 18:44:25 -0800 Subject: randr: Fix rotation check in ProcRRSetScreenSize() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ProcRRSetScreenSize() does bounds checking to ensure that none of the CRTCs have a viewport that extends beyond the new screen size. In doing so, it accounts for if the CRTC is rotated 90 or 270 degrees, swapping width and height. However, it does so by testing if crtc->rotation is equal to RR_Rotate_90 or RR_Rotate_270. crtc->rotation is a bit mask, and it includes reflection as well as rotation. If a CRTC is reflected as well as rotated, it will incorrectly fail this test, resulting in incorrect dimensions being used to verify the validity of the new screen size. In some cases, this can cause valid uses of ProcRRSetScreenSize() to fail with BadMatch. This patch fixes the issue by testing that the bits RR_Rotate_90 or RR_Rotate_270 are set, rather than testing for equality. Signed-off-by: Alex Goins Reviewed-by: Michel Dänzer (cherry picked from commit 6b26a7bda9efa93440734ede0382a3e9a6761365) --- randr/rrscreen.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'randr') diff --git a/randr/rrscreen.c b/randr/rrscreen.c index 0c70b28dd..d059ce74a 100644 --- a/randr/rrscreen.c +++ b/randr/rrscreen.c @@ -272,7 +272,7 @@ ProcRRSetScreenSize(ClientPtr client) int source_height = mode->mode.height; Rotation rotation = crtc->rotation; - if (rotation == RR_Rotate_90 || rotation == RR_Rotate_270) { + if (rotation & (RR_Rotate_90 | RR_Rotate_270)) { source_width = mode->mode.height; source_height = mode->mode.width; } -- cgit v1.2.3