rootless: fix uninitialized private key assert in non-rootless modes in Cygwin/X

IsFramedWindow() is called from miPaintWindow() if the server has
been built with ROOTLESS defined, irrespective of if RootlessInit()
has ever been called, or not.

Add a check to IsFramedWindow() to check if rootlessWindowPrivateKey
has been registered (as a proxy for checking if the rootless extension
has been initialized) so we don't go on to try to use that key,
triggering an assert.

This bug exposes what appears to be a difference in opinion about
the rootless extension between XQuartz and XWin.  XQuartz always
initializes the rootless extension, whereas XWin offers several modes
of operation, and the rootless extension is only used for one of them

That probably means that the all code under compile time guard for
ROOTLESS should be carefully checked that it doesn't also need to be
under a run-time guard

(I've reviewed the other ROOTLESS blocks in dix/events.c and
dix/window.c and they look ok -- keithp)

Signed-off-by: Jon TURNEY <jon.turney@dronecode.org.uk>
Reviewed-by: Keith Packard <keithp@keithp.com>
Signed-off-by: Keith Packard <keithp@keithp.com>

1 files changed, 3 insertions, 0 deletions

diff --git a/miext/rootless/rootlessCommon.c b/miext/rootless/rootlessCommon.c
index 82fe0b936..f91cfb432 100644
--- a/miext/rootless/rootlessCommon.c
+++ b/miext/rootless/rootlessCommon.c
@@ -92,6 +92,9 @@ IsFramedWindow(WindowPtr pWin)
 {
     WindowPtr top;
 
+    if (!dixPrivateKeyRegistered(&rootlessWindowPrivateKeyRec))
+        return FALSE;
+
     if (!pWin->realized)
         return FALSE;
     top = TopLevelParent(pWin);