Planar device memory setup tweaks.

When setting up a memory planar device, we currently build a mask
out of the plane information supplied to us about how to pack
colors into a gx_color_index. This seems like a reasonable thing to
do as we will always need to pack colors into a gx_color_index at
some point.

As part of his work on tiffsep, Michael is about to lift this
requirement though, enabling us to cope with larger numbers of
spots. The code as is fails on certain machines/compilers due to
C's undefined behaviour when shifting by more bits than are in the
variable. We spot this case explicitly, and don't bother checking
for overlap in this case.

1 files changed, 8 insertions, 4 deletions

diff --git a/gs/base/gdevmpla.c b/gs/base/gdevmpla.c
index 8c6bf2892..5459d5126 100644
--- a/gs/base/gdevmpla.c
+++ b/gs/base/gdevmpla.c
@@ -88,10 +88,14 @@ gdev_mem_set_planar(gx_device_memory * mdev, int num_planes,
         if (shift < 0 || plane_depth > 16 ||
             !gdev_mem_device_for_bits(plane_depth))
             return_error(gs_error_rangecheck);
-        mask = (((gx_color_index)1 << plane_depth) - 1) << shift;
-        if (covered & mask)
-            return_error(gs_error_rangecheck);
-        covered |= mask;
+        /* Don't test overlap if shift is too large to fit in the variable */
+        if (shift < 8*sizeof(gx_color_index))
+        {
+            mask = (((gx_color_index)1 << plane_depth) - 1) << shift;
+            if (covered & mask)
+                return_error(gs_error_rangecheck);
+            covered |= mask;
+        }
         if (plane_depth != same_depth)
             same_depth = 0;
         total_depth += plane_depth;