utils.c: Increase acceptable deviation to 0.0064 in pixel_checker_tcheck-formats

The check-formats programs reveals that the 8 bit pipeline cannot meet
the current 0.004 acceptable deviation specified in utils.c, so we
have to increase it. Some of the failing pixels were captured in
pixel-test, which with this commit now passes.

== a4r4g4b4 DISJOINT_XOR a8r8g8b8 ==

The DISJOINT_XOR operator applied to an a4r4g4b4 source pixel of
0xd0c0 and a destination pixel of 0x5300ea00 results in the exact
value:

    fa = (1 - da) / sa = (1 - 0x53 / 255.0) / (0xd / 15.0) = 0.7782
    fb = (1 - sa) / da = (1 - 0xd / 15.0) / (0x53 / 255.0) = 0.4096

    r = fa * (0xc / 15.0) + fb * (0xea / 255.0) = 0.99853

But when computing in 8 bits, we get:

    fa8 = ((255 - 0x53) * 255 + 0xdd / 2) / 0xdd = 0xc6
    fb8 = ((255 - 0xdd) * 255 + 0x53 / 3) / 0x53 = 0x68

    r8 = (fa8 * 0xcc + 127) / 255 + (fb8 * 0xea + 127) / 255 = 0xfd

and

    0xfd / 255.0 = 0.9921568627450981

for a deviation of 0.00637118610187, which we then have to consider
acceptable given the current implementation.

By switching to computing the result with

   r = (fa * s + fb * d + 127) / 255

rather than

   r = (fa * s + 127) / 255 + (fb * d + 127) / 255

the deviation would be only 0.00244961747442, so at some point it may
be worth doing either this, or switching to floating point for
operators that involve divisions.

Note that the conversion from 4 bits to 8 bits does not cause any
error in this case because both rounding and bit replication produces
an exact result when the number of from-bits divide the number of
to-bits.

== a8r8g8b8 OVER r5g6b5 ==

When OVER compositing the a8r8g8b8 pixel 0x0f00c300 with the x14r5g6b5
pixel 0x03c0, the true floating point value of the resulting green
channel is:

   0xc3 / 255.0 + (1.0 - 0x0f / 255.0) * (0x0f / 63.0) = 0.9887955

but when compositing 8 bit values, where the 6-bit green channel is
converted to 8 bit through bit replication, the 8-bit result is:

   0xc3 + ((255 - 0x0f) * 0x3c + 127) / 255 = 251

which corresponds to a real value of 0.984314. The difference from the
true value is 0.004482 which is bigger than the acceptable deviation
of 0.004. So, if we were to compute all the CONJOINT/DISJOINT
operators in floating point, or otherwise make them more accurate, the
acceptable deviation could be set at 0.0045.

If we were doing the 6-bit conversion with rounding:

   (x / 63.0 * 255.0 + 0.5)

instead of bit replication, the deviation in this particular case
would be only 0.0005, so we may want to consider this at some
point.

1 files changed, 1 insertions, 1 deletions

diff --git a/test/utils.c b/test/utils.c
index 9a8decfa..3d1ba22a 100644
--- a/test/utils.c
+++ b/test/utils.c
@@ -1513,7 +1513,7 @@ get_limits (const pixel_checker_t *checker, double limit,
 
 /* The acceptable deviation in units of [0.0, 1.0]
  */
-#define DEVIATION (0.004)
+#define DEVIATION (0.0064)
 
 void
 pixel_checker_get_max (const pixel_checker_t *checker, color_t *color,