path: root/blend.txt

Outstanding issues:

	    - Find out what's up with HSL blend modes
	    - Rethink structure of the document

Porter/Duff and blend modes

Porter/Duff

In the Porter/Duff compositing algebra, images are equipped with an
alpha channel that determines on a per-pixel basis whether the image
is there or not. When the alpha channel is 1, the image is fully
there, when it is 0, the image isn't there at all. When it is in
between, the imgae is partially there. In other words, the alpha
channel describes the the shape of the image, it does not describe
opacity. The way to think of images with an alpha channel is as
irregularly shaped pieces of cardboard, not as colored glass.

Consider these two images:

	 SRC ("triangle")  	DEST ("ampersand")

When we combine them, each pixel can be divided into four regions:

              	      	 \    /
	         	  \  /
		           \/
      SRC x DEST  	   /\
         		  /  \
	        	 /    \

One region where only the source is present, one where only the
destination is present, one where both are present, and one where
neither is present.

By deciding on what happens in each of the four regions, various
effects can be generated. For example, if the destination-only region
is treated as blank, the source-only region is filled with the source
color, and the 'both' region is filled with the destination color like
this:

			\/
			/\			

the effect is as if the destination image is trimmed to match the
source image, and then held up in front of it:

	    	   SRC dest_atop DEST

The Porter/Duff operator that does this is called "Dest Atop".

There are twelve of these operators, each one characterized by its
behavior in the three regions: source, destination and both (the
'neither' region is always blank). The source-only region can be
either 'source' or 'blank'. The destination-only region can be either
'destination' or 'blank'. The 'both' region can be either 'blank',
'source', or 'destination'.

The formula for the operators is a linear combination of the contents
of the four regions, where the weights are the areas of each region:

    A_src * { C_s, 0 } + A_dest * { C_d, 0 } + A_both * { C_s, C_d, 0 }

Where C_s is the source pixel and C_d is the destination pixel. The
areas are given by these formulas:

    A_src = as * (1 - ad)
    A_dst = ad * (1 - as)
    A_both = as * ad

The resulting alpha channel is computed in a similar way. There is
full coverage in all regions that are not blank:

    A_src * { 1, 0 } + A_dest * ( 1, 0 } + A_both * { 1, 1, 0 }

FIXME: something about alpha channels and premultiplication.

It is important to understand that despite being referred to as alpha
"blending", and despite alpha often being used to model opacity,
conceptually Porter/Duff is not a way to blend the source and
destination shapes. It is a way to overlay, combine and cut them as if
they were pieces of cardboard. The only places where source and
destination pixels are actually blended is where the antialiased edges
meet.

Here are all of the twelve Porter/Duff operators:

     [table.png]


Blending

Photoshop and the Gimp have the concept of layers which are images
stacked on top of each other. In Porter/Duff, stacking images on top
of each other is done with the "Over" operator, which is also what
Photoshop/Gimp use by default to composite layers:

	   <over>

Conceptually, two pieces of cardboard are held up with one in front of
the other. Neither shape is trimmed, and in places where both are
present, only the top layer is visible.

A layer in these programs also has an associated *Blend Mode* which
can be used to modify what happens in places where both are
visible. For example, the 'Color Dodge' blend mode computes a mix of
source and destination according to this formula:

	    B(s, d) = <color dodge formula>

The pixel diagram is this:

       	   Zoomed Color Dodge

And the result looks like this:

     	   src ColorDodge dest

Unlike the regular Over operator, in this case there is a substantial
piece of the output where the source and the destination are actually
mixed.

Layers in Photoshop and Gimp are not tailored to each other (except
for layer masks, which we will ignore here), so the compositing of the
layer stack is done with the source-only and destination-only region
fixed to source and destination respectively. However, there is
nothing in principle stopping us from setting the source-only and
destination-only regions to blank, but keep the blend mode in the
'both' region so that tailoring could be supported alongside
blending. For example, we could set the 'source' region to blank, the
'destination' region to destination, and the 'both' region to
ColorDodge:

	colordodge-dest-diagram.png

The result looks like this:

           ColorDodge dest

The default blend mode in Photoshop/Gimp is called 'Normal' and the
"blend" it computes is simply the top layer, which results in the
'Over' operator that corresponds to stacking opaque images on top of
each other.

This idea naturally leads to an extension of the Porter/Duff algebra,
where the 'both' behavior is selected from a large set of blend
modes. By using these three simple blend modes:

       Source:	B(s,d) = s
       Dest:	B(s,d) = d
       Zero:	B(s,d) = 0

we get the twelve original Porter/Duff operators. By selecting other
blend modes a large number of operators become available that allows
various trimmings and mixings of the source and destination images.
Each of these operators is characterized by the answers to three
questions:

    - Is the source tailored to the destination?
    - Is the destination tailored to the source?
    - What happens in the area where both source and destination are present?

By answering 'Zero', 'Source', or 'Dest' to the third question the
original Porter/Duff operators fall out as special cases.

The general formula is still an area weighted average:

       A_src * { C_s, 0 } + A_dest * { C_d, 0 } + A_both * B(C_s, C_d)

where instead of { C_s, C_d, 0 }, the blend mode can now by chosen
from a large set of formulas. In this way, each new blend modes gives
rise to four new compositing operators defined by whether the source
and destination regions are blank or contain the corresponding pixel
value.

Here is a table of the operators that are generated by various blend
modes:

	[ blend table.png ]

The three first rows are the original Porter/Duff operators.


Alpha Channels

With the classic Porter/Duff operators the output alpha channel is
generally computed by plugging 1 into the area weighted average
whenever a region is filled with either source or destination, and 0
whenever a region is blank.

With most blend modes, the alpha value of the 'both' area can be
considered 1, ie. 'covered', and so the formula still works. However,
there are some exceptions. The first one is the blend mode 'Plus':

     B(s, d) = s + d

The 'both' variant of this blend mode is almost equivalent to the
useful operator Add that simply adds all channels together. The reason
it isn't exactly equivalent is that if we consider the alpha channel 1
in the 'both' area, the output alpha value will be:

   a_r = a_s + a_d - a_s * a_d

where the Add operator has a_s * a_d. We can fix this by defining that
the alpha value of the 'both' area to be B(1, 1). That is, plug s=1
and d=1 into the blend mode formula. Most blend modes produce 1 when
s=1 and d=1, but Plus produces 2, which means the resulting alpha
channel is a_s + a_d as desired. This definition also works for the
'Zero' blend mode: it produces 0 as desired. Using this definition
also means that when computing with premultiplied pixels, the
computation is exactly the same for all four channels. To see this,
consider the blend computation with premultiplied pixels:

   B_{pre} (s, d) = a_s * a_d * B (C_s / a_s, C_d / a_s)

and see what happens when you plug in the alpha channel:

   BP(as, ad) = a_s * a_d * B (a_s / a_s, a_d / a_d) = a_s * a_d * B(1, 1)

There are still two annoying exceptions: The blend modes Exclusion and
Difference do not produce 1 when you plug in (s = 1, d = 1), they
produce 0. There is some logic to that, because these blend modes are
concerned with the difference between source and destination, and the
difference in coverage in the 'both' region (where source and
destination are both present by definition) would in fact be
zero. Nonetheless, making the alpha channel 0 in the 'both' region
would mean that you couldn't see the blended result, so that's not a
useful option.

Finally, the HSL blend modes don't work on a channel-by-channel basis,
so it will never be possible to do the same computation on all four
channels. These will just have to be special cased.

Instead, if we were to actually extend Pixman / cairo / Xrender along
the lines of this post, it might be better to leave out those blend
modes and instead add the inverted versions:

	Inverted Exclusion:	B(s,d) = 1 - Exclusion(s,d)
	Inverted Difference:	B(s,d) = 1 - Difference(s,d)

These will produce the correct alpha channel, and users can still
achieve the non-inverted blend modes by using solid white with the
inverted blend modes using two passes if necessary.


Conclusion

It is possible to generalize the Porter/Duff compositing algebra by
allowing more behaviors in the 'both' region of the Porter/Duff
pixel. If the effect on the alpha channel is defined to be B(1,1),
then all four channels can be computed with the same formula:

  (1 - ad) * [s] * S + (1 - as) * [d] * D + as * ad * B(S/as, D/ad)

where [s] and [d] are 1 or 0 depending on whether the source and
destination are drawn, and B is the blend mode. When B(s,d) is chosen
to be 0, s, or d, the formula reduces to the twelve original
Porter/Duff operators.


<!--

CSS compositing spec

There is a proposal for extending CSS
[https://dvcs.w3.org/hg/FXTF/rawfile/tip/compositing/index.html] with
support for blending and compositing that also tries to unify
porter/duff compositing with blend modes. In this scheme a Porter/Duff
operator and a blend mode are chosen orthogonally. 

For a given pixel, the source and the destination are blended
according to the chosen blendmode, and then a linear interpolation
between the source and the blended result is computed with the
destination alpha as the interpolation weight. This result is then
used as the source in compositing with the chosen Porter/Duff
operator. The full formula is this:

      Fa * as * [ (1 - ad) * Cs + ad * B(s,d) ] + Fb * ad * Cd

where Fa and Fb are weight factors corresponding to the chosen
Porter/Duff operator.

This formula has some useful properties: If the blend mode is chosen
to be Normal, that is, B(s,d) = Cs, then the formula becomes identical
to whatever Porter/Duff operator is involved. If the Porter/Duff
operator is chosen to be Over, then the formula becomes identical to
the Photoshop/Gimp layer blending. When the operator is Source, the
formula produces the atop-like result shown above.

However, there is no conceptual underlying reason that the formula has
these properties. You can't give a graphical interpretation of it the
way you can with Porter/Duff; the useful properties the formula has
might as well be numerological accidents. In fact, for many operators,
the formula produces nonsensical results.

For example, here is what it looks like when the operator is DEST_OVER
and the blend mode is ColorDodge:

    DEST_OVER / COLOR_DODGE

What is this? It's DEST_OVER, except that the antialiasing is messed
up with a weird purple shadow. There is no situation where this is the
operation you need.

There is also a bunch of operators where the blend mode is simply
ignored: DEST, DEST_IN, DEST_OUT, CLEAR.

The fundamental problem is that it doesn't make any sense to select a
Porter/Duff compositing operators and a blend mode orthogonally. The
Porter/Duff operator and the blend mode both specify what happens in
the region where source and destination overlaps, and if they disagree
on what happens there, it is not clear what happens. The answer to
this is not to pick some formula that happens to produce useful
results in certain special cases.

-->