path: root/docs/design/draft-subtitle-overlays.txt

===============================================================
 Subtitle overlays, hardware-accelerated decoding and playbin
===============================================================

Status: EARLY DRAFT / BRAINSTORMING

 === 1. Background ===

Subtitles can be muxed in containers or come from an external source.

Subtitles come in many shapes and colours. Usually they are either
text-based (incl. 'pango markup'), or bitmap-based (e.g. DVD subtitles
and the most common form of DVB subs). Bitmap based subtitles are
usually compressed in some way, like some form of run-length encoding.

Subtitles are currently decoded and rendered in subtitle-format-specific
overlay elements. These elements have two sink pads (one for raw video
and one for the subtitle format in question) and one raw video source pad.

They will take care of synchronising the two input streams, and of
decoding and rendering the subtitles on top of the raw video stream.

Digression: one could theoretically have dedicated decoder/render elements
that output an AYUV or ARGB image, and then let a videomixer element do
the actual overlaying, but this is not very efficient, because it requires
us to allocate and blend whole pictures (1920x1080 AYUV = 8MB,
1280x720 AYUV = 3.6MB, 720x576 AYUV = 1.6MB) even if the overlay region
is only a small rectangle at the bottom. This wastes memory and CPU.
We could do something better by introducing a new format that only
encodes the region(s) of interest, but we don't have such a format yet, and
are not necessarily keen to rewrite this part of the logic in playbin
at this point - and we can't change existing elements' behaviour, so would
need to introduce new elements for this.

Playbin2 supports outputting compressed formats, i.e. it does not
force decoding to a raw format, but is happy to output to a non-raw
format as long as the sink supports that as well.

In case of certain hardware-accelerated decoding APIs, we will make use
of that functionality. However, the decoder will not output a raw video
format then, but some kind of hardware/API-specific format (in the caps)
and the buffers will reference hardware/API-specific objects that
the hardware/API-specific sink will know how to handle.


 === 2. The Problem ===

In the case of such hardware-accelerated decoding, the decoder will not
output raw pixels that can easily be manipulated. Instead, it will
output hardware/API-specific objects that can later be used to render
a frame using the same API.

Even if we could transform such a buffer into raw pixels, we most
likely would want to avoid that, in order to avoid the need to
map the data back into system memory (and then later back to the GPU).
It's much better to upload the much smaller encoded data to the GPU/DSP
and then leave it there until rendered.

Currently playbin only supports subtitles on top of raw decoded video.
It will try to find a suitable overlay element from the plugin registry
based on the input subtitle caps and the rank. (It is assumed that we
will be able to convert any raw video format into any format required
by the overlay using a converter such as videoconvert.)

It will not render subtitles if the video sent to the sink is not
raw YUV or RGB or if conversions have been disabled by setting the
native-video flag on playbin.

Subtitle rendering is considered an important feature. Enabling
hardware-accelerated decoding by default should not lead to a major
feature regression in this area.

This means that we need to support subtitle rendering on top of
non-raw video.


 === 3. Possible Solutions ===

The goal is to keep knowledge of the subtitle format within the
format-specific GStreamer plugins, and knowledge of any specific
video acceleration API to the GStreamer plugins implementing
that API. We do not want to make the pango/dvbsuboverlay/dvdspu/kate
plugins link to libva/libvdpau/etc. and we do not want to make
the vaapi/vdpau plugins link to all of libpango/libkate/libass etc.


Multiple possible solutions come to mind:

  (a) backend-specific overlay elements

      e.g. vaapitextoverlay, vdpautextoverlay, vaapidvdspu, vdpaudvdspu,
      vaapidvbsuboverlay, vdpaudvbsuboverlay, etc.

      This assumes the overlay can be done directly on the backend-specific
      object passed around.

      The main drawback with this solution is that it leads to a lot of
      code duplication and may also lead to uncertainty about distributing
      certain duplicated pieces of code. The code duplication is pretty
      much unavoidable, since making textoverlay, dvbsuboverlay, dvdspu,
      kate, assrender, etc. available in form of base classes to derive
      from is not really an option. Similarly, one would not really want
      the vaapi/vdpau plugin to depend on a bunch of other libraries
      such as libpango, libkate, libtiger, libass, etc.

      One could add some new kind of overlay plugin feature though in
      combination with a generic base class of some sort, but in order
      to accommodate all the different cases and formats one would end
      up with quite convoluted/tricky API.

      (Of course there could also be a GstFancyVideoBuffer that provides
      an abstraction for such video accelerated objects and that could
      provide an API to add overlays to it in a generic way, but in the
      end this is just a less generic variant of (c), and it is not clear
      that there are real benefits to a specialised solution vs. a more
      generic one).


  (b) convert backend-specific object to raw pixels and then overlay

      Even where possible technically, this is most likely very
      inefficient.


  (c) attach the overlay data to the backend-specific video frame buffers
      in a generic way and do the actual overlaying/blitting later in
      backend-specific code such as the video sink (or an accelerated
      encoder/transcoder)

      In this case, the actual overlay rendering (i.e. the actual text
      rendering or decoding DVD/DVB data into pixels) is done in the
      subtitle-format-specific GStreamer plugin. All knowledge about
      the subtitle format is contained in the overlay plugin then,
      and all knowledge about the video backend in the video backend
      specific plugin.

      The main question then is how to get the overlay pixels (and
      we will only deal with pixels here) from the overlay element
      to the video sink.

      This could be done in multiple ways: One could send custom
      events downstream with the overlay data, or one could attach
      the overlay data directly to the video buffers in some way.

      Sending inline events has the advantage that is is fairly
      transparent to any elements between the overlay element and
      the video sink: if an effects plugin creates a new video
      buffer for the output, nothing special needs to be done to
      maintain the subtitle overlay information, since the overlay
      data is not attached to the buffer. However, it slightly
      complicates things at the sink, since it would also need to
      look for the new event in question instead of just processing
      everything in its buffer render function.

      If one attaches the overlay data to the buffer directly, any
      element between overlay and video sink that creates a new
      video buffer would need to be aware of the overlay data
      attached to it and copy it over to the newly-created buffer.

      One would have to do implement a special kind of new query
      (e.g. FEATURE query) that is not passed on automatically by
      gst_pad_query_default() in order to make sure that all elements
      downstream will handle the attached overlay data. (This is only
      a problem if we want to also attach overlay data to raw video
      pixel buffers; for new non-raw types we can just make it
      mandatory and assume support and be done with it; for existing
      non-raw types nothing changes anyway if subtitles don't work)
      (we need to maintain backwards compatibility for existing raw
      video pipelines like e.g.:  ..decoder ! suboverlay ! encoder..)

      Even though slightly more work, attaching the overlay information
      to buffers seems more intuitive than sending it interleaved as
      events. And buffers stored or passed around (e.g. via the
      "last-buffer" property in the sink when doing screenshots via
      playbin) always contain all the information needed.


  (d) create a video/x-raw-*-delta format and use a backend-specific videomixer

      This possibility was hinted at already in the digression in
      section 1. It would satisfy the goal of keeping subtitle format
      knowledge in the subtitle plugins and video backend knowledge
      in the video backend plugin. It would also add a concept that
      might be generally useful (think ximagesrc capture with xdamage).
      However, it would require adding foorender variants of all the
      existing overlay elements, and changing playbin to that new
      design, which is somewhat intrusive. And given the general
      nature of such a new format/API, we would need to take a lot
      of care to be able to accommodate all possible use cases when
      designing the API, which makes it considerably more ambitious.
      Lastly, we would need to write videomixer variants for the
      various accelerated video backends as well.


Overall (c) appears to be the most promising solution. It is the least
intrusive and should be fairly straight-forward to implement with
reasonable effort, requiring only small changes to existing elements
and requiring no new elements.

Doing the final overlaying in the sink as opposed to a videomixer
or overlay in the middle of the pipeline has other advantages:

 - if video frames need to be dropped, e.g. for QoS reasons,
   we could also skip the actual subtitle overlaying and
   possibly the decoding/rendering as well, if the
   implementation and API allows for that to be delayed.

 - the sink often knows the actual size of the window/surface/screen
   the output video is rendered to. This *may* make it possible to
   render the overlay image in a higher resolution than the input
   video, solving a long standing issue with pixelated subtitles on
   top of low-resolution videos that are then scaled up in the sink.
   This would require for the rendering to be delayed of course instead
   of just attaching an AYUV/ARGB/RGBA blog of pixels to the video buffer
   in the overlay, but that could all be supported.

 - if the video backend / sink has support for high-quality text
   rendering (clutter?) we could just pass the text or pango markup
   to the sink and let it do the rest (this is unlikely to be
   supported in the general case - text and glyph rendering is
   hard; also, we don't really want to make up our own text markup
   system, and pango markup is probably too limited for complex
   karaoke stuff).


 === 4. API needed ===

  (a) Representation of subtitle overlays to be rendered

      We need to pass the overlay pixels from the overlay element to the
      sink somehow. Whatever the exact mechanism, let's assume we pass
      a refcounted GstVideoOverlayComposition struct or object.

      A composition is made up of one or more overlays/rectangles.

      In the simplest case an overlay rectangle is just a blob of
      RGBA/ABGR [FIXME?] or AYUV pixels with positioning info and other
      metadata, and there is only one rectangle to render.

      We're keeping the naming generic ("OverlayFoo" rather than
      "SubtitleFoo") here, since this might also be handy for
      other use cases such as e.g. logo overlays or so. It is not
      designed for full-fledged video stream mixing though.

        // Note: don't mind the exact implementation details, they'll be hidden

        // FIXME: might be confusing in 0.11 though since GstXOverlay was
        //        renamed to GstVideoOverlay in 0.11, but not much we can do,
        //        maybe we can rename GstVideoOverlay to something better

        struct GstVideoOverlayComposition
        {
            guint                          num_rectangles;
            GstVideoOverlayRectangle    ** rectangles;

            /* lowest rectangle sequence number still used by the upstream
             * overlay element. This way a renderer maintaining some kind of
             * rectangles <-> surface cache can know when to free cached
             * surfaces/rectangles. */
            guint                          min_seq_num_used;

            /* sequence number for the composition (same series as rectangles) */
            guint                          seq_num;
        }

        struct GstVideoOverlayRectangle
        {
            /* Position on video frame and dimension of output rectangle in
             * output frame terms (already adjusted for the PAR of the output
             * frame). x/y can be negative (overlay will be clipped then) */
            gint  x, y;
            guint render_width, render_height;

            /* Dimensions of overlay pixels */
            guint width, height, stride;

            /* This is the PAR of the overlay pixels */
            guint par_n, par_d;

            /* Format of pixels, GST_VIDEO_FORMAT_ARGB on big-endian systems,
             * and BGRA on little-endian systems (i.e. pixels are treated as
             * 32-bit values and alpha is always in the most-significant byte,
             * and blue is in the least-significant byte).
             *
             * FIXME: does anyone actually use AYUV in practice? (we do
             * in our utility function to blend on top of raw video)
             * What about AYUV and endianness? Do we always have [A][Y][U][V]
             * in memory? */
            /* FIXME: maybe use our own enum? */
            GstVideoFormat format;

            /* Refcounted blob of memory, no caps or timestamps */
            GstBuffer *pixels;

            // FIXME: how to express source like text or pango markup?
            //        (just add source type enum + source buffer with data)
            //
            // FOR 0.10: always send pixel blobs, but attach source data in
            // addition (reason: if downstream changes, we can't renegotiate
            // that properly, if we just do a query of supported formats from
            // the start). Sink will just ignore pixels and use pango markup
            // from source data if it supports that.
            //
            // FOR 0.11: overlay should query formats (pango markup, pixels)
            // supported by downstream and then only send that. We can
            // renegotiate via the reconfigure event.
            //

            /* sequence number: useful for backends/renderers/sinks that want
             * to maintain a cache of rectangles <-> surfaces. The value of
             * the min_seq_num_used in the composition tells the renderer which
             * rectangles have expired. */
            guint      seq_num;

            /* FIXME: we also need a (private) way to cache converted/scaled
             * pixel blobs */
        }

      (a1) Overlay consumer API:

        How would this work in a video sink that supports scaling of textures:

        gst_foo_sink_render () {
          /* assume only one for now */
          if video_buffer has composition:
            composition = video_buffer.get_composition()

            for each rectangle in composition:
              if rectangle.source_data_type == PANGO_MARKUP
                actor = text_from_pango_markup (rectangle.get_source_data())
              else
                pixels = rectangle.get_pixels_unscaled (FORMAT_RGBA, ...)
                actor = texture_from_rgba (pixels, ...)

              .. position + scale on top of video surface ...
        }

      (a2) Overlay producer API:

        e.g. logo or subpicture overlay: got pixels, stuff into rectangle:

         if (logoverlay->cached_composition == NULL) {
           comp = composition_new ();

           rect = rectangle_new (format, pixels_buf,
                                 width, height, stride, par_n, par_d,
                                 x, y, render_width, render_height);

           /* composition adds its own ref for the rectangle */
           composition_add_rectangle (comp, rect);
           rectangle_unref (rect);

           /* buffer adds its own ref for the composition */
           video_buffer_attach_composition (comp);

           /* we take ownership of the composition and save it for later */
           logoverlay->cached_composition = comp;
         } else {
           video_buffer_attach_composition (logoverlay->cached_composition);
         }

      FIXME: also add some API to modify render position/dimensions of
      a rectangle (probably requires creation of new rectangle, unless
      we handle writability like with other mini objects).

  (b) Fallback overlay rendering/blitting on top of raw video

      Eventually we want to use this overlay mechanism not only for
      hardware-accelerated video, but also for plain old raw video,
      either at the sink or in the overlay element directly.

      Apart from the advantages listed earlier in section 3, this
      allows us to consolidate a lot of overlaying/blitting code that
      is currently repeated in every single overlay element in one
      location. This makes it considerably easier to support a whole
      range of raw video formats out of the box, add SIMD-optimised
      rendering using ORC, or handle corner cases correctly.

      (Note: side-effect of overlaying raw video at the video sink is
      that if e.g. a screnshotter gets the last buffer via the last-buffer
      property of basesink, it would get an image without the subtitles
      on top. This could probably be fixed by re-implementing the
      property in GstVideoSink though. Playbin2 could handle this
      internally as well).

        void
        gst_video_overlay_composition_blend (GstVideoOverlayComposition * comp
                                             GstBuffer                  * video_buf)
        {
          guint n;

          g_return_if_fail (gst_buffer_is_writable (video_buf));
          g_return_if_fail (GST_BUFFER_CAPS (video_buf) != NULL);

          ... parse video_buffer caps into BlendVideoFormatInfo ...

          for each rectangle in the composition: {

                 if (gst_video_format_is_yuv (video_buf_format)) {
                   overlay_format = FORMAT_AYUV;
                 } else if (gst_video_format_is_rgb (video_buf_format)) {
                   overlay_format = FORMAT_ARGB;
                 } else {
                   /* FIXME: grayscale? */
                   return;
                 }

                 /* this will scale and convert AYUV<->ARGB if needed */
                 pixels = rectangle_get_pixels_scaled (rectangle, overlay_format);

                 ... clip output rectangle ...

                 __do_blend (video_buf_format, video_buf->data,
                             overlay_format, pixels->data,
                             x, y, width, height, stride);

                 gst_buffer_unref (pixels);
          }
        }


  (c) Flatten all rectangles in a composition

      We cannot assume that the video backend API can handle any
      number of rectangle overlays, it's possible that it only
      supports one single overlay, in which case we need to squash
      all rectangles into one.

      However, we'll just declare this a corner case for now, and
      implement it only if someone actually needs it. It's easy
      to add later API-wise. Might be a bit tricky if we have
      rectangles with different PARs/formats (e.g. subs and a logo),
      though we could probably always just use the code from (b)
      with a fully transparent video buffer to create a flattened
      overlay buffer.

  (d) core API: new FEATURE query

      For 0.10 we need to add a FEATURE query, so the overlay element
      can query whether the sink downstream and all elements between
      the overlay element and the sink support the new overlay API.
      Elements in between need to support it because the render
      positions and dimensions need to be updated if the video is
      cropped or rescaled, for example.

      In order to ensure that all elements support the new API,
      we need to drop the query in the pad default query handler
      (so it only succeeds if all elements handle it explicitly).

      Might want two variants of the feature query - one where
      all elements in the chain need to support it explicitly
      and one where it's enough if some element downstream
      supports it.

      In 0.11 this could probably be handled via GstMeta and
      ALLOCATION queries (and/or we could simply require
      elements to be aware of this API from the start).

      There appears to be no issue with downstream possibly
      not being linked yet at the time when an overlay would
      want to do such a query.


Other considerations:

 - renderers (overlays or sinks) may be able to handle only ARGB or only AYUV
   (for most graphics/hw-API it's likely ARGB of some sort, while our
   blending utility functions will likely want the same colour space as
   the underlying raw video format, which is usually YUV of some sort).
   We need to convert where required, and should cache the conversion.

 - renderers may or may not be able to scale the overlay. We need to
   do the scaling internally if not (simple case: just horizontal scaling
   to adjust for PAR differences; complex case: both horizontal and vertical
   scaling, e.g. if subs come from a different source than the video or the
   video has been rescaled or cropped between overlay element and sink).

 - renderers may be able to generate (possibly scaled) pixels on demand
   from the original data (e.g. a string or RLE-encoded data). We will
   ignore this for now, since this functionality can still be added later
   via API additions. The most interesting case would be to pass a pango
   markup string, since e.g. clutter can handle that natively.

 - renderers may be able to write data directly on top of the video pixels
   (instead of creating an intermediary buffer with the overlay which is
   then blended on top of the actual video frame), e.g. dvdspu, dvbsuboverlay

   However, in the interest of simplicity, we should probably ignore the
   fact that some elements can blend their overlays directly on top of the
   video (decoding/uncompressing them on the fly), even more so as it's
   not obvious that it's actually faster to decode the same overlay
   70-90 times (say) (ie. ca. 3 seconds of video frames) and then blend
   it 70-90 times instead of decoding it once into a temporary buffer
   and then blending it directly from there, possibly SIMD-accelerated.
   Also, this is only relevant if the video is raw video and not some
   hardware-acceleration backend object.

   And ultimately it is the overlay element that decides whether to do
   the overlay right there and then or have the sink do it (if supported).
   It could decide to keep doing the overlay itself for raw video and
   only use our new API for non-raw video.

 - renderers may want to make sure they only upload the overlay pixels once
   per rectangle if that rectangle recurs in subsequent frames (as part of
   the same composition or a different composition), as is likely. This caching
   of e.g. surfaces needs to be done renderer-side and can be accomplished
   based on the sequence numbers. The composition contains the lowest
   sequence number still in use upstream (an overlay element may want to
   cache created compositions+rectangles as well after all to re-use them
   for multiple frames), based on that the renderer can expire cached
   objects. The caching needs to be done renderer-side because attaching
   renderer-specific objects to the rectangles won't work well given the
   refcounted nature of rectangles and compositions, making it unpredictable
   when a rectangle or composition will be freed or from which thread
   context it will be freed. The renderer-specific objects are likely bound
   to other types of renderer-specific contexts, and need to be managed
   in connection with those.

 - composition/rectangles should internally provide a certain degree of
   thread-safety. Multiple elements (sinks, overlay element) might access
   or use the same objects from multiple threads at the same time, and it
   is expected that elements will keep a ref to compositions and rectangles
   they push downstream for a while, e.g. until the current subtitle
   composition expires.

 === 5. Future considerations ===

 - alternatives: there may be multiple versions/variants of the same subtitle
   stream. On DVDs, there may be a 4:3 version and a 16:9 version of the same
   subtitles. We could attach both variants and let the renderer pick the best
   one  for the situation (currently we just use the 16:9 version). With totem,
   it's ultimately totem that adds the 'black bars' at the top/bottom, so totem
   also knows if it's got a 4:3 display and can/wants to fit 4:3 subs (which
   may render on top of the bars) or not, for example.

 === 6. Misc. FIXMEs ===

TEST: should these look (roughly) alike (note text distortion) - needs fixing in textoverlay

gst-launch-0.10 \
    videotestsrc ! video/x-raw,width=640,height=480,pixel-aspect-ratio=1/1 ! textoverlay text=Hello font-desc=72 ! xvimagesink \
    videotestsrc ! video/x-raw,width=320,height=480,pixel-aspect-ratio=2/1 ! textoverlay text=Hello font-desc=72 ! xvimagesink \
    videotestsrc ! video/x-raw,width=640,height=240,pixel-aspect-ratio=1/2 ! textoverlay text=Hello font-desc=72 ! xvimagesink

 ~~~ THE END ~~~