Add gapless-rate-change.mdinstant-rate-change

1 files changed, 776 insertions, 0 deletions

diff --git a/markdown/design/gapless-rate-change.md b/markdown/design/gapless-rate-change.md
new file mode 100644
index 0000000..462f968
--- /dev/null
+++ b/markdown/design/gapless-rate-change.md
@@ -0,0 +1,776 @@
+#+TITLE: Design and API for fast seamless playback rate change
+#+SUBTITLE: v1.2
+#+AUTHOR: Edward Hervey, Sebastian Dröge
+#+OPTIONS: H:5
+#+LATEX_CLASS_OPTIONS: [ a4paper ]
+#+LATEX_HEADER: \usepackage[a4paper,top=2.5cm,bottom=2.5cm,left=3.5cm,right=3.5cm]{geometry}
+#+LATEX_HEADER: \hypersetup{colorlinks=true,urlcolor=blue}
+#+LATEX_HEADER: \usepackage{parskip}
+#+LATEX_HEADER: \usepackage{placeins}
+
+* Fast Seamless playback rate change
+
+ The goal is to be able to change the rate of playback:
+ * Without disrupting playback:
+   * No position change (anywhere, including upstream of seek handler)
+   * No audio/video corruption
+ * As fast as possible
+ * In a backwards-compatible way (meaning elements that don't handle
+   seeks or do synchronization are not disrupted by the new API and
+   design).
+
+* History and changes
+
+  * v1.0 : April 23rd 2018 : Initial document
+  * v1.1 : May 11th 2018 : New design
+  * v1.2 : May 25th 2018 :
+    * Add summary, more workflows and examples.
+    * Clarify usage of events and algorithms
+    * Add figures from simulator
+  * v1.3 : June 19th 2018 :
+    * Change segment calculations to make use of ~GstSegment~ API
+    * Use ~segment.position~ to get the current position instead
+      of working with ~segment.base~ alone, called ~anchor~
+    * Add new issue to solve about ~running_time_offset~ in events
+    * Mention that the correction offset is only valid at the time
+      of a segment change
+
+* Summary of Flow of events
+
+  This is a summary of the typical flow of actions/events that happens
+  when using the /New API/ explained in the following chapters.
+
+  1. An application sends a non-flushing ~GST_EVENT_SEEK~ with the \\
+     ~GST_SEEK_FLAG_INSTANT_RATE_CHANGE~ flag on it.
+   * A seek handler (typically a demuxer) sends a non-serialized \\
+     ~GST_EVENT_INSTANT_RATE_CHANGE~ event downstream indicating the
+     rate from the seek event
+   * The seek handler will also eventually send (from the streaming
+     thread) the /regular ~GstSegment~/ (i.e. what happens currently
+     without the new API).
+  2. When sinks receive the =GST_EVENT_INSTANT_RATE_CHANGE= they send
+     a \\
+     =GST_MESSAGE_INSTANT_RATE_REQUEST= upstream with the specified rate.
+  3. When the pipeline receives identical
+     ~GST_MESSAGE_INSTANT_RATE_REQUEST~ for all sinks, it sends to all
+     sinks the requested rate and the current pipeline running time
+     via the ~GST_EVENT_INSTANT_RATE_SYNC_TIME~. Sinks also send that
+     event upstream.
+  4. Sinks modify their internal ~GstSegment~ (which they use for
+     synchronization) using the specified running time (as the
+     position at which they should switch) and rate from the
+     ~GST_EVENT_INSTANT_RATE_SYNC_TIME~.
+  5. When the /regular ~GstSegment~/ from the seek handler arrives in
+     the sinks, the sinks calculated the /running time/ difference
+     between that /regular ~GstSegment~/ and the overridden
+     ~GstSegment~ they used in the meantime. The sinks use that
+     difference to adjust synchronization and ~QoS~ events.
+
+* New API for application and seek handlers
+
+ The new API introduced by this design is:
+ * A new ~GstSeekFlags~ to be used by applications.
+ * A new ~GST_EVENT_INSTANT_RATE_CHANGE~ non-serialized event to be
+   sent by seek handlers downstream to sink elements.
+ * A new ~GST_MESSAGE_INSTANT_RATE_REQUEST~ which ~GstBaseSink~ sends
+   to upstream ~GstBin~ and ~GstPipeline~ whenever they receive the
+   non-serialized event.
+ * A new ~GST_EVENT_INSTANT_RATE_SYNC_TIME~ send upstream by the
+   pipeline and notifying all elements doing synchronization that they
+   should override the ~GstSegment~ they are using for synchronization
+   to the /running time/ and /rate/ specified within that ~GstEvent~.
+
+** New GstSeekFlag : ~GST_SEEK_FLAG_INSTANT_RATE_CHANGE~
+
+   When used in a ~GST_EVENT_SEEK~, seek handlers send a
+   non-serialized \\
+   ~GST_EVENT_INSTANT_RATE_CHANGE~ with the rate of the seek event
+   downstream. That event shall have the same seqnum as the seek event
+   and resulting ~GstSegment~.
+
+   This flag can only be used if all of these conditions are met:
+   * The position is *NOT* changed,
+   * The direction is *NOT* changed,
+   * The seek is not flushing[fn:flush]
+
+[fn:flush] The reason why that is refused is that doing a flushing
+seek would render all the instant-rate handling useless (it would have
+the same end-result at the expense of more calculation).
+
+** New event : ~GST_EVENT_INSTANT_RATE_CHANGE~
+
+   A new downstream non-serialized event with the new rate.
+
+#+CAPTION: New "instant rate change" ~GstEvent~
+#+BEGIN_SRC c
+  /* Create a new instant rate change event with the specified rate */
+  GstEvent *gst_event_new_instant_rate_change(gdouble rate);
+
+  /* Parse the rate value contained in a GST_EVENT_INSTANT_RATE_CHANGE */
+  void      gst_event_parse_instant_rate_change(GstEvent *event,
+                                                gdouble  *rate);
+#+END_SRC
+
+* Usage in seek handlers
+** Standard usage
+
+  In order to do a seamless playback rate change, the official API is
+  to send a ~GST_EVENT_SEEK~ with:
+  * the ~start_type~ and ~stop_type~ fields set to ~GST_SEEK_TYPE_NONE~
+  * the flags field does *NOT* contain ~GST_SEEK_FLAG_FLUSH~
+  * the rate field set to the new playback rate
+
+** Fast usage
+
+  The only additional requirement to cause a seamless playback rate
+  change to happen "immediately" is to use the
+  ~GST_SEEK_FLAG_INSTANT_RATE_CHANGE~ in the seek event.
+
+  If that flag is present, seek handlers handle the seek by returning
+  two events (both with the same ~seqnum~ as the ~GST_EVENT_SEEK~):
+  * The non-serialized ~GST_EVENT_INSTANT_RATE_CHANGE~ with the
+    requested rate,
+  * The regular serialized ~GST_EVENT_SEGMENT~ 
+
+* New API to handle ~GST_EVENT_INSTANT_RATE_CHANGE~
+
+ The ~GstBaseSink~ parent class is the sole handler of this event.
+
+ #+BEGIN_QUOTE
+ /Note:/ Elements can safely ignore that event but *must* forward it
+ immediately (which should already be the case).
+ #+END_QUOTE
+
+** New API : Synchronizing rate-switching across sinks
+
+ Various issues might cause sinks to apply the rate change at
+ different point in time. In order to ensure a common switching time,
+ and to reduce the time to apply that delay to a minimum, a new API is
+ proposed to allow the pipeline to synchronize that switch.
+
+ Another issue that needs to be avoided is having a pipeline where one
+ (of the many) seek handlers doesn't handle the
+ ~GST_SEEK_FLAG_INSTANT_RATE_CHANGE~ and therefore doesn't send the
+ ~GST_EVENT_INSTANT_RATE_CHANGE~ event. If that happens no rate change
+ should happen in the sinks.
+
+ When a ~GstBaseSink~ receives a ~GST_EVENT_INSTANT_RATE_CHANGE~ it
+ posts a \\
+ ~GST_MESSAGE_INSTANT_RATE_REQUEST~ on the bus and
+ returns. That message contains the /requested rate/ and shall have
+ the same ~seqnum~ as the originating event.
+
+ When the pipeline sees that all sinks present in the pipeline have
+ posted a \\
+ ~GST_MESSAGE_INSTANT_RATE_REQUEST~ for the same ~seqnum~ and rate it
+ will to all sinks an upstream ~GST_EVENT_INSTANT_RATE_SYNC_TIME~
+ event with:
+ * The same ~seqnum~ as the message
+ * The requested rate
+ * The /current pipeline running-time/
+
+ The sinks will be able to /apply the instant rate change/ when
+ receiving that event (see below).
+
+#+CAPTION: New sink rate switching synchronization API
+#+BEGIN_SRC c
+  /* Create a new message to notify parents of a instant rate change
+   * event */
+  GstMessage *
+  gst_message_new_instant_rate_request(GstObject *src,
+                                       gdouble    rate);
+
+  /* Parse the contents of an instant rate request message */
+  void
+  gst_message_parse_instant_rate_request(GstMessage *message,
+                                         gdouble    *rate);
+
+  /* Create a new upstream event to notify sinks of the
+   * current running-time and rate to use for a given instant
+   * rate change (identified by seqnum).
+   */
+  GstEvent *
+  gst_event_new_instant_rate_sync_time(gdouble      rate,
+                                       GstClockTime switchtime);
+
+  /* Parse the contents of an instant rate sync time event */
+  void
+  gst_event_parse_instant_rate_sync_time(GstEvent     *event
+                                         gdouble      *rate,
+					 GstClockTime *switchtime);
+
+#+END_SRC
+
+** New API : Applying instant rate changes
+
+ An element wanting to handle ~GST_EVENT_INSTANT_RATE_SYNC_TIME~ (such
+ as ~GstBaseSink~, but also any other element that does rate-based
+ operations or care about the relationship between running-time and
+ clock-time) will need to keep track of:
+ * the /instant rate/ that might need to be applied.
+ * the /seqnum/ of the ~GST_EVENT_SEGMENT~ at which the /instant rate
+   override/ should no longer be applied.
+ * the ~GstSegment~ currently used for synchronization (note that this
+   is the current/legacy behavior). This is called ~segment~ below.
+ * the /uncorrected upstream ~GstSegment~/ (i.e. last received with
+   ~GST_EVENT_SEGMENT~). This is called ~upstream_segment~ below. It
+   is not used for synchronization but to keep track of what upstream
+   is using. It always contains the latest received segment.
+ * the /anchor/ running time when the last segment change happened, i.e. a new
+   segment event arrived or an instant-rate change was applied. It is taken from
+   the event directly in case of ~GST_EVENT_INSTANT_RATE_SYNC_TIME~ and in case
+   of ~GST_EVENT_SEGMENT~ the running time of ~segment.position~. Its meaning is that
+   we already compensated for any upstream/internal running time differences up to this
+   running time and stored the difference up to that point.
+   This value is based on the internal segment used for synchronization and
+   *not* the upstream segment.
+ * A /correction offset/ that will be used to convert running times
+   to/from the /applied corrected ~GstSegment~/ and the
+   /uncorrected upstream ~GstSegment~/. That value is initially 0.
+   Subtracting this offset from the ~anchor~ running time of the last segment change
+   in the internal segment will give the corresponding running time in the
+   upstream segment.
+   This offset is only valid for the ~anchor~ running time when the last segment change
+   happened as afterwards the upstream and internal running time will potentially
+   drift further.
+
+ For *all* ~GST_EVENT_SEGMENT~ that arrives in such elements:
+ * the /correction offset/ is calculated as detailed in the following
+   chapter, and stored together with the /anchor running time of the current segment/.
+   Note that the offset will not change for any segments with a ~seqnum~ that is
+   greater than the ~seqnum~ of the last ~GST_EVENT_INSTANT_RATE_SYNC_TIME~.
+ * The new segment is stored as both the ~segment~ and
+   ~upstream_segment~.
+ * If an ~INSTANT_RATE~ is in effect (i.e. we still haven't seen the
+   ~GST_EVENT_SEGMENT~ with a /seqnum/ equal or greater to the one
+   provided in the ~GST_EVENT_INSTANT_RATE_SYNC_TIME~) then the
+   ~segment.rate~ is overridden with the /instant rate/
+ * The segment is offsetted by the /correction offset/ via ~gst_segment_offset_running_time()~
+   (taking into account the effective difference between assumed and
+   real running-time that elapsed at the /anchor/ running time of the segment).
+
+ When a ~GST_EVENT_INSTANT_RATE_SYNC_TIME~ arrives on such elements
+ they will first check if the current segment ~seqnum~ is greater than
+ the ~seqnum~ in the event. If it greater, this means the /resulting
+ segment/ (i.e. the one that the seek handler sends) has already arrived in
+ the sink. This could happen if the buffering delay between the seek handler
+ and the sinks is so small that all data from before that segment was already
+ consumed.
+
+ #+BEGIN_QUOTE
+ /Note:/ All seqnum comparisions should be done using
+ ~gst_util_seqnum_compare()~ to handle potential wraparound.
+ #+END_QUOTE
+
+ If the /current segment seqnum/ is *lower* than the /seqnum at
+ which we should stop applying corrections/, then the element will
+ adjust the values it is tracking (mentioned above) in the following
+ way:
+ 1. Store the provided ~seqnum~, which is the ~seqnum~ of the future
+    ~GST_EVENT_SEGMENT~ at which it will stop adjusting rates (since
+    it will already be applied in that one).
+ 2. Store the rate to be applied until that event is received.
+ 3. Update the /correction offset/ and /anchor running time of the current segment/ (see /Correction Algorithm/ below).
+ 4. Calculate the position of the ~switchtime~ in the current
+    ~GstSegment~ using \\
+    ~gst_segment_position_from_running_time()~ and store it in
+    ~segment.position~.
+ 5. Apply the /instant rate/ to the ~segment.rate~.
+ 6. Call
+    #+BEGIN_SRC c
+    gst_segment_do_seek(&sink->segment, rate, GST_FORMAT_TIME,
+         sink->segment.flags & (~GST_SEEK_FLAG_FLUSH) & GST_SEEK_FLAG_INSTANT_RATE_CHANGE,
+         GST_SEEK_TYPE_NONE, -1, GST_SEEK_TYPE_NONE, -1, NULL)
+    #+END_SRC
+    This is the same a seek handler (i.e. demuxer) is doing when
+    receiving the ~GST_EVENT_SEEK~ for the instant-rate change, the
+    only difference being the ~segment.position~ due to buffering
+    between both elements.
+
+ #+BEGIN_QUOTE
+ /Note:/ When a ~GST_EVENT_INSTANT_RATE_SYNC_TIME~ is received, the
+ ~upstream_segment~ is *never* modified. It is always the _exact_
+ values received in the last ~GST_EVENT_SEGMENT~.
+
+ /Note:/ If the element handling ~GST_EVENT_INSTANT_RATE_SYNC_TIME~ is
+ *not* a sink element is must use the calculated ~segment~ in all
+ downstream ~GST_EVENT_SEGMENT~. And it must send that ~segment~ every
+ time it is modified. Doing this ensures that 1) upstream elements
+ can apply rate corrections and 2) sinks don't double-correct the
+ rate.
+
+ /Note:/ The /calculated offset/ and /anchor running time of the segment/ will be
+ also used to adjust the running-time value sent in ~GST_EVENT_QOS~ and the
+ ~running_time_offset~ of ~GstEvent~ (adjusting them from the /actual/
+ running-time to the running-time /assumed/ by upstream). For this the same
+ calculations for calculating a new offset are done with the /anchor running time of the event/
+ but the new offset is not stored anywhere else. The reason for this is that
+ the offset is drifting over time while we are still applying a different rate
+ than upstream.
+
+ /Note:/ When a ~GST_EVENT_FLUSH_STOP~ with ~reset-time=True~ arrives
+ in those elements, or when the element goes back to ~READY~, all
+ stored values shall be reset.
+ #+END_QUOTE
+
+*** Correction Algorithm
+
+ The goal of the correction algorithm is to figure out the proper
+ value by which we need to correct the /anchor running time of the segment/ which we
+ use for synchronization in order to be coherent with what upstream
+ sends us. The problem to solve here is that during the time period in which
+ an instant-rate change is applied, the actual running time in the sink and
+ the running time that upstream calculates are drifting apart, and we need to
+ calculate this amount of drift for each segment update to be able to compensate
+ for it.
+
+ /Note:/ This /correction offset/ is only valid for the /anchor running time
+ of the segment/ and will drift at later time as long as the rate between the
+ upstream segment and the sink segment are different.
+
+ For calculating the current /correction offset/, we will calculate in both the
+ internal and the upstream segment how much running time has passed since the last
+ time we calculated it, and then accumulate their difference in the /correction offset/.
+
+ The formula for the correction, when based on a new actual running time,
+ is:
+
+#+BEGIN_CENTER
+
+ ~actual_duration~ = ~current_actual_running_time~ - ~last_anchor_running_time~
+
+ ~upstream_duration~ = (~actual_duration~ * ~sink->segment.rate~) / ~upstream_segment.rate~
+
+ ~new_correction~ = ~last_correction~ + (~actual_duration~ - ~upstream_duration~)
+
+ ~new_anchor_running_time~ = ~current_running_time~
+
+#+END_CENTER
+
+ or alternatively, when based on a new upstream running time, is:
+
+#+BEGIN_CENTER
+
+ ~upstream_duration~ = ~current_upstream_segment_running_time~ - (~last_anchor_running_time~ - ~last_correction~)
+
+ ~actual_duration~ = (~upstream_duration~ * ~upstream_segment.rate~) / ~sink->segment.rate~
+
+ ~new_correction~ = ~last_correction~ + (~actual_duration~ - ~upstream_duration~)
+
+ ~new_anchor_running_time~ = ~current_upstream_segment_running_time~ + ~new_correction~
+
+#+END_CENTER
+
+ which gives ~new_anchor_running_time~ as an anchor in the current segment
+ and ~new_anchor_running_time - correction~ as the corresponding running time
+ that upstream believes we are at at that time.
+
+ This allows us to do conversions between what /upstream/ believes is
+ the running time and what /actually/ is the running time for any position
+ inside the current upstream segment and the internal segment.
+
+ We have to calculate the /correction offset/ in two places:
+ * When we get a new ~GST_EVENT_SEGMENT~
+ * When we get a ~GST_EVENT_INSTANT_RATE_SYNC_TIME~
+
+ When upstream elements push a new ~GstSegment~ they set a ~segment.position~
+ value which they consider the current position of the ~GstSegment~, and we can
+ calculate the current running time based on that. While the documentation
+ of ~GstSegment~ notes that this value is only for internal use, it must be set
+ correctly for instant-rate changes to work and generally will be set correctly
+ as otherwise ~GST_EVENT_SEEK~ without a start/stop position wouldn't work in any
+ of the seek handlers.
+
+ The implementation of the following algorithms can be see in
+ [[fig:correction-code][Listing 3: Correction calculation]].
+
+#+CAPTION: Correction calculation
+#+NAME: fig:correction-code
+#+BEGIN_SRC c
+
+ /* The sink stores two GstSegment :
+  * * sink->segment : the corrected segment
+  * * sink->upstream_segment : the uncorrected segment
+  *
+  * The following function calculates the correction to be applied (stored
+  * in (GstClockTimeDiff) sink->priv->instant_rate_offset;
+  *
+  * This function is called in the sinkpad event handler *before* the
+  * GstSegment is stored in sink->segment *if and only if* there is an
+  * INSTANT_RATE correction that is currently being applied.
+  */
+ static void
+ correct_for_new_segment(GstBaseSink *sink, GstSegment *new_segment)
+ {
+   GstClockTime upstream_duration;
+   GstClockTime actual_duration;
+   GstClockTime new_segment_running_time;
+   GstClockTimeDiff difference;
+   gboolean negative_duration;
+
+   /* Make sure that the position stays between start and stop */
+   new_segment->position = CLAMP (new_segment->position, new_segment->start,
+                                  new_segment->stop);
+
+   /* Calculate how much running time upstream believes has passed since
+    * the last switch/segment */
+   new_segment_running_time =
+       gst_segment_to_running_time (&new_segment, GST_FORMAT_TIME,
+       new_segment->position);
+
+   /* Due to rounding errors and other inaccuracies, it can happen
+    * that our calculated internal running time is before the upstream
+    * running time. We need to compensate for that */
+   if (new_segment_running_time <
+       sink->priv->last_anchor_running_time -
+       sink->priv->instant_rate_offset) {
+     upstream_duration =
+         sink->priv->last_anchor_running_time -
+         sink->priv->instant_rate_offset - new_segment_running_time;
+     negative_duration = TRUE;
+   } else {
+     upstream_duration =
+         new_segment_running_time -
+         sink->priv->last_anchor_running_time +
+         sink->priv->instant_rate_offset;
+     negative_duration = FALSE;
+   }
+
+   /* Calculate the actual running-time duration of the previous segment */
+   actual_duration =
+       (upstream_duration *
+       sink->priv->upstream_segment.rate) / sink->segment.rate;
+
+   if (negative_duration)
+     difference = upstream_duration - actual_duration;
+   else
+     difference = actual_duration - upstream_duration;
+
+   /* Add the difference to the previously accumulated correction. */
+   sink->priv->instant_rate_offset += difference;
+
+   if (sink->priv->instant_rate_offset < 0 &&
+       new_segment_running_time < -sink->priv->instant_rate_offset) {
+     sink->priv->last_anchor_running_time = 0;
+     sink->priv->instant_rate_offset = 0;
+   } else {
+     sink->priv->last_anchor_running_time =
+         new_segment_running_time + sink->priv->instant_rate_offset;
+   }
+}
+ }
+
+ /* This function is called in the instant-rate-sync-time event handler
+  * *before* the sink->segment is corrected for the new switchtime and rate
+  */
+ static void
+ correct_for_instant_rate(GstBaseSink *sink, GstClockTime switch_running_time)
+ {
+   GstClockTime actual_duration;
+   GstClockTime upstream_duration;
+   GstClockTime switch_time;
+   GstClockTimeDiff difference;
+   gboolean negative_duration;
+
+   /* Calculate how much running time was spent since the last switch/segment
+    * in the "corrected upstream segment", our segment */
+
+   /* Due to rounding errors and other inaccuracies, it can happen
+    * that our calculated internal running time is before the upstream
+    * running time. We need to compensate for that */
+   if (running_time < sink->priv->last_anchor_running_time) {
+     actual_duration = sink->priv->last_anchor_running_time - running_time;
+     negative_duration = TRUE;
+   } else {
+     actual_duration = running_time - sink->priv->last_anchor_running_time;
+     negative_duration = FALSE;
+   }
+
+   /* Transpose that duration (i.e. what upstream beliefs) */
+   upstream_duration =
+       (actual_duration * sink->segment.rate) /
+       sink->priv->upstream_segment.rate;
+
+   /* Add the difference to the previously accumulated correction */
+   if (negative_duration)
+     difference = upstream_duration - actual_duration;
+   else
+     difference = actual_duration - upstream_duration;
+   sink->priv->instant_rate_offset += difference;
+
+   if (sink->priv->instant_rate_offset > 0 &&
+       running_time < sink->priv->instant_rate_offset) {
+     sink->priv->last_anchor_running_time = 0;
+     sink->priv->instant_rate_offset = 0;
+   } else {
+     sink->priv->last_anchor_running_time = running_time;
+   }
+ }
+#+END_SRC
+
+\FloatBarrier
+# The float barrier above is to prevent figures to be placed further
+
+* Example sink usage
+
+ * Initial playback from 0s
+ * There is a 2s buffering between the seek handler (demuxer) and the sink.
+ * At 10s, user wants to switch to rate of 2.0
+
+** Initial stage
+
+ * Demuxer outputs the initial SEGMENT:
+#+BEGIN_QUOTE
+|-------+------------+------+------+------+--------+----------+------------|
+| start | stop       | rate | time | base | offset | position | duration   |
+|-------+------------+------+------+------+--------+----------+------------|
+| 0s    | <duration> |  1.0 | 0s   | 0s   | 0s     | 0s       | <duration> |
+|-------+------------+------+------+------+--------+----------+------------|
+#+END_QUOTE
+ * Sink receives segment:
+   * Stores that segment to both ~segment~ and ~upstream_segment~
+   * /correction offset/ is initially 0s.
+ * Demuxer pushes out buffers starting from 0.0s
+ 
+#+BEGIN_EXAMPLE
+   buffer running_time = gst_segment_to_running_time(segment, buf_ts)
+      = ((buf_ts - (s->start + s->offset)) / ABS(s->rate)) + s->base
+
+   clock time = buffer running_time + sink base_time
+#+END_EXAMPLE
+
+   | Buffer ts | buffer running time | sync running time |
+   |-----------+----------------------+-------------------|
+   |       0.0 |                  0.0 |               0.0 |
+   |       0.1 |                  0.1 |               0.1 |
+   |       ... |                 .... |               ... |
+   |       9.9 |                  9.9 |               9.9 |
+
+** User sends instant rate seek 2.0 at stream time 10s
+
+ The demuxer receives a seek event for rate 2.0 and with the
+ ~GST_SEEK_FLAG_INSTANT_RATE_CHANGE~:
+  * Due to the buffering between demuxer and sink, it is currently at
+    stream time 12s
+  * Due to the ~GST_SEEK_FLAG_INSTANT_RATE_CHANGE~ presence, the
+    demuxer sends out a ~GST_EVENT_INSTANT_RATE_CHANGE~ to the sink
+    with the same seqnum as the seek event containing the rate from
+    the seek event (2.0).
+  * The demuxer does the legacy handling which is to apply the seek
+    on the ~GstSegment~ which, depending on the demuxer, could become:
+#+BEGIN_QUOTE
+|-------+------------+------+------+------+--------+----------+------------|
+| start | stop       | rate | time | base | offset | position | duration   |
+|-------+------------+------+------+------+--------+----------+------------|
+| 0s    | <duration> |  2.0 | 0s   | 12s  | 12s    | 12s      | <duration> |
+|-------+------------+------+------+------+--------+----------+------------|
+#+END_QUOTE
+  * The demuxer sends that event downstream with the same seqnum as
+    the seek event.
+
+** Sink receives confirmation switch time
+
+  The sink receives a ~GST_EVENT_INSTANT_RATE_CHANGE~ for rate 2.0 and
+  sends a \\
+  ~GST_MESSAGE_INSTANT_RATE_REQUEST~ with that rate and the same
+  seqnum as the event.
+
+  The pipeline receives that message, sees that all sinks (there is
+  only one here) have posted the same message, so gets the current
+  pipeline running-time (here 10s) and sends a \\
+  ~GST_EVENT_INSTANT_RATE_SYNC_TIME~ upstream with:
+  * that pipeline running-time as the ~switchtime~ : 10s,
+  * the same seqnum as the originating message,
+  * a rate of 2.0
+
+  The sink receives that ~GST_EVENT_INSTANT_RATE_SYNC_TIME~ and
+  handles it:
+  * Stores the ~seqnum~ at which the /instant rate correction/ shall
+    stop.
+  * Stores the /instant rate/ to apply
+  * Update the correction offset (~correct_for_instant_rate()~)
+    #+BEGIN_QUOTE
+    ~actual_duration~ = ~switchtime~ - ~anchor_running_time~ \\
+    ~actual_duration~ = 10s - 0s \\
+    ~actual_duration~ = 10s
+
+    ~upstream_duration~ = ~actual_duration~ * ~upstream_segment.rate~ /
+    ~segment.rate~ \\
+    ~upstream_duration~ = 10s * 1.0 / 1.0 \\
+    ~upstream_duration~ = 10s
+
+    ~correction~ += (~actual_duration~ - ~upstream_duration~) \\
+    ~correction~ += 0
+
+    ~anchor_running_time~ = ~switchtime~
+    #+END_QUOTE
+
+  * Updates the ~segment~ values:
+    * The sink calculates the /position in ~segment~/ for the
+      switchtime, which is 10.0 and sets it to ~segment.position~.
+
+    * The sinks calls
+      #+BEGIN_SRC c
+        gst_segment_do_seek(segment, rate, GST_FORMAT_TIME,
+            sink->segment.flags & (~GST_SEEK_FLAG_FLUSH) &
+            GST_SEEK_FLAG_INSTANT_RATE_CHANGE, GST_SEEK_TYPE_NONE, -1,
+            GST_SEEK_TYPE_NONE, -1, NULL);
+      #+END_SRC
+      It becomes:
+
+      #+BEGIN_QUOTE
+|-------+------------+------+------+------+--------+----------+------------|
+| start | stop       | rate | time | base | offset | position | duration   |
+|-------+------------+------+------+------+--------+----------+------------|
+| 0s    | <duration> |  2.0 | 0s   | 10s  | 10s    | 10s      | <duration> |
+|-------+------------+------+------+------+--------+----------+------------|
+      #+END_QUOTE
+
+  * The ~upstream_segment~ remains untouched.
+
+  The buffer synchronization carries on as follows:
+
+#+BEGIN_QUOTE
+   | Buffer ts | buffer running time | sync running time |
+   |-----------+----------------------+-------------------|
+   |      10.0 |                 10.0 |              10.0 |
+   |      10.1 |                10.05 |             10.05 |
+   |       ... |                 .... |               ... |
+   |      11.9 |                10.95 |             10.95 |
+#+END_QUOTE
+
+** Sink receives /regular/ segment
+
+ Eventually the regular segment created by the seek handler arrives in
+ the sink, which depending on the demuxer could be:
+ #+BEGIN_QUOTE
+|-------+------------+------+------+------+--------+----------+------------|
+| start | stop       | rate | time | base | offset | position | duration   |
+|-------+------------+------+------+------+--------+----------+------------|
+| 0s    | <duration> |  2.0 | 0s   | 12s  | 12s    | 12s      | <duration> |
+|-------+------------+------+------+------+--------+----------+------------|
+ #+END_QUOTE
+
+ The first thing the sink does is to calculate the /correction offset/
+ and new /anchor/ running time for the elapsed segment using the base value
+ from this new incoming segment (12s) and the other stored values.
+
+ According to the /correction offset algorithm/
+ (~correct_for_new_segment()~), we get:
+ * The *assumed* duration of the previous segment
+   #+BEGIN_QUOTE
+   ~new_segment_running_time~ =
+              ~gst_segment_to_running_time~(~new_segment~, ~GST_FORMAT_TIME~,
+              ~new_segment.position~);
+
+   ~assumed_duration~ = ~new_segment_running_time~ - (~anchor_running_time~ -
+   ~correction~)\\
+   ~assumed_duration~ = 12s - (10s - 0s) \\
+   ~assumed_duration~ = 2s
+
+   ~actual_duration~ = ~assumed_duration~ * ~upstream_segment.rate~ /
+   ~segment.rate~ \\
+   ~actual_duration~ = 2s * 1.0 / 2.0 \\
+   ~actual_duration~ = 1s
+
+   ~correction~ += (~actual_duration~ - ~assumed_duration~) \\
+   ~correction~ += (1s - 2s)
+   ~correction~ = -1s
+
+   ~anchor_running_time~ = ~new_segment_running_time~ + ~correction~
+   #+END_QUOTE
+
+ As a next step, the sink sees that the incoming
+ ~GST_EVENT_SEGMENT~ has the same /seqnum/ as the one provided in the
+ ~GST_EVENT_INSTANT_RATE_SYNC_TIME~, it therefore does *not* override
+ the rate field of the stored ~segment~.
+
+ Finally as a last step the sink does is to store that new incoming segment
+ (from the event) as both ~segment~ and ~upstream_segment~. And then
+ adds the correction offset to the ~segment~ by calling
+ ~gst_segment_offset_running_time(segment, GST_FORMAT_TIME, correction)~:
+
+ The values of ~upstream_segment~ are the same as the segment received
+ in the ~GST_EVENT_SEGMENT~, whereas the values of ~segment~ are now:
+ #+BEGIN_QUOTE
+|-------+------------+------+------+------+--------+----------+------------|
+| start | stop       | rate | time | base | offset | position | duration   |
+|-------+------------+------+------+------+--------+----------+------------|
+| 0s    | <duration> |  2.0 | 0s   | 11s  | 12s    | 12s      | <duration> |
+|-------+------------+------+------+------+--------+----------+------------|
+ #+END_QUOTE
+
+ Finally the sink can resume synchronizing the incoming data stream.
+
+   | Buffer ts | buffer running time | sync running time |
+   |-----------+---------------------+-------------------|
+   |      12.0 |                11.0 |              11.0 |
+   |      12.1 |               11.05 |             10.05 |
+   |       ... |                .... |               ... |
+
+* Other examples
+
+  For all following examples, the graph (produced by a simulator)
+  shows the synchronization of buffers for:
+  * /Regular seamless seek/ : This is what synchronization would have
+    happened without usage of the ~GST_SEEK_FLAG_INSTANT_RATE_CHANGE~.
+  * /Fast seamless seek/ : This is the synchronization that happens
+    when the \\
+    ~GST_SEEK_FLAG_INSTANT_RATE_CHANGE~ is used.
+
+  Note that:
+  * The changes do get applied as early as possible when using the
+    special flag.
+  * The /horizontal difference/ between /Fast/ and /Regular/ is the
+    value calculated by the /Correction algorithm/.
+
+** Seeking to 2.0 and then back to 1.0
+
+#+CAPTION: Seeking to 2.0 and then back to 1.0
+#+NAME: fig:ex1
+[[./Seek_to_2.0_then_back_to_1.0.png]]
+
+  See figure [[fig:ex1][Seeking to 2.0 and then back to 1.0]].
+
+  In that example, at stream-time 10s we seek to 2.0, the actual
+  running-time at which it happens is 8s. At stream-time 18s we then
+  seek back to 1.0.
+
+** Seeking to 2.0, 1.0, 2.0, 1.0
+
+#+CAPTION: Seeking to 2.0 / 1.0 / 2.0 / 1.0
+#+NAME: fig:ex2
+[[./Seek_to_2.0_then_1.0_then_2.0_then_1.0.png]]
+
+  See figure [[fig:ex2][Seek to 2.0 then 1.0 then 2.0 then 1.0]].
+
+  This is a bit more complex example. Furthermore the amount of
+  buffering between seek handler and sinks vary through time.
+
+** Instant-rate overrides
+
+  See figure [[fig:ex3][Multiple instant seeks]].
+
+  In this example, the user quickly makes two instant-rate seeks to
+  2.0 then back to 1.0. The changes happen before the /initial/ 2.0
+  segment has arrived in the sink. And then later switches to 2.0
+  again. This shows that the algorithm can support such fast changes
+  also.
+
+#+CAPTION: Instant-rate switch before target segment arrives
+#+NAME: fig:ex3
+[[./Multiple_fast_seeks.png]]
+\FloatBarrier
+# The float barrier above is to prevent figures to be placed further
+
+* Issues to solve
+
+ * Replacing sinks without flushing. There are states that are kept in
+   each sink which wouldn't be present in new sinks.
+
+ * Avoiding "noise" or "glitches" in baseaudiosink.
+
+ * Adjusting the ~running_time_offset~ of other events. The offset we calculated
+   is only ever valid for the moment a segment change happened and after that the
+   offset can continue to drift. For the time being, we use the current running time
+   in the sink at the time the event arrives but this won't work in general.