Application development manual

1 files changed, 201 insertions, 0 deletions

diff --git a/manual-clocks.md b/manual-clocks.md
new file mode 100644
index 0000000..80c159a
--- /dev/null
+++ b/manual-clocks.md
@@ -0,0 +1,201 @@
+---
+title: Clocks and synchronization in GStreamer
+...
+
+# Clocks and synchronization in GStreamer
+
+When playing complex media, each sound and video sample must be played
+in a specific order at a specific time. For this purpose, GStreamer
+provides a synchronization mechanism.
+
+GStreamer provides support for the following use cases:
+
+  - Non-live sources with access faster than playback rate. This is the
+    case where one is reading media from a file and playing it back in a
+    synchronized fashion. In this case, multiple streams need to be
+    synchronized, like audio, video and subtitles.
+
+  - Capture and synchronized muxing/mixing of media from multiple live
+    sources. This is a typical use case where you record audio and video
+    from a microphone/camera and mux it into a file for storage.
+
+  - Streaming from (slow) network streams with buffering. This is the
+    typical web streaming case where you access content from a streaming
+    server with http.
+
+  - Capture from live source and and playback to live source with
+    configurable latency. This is used when, for example, capture from a
+    camera, apply an effect and display the result. It is also used when
+    streaming low latency content over a network with UDP.
+
+  - Simultaneous live capture and playback from prerecorded content.
+    This is used in audio recording cases where you play a previously
+    recorded audio and record new samples, the purpose is to have the
+    new audio perfectly in sync with the previously recorded data.
+
+GStreamer uses a `GstClock` object, buffer timestamps and a SEGMENT
+event to synchronize streams in a pipeline as we will see in the next
+sections.
+
+# Clock running-time
+
+In a typical computer, there are many sources that can be used as a time
+source, e.g., the system time, soundcards, CPU performance counters, ...
+For this reason, there are many `GstClock` implementations available in
+GStreamer. The clock time doesn't always start from 0 or from some known
+value. Some clocks start counting from some known start date, other
+clocks start counting since last reboot, etc...
+
+A `GstClock` returns the **absolute-time** according to that clock with
+`gst_clock_get_time ()`. The absolute-time (or clock time) of a clock is
+monotonically increasing. From the absolute-time is a **running-time**
+calculated, which is simply the difference between a previous snapshot
+of the absolute-time called the **base-time**. So:
+
+running-time = absolute-time - base-time
+
+A GStreamer `GstPipeline` object maintains a `GstClock` object and a
+base-time when it goes to the PLAYING state. The pipeline gives a handle
+to the selected `GstClock` to each element in the pipeline along with
+selected base-time. The pipeline will select a base-time in such a way
+that the running-time reflects the total time spent in the PLAYING
+state. As a result, when the pipeline is PAUSED, the running-time stands
+still.
+
+Because all objects in the pipeline have the same clock and base-time,
+they can thus all calculate the running-time according to the pipeline
+clock.
+
+# Buffer running-time
+
+To calculate a buffer running-time, we need a buffer timestamp and the
+SEGMENT event that preceeded the buffer. First we can convert the
+SEGMENT event into a `GstSegment` object and then we can use the
+`gst_segment_to_running_time ()` function to perform the calculation of
+the buffer running-time.
+
+Synchronization is now a matter of making sure that a buffer with a
+certain running-time is played when the clock reaches the same
+running-time. Usually this task is done by sink elements. Sink also have
+to take into account the latency configured in the pipeline and add this
+to the buffer running-time before synchronizing to the pipeline clock.
+
+Non-live sources timestamp buffers with a running-time starting from 0.
+After a flushing seek, they will produce buffers again from a
+running-time of 0.
+
+Live sources need to timestamp buffers with a running-time matching the
+pipeline running-time when the first byte of the buffer was captured.
+
+# Buffer stream-time
+
+The buffer stream-time, also known as the position in the stream, is
+calculated from the buffer timestamps and the preceding SEGMENT event.
+It represents the time inside the media as a value between 0 and the
+total duration of the media.
+
+The stream-time is used in:
+
+  - Report the current position in the stream with the POSITION query.
+
+  - The position used in the seek events and queries.
+
+  - The position used to synchronize controlled values.
+
+The stream-time is never used to synchronize streams, this is only done
+with the running-time.
+
+# Time overview
+
+Here is an overview of the various timelines used in GStreamer.
+
+The image below represents the different times in the pipeline when
+playing a 100ms sample and repeating the part between 50ms and 100ms.
+
+![GStreamer clock and various times](images/clocks.png "fig:")
+
+You can see how the running-time of a buffer always increments
+monotonically along with the clock-time. Buffers are played when their
+running-time is equal to the clock-time - base-time. The stream-time
+represents the position in the stream and jumps backwards when
+repeating.
+
+# Clock providers
+
+A clock provider is an element in the pipeline that can provide a
+`GstClock` object. The clock object needs to report an absolute-time
+that is monotonically increasing when the element is in the PLAYING
+state. It is allowed to pause the clock while the element is PAUSED.
+
+Clock providers exist because they play back media at some rate, and
+this rate is not necessarily the same as the system clock rate. For
+example, a soundcard may playback at 44,1 kHz, but that doesn't mean
+that after *exactly* 1 second *according to the system clock*, the
+soundcard has played back 44.100 samples. This is only true by
+approximation. In fact, the audio device has an internal clock based on
+the number of samples played that we can expose.
+
+If an element with an internal clock needs to synchronize, it needs to
+estimate when a time according to the pipeline clock will take place
+according to the internal clock. To estimate this, it needs to slave its
+clock to the pipeline clock.
+
+If the pipeline clock is exactly the internal clock of an element, the
+element can skip the slaving step and directly use the pipeline clock to
+schedule playback. This can be both faster and more accurate. Therefore,
+generally, elements with an internal clock like audio input or output
+devices will be a clock provider for the pipeline.
+
+When the pipeline goes to the PLAYING state, it will go over all
+elements in the pipeline from sink to source and ask each element if
+they can provide a clock. The last element that can provide a clock will
+be used as the clock provider in the pipeline. This algorithm prefers a
+clock from an audio sink in a typical playback pipeline and a clock from
+source elements in a typical capture pipeline.
+
+There exist some bus messages to let you know about the clock and clock
+providers in the pipeline. You can see what clock is selected in the
+pipeline by looking at the NEW\_CLOCK message on the bus. When a clock
+provider is removed from the pipeline, a CLOCK\_LOST message is posted
+and the application should go to PAUSED and back to PLAYING to select a
+new clock.
+
+# Latency
+
+The latency is the time it takes for a sample captured at timestamp X to
+reach the sink. This time is measured against the clock in the pipeline.
+For pipelines where the only elements that synchronize against the clock
+are the sinks, the latency is always 0 since no other element is
+delaying the buffer.
+
+For pipelines with live sources, a latency is introduced, mostly because
+of the way a live source works. Consider an audio source, it will start
+capturing the first sample at time 0. If the source pushes buffers with
+44100 samples at a time at 44100Hz it will have collected the buffer at
+second 1. Since the timestamp of the buffer is 0 and the time of the
+clock is now \>= 1 second, the sink will drop this buffer because it is
+too late. Without any latency compensation in the sink, all buffers will
+be dropped.
+
+## Latency compensation
+
+Before the pipeline goes to the PLAYING state, it will, in addition to
+selecting a clock and calculating a base-time, calculate the latency in
+the pipeline. It does this by doing a LATENCY query on all the sinks in
+the pipeline. The pipeline then selects the maximum latency in the
+pipeline and configures this with a LATENCY event.
+
+All sink elements will delay playback by the value in the LATENCY event.
+Since all sinks delay with the same amount of time, they will be
+relative in sync.
+
+## Dynamic Latency
+
+Adding/removing elements to/from a pipeline or changing element
+properties can change the latency in a pipeline. An element can request
+a latency change in the pipeline by posting a LATENCY message on the
+bus. The application can then decide to query and redistribute a new
+latency or not. Changing the latency in a pipeline might cause visual or
+audible glitches and should therefore only be done by the application
+when it is allowed.
+