diff options
| author | Stefan Kost <ensonic@users.sf.net> | 2010-11-01 15:32:43 +0200 |
|---|---|---|
| committer | Stefan Kost <ensonic@users.sf.net> | 2010-12-03 09:50:31 +0200 |
| commit | 16ce2d4ea47f48b25c83ef6c8df81b01a986326c (patch) | |
| tree | 641d17da8121cf416376953bd96b7fb24f763127 | |
| parent | 1c50dcd54fb1cd07988b5d928bea9840f80a603c (diff) | |
design-docs: add html output using asciidoc
Unify the ad-hoc markup to be asciidoc style in many places. Add a "html" target
to Makefile to generate the output.
46 files changed, 1681 insertions, 1633 deletions
diff --git a/docs/design/Makefile.am b/docs/design/Makefile.am index 1b8b4fba2..1433ee17f 100644 --- a/docs/design/Makefile.am +++ b/docs/design/Makefile.am @@ -47,6 +47,22 @@ EXTRA_DIST = \ part-TODO.txt \ part-trickmodes.txt +CLEANFILES = index.html index.txt + +html: + if ! test -z `which asciidoc`; then \ + echo >index.txt "GStreamer design"; \ + echo >>index.txt "================"; \ + echo >>index.txt "The Gstreamer developers"; \ + echo >>index.txt "Version $(PACKAGE_VERSION)"; \ + echo >>index.txt ""; \ + ( cd $(srcdir) && \ + cat >>$(abs_builddir)/index.txt $(EXTRA_DIST) ); \ + asciidoc -o index.html index.txt; \ + else \ + echo "need asciidoc to generate html"; \ + fi; + upload: @echo nothing to upload diff --git a/docs/design/draft-buffer2.txt b/docs/design/draft-buffer2.txt index 7fbe41265..20a03ae55 100644 --- a/docs/design/draft-buffer2.txt +++ b/docs/design/draft-buffer2.txt @@ -23,7 +23,7 @@ Some examples of metadata: Requirements ------------- +~~~~~~~~~~~~ - It must be fast * allocation, free, low fragmentation @@ -41,7 +41,7 @@ Requirements GstMiniObject -------------- +~~~~~~~~~~~~~ We make GstMiniObject a simple refcounted C structure and also a GLib boxed type. The following fields will be in the structure: @@ -66,7 +66,7 @@ objects. GstEvent, GstCaps, GstQuery, GstMessage ---------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Have the new GstMiniObject be the first field in these objects. They will probably also replace the copy and free functions with their own implementations. @@ -77,7 +77,7 @@ the custom functions. GstBuffer ---------- +~~~~~~~~~ A GstMiniObject will be the parent instance of the GstBuffer object, which is a regular C structure. @@ -105,7 +105,7 @@ Buffers point to a GstCaps structure that contains the caps of the buffer data. GstBufferMeta -------------- +~~~~~~~~~~~~~ A GstBufferMeta is a structure as follows: @@ -208,7 +208,7 @@ GstBufferMetaMemory | | next ---+ . . API examples ------------- +~~~~~~~~~~~~ Buffers are created using the normal gst_buffer_new functions. The standard fields are initialized as usual. A memory area that is bigger than the structure size @@ -276,7 +276,7 @@ possible simple API would look like this: Memory management ------------------ +~~~~~~~~~~~~~~~~~ * allocation @@ -316,7 +316,7 @@ Memory management Subbuffers ----------- +~~~~~~~~~~ Subbuffers are a first class feature of the GstBuffer. @@ -334,7 +334,7 @@ timing metadata needs to be reset to NONE when the start offset is different. Serialization -------------- +~~~~~~~~~~~~~ When buffer should be sent over the wire or be serialized in GDP, we need a way to perform custom serialization and deserialization on the metadata. @@ -345,7 +345,7 @@ and endianness. Transformations ---------------- +~~~~~~~~~~~~~~~ After certain transformations, the metadata on a buffer might not be relevant anymore. @@ -368,7 +368,7 @@ so on). Other use cases ---------------- +~~~~~~~~~~~~~~~ Making the GstBufferMetaMemory (for making the buffer point to the associated memory region) as metadata on a GstBuffer, as opposed to making it an integral @@ -397,7 +397,7 @@ implementations add (or use, in the case of a file reader) the memory metadata. Relationship with GstCaps -------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~ The difference between GstCaps, used in negotiation, and the metadata is not clearly defined. @@ -415,7 +415,7 @@ video resolution while the implementation details would be inside the metadata. Compatibility -------------- +~~~~~~~~~~~~~ We need to make sure that elements exchange metadata that they both understand, This is particulary important when the metadata describes the data layout in @@ -436,16 +436,16 @@ that are metadata aware to set a flag on their pads; any buffer passed on that pad will be converted to the old layout when the flag is not set. -Notes: ------- +Notes +~~~~~ Some structures that we need to be able to add to buffers. - Clean Aperture - Arbitrary Matrix Transform - Aspect ratio - Pan/crop/zoom - Video strides +* Clean Aperture +* Arbitrary Matrix Transform +* Aspect ratio +* Pan/crop/zoom +* Video strides Some of these overlap, we need to find a minimal set of metadata structures that allows us to define all use cases. diff --git a/docs/design/draft-klass.txt b/docs/design/draft-klass.txt index 278ec1a3f..cfbfaff50 100644 --- a/docs/design/draft-klass.txt +++ b/docs/design/draft-klass.txt @@ -1,189 +1,187 @@ Element Klass definition ------------------------ -Status: - - DRAFT. - - -Purpose: - - Applications should be able to retrieve elements from the registry of existing - elements based on specific capabilities or features of the element. - - A playback application might want to retrieve all the elements that can be - used for visualisation, for example, or a video editor might want to select - all video effect filters. - - The topic of defining the klass of elements should be based on use cases. - - A list of classes that are used in a installation can be generated using: - gst-inspect-0.10 -a | grep -ho Class:.* | cut -c8- | sed "s/\//\\n/g" | sort | uniq - -Proposal: - - The GstElementDetails contains a field named klass that is a pointer to a - string describing the element type. - - In this document we describe the format and contents of the string. Elements - should adhere to this specification although that is not enforced to allow - for wild (application specific) customisation. - - 1) string format - - <keyword>['/'<keyword]* - - The string consists of an _unordered_ list of keywords separated with a '/' - character. While the / suggests a hierarchy, this is _not_ the case. - - 2) keyword categories - - - functional - - Categories are base on _intended usage_ of the element. Some elements - might have other side-effects (especially for filers/effects). The purpose - is to list enough keywords so that applications can do meaningfull filtering, - not to completely describe the functionality, that is expressed in caps etc.. - - * Source : produces data - * Sink : consumes data - * Filter : filters/transforms data, no modification on the data is - intended (although it might be unavoidable). The - filter can decide on input and output caps independently - of the stream contents (GstBaseTransform). - * Effect : applies an effect to some data, changes to data are - intended. Examples are colorbalance, volume. These - elements can also be implemented with GstBaseTransform. - * Demuxer : splits audio, video, ... from a stream - * Muxer : interleave audio, video, ... into one stream, this is - like mixing but without losing or degrading each separate - input stream. The reverse operation is possible with a - Demuxer that reproduces the exact same input streams. - * Decoder : decodes encoded data into a raw format, there is typically - no relation between input caps and output caps. The output - caps are defined in the stream data. This separates the - Decoder from the Filter and Effect. - * Encoder : encodes raw data into an encoded format. - * Mixer : combine audio, video, .. this is like muxing but with - applying some algorithm so that the individual streams - are not extractable anymore, there is therefore no - reverse operation to mixing. (audio mixer, video mixer, ...) - * Converter : convert audio into video, text to audio, ... The converter - typically works on raw types only. The source media type - is listed first. - * Analyzer : reports about the stream contents. - * Control : controls some aspect of a hardware device - * Extracter : extracts tags/headers from a stream - * Formatter : adds tags/headers to a stream - * Connector : allows for new connections in the pipeline. (tee, ...) - * ... - - - Based on media type - - Purpose is to make a selection for elements operating on the different - types of media. An audio application must be able to filter out the - elements operating on audio, for example. - - * Audio : operates on audio data - * Video : operates on video data - * Image : operates on image data. Usually this media type can also - be used to make a video stream in which case it is added - together with the Video media type. - * Text : operates on text data - * Metadata : operates on metadata - * ... - - - Extra features - - The purpose is to further specialize the element, mostly for - application specific needs. - - * Network : element is used in networked situations - * Protocol : implements some protocol (RTSP, HTTP, ...) - * Payloader : encapsulate as payload (RTP, RDT,.. ) - * Depayloader : strip a payload (RTP, RDT,.. ) - * RTP : intended to be used in RTP applications - * Device : operates on some hardware device (disk, network, - audio card, video card, usb, ...) - * Visualisation : intended to be used for audio visualisation - * Debug : intended usage is more for debugging purposes. - - - Categories found, but not yet in one of the above lists - - * Bin : playbin, decodebin, bin, pipeline - * Codec : lots of decoders, encoder, demuxers - should be removed? - * Generic : should be removed? - * File : like network, should go to Extra? - * Editor : gnonlin, textoverlays - * DVD, GDP, LADSPA, Parser, Player, Subtitle, Testing, ... - - 3) suggested order: - - <functional>[/<media type>]*[/<extra...>]* - - 4) examples: - - apedemux : Extracter/Metadata - audiotestsrc : Source/Audio +Purpose +~~~~~~~ + +Applications should be able to retrieve elements from the registry of existing +elements based on specific capabilities or features of the element. + +A playback application might want to retrieve all the elements that can be +used for visualisation, for example, or a video editor might want to select +all video effect filters. + +The topic of defining the klass of elements should be based on use cases. + +A list of classes that are used in a installation can be generated using: +gst-inspect-0.10 -a | grep -ho Class:.* | cut -c8- | sed "s/\//\\n/g" | sort | uniq + +Proposal +~~~~~~~~ + +The GstElementDetails contains a field named klass that is a pointer to a +string describing the element type. + +In this document we describe the format and contents of the string. Elements +should adhere to this specification although that is not enforced to allow +for wild (application specific) customisation. + +1) string format + + <keyword>['/'<keyword]* + + The string consists of an _unordered_ list of keywords separated with a '/' + character. While the / suggests a hierarchy, this is _not_ the case. + +2) keyword categories + + - functional + + Categories are base on _intended usage_ of the element. Some elements + might have other side-effects (especially for filers/effects). The purpose + is to list enough keywords so that applications can do meaningfull filtering, + not to completely describe the functionality, that is expressed in caps etc.. + + * Source : produces data + * Sink : consumes data + * Filter : filters/transforms data, no modification on the data is + intended (although it might be unavoidable). The + filter can decide on input and output caps independently + of the stream contents (GstBaseTransform). + * Effect : applies an effect to some data, changes to data are + intended. Examples are colorbalance, volume. These + elements can also be implemented with GstBaseTransform. + * Demuxer : splits audio, video, ... from a stream + * Muxer : interleave audio, video, ... into one stream, this is + like mixing but without losing or degrading each separate + input stream. The reverse operation is possible with a + Demuxer that reproduces the exact same input streams. + * Decoder : decodes encoded data into a raw format, there is typically + no relation between input caps and output caps. The output + caps are defined in the stream data. This separates the + Decoder from the Filter and Effect. + * Encoder : encodes raw data into an encoded format. + * Mixer : combine audio, video, .. this is like muxing but with + applying some algorithm so that the individual streams + are not extractable anymore, there is therefore no + reverse operation to mixing. (audio mixer, video mixer, ...) + * Converter : convert audio into video, text to audio, ... The converter + typically works on raw types only. The source media type + is listed first. + * Analyzer : reports about the stream contents. + * Control : controls some aspect of a hardware device + * Extracter : extracts tags/headers from a stream + * Formatter : adds tags/headers to a stream + * Connector : allows for new connections in the pipeline. (tee, ...) + * ... + + - Based on media type + + Purpose is to make a selection for elements operating on the different + types of media. An audio application must be able to filter out the + elements operating on audio, for example. + + * Audio : operates on audio data + * Video : operates on video data + * Image : operates on image data. Usually this media type can also + be used to make a video stream in which case it is added + together with the Video media type. + * Text : operates on text data + * Metadata : operates on metadata + * ... + + - Extra features + + The purpose is to further specialize the element, mostly for + application specific needs. + + * Network : element is used in networked situations + * Protocol : implements some protocol (RTSP, HTTP, ...) + * Payloader : encapsulate as payload (RTP, RDT,.. ) + * Depayloader : strip a payload (RTP, RDT,.. ) + * RTP : intended to be used in RTP applications + * Device : operates on some hardware device (disk, network, + audio card, video card, usb, ...) + * Visualisation : intended to be used for audio visualisation + * Debug : intended usage is more for debugging purposes. + + - Categories found, but not yet in one of the above lists + + * Bin : playbin, decodebin, bin, pipeline + * Codec : lots of decoders, encoder, demuxers + should be removed? + * Generic : should be removed? + * File : like network, should go to Extra? + * Editor : gnonlin, textoverlays + * DVD, GDP, LADSPA, Parser, Player, Subtitle, Testing, ... + +3) suggested order: + + <functional>[/<media type>]*[/<extra...>]* + +4) examples: + + apedemux : Extracter/Metadata + audiotestsrc : Source/Audio + autoaudiosink : Sink/Audio/Device + cairotimeoverlay : Mixer/Video/Text + dvdec : Decoder/Video + dvdemux : Demuxer + goom : Converter/Audio/Video + id3demux : Extracter/Metadata + udpsrc : Source/Network/Protocol/Device + videomixer : Mixer/Video + ffmpegcolorspace : Filter/Video (intended use to convert video with as little + visible change as possible) + vertigotv : Effect/Video (intended use is to change the video) + volume : Effect/Audio (intended use is to change the audio data) + vorbisdec : Decoder/Audio + vorbisenc : Encoder/Audio + oggmux : Muxer + adder : Mixer/Audio + videobox : Effect/Video + alsamixer : Control/Audio/Device + audioconvert : Filter/Audio + audioresample : Filter/Audio + xvimagesink : Sink/Video/Device + navseek : Filter/Debug + decodebin : Decoder/Demuxer + level : Filter/Analyzer/Audio + tee : Connector/Debug + + 5) open issues: + + - how to differencial physical devices from logical ones? autoaudiosink : Sink/Audio/Device - cairotimeoverlay : Mixer/Video/Text - dvdec : Decoder/Video - dvdemux : Demuxer - goom : Converter/Audio/Video - id3demux : Extracter/Metadata - udpsrc : Source/Network/Protocol/Device - videomixer : Mixer/Video - ffmpegcolorspace : Filter/Video (intended use to convert video with as little - visible change as possible) - vertigotv : Effect/Video (intended use is to change the video) - volume : Effect/Audio (intended use is to change the audio data) - vorbisdec : Decoder/Audio - vorbisenc : Encoder/Audio - oggmux : Muxer - adder : Mixer/Audio - videobox : Effect/Video - alsamixer : Control/Audio/Device - audioconvert : Filter/Audio - audioresample : Filter/Audio - xvimagesink : Sink/Video/Device - navseek : Filter/Debug - decodebin : Decoder/Demuxer - level : Filter/Analyzer/Audio - tee : Connector/Debug + alsasink : Sink/Audio/Device - 5) open issues: +Use cases +~~~~~~~~~ - - how to differencial physical devices from logical ones? - autoaudiosink : Sink/Audio/Device - alsasink : Sink/Audio/Device +- get a list of all elements implementing a video effect (pitivi): -Use cases: + klass.contains (Effect & Video) - - get a list of all elements implementing a video effect (pitivi): +- get list of muxers (pitivi): - klass.contains (Effect & Video) + klass.contains (Muxer) - - get list of muxers (pitivi): +- get list of video encoders (pitivi): - klass.contains (Muxer) + klass.contains (Encoder & video) - - get list of video encoders (pitivi): +- Get a list of all audio/video visualisations (totem): - klass.contains (Encoder & video) + klass.contains (Visualisation) - - Get a list of all audio/video visualisations (totem): +- Get a list of all decoders/demuxer/metadata parsers/vis (playbin): - klass.contains (Visualisation) + klass.contains (Visualisation | Demuxer | Decoder | (Extractor & Metadata)) - - Get a list of all decoders/demuxer/metadata parsers/vis (playbin): +- Get a list of elements that can capture from an audio device (gst-properties): - klass.contains (Visualisation | Demuxer | Decoder | (Extractor & Metadata)) + klass.contains (Source & Audio & Device) - - Get a list of elements that can capture from an audio device (gst-properties): - - klass.contains (Source & Audio & Device) - - * filters out audiotestsrc, since it is not a device + * filters out audiotestsrc, since it is not a device diff --git a/docs/design/draft-metadata.txt b/docs/design/draft-metadata.txt index 68d28142b..ffc4df659 100644 --- a/docs/design/draft-metadata.txt +++ b/docs/design/draft-metadata.txt @@ -6,7 +6,7 @@ additions. Supported Metadata standards ----------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The paragraphs below list supported native metadata standards sorted by type and then in alphabetical order. Some standards have been extended to support @@ -94,7 +94,7 @@ http://wiki.creativecommons.org/Tracker_CC_Indexing Current Metadata handling -------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~ When reading files, demuxers or parsers extract the metadata. It will be sent as a GST_MESSAGE_TAG to the GstBus and GST_EVENT_TAG to downstream elements. @@ -130,10 +130,11 @@ metadata. Issues ------- +~~~~~~ Unknown/Unmapped metadata -- - - - - - - - - - - - - +^^^^^^^^^^^^^^^^^^^^^^^^^ + Right now GStreamer can lose metadata when transcoding, remuxing content. This can happend as we don't map all metadata fields to generic ones. @@ -159,7 +160,8 @@ We would basically need this for each container format. See also https://bugzilla.gnome.org/show_bug.cgi?id=345352 Lost metadata -- - - - - - - +^^^^^^^^^^^^^ + A case slighly different from the previous is that when an application sets a GstTagList on a pipeline. Right elements consuming tags do not report which tags have been consumed. Especially when using elements that make metadata @@ -181,7 +183,8 @@ of elements and their tags. As a convenience we could flatten the list of tags for the top-level element (if the query was sent to a bin) and add that. Tags are per Element -- - - - - - - - - - +^^^^^^^^^^^^^^^^^^^^ + In many cases we want tags per stream. Even metadata standards like mp4/3gp metadata supports that. Right now GST_MESSAGE_SRC(tags) is the element. We tried changing that to the pad, but that broke applications. @@ -189,7 +192,7 @@ Also we miss the symmetric functionality in GstTagSetter. This interface is usually implemented by elements. Open bugs -- - - - - +^^^^^^^^^ https://bugzilla.gnome.org/buglist.cgi?query_format=advanced;short_desc=tag;bug_status=UNCONFIRMED;bug_status=NEW;bug_status=ASSIGNED;bug_status=REOPENED;bug_status=NEEDINFO;short_desc_type=allwordssubstr;product=GStreamer diff --git a/docs/design/draft-tagreading.txt b/docs/design/draft-tagreading.txt index 89308f08c..7d0262934 100644 --- a/docs/design/draft-tagreading.txt +++ b/docs/design/draft-tagreading.txt @@ -20,7 +20,7 @@ peeked buffers (a queue element after sink, but that would change pull to push). Design ------- +~~~~~~ The plan is that applications can do the following: pipeline = "filesrc ! tagbin" @@ -38,7 +38,8 @@ The plan is that applications can do the following: decodebin Interface ---------- +~~~~~~~~~ + * gboolean iface property "tag-reading" Switches the element to tagreading mode. Needed if normal element implement that behaviour. Elements will skip parsing unneeded data, don't build a @@ -47,7 +48,7 @@ Interface Equivalent of EOS. Use Cases ---------- +~~~~~~~~~ * mp3 with id3- and apetags * plug id3demux ! apedemux @@ -57,7 +58,8 @@ Use Cases * plug vorbisdec or special vorbiscomment reader Additional Thoughts -------------------- +~~~~~~~~~~~~~~~~~~~ + * would it make sense to have 2-phase tag-reading (property on tagbin and/or tagread elements) * 1st phase: get tag-data that are directly embedded in the data @@ -73,23 +75,25 @@ Additional Thoughts gst_tag_cache_store_tag_data (GstTagCache *self, const gchar *uri, GstTagList *tags); Tests ------ +~~~~~ + * write a generic test for parsers/demuxers to ensure they send tags until they reached PAUSED (elements need to parse file for prerolling anyway): set pipeline to paused, check for tags, set to playing, error out if tags come after paused Code Locations --------------- +~~~~~~~~~~~~~~ + * tagreadbin -> gst-plugins-base/gst/tagread * tagreaderiface -> gst-plugins-base/gst-libs/gst/tag Reuse ------ +~~~~~ -ogg : gst-plugins-base/ext/ogg -avi : gst-plugins-good/gst/avi -mp3 : gst-plugins-good/gst/id3demux -wav : gst-plugins-good/gst/wavparse -qt : gst-plugins-bad/gst/qtdemux +* ogg : gst-plugins-base/ext/ogg +* avi : gst-plugins-good/gst/avi +* mp3 : gst-plugins-good/gst/id3demux +* wav : gst-plugins-good/gst/wavparse +* qt : gst-plugins-bad/gst/qtdemux diff --git a/docs/design/part-MT-refcounting.txt b/docs/design/part-MT-refcounting.txt index e992da294..a6e652935 100644 --- a/docs/design/part-MT-refcounting.txt +++ b/docs/design/part-MT-refcounting.txt @@ -10,7 +10,7 @@ This design decision has implications for the usage of the API and the objects which this document explains. MT safety techniques --------------------- +~~~~~~~~~~~~~~~~~~~~ Several design patterns are used to guarantee object consistency in GStreamer. This is an overview of the methods used in various GStreamer subsystems. @@ -140,7 +140,7 @@ Compare and swap Objects -------- +~~~~~~~ * Locking involved: @@ -414,3 +414,4 @@ Objects } } gst_iterator_free (it); + diff --git a/docs/design/part-TODO.txt b/docs/design/part-TODO.txt index 4bd0ae94e..32ea19050 100644 --- a/docs/design/part-TODO.txt +++ b/docs/design/part-TODO.txt @@ -1,5 +1,8 @@ +TODO - Future Development +------------------------- + API/ABI -------- +~~~~~~~ - implement return values from events in addition to the gboolean. This should be done by making the event contain a GstStructure with input/output values, @@ -66,7 +69,7 @@ API/ABI IMPLEMENTATION --------------- +~~~~~~~~~~~~~~ - implement more QOS, see part-qos.txt. @@ -76,7 +79,7 @@ IMPLEMENTATION DESIGN ------- +~~~~~~ - unlinking pads in the PAUSED state needs to make sure the stream thread is not executing code. Can this be done with a flush to unlock all downstream chain diff --git a/docs/design/part-activation.txt b/docs/design/part-activation.txt index 6a56ff03b..4d7a2193a 100644 --- a/docs/design/part-activation.txt +++ b/docs/design/part-activation.txt @@ -1,5 +1,8 @@ -Pad activation --------------- +Pad (de)activation +------------------ + +Activation +~~~~~~~~~~ When changing states, a bin will set the state on all of its children in sink-to-source order. As elements undergo the READY->PAUSED transition, @@ -73,7 +76,7 @@ already in PAUSED with an active sink pad by the time fakesrc starts pushing data. Deactivation ------------- +~~~~~~~~~~~~ Pad deactivation occurs when its parent goes into the READY state or when the pad is deactivated explicitly by the application or element. @@ -82,7 +85,8 @@ activate_push() or activate_pull() with a FALSE argument, depending on the activation mode of the pad. Mode switching --------------- +~~~~~~~~~~~~~~ Changing from push to pull modes needs a bit of thought. This is actually possible and implemented but not yet documented here. + diff --git a/docs/design/part-block.txt b/docs/design/part-block.txt index e7a8561c3..5a1cf8068 100644 --- a/docs/design/part-block.txt +++ b/docs/design/part-block.txt @@ -18,66 +18,69 @@ and will succeed when the following events happen on the pad: Flushing --------- +~~~~~~~~ - The flushing event is used to clear any data out of the - downstream elements. +The flushing event is used to clear any data out of the +downstream elements. -* Generic case +Generic case +^^^^^^^^^^^^ - Consider the following pipeline: +Consider the following pipeline: .-----. .-------. .-------. | src | | elem1 |\/ | elem2 | | src -> sink src -> sink src .... '-----' '-------'/\ '-------' - Where elem1.src is blocked. If the pad block is taken (the callback - is called or the sync block returned) no data is flowing in elem2.sink. - In this situation, the streaming thread is blocked on a GCond and is - waiting to be unblocked. +Where elem1.src is blocked. If the pad block is taken (the callback +is called or the sync block returned) no data is flowing in elem2.sink. +In this situation, the streaming thread is blocked on a GCond and is +waiting to be unblocked. - When sending a flushing seek upstream on elem1.src, the FLUSH_START and - will temporary unblock the streaming thread and make all pad functions that - triggers a block (_push/_alloc_buffer/_push_event/_pull_range) return - GST_FLOW_WRONG_STATE. This will then eventually pause the streaming thread - and release the STREAM_LOCK. - - Since no STREAM lock is taken after the pad block it is not needed to send - the FLUSH_START event further downstream. +When sending a flushing seek upstream on elem1.src, the FLUSH_START and +will temporary unblock the streaming thread and make all pad functions that +triggers a block (_push/_alloc_buffer/_push_event/_pull_range) return +GST_FLOW_WRONG_STATE. This will then eventually pause the streaming thread +and release the STREAM_LOCK. - The FLUSH_STOP will set the srcpad to non-flushing again and is dropped - for the same reason. From then on, the new data after the flushing seek - will be queued when the pad block is taken again. +Since no STREAM lock is taken after the pad block it is not needed to send +the FLUSH_START event further downstream. -* Case where the stream is blocking downstream +The FLUSH_STOP will set the srcpad to non-flushing again and is dropped +for the same reason. From then on, the new data after the flushing seek +will be queued when the pad block is taken again. - The example above is only valid if the elem1.src pad is really blocking - (callback called or sync block returned). - - In the case where the stream is blocking further downstream (on elem2.src - for example, or on a blocking queue), extra care has to be taken. +Case where the stream is blocking downstream +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Consider the following pipeline: +The example above is only valid if the elem1.src pad is really blocking +(callback called or sync block returned). + +In the case where the stream is blocking further downstream (on elem2.src +for example, or on a blocking queue), extra care has to be taken. + +Consider the following pipeline: .-----. .-------. .-------. | src | | elem1 |\/ | elem2 | | src -> sink src -> sink src .... Blocking somewhere downstream '-----' '-------'/\ '-------' - A pad block has been requested by the user on elem1.src , but since the stream - is blocking somewhere downstream, the callback is not called or the sync block - does not return. +A pad block has been requested by the user on elem1.src , but since the stream +is blocking somewhere downstream, the callback is not called or the sync block +does not return. - In order for the block to happen, a FLUSH_START needs to be sent directly on - the downstream blocking element/pad so that it releases the stream lock, and it - gives a chance for the elem1.src pad to block. +In order for the block to happen, a FLUSH_START needs to be sent directly on +the downstream blocking element/pad so that it releases the stream lock, and it +gives a chance for the elem1.src pad to block. -Use cases: ----------- +Use cases +~~~~~~~~~ -* Prerolling a partial pipeline +Prerolling a partial pipeline +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .---------. .---------. .----------. | filesrc | | demuxer | .-----. | decoder1 | @@ -89,27 +92,28 @@ Use cases: '---------' '-----' '----------' X - The purpose is to create the pipeline dynamically up to the - decoders but not yet connect them to a sink and without losing - any data. +The purpose is to create the pipeline dynamically up to the +decoders but not yet connect them to a sink and without losing +any data. - To do this, the source pads of the decoders is blocked so that no - events or buffers can escape and we don't interrupt the stream. +To do this, the source pads of the decoders is blocked so that no +events or buffers can escape and we don't interrupt the stream. - When all of the dynamic pad are created (no-more-pads emited by the - branching point, ie, the demuxer or the queues filled) and the pads - are blocked (blocked callback received) the pipeline is completely - prerolled. +When all of the dynamic pad are created (no-more-pads emited by the +branching point, ie, the demuxer or the queues filled) and the pads +are blocked (blocked callback received) the pipeline is completely +prerolled. - It should then be possible to perform the following actions on the - prerolled pipeline: +It should then be possible to perform the following actions on the +prerolled pipeline: - - query duration/position - - perform a flushing seek to preroll a new position - - connect other elements and unblock the blocked pads. +- query duration/position +- perform a flushing seek to preroll a new position +- connect other elements and unblock the blocked pads. -* dynamically switching an element in a PLAYING pipeline. +dynamically switching an element in a PLAYING pipeline +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .----------. .----------. .----------. @@ -121,8 +125,8 @@ Use cases: sink src '----------' - The purpose is to replace element2 with element4 in the PLAYING - pipeline. +The purpose is to replace element2 with element4 in the PLAYING +pipeline. 1) block element1 src pad. This can be done async. 2) wait for block to happen. at that point nothing is flowing between @@ -144,18 +148,18 @@ Use cases: element should at least be PAUSED. 9) unblock element1 src - The same flow can be used to replace an element in a PAUSED pipeline. Only - special care has to be taken when performing step 2) which has to be done - async or it might deadlock. In the async callback one can then perform the - steps from 3). In a playing pipeline one can of course use the async block - as well, so that there is a generic method for both PAUSED and PLAYING. +The same flow can be used to replace an element in a PAUSED pipeline. Only +special care has to be taken when performing step 2) which has to be done +async or it might deadlock. In the async callback one can then perform the +steps from 3). In a playing pipeline one can of course use the async block +as well, so that there is a generic method for both PAUSED and PLAYING. - The same flow works as well for any chain of multiple elements and might - be implemented with a helper function in the future. +The same flow works as well for any chain of multiple elements and might +be implemented with a helper function in the future. Issues ------- +~~~~~~ When an EOS event has passed a pad and the pad is set to blocked, the block will never happen because no data is going to flow anymore. One possibility is to diff --git a/docs/design/part-buffering.txt b/docs/design/part-buffering.txt index bacd6259e..eb9d807bd 100644 --- a/docs/design/part-buffering.txt +++ b/docs/design/part-buffering.txt @@ -21,7 +21,7 @@ We want to be able to implement the following features: - easy (backward compatible) application notification of buffering - the possibility for the application to do more complex buffering - Some use cases: +Some use cases: * Stream buffering: @@ -112,7 +112,7 @@ We want to be able to implement the following features: Messages --------- +~~~~~~~~ A GST_MESSAGE_BUFFERING must be posted on the bus when playback temporarily stops to buffer and when buffering finishes. When percentage field in the @@ -137,27 +137,24 @@ are added to the buffering message: "avg-in-rate", G_TYPE_INT - gives the average input buffering speed in bytes/second. -1 is unknown. - This is the average number of bytes per second that is received on the buffering element input (sink) pads. It is a measurement of the network speed in most cases. "avg-out-rate", G_TYPE_INT - gives the average consumption speed in bytes/second. -1 is unknown. - This is the average number of bytes per second that is consumed by the downstream element of the buffering element. "buffering-left", G_TYPE_INT64 - gives the estimated time that bufferring will take in milliseconds. -1 unknown. - - This is measured based on the avg-in-rate and the filled level of the - queue. The application can use this hint to update the GUI about the - estimated remaining time that buffering will take. + This is measured based on the avg-in-rate and the filled level of the + queue. The application can use this hint to update the GUI about the + estimated remaining time that buffering will take. Application ------------ +~~~~~~~~~~~ While data is buffered, the pipeline should remain in the PAUSED state. It is also possible that more data should be buffered while the pipeline is PLAYING, @@ -180,7 +177,7 @@ time to resume playback to get uninterrupted playback. Buffering Query ---------------- +~~~~~~~~~~~~~~~ In addition to the BUFFERING messages posted by the buffering elements we want to be able to query the same information from the application. We also want to @@ -222,7 +219,7 @@ and newest item (expressed in "format") in the buffer. Defaults --------- +~~~~~~~~ Some defaults for common elements: diff --git a/docs/design/part-bufferlist.txt b/docs/design/part-bufferlist.txt index bc82ec991..91674509a 100644 --- a/docs/design/part-bufferlist.txt +++ b/docs/design/part-bufferlist.txt @@ -21,17 +21,18 @@ for the following extra functionality: Use cases ---------- +~~~~~~~~~ A typical use case for multimedia pipelines is to append or remove 'headers' from packets of data. -* Generating RTP packets from h264 video +Generating RTP packets from h264 video +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - We receive as input a GstBuffer with an encoded h264 image and we need to - create RTP packets containing this h264 data as the payload. We typically need - to fragment the h264 data into multiple packets, each with their own RTP and - payload specific header. +We receive as input a GstBuffer with an encoded h264 image and we need to +create RTP packets containing this h264 data as the payload. We typically need +to fragment the h264 data into multiple packets, each with their own RTP and +payload specific header. +-------+-------+---------------------------+--------+ input H264 buffer: | NALU1 | NALU2 | ..... | NALUx | @@ -41,25 +42,25 @@ from packets of data. +-+ +-------+ +-+ +-------+ +-+ +-------+ output bufferlist: | | | NALU1 | | | | NALU2 | .... | | | NALUx | +-+ +-------+ +-+ +-------+ +-+ +-------+ - + : : : : \-----------/ \-----------/ group 1 group 2 - The output bufferlist consists of x groups consisting of an RTP payload header - and a subbuffer of the original input H264 buffer. Since the rtp headers and - the h264 data don't need to be contiguous in memory, we can avoid to memcpy the - h264 data into the rtp packets. +The output bufferlist consists of x groups consisting of an RTP payload header +and a subbuffer of the original input H264 buffer. Since the rtp headers and +the h264 data don't need to be contiguous in memory, we can avoid to memcpy the +h264 data into the rtp packets. - Since we can generate a bufferlist with multiple groups, we can push all the - RTP packets for the input data to the next element in one operation. +Since we can generate a bufferlist with multiple groups, we can push all the +RTP packets for the input data to the next element in one operation. - A typical udpsink will then use something like sendmsg to send the groups on - the network inside one UDP packet. This will further avoid having to memcpy - data into contiguous memory. +A typical udpsink will then use something like sendmsg to send the groups on +the network inside one UDP packet. This will further avoid having to memcpy +data into contiguous memory. API ---- +~~~ The GstBufferList is an opaque data structure and is operated on using an iterator. It derives from GstMiniObject so that it has basic refcounting and @@ -78,7 +79,7 @@ group. Metadata --------- +~~~~~~~~ Each of the buffers inside the bufferlist can have metadata assiociated with it. @@ -92,3 +93,4 @@ first group in the bufferlist. This means that: This allows for efficient (re)timestamping and re-typing (caps) of a group of buffers without having to modify each of the buffer's metadata. + diff --git a/docs/design/part-clocks.txt b/docs/design/part-clocks.txt index 032ca3c20..5f080c5d9 100644 --- a/docs/design/part-clocks.txt +++ b/docs/design/part-clocks.txt @@ -11,7 +11,7 @@ The time reported by the clock is called the absolute_time. Clock Selection ---------------- +~~~~~~~~~~~~~~~ To synchronize the different elements, the GstPipeline is responsible for selecting and distributing a global GstClock for all the elements in it. @@ -35,7 +35,7 @@ PLAYING and is described in part-states.txt. Clock features --------------- +~~~~~~~~~~~~~~ The clock supports periodic and single shot clock notifications both synchronous and asynchronous. @@ -73,7 +73,7 @@ the clock is not PLAYING. Clock implementations ---------------------- +~~~~~~~~~~~~~~~~~~~~~ The GStreamer core provides a GstSystemClock based on the system time. Asynchronous callbacks are scheduled from an internal thread. diff --git a/docs/design/part-conventions.txt b/docs/design/part-conventions.txt index 5569644d6..9de333e58 100644 --- a/docs/design/part-conventions.txt +++ b/docs/design/part-conventions.txt @@ -1,5 +1,5 @@ Documentation conventions -========================= +------------------------- Due to the potential for exponential growth, several abbreviating conventions will be used throughout this documentation. These conventions have grown primarily from extremely in-depth discussions of the architecure in IRC. @@ -7,21 +7,21 @@ This has verified the safety of these conventions, if used properly. There are context is rigorously observed. Object classes --------------- +~~~~~~~~~~~~~~ Since everything starts with Gst, we will generally refer to objects by the shorter name, i.e. Element or Pad. These names will always have their first letter capitalized. Function names --------------- +~~~~~~~~~~~~~~ Within the context of a given object, functions defined in that object's header and/or source file will have their object-specific prefix stripped. For instance, gst_element_add_pad() would be referred to as simply _add_pad(). Note that the trailing parentheses should always be present, but sometimes may not be. A prefixing underscore (_) will always tell you it's a function, however, regardless of the presence or absence of the trailing parentheses. -#defines and enums ------------------- +defines and enums +~~~~~~~~~~~~~~~~~ Values and macros defined as enums and preprocessor macros will be referred to in all capitals, as per their definition. This includes object flags and element states, as well as general enums. Examples are the states NULL, @@ -31,23 +31,25 @@ element flags should be cross-checked with the header, as there are currently tw _FLAGS_ in the middle. Drawing conventions -=================== +~~~~~~~~~~~~~~~~~~~ When drawing pictures the folowing conventions apply: objects -------- +^^^^^^^ + +Objects are drawn with a box like: -Objects are drawn with a box like +------+ | | +------+ pointers --------- +^^^^^^^^ a pointer to an object. + +-----+ *--->| | +-----+ @@ -58,7 +60,7 @@ an invalid pointer, this is a pointer that should not be used. elements --------- +^^^^^^^^ +----------+ | name | @@ -66,16 +68,10 @@ elements +----------+ pad links ---------- +^^^^^^^^^ -----+ +--- | | src--sink -----+ +--- - - - - - - diff --git a/docs/design/part-dynamic.txt b/docs/design/part-dynamic.txt index ab6874b85..2a85924b1 100644 --- a/docs/design/part-dynamic.txt +++ b/docs/design/part-dynamic.txt @@ -11,3 +11,4 @@ following features of gstreamer: - pad blocking (part-block.txt) - playback segments. - streaming vs application threads. + diff --git a/docs/design/part-element-sink.txt b/docs/design/part-element-sink.txt index e5932c75c..edc8dd4d5 100644 --- a/docs/design/part-element-sink.txt +++ b/docs/design/part-element-sink.txt @@ -13,7 +13,7 @@ Sinks are harder to construct than other element types as they are treated specially by the GStreamer core. state changes -------------- +~~~~~~~~~~~~~ A sink always returns ASYNC from the state change to PAUSED, this includes a state change from READY->PAUSED and PLAYING->PAUSED. The @@ -29,7 +29,7 @@ and take no part in the preroll procedure. Events other than EOS do not complete the preroll stage. sink overview -------------- +~~~~~~~~~~~~~ - TODO: PREROLL_LOCK can be removed and we can safely use the STREAM_LOCK. diff --git a/docs/design/part-element-source.txt b/docs/design/part-element-source.txt index 46b6eb935..fc7f13dd1 100644 --- a/docs/design/part-element-source.txt +++ b/docs/design/part-element-source.txt @@ -13,7 +13,7 @@ Typical source elements include: Live sources ------------- +~~~~~~~~~~~~ A source is said to be a live source when it has the following property: @@ -52,7 +52,7 @@ Let's look at some example sources. Source types ------------- +~~~~~~~~~~~~ A source element can operate in three ways: @@ -90,36 +90,36 @@ source's state change function is called. Source base classes -------------------- +~~~~~~~~~~~~~~~~~~~ GstBaseSrc: - This base class provides an implementation of a random access source and - is very well suited for file reader like sources. +This base class provides an implementation of a random access source and +is very well suited for file reader like sources. GstPushSrc: - Base class for block-based sources. This class is mostly useful for - elements that cannot do random access, or at least very slowly. The - source usually prefers to push out a fixed size buffer. +Base class for block-based sources. This class is mostly useful for +elements that cannot do random access, or at least very slowly. The +source usually prefers to push out a fixed size buffer. - Classes extending this base class will usually be scheduled in a push - based mode. It the peer accepts to operate without offsets and withing - the limits of the allowed block size, this class can operate in getrange - based mode automatically. +Classes extending this base class will usually be scheduled in a push +based mode. It the peer accepts to operate without offsets and withing +the limits of the allowed block size, this class can operate in getrange +based mode automatically. - The subclass should extend the methods from the baseclass in - addition to the create method. If the source is seekable, it - needs to override GstBaseSrc::event() in addition to - GstBaseSrc::is_seekable() in order to retrieve the seek offset, - which is the offset of the next buffer to be requested. +The subclass should extend the methods from the baseclass in +addition to the create method. If the source is seekable, it +needs to override GstBaseSrc::event() in addition to +GstBaseSrc::is_seekable() in order to retrieve the seek offset, +which is the offset of the next buffer to be requested. - Flushing, scheduling and sync is all handled by this base class. +Flushing, scheduling and sync is all handled by this base class. Timestamps ----------- +~~~~~~~~~~ A non-live source should timestamp the buffers it produces starting from 0. If it is not possible to timestamp every buffer (filesrc), the source is allowed to @@ -134,3 +134,4 @@ the pipeline, which is expressed as: With absolute_time the time obtained from the global pipeline with gst_clock_get_time() and base_time being the time of that clock when the pipeline was last set to PLAYING. + diff --git a/docs/design/part-element-transform.txt b/docs/design/part-element-transform.txt index 3eb6e95cd..783b68889 100644 --- a/docs/design/part-element-transform.txt +++ b/docs/design/part-element-transform.txt @@ -49,7 +49,7 @@ each of the above use cases. We focus mostly on the buffer allocation strategies and caps negotiation. Processing ----------- +~~~~~~~~~~ A transform has 2 main processing functions: @@ -94,10 +94,10 @@ When no functions are provided, we can only process in passthrough mode. Negotiation ------------ +~~~~~~~~~~~ - Typical (re)negotiation of the transform element in push mode always goes from - sink to src, this means triggers the following sequence: +Typical (re)negotiation of the transform element in push mode always goes from +sink to src, this means triggers the following sequence: - the sinkpad receives a buffer with new caps, this triggers the setcaps function on the sinkpad before handing the buffer to transform. @@ -108,7 +108,7 @@ Negotiation target src caps - the transform processes and sets the output caps on the src pad - We call this downstream negotiation (DN) and it goes roughly like this: +We call this downstream negotiation (DN) and it goes roughly like this: sinkpad transform srcpad setcaps() | | | @@ -119,10 +119,10 @@ Negotiation | <configure caps> <-| | - These steps configure the element for a transformation from the input caps to - the output caps. +These steps configure the element for a transformation from the input caps to +the output caps. - The transform has 3 function to perform the negotiation: +The transform has 3 function to perform the negotiation: - transform_caps(): @@ -140,17 +140,17 @@ Negotiation Configure the transform for a transformation between src caps and dest caps. Both caps are guaranteed to be fixed caps. - If no transform_caps() is defined, we can only perform the identity transform, - by default. +If no transform_caps() is defined, we can only perform the identity transform, +by default. - If no set_caps() is defined, we don't care about caps. In that case we also - assume nothing is going to write to the buffer and we don't enforce a writable - buffer for the transform_ip function, when present. +If no set_caps() is defined, we don't care about caps. In that case we also +assume nothing is going to write to the buffer and we don't enforce a writable +buffer for the transform_ip function, when present. - One common function that we need for the transform element is to find the best - transform from one format (src) to another (dest). Since the function is - bidirectional, we will use the src->dest negotiation. Some requirements of this - function are: +One common function that we need for the transform element is to find the best +transform from one format (src) to another (dest). Since the function is +bidirectional, we will use the src->dest negotiation. Some requirements of this +function are: - has a fixed src caps - finds a fixed dest caps that the transform element can transform to @@ -158,7 +158,7 @@ Negotiation - the transform function prefers to make src caps == dest caps - the transform function can optionally fixate dest caps. - The find_transform() function goes like this: +The find_transform() function goes like this: - start from src aps, these caps are fixed. - check if the caps are acceptable for us as src caps. This is usually @@ -177,8 +177,8 @@ Negotiation - if we run out of caps, we fail to find a transform. - if we found a destination caps, configure the transform with set_caps(). - After this negotiation process, the transform element is usually in a steady - state. We can identify these steady states: +After this negotiation process, the transform element is usually in a steady +state. We can identify these steady states: - src and sink pads both have the same caps. Note that when the caps are equal on both pads, the input and output buffers automatically have the same size. @@ -266,14 +266,14 @@ Negotiation input buffer is transformed into the output buffer. The flow is exactly the same as the case with the same-caps negotiation. (DCC) - We can immeditatly observe that the copy transform states will need to - allocate a buffer from a downstream element using pad-alloc. When the transform - element is receiving a non-writable buffer in the in-place state, it will also - need to perform a pad-alloc. There is no reason why the passthrough state would - perform a pad-alloc. This is important because upstream re-negotiation can only - happen when the transform uses pad-alloc for all outgoing buffers. +We can immeditatly observe that the copy transform states will need to +allocate a buffer from a downstream element using pad-alloc. When the transform +element is receiving a non-writable buffer in the in-place state, it will also +need to perform a pad-alloc. There is no reason why the passthrough state would +perform a pad-alloc. This is important because upstream re-negotiation can only +happen when the transform uses pad-alloc for all outgoing buffers. - This steady state changes when one of the following actions occur: +This steady state changes when one of the following actions occur: - the sink pad receives new caps, this triggers the above downstream renegotation process, see above for the flow. @@ -284,9 +284,9 @@ Negotiation for example). This essentially clears the current steady state and triggers the downstream and upstream renegotiation process. - Parallel to the downstream negotiation process there is an upstream negotiation - process. The handling and proxy of buffer-alloc is the most comple part of the - transform element. This upstream negotiation process has 3 cases: (UN) +Parallel to the downstream negotiation process there is an upstream negotiation +process. The handling and proxy of buffer-alloc is the most comple part of the +transform element. This upstream negotiation process has 3 cases: (UN) - upstream calls the buffer-alloc function of the transform sinkpad and this call is proxied downstream (UNP) @@ -295,13 +295,13 @@ Negotiation - the transform calls the pad-alloc function downstream to allocate a new output buffer (but not because of a proxied buffer-alloc) (UNA) - The case where the pad-alloc is called because an output buffer must be - generated in the chain function is handled above in the copy-transform and the - in-place transform when the input buffer is not writable or the input buffer - size is smaller than the output size. +The case where the pad-alloc is called because an output buffer must be +generated in the chain function is handled above in the copy-transform and the +in-place transform when the input buffer is not writable or the input buffer +size is smaller than the output size. - We are left with the last case (proxy an incomming pad-alloc or not). We have 2 - possibilities here: +We are left with the last case (proxy an incomming pad-alloc or not). We have 2 +possibilities here: - pad-alloc is called with the same caps as are currently being handled by the transform on the sinkcaps. Note that this will only be true when the @@ -400,14 +400,14 @@ Negotiation <----------------------------------| | | | | - In order to perform passthrough buffer-alloc or pad-alloc, we need to be able - to get the size of the output buffer after the transform. - - For passthrough buffer-alloc, this is trivial: the input size equals the output - size. +In order to perform passthrough buffer-alloc or pad-alloc, we need to be able +to get the size of the output buffer after the transform. + +For passthrough buffer-alloc, this is trivial: the input size equals the output +size. - For the copy transform or the in-place transform we need additional function to - retrieve the size. There are two functions: +For the copy transform or the in-place transform we need additional function to +retrieve the size. There are two functions: - transform_size() @@ -426,35 +426,35 @@ Negotiation Issues ------- +~~~~~~ - passthrough and in-place transforms (with writable buffers) never need to - perform a pad-alloc on the srcpad. This means that if upstream negotiation - happens, the transform element will never know about it. +passthrough and in-place transforms (with writable buffers) never need to +perform a pad-alloc on the srcpad. This means that if upstream negotiation +happens, the transform element will never know about it. - The transform element will keep therefore track of the allocation pattern of - the peer elements. We can see the following cases: +The transform element will keep therefore track of the allocation pattern of +the peer elements. We can see the following cases: - - upstream peer calls buffer-alloc on the sinkpad of the transform. In some - cases (see above) this call gets proxied or not. +- upstream peer calls buffer-alloc on the sinkpad of the transform. In some + cases (see above) this call gets proxied or not. - - upstream peer does never call buffer-alloc. +- upstream peer does never call buffer-alloc. - We will keeps state about this allocation pattern and perform the following in - each case respectively: +We will keeps state about this allocation pattern and perform the following in +each case respectively: - - Upstream calls buffer-alloc: In passthrough and (some) in-place we proxy - this call onto the downstream element. If the caps are changed, we mark - a flag that we will require a new pad-alloc for the output of the next - output buffer. +- Upstream calls buffer-alloc: In passthrough and (some) in-place we proxy + this call onto the downstream element. If the caps are changed, we mark + a flag that we will require a new pad-alloc for the output of the next + output buffer. - - upstream peer does not call buffer-alloc: We always perform a pad-alloc - when processing buffers. We can further optimize by only looking at the - returned caps instead of doing a full, needless buffer copy. +- upstream peer does not call buffer-alloc: We always perform a pad-alloc + when processing buffers. We can further optimize by only looking at the + returned caps instead of doing a full, needless buffer copy. Use cases ---------- +~~~~~~~~~ videotestsrc ! ximagesink @@ -478,7 +478,8 @@ Use cases - resizing the videosink makes videoscale perform the scaling. Problematic ------------ +~~~~~~~~~~~ filesrc location=~/media/moveyourfeet.mov ! decodebin ! ffmpegcolorspace ! videoscale ! ffmpegcolorspace ! ximagesink -v + diff --git a/docs/design/part-events.txt b/docs/design/part-events.txt index c029af308..c2fd6506f 100644 --- a/docs/design/part-events.txt +++ b/docs/design/part-events.txt @@ -27,7 +27,7 @@ Different types of events exist to implement various functionalities. in DVD. FLUSH_START/STOP ----------------- +~~~~~~~~~~~~~~~~ A flush event is sent both downstream and upstream to clear any pending data from the pipeline. This might be needed to make the graph more responsive @@ -65,7 +65,7 @@ base_time (see part-clocks.txt and part-synchronisation.txt). EOS ---- +~~~ The EOS event can only be sent on a sinkpad. It is typically emited by the source element when it has finished sending data. This event is mainly sent @@ -107,7 +107,7 @@ A FLUSH_STOP event on an element flushes the EOS state and all pending EOS messa NEWSEGMENT -------------- +~~~~~~~~~~ A newsegment event is sent downstream by an element to indicate that the following group of buffers start and end at the specified positions. The newsegment event @@ -146,7 +146,7 @@ make the buffer timestamps increasing (part-segments.txt). TAG ---- +~~~ The tag event is sent downstream when an element has discovered metadata tags in a media file. Encoders can use this event to adjust their tagging @@ -154,15 +154,17 @@ system. A tag is serialized with buffers. BUFFERSIZE ----------- +~~~~~~~~~~ + +NOTE: This event is not yet implemented. An element can suggest a buffersize for downstream elements. This is typically done by elements that produce data on multiple source pads -such as demuxers. This event is currently not yet defined nor used. +such as demuxers. QOS ---- +~~~ A QOS, or quality of service message, is generated in an element to report to the upstream elements about the current quality of real-time performance @@ -171,7 +173,7 @@ of framedrops they have. (see part-qos.txt) SEEK ----- +~~~~ A seek event is issued by the application to configure the playback range of a stream. It is called form the application thread and travels upstream. @@ -227,7 +229,7 @@ part-seeking.txt. NAVIGATION ----------- +~~~~~~~~~~~ A navigation event is generated by a sink element to signal the elements of a navigation event such as a mouse movement or button click. @@ -235,7 +237,7 @@ Navigation events travel upstream. LATENCY -------- +~~~~~~~ A latency event is used to configure a certain latency in the pipeline. It contains a single GstClockTime with the required latency. The latency value is @@ -246,9 +248,9 @@ timestamps of the buffer in order to delay their presentation. DRAIN ------ +~~~~~ -This event is not yet implemented. +NOTE: This event is not yet implemented. Drain event indicates that upstream is about to perform a real-time event, such as pausing to present an interactive menu or such, and needs to wait for all diff --git a/docs/design/part-framestep.txt b/docs/design/part-framestep.txt index b3080fc2b..3506fd13b 100644 --- a/docs/design/part-framestep.txt +++ b/docs/design/part-framestep.txt @@ -26,7 +26,7 @@ segments of data in PLAYING. Use Cases ---------- +~~~~~~~~~ * frame stepping in video only pipeline in PAUSED @@ -92,10 +92,10 @@ Use Cases events ------- +~~~~~~ - A new GST_EVENT_STEP event is introduced to start the step operation. - The step event is created with the following fields in the structure: +A new GST_EVENT_STEP event is introduced to start the step operation. +The step event is created with the following fields in the structure: "format", GST_TYPE_FORMAT The format of the step units @@ -146,29 +146,29 @@ events flag is passed to the corresponding GST_MESSAGE_STEP_DONE. - The application will create a STEP event to start or stop the stepping - operation. Both stepping in PAUSED and PLAYING can be performed by means of - the flush flag. +The application will create a STEP event to start or stop the stepping +operation. Both stepping in PAUSED and PLAYING can be performed by means of +the flush flag. - The event is usually sent to the pipeline, which will typically distribute the - event to all of its sinks. For some use cases, like frame stepping on video - frames only, the event should only be sent to the video sink and upon reception - of the STEP_DONE message, one can step the other sinks to align the streams - again. +The event is usually sent to the pipeline, which will typically distribute the +event to all of its sinks. For some use cases, like frame stepping on video +frames only, the event should only be sent to the video sink and upon reception +of the STEP_DONE message, one can step the other sinks to align the streams +again. - For large stepping amounts, there needs to be enough queueing in front of all - the sinks. If large steps need to be performed, they can be split up into - smaller step operations using the "intermediate" flag on the step. +For large stepping amounts, there needs to be enough queueing in front of all +the sinks. If large steps need to be performed, they can be split up into +smaller step operations using the "intermediate" flag on the step. - Since the step event does not update the base_time of any of the elements, the - sinks should keep track of the amount of stepped data in order to remain - synchronized against the clock. +Since the step event does not update the base_time of any of the elements, the +sinks should keep track of the amount of stepped data in order to remain +synchronized against the clock. messages --------- +~~~~~~~~ - A GST_MESSAGE_STEP_START is created. It contains the following fields. +A GST_MESSAGE_STEP_START is created. It contains the following fields. "active" If the step was queued or activated. @@ -188,7 +188,7 @@ messages "intermediate", G_TYPE_BOOLEAN If this is an intermediate step operation that queued/activated. - The STEP_START message is emited 2 times: +The STEP_START message is emited 2 times: * first when an element received the STEP event and queued it. The "active" field will be FALSE in this case. @@ -197,12 +197,12 @@ messages field is TRUE in this case. After this message is emited, the application can queue a new step operation. - The purpose of this message is to find out how many elements participate in the - step operation and to queue new step operations at the earliest possible - moment. +The purpose of this message is to find out how many elements participate in the +step operation and to queue new step operations at the earliest possible +moment. - A new GST_MESSAGE_STEP_DONE message is created. It contains the following - fields: +A new GST_MESSAGE_STEP_DONE message is created. It contains the following +fields: "format", GST_TYPE_FORMAT The format of the step units that completed. @@ -225,25 +225,24 @@ messages "eos", G_TYPE_BOOLEAN The step ended because of EOS. - The message is emited by the element that performs the step operation. The - purpose is to return the duration in GST_FORMAT_TIME of the stepped media. This - especially interesting to align other stream in case of stepping frames on the - video sink element. +The message is emited by the element that performs the step operation. The +purpose is to return the duration in GST_FORMAT_TIME of the stepped media. This +especially interesting to align other stream in case of stepping frames on the +video sink element. Direction switch ----------------- +~~~~~~~~~~~~~~~~ - When quickly switching between a forwards and a backwards step of, for example, - one video frame, we need either: +When quickly switching between a forwards and a backwards step of, for example, +one video frame, we need either: a) issue a new seek to change the direction from the current position. b) cache a certain number of stepped frames and walk the cache. - option a) might be very slow. - For option b) we would ideally like to offload this caching functionality to a - separate element, which means that we need to forward the STEP event upstream. - It's unclear how this could work in a generic way. What is a demuxer supposed - to do when it received a step event? a flushing seek to what stream position? - +option a) might be very slow. +For option b) we would ideally like to offload this caching functionality to a +separate element, which means that we need to forward the STEP event upstream. +It's unclear how this could work in a generic way. What is a demuxer supposed +to do when it received a step event? a flushing seek to what stream position? diff --git a/docs/design/part-gstbin.txt b/docs/design/part-gstbin.txt index 404093f88..6cd911f24 100644 --- a/docs/design/part-gstbin.txt +++ b/docs/design/part-gstbin.txt @@ -7,7 +7,7 @@ GstElement. A GstBin provides a GstBus for the children and collates messages from them. Add/removing elements ---------------------- +~~~~~~~~~~~~~~~~~~~~~ The basic functionality of a bin is to add and remove GstElements to/from it. gst_bin_add() and gst_bin_remove() perform these operations respectively. @@ -17,7 +17,7 @@ relations.txt). Retrieving elements -------------------- +~~~~~~~~~~~~~~~~~~~ GstBin provides a number of functions to retrieve one or more children from itself. A few examples of the provided functions: @@ -28,7 +28,7 @@ gst_bin_iterate_elements() returns an iterator to all the children. element management ------------------- +~~~~~~~~~~~~~~~~~~ The most important function of the GstBin is to distribute all GstElement operations on itself to all of its children. This includes: @@ -41,7 +41,7 @@ The state change distribution is the most complex and is explained in part-states.txt. GstBus ------- +~~~~~~ The GstBin creates a GstBus for its children and distributes it when child elements are added to the bin. The bin attaches a sync handler to receive @@ -63,7 +63,7 @@ When a bin goes to READY it will clear all cached messages. EOS ---- +~~~ The sink elements will post an EOS message on the bus when they reach EOS. The EOS message is only posted to the bus when the sink element is in PLAYING. @@ -77,7 +77,7 @@ to PLAYING again. SEGMENT_START/DONE ------------------- +~~~~~~~~~~~~~~~~~~ A bin collects SEGMENT_START messages but does not post them to the application. It counts the number of SEGMENT_START messages and posts a SEGMENT_STOP message @@ -87,7 +87,7 @@ The cached SEGMENT_START/STOP messages are cleared when going to READY. DURATION --------- +~~~~~~~~ When a DURATION query is performed on a bin, it will forward the query to all its sink elements. The bin will calculate the total duration as the MAX of all @@ -103,7 +103,7 @@ posted to the application, which can then fetch the updated DURATION. Subclassing ------------ +~~~~~~~~~~~ Subclasses of GstBin are free to implement their own add/remove implementations. It is a good idea to update the GList of children so that the _iterate() functions diff --git a/docs/design/part-gstbus.txt b/docs/design/part-gstbus.txt index e246605c8..8e8064e8d 100644 --- a/docs/design/part-gstbus.txt +++ b/docs/design/part-gstbus.txt @@ -38,3 +38,4 @@ to flushing, ie. the bus will drop all existing and new messages on the bus, This is necessary because bus messages hold references to the bin/pipeline or its elements, so there are circular references that need to be broken if one ever wants to be able to destroy a bin or pipeline properly. + diff --git a/docs/design/part-gstelement.txt b/docs/design/part-gstelement.txt index 7c104d9e8..ad3951a40 100644 --- a/docs/design/part-gstelement.txt +++ b/docs/design/part-gstelement.txt @@ -1,14 +1,5 @@ - gstelement - name - pads - state - loopfunction - threadstate - manager - ghostpads - GstElement -========== +---------- The Element is the most important object in the entire GStreamer system, as it defines the structure of the pipeline. Elements include sources, filters, @@ -26,7 +17,7 @@ Element. This allows deeply nested pipelines, and the possibility of "black-box" meta-elements. Name ----- +~~~~ All elements are named, and while they should ideally be unique in any given pipeline, they do not have to be. The only guaranteed unique name for an @@ -39,7 +30,7 @@ or name of an element is changed. Pads ----- +~~~~ GstPads are the property of a given GstElement. They provide the connection capability, with allowing arbitrary structure in the graph. For any Element @@ -67,10 +58,12 @@ pad = gst_element_get_pad(element,"padname"): Ghost Pads ----------- +~~~~~~~~~~ +More info in part-gstghostpad.txt. State ------ +~~~~~ An element has a state. More info in part-states.txt. + diff --git a/docs/design/part-gstghostpad.txt b/docs/design/part-gstghostpad.txt index d5c987c65..919677405 100644 --- a/docs/design/part-gstghostpad.txt +++ b/docs/design/part-gstghostpad.txt @@ -1,11 +1,11 @@ Ghostpads --------- - GhostPads are used to build complex compound elements out of - existing elements. They are used to expose internal element pads - on the complex element. +GhostPads are used to build complex compound elements out of +existing elements. They are used to expose internal element pads +on the complex element. - Some design requirements +Some design requirements - Must look like a real GstPad on both sides. - target of Ghostpad must be changeable @@ -346,23 +346,22 @@ Ghostpads Activation -========== +~~~~~~~~~~ Sometimes ghost pads should proxy activation functions. This thingie attempts to explain how it should work in the different cases. - -+---+ +----+ +----+ +----+ -| A +-----+ B | | C |-------+ D | -+---+ +---=+ +=---+ +----+ - +--=-----------------------------=-+ - | +=---+ +----+ +----+ +---=+ | - | | a +---+ b ==== c +--+ d | | - | +----+ +----+ +----+ +----+ | - | | - +----------------------------------+ - state change goes from right to left - <----------------------------------------------------------- + +---+ +----+ +----+ +----+ + | A +-----+ B | | C |-------+ D | + +---+ +---=+ +=---+ +----+ + +--=-----------------------------=-+ + | +=---+ +----+ +----+ +---=+ | + | | a +---+ b ==== c +--+ d | | + | +----+ +----+ +----+ +----+ | + | | + +----------------------------------+ + state change goes from right to left + <----------------------------------------------------------- All of the labeled boxes are pads. The dashes (---) show pad links, and the double-lines (===) are internal connections. The box around a, b, c, @@ -375,10 +374,10 @@ Now, in the state change from READY to PAUSED, assuming the pipeline does not have a live source, all of the pads will end up activated at the end. There are 4 possible activation modes: -1) AD and ab in PUSH, cd and CD in PUSH -2) AD and ab in PUSH, cd and CD in PULL -3) AD and ab in PULL, cd and CD in PUSH -4) AD and ab in PULL, cd and CD in PULL + 1) AD and ab in PUSH, cd and CD in PUSH + 2) AD and ab in PUSH, cd and CD in PULL + 3) AD and ab in PULL, cd and CD in PUSH + 4) AD and ab in PULL, cd and CD in PULL When activating (1), the state change algorithm will first visit the parent of D and activate D in push mode. Then it visits the bin. The bin diff --git a/docs/design/part-gstobject.txt b/docs/design/part-gstobject.txt index 3ebd83145..b84607252 100644 --- a/docs/design/part-gstobject.txt +++ b/docs/design/part-gstobject.txt @@ -1,10 +1,10 @@ GstObject -========= +--------- The base class for the entire GStreamer hierarchy is the GstObject. Parentage ---------- +~~~~~~~~~ A pointer is available to store the current parent of the object. This is one of the two fundamental requirements for a hierarchical system such as GStreamer @@ -20,7 +20,8 @@ allows for new additions later. Naming ------- +~~~~~~ + - names of objects cannot be changed when they are parented - names of objects should be unique across parent - set_name() can fail because of this @@ -36,7 +37,7 @@ Naming Locking -------- +~~~~~~~ The GstObject contains the necessary primitives to lock the object in a thread-safe manner. This will be used to provide general thread-safety as @@ -62,7 +63,7 @@ GstObject. Locking order -------------- +~~~~~~~~~~~~~ In parent-child situations the lock of the parent must always be taken first before taking the lock of the child. It is NOT allowed to hold the child @@ -76,7 +77,7 @@ parent-child relation (eg. pads), an explictic locking order has to be defined. Path Generation ---------------- +~~~~~~~~~~~~~~~ Due to the base nature of the GstObject, it becomes the only reasonable place to put this particular function (_get_path_string). It will generate a string @@ -84,7 +85,7 @@ describing the parent hierarchy of a given GstObject. Flags ------ +~~~~~ Each object in the GStreamer object hierarchy can have flags associated with it, which are used to describe a state or a feature of the object. @@ -92,7 +93,7 @@ GstObject has flags to mark its lifecycle: FLOATING and DISPOSING. Class signals -------------- +~~~~~~~~~~~~~ It is possible to know when a new object is loaded by connecting to the GstObjectClass signal. This feature is not very much used and might be removed diff --git a/docs/design/part-gstpipeline.txt b/docs/design/part-gstpipeline.txt index 04903b2a1..694a46b32 100644 --- a/docs/design/part-gstpipeline.txt +++ b/docs/design/part-gstpipeline.txt @@ -13,7 +13,7 @@ The pipeline will calculate a global latency for the elements in the pipeline. (See also part-latency.txt). State changes -------------- +~~~~~~~~~~~~~ In addition to the normal state change procedure of its parent class GstBin, the pipeline performs the following actions during a state change: @@ -43,7 +43,7 @@ The running_time is set to 0 after a flushing seek. Clock selection ---------------- +~~~~~~~~~~~~~~~ Since all of the children of a GstPipeline must use the same clock, the pipeline must select a clock. This clock selection happens when the pipeline @@ -80,7 +80,7 @@ matic clock selection algorithm described above. GstBus ------- +~~~~~~ A GstPipeline provides a GstBus to the application. The bus can be retrieved with gst_pipeline_get_bus() and can then be used to retrieve messages posted by diff --git a/docs/design/part-latency.txt b/docs/design/part-latency.txt index ce6f97108..637d58405 100644 --- a/docs/design/part-latency.txt +++ b/docs/design/part-latency.txt @@ -35,7 +35,7 @@ application needs (required minimal latency). Pipelines without latency compensation --------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We show some examples to demonstrate the problem of latency in typical capture pipelines. @@ -196,7 +196,7 @@ capture pipelines. State Changes revised ---------------------- +~~~~~~~~~~~~~~~~~~~~~ As a first step in a generic solution we propose to modify the state changes so that no sink is set to PLAYING before it is prerolled. @@ -229,7 +229,7 @@ implications: Latency compensation --------------------- +~~~~~~~~~~~~~~~~~~~~ As an extension to the revised state changes we can perform latency calculation and compensation before we proceed to the PLAYING state. @@ -277,7 +277,7 @@ the same for all sinks, all sinks will render data relatively synchronised. Flushing a playing pipeline ---------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~ Using the new state change mechanism we can implement resynchronisation after an uncontrolled FLUSH in (part of) a pipeline. Indeed, when a flush is performed on @@ -306,7 +306,7 @@ to perform additional latency calculations and adjustments before doing this. Dynamically adjusting latency ------------------------------ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ An element that want to change the latency in the pipeline can do this by posting a LATENCY message on the bus. This message instructs the pipeline to: diff --git a/docs/design/part-live-source.txt b/docs/design/part-live-source.txt index 0e472ffa0..47055a5a9 100644 --- a/docs/design/part-live-source.txt +++ b/docs/design/part-live-source.txt @@ -28,14 +28,14 @@ a blocking wait. Scheduling ----------- +~~~~~~~~~~ Live sources will not produce data in the paused state. They block in the getrange function or in the loop function until they go to PLAYING. Latency -------- +~~~~~~~ The live source timestamps its data with the time of the clock at the time the data was captured. Normally it will take some time to capture @@ -49,7 +49,7 @@ See part-latency.txt. Timestamps ----------- +~~~~~~~~~~ Live sources always timestamp their buffers with the running_time of the pipeline. This is needed to be able to match the timestamps of different live diff --git a/docs/design/part-messages.txt b/docs/design/part-messages.txt index cd53e106e..a0eef639f 100644 --- a/docs/design/part-messages.txt +++ b/docs/design/part-messages.txt @@ -14,122 +14,123 @@ application using the GstBus (See also part-gstbus.txt and part-gstpipeline.txt) Message types -------------- +~~~~~~~~~~~~~ - GST_MESSAGE_EOS: +GST_MESSAGE_EOS: - Posted by sink elements. This message is posted to the application when - all the sinks in a pipeline posted an EOS message. When performing a seek, - the EOS state of the pipeline and sinks is undone. - - GST_MESSAGE_ERROR: - - An element in the pipeline got into an error state. The message carries - a GError and a debug string describing the error. This usually means that - part of the pipeline is not streaming anymore. - - GST_MESSAGE_WARNING: - - An element in the pipeline encountered a condition that made it produce a - warning. This could be a recoverable decoding error or some other non fatal - event. The pipeline continues streaming after a warning. - - GST_MESSAGE_INFO: + Posted by sink elements. This message is posted to the application when + all the sinks in a pipeline posted an EOS message. When performing a seek, + the EOS state of the pipeline and sinks is undone. - An element produced an informational message. - - GST_MESSAGE_TAG: +GST_MESSAGE_ERROR: - An element decoded metadata about the stream. The message carries a GstTagList - with the tag information. - - GST_MESSAGE_BUFFERING: - - An element is buffering data and that could potentially take some time. This - message is typically emited by elements that perform some sort of network - buffering. While the pipeline is buffering it should remain in the PAUSED - state. When the buffering is finished, it can resume PLAYING. - - GST_MESSAGE_STATE_CHANGED: + An element in the pipeline got into an error state. The message carries + a GError and a debug string describing the error. This usually means that + part of the pipeline is not streaming anymore. - An element changed state in the pipeline. The message carries the old, new - and pending state of the element. - - GST_MESSAGE_STATE_DIRTY: - - An internal message used to instruct a pipeline hierarchy that a state - recalculation must be performed because of an ASYNC state change completed. - This message is not used anymore. +GST_MESSAGE_WARNING: + + An element in the pipeline encountered a condition that made it produce a + warning. This could be a recoverable decoding error or some other non fatal + event. The pipeline continues streaming after a warning. + +GST_MESSAGE_INFO: - GST_MESSAGE_STEP_DONE: + An element produced an informational message. - An element stepping frames has finished. This is currently not used. +GST_MESSAGE_TAG: + + An element decoded metadata about the stream. The message carries a GstTagList + with the tag information. + +GST_MESSAGE_BUFFERING: + + An element is buffering data and that could potentially take some time. This + message is typically emited by elements that perform some sort of network + buffering. While the pipeline is buffering it should remain in the PAUSED + state. When the buffering is finished, it can resume PLAYING. + +GST_MESSAGE_STATE_CHANGED: + + An element changed state in the pipeline. The message carries the old, new + and pending state of the element. + +GST_MESSAGE_STATE_DIRTY: - GST_MESSAGE_CLOCK_PROVIDE: + An internal message used to instruct a pipeline hierarchy that a state + recalculation must be performed because of an ASYNC state change completed. + This message is not used anymore. - An element notifies it capability of providing a clock for the pipeline. +GST_MESSAGE_STEP_DONE: - GST_MESSAGE_CLOCK_LOST: + An element stepping frames has finished. This is currently not used. - The current clock as selected by the pipeline became unusable. The pipeline - will select a new clock on the next PLAYING state change. +GST_MESSAGE_CLOCK_PROVIDE: - GST_MESSAGE_NEW_CLOCK: + An element notifies it capability of providing a clock for the pipeline. - A new clock was selected for the pipeline. +GST_MESSAGE_CLOCK_LOST: - GST_MESSAGE_STRUCTURE_CHANGE: + The current clock as selected by the pipeline became unusable. The pipeline + will select a new clock on the next PLAYING state change. - The pipeline changed of structure, This means elements were added or removed or - pads were linked or unlinked. This messages is not yet used. +GST_MESSAGE_NEW_CLOCK: - GST_MESSAGE_STREAM_STATUS: + A new clock was selected for the pipeline. + +GST_MESSAGE_STRUCTURE_CHANGE: + + The pipeline changed of structure, This means elements were added or removed or + pads were linked or unlinked. This messages is not yet used. + +GST_MESSAGE_STREAM_STATUS: + + Posted by an element when it start/stop/pauses a streaming task. It + contains information about the reason why the stream state changed along + with the thread id. The application can use this information to detect + failures in streaming threads. It can also be used to adjust streaming + thread priorities by the application. + +GST_MESSAGE_APPLICATION: - Posted by an element when it start/stop/pauses a streaming task. It - contains information about the reason why the stream state changed along - with the thread id. The application can use this information to detect - failures in streaming threads. It can also be used to adjust streaming - thread priorities by the application. - - GST_MESSAGE_APPLICATION: + The application posted a message. This message must be used when the + application posts a message on the bus. - The application posted a message. This message must be used when the - application posts a message on the bus. +GST_MESSAGE_ELEMENT: - GST_MESSAGE_ELEMENT: + Element-specific message, see the specific element's documentation - Element-specific message, see the specific element's documentation +GST_MESSAGE_SEGMENT_START: - GST_MESSAGE_SEGMENT_START: + An element started playback of a new segment. This message is not forwarded + the the application but is used internally to schedule SEGMENT_DONE messages. - An element started playback of a new segment. This message is not forwarded - the the application but is used internally to schedule SEGMENT_DONE messages. +GST_MESSAGE_SEGMENT_DONE: - GST_MESSAGE_SEGMENT_DONE: + An element or bin completed playback of a segment. This message is only posted + on the bus if a SEGMENT seek is performed on a pipeline. - An element or bin completed playback of a segment. This message is only posted - on the bus if a SEGMENT seek is performed on a pipeline. +GST_MESSAGE_DURATION: - GST_MESSAGE_DURATION: + An element posts this message when it has detected or updated the stream duration. - An element posts this message when it has detected or updated the stream duration. +GST_MESSAGE_ASYNC_START: - GST_MESSAGE_ASYNC_START: + Posted by sinks when they start an asynchronous state change. - Posted by sinks when they start an asynchronous state change. +GST_MESSAGE_ASYNC_DONE: - GST_MESSAGE_ASYNC_DONE: + Posted by sinks when they receive the first data buffer and complete the + asynchronous state change. - Posted by sinks when they receive the first data buffer and complete the - asynchronous state change. +GST_MESSAGE_LATENCY: - GST_MESSAGE_LATENCY: + Posted by elements when the latency in a pipeline changed and a new global + latency should be calculated by the pipeline or application. - Posted by elements when the latency in a pipeline changed and a new global - latency should be calculated by the pipeline or application. +GST_MESSAGE_REQUEST_STATE: - GST_MESSAGE_REQUEST_STATE: + Posted by elements when they want to change the state of the pipeline they + are in. A typical use case would be an audio sink that requests the pipeline + to pause in order to play a higher priority stream. - Posted by elements when they want to change the state of the pipeline they - are in. A typical use case would be an audio sink that requests the pipeline - to pause in order to play a higher priority stream. diff --git a/docs/design/part-missing-plugins.txt b/docs/design/part-missing-plugins.txt index e235fb617..eae1de42d 100644 --- a/docs/design/part-missing-plugins.txt +++ b/docs/design/part-missing-plugins.txt @@ -1,5 +1,5 @@ What to do when a plugin is missing -=================================== +----------------------------------- The mechanism and API described in this document requires GStreamer core and gst-plugins-base versions >= 0.10.12. Further information on some aspects of diff --git a/docs/design/part-negotiation.txt b/docs/design/part-negotiation.txt index 20f7feee8..2d8d8377c 100644 --- a/docs/design/part-negotiation.txt +++ b/docs/design/part-negotiation.txt @@ -13,7 +13,7 @@ themselves to different mechanisms to achieve this goal. As it is more common we describe push mode negotiation first. Push-mode negotiation ---------------------- +~~~~~~~~~~~~~~~~~~~~~ Push-mode negotiation happens when elements want to push buffers and need to decide on the format. This is called downstream negotiation @@ -168,10 +168,10 @@ videotestsrc ! queue ! xvimagesink Pull-mode negotiation ---------------------- +~~~~~~~~~~~~~~~~~~~~~ Rationale -......... +^^^^^^^^^ A pipeline in pull mode has different negotiation needs than one activated in push mode. Push mode is optimized for two use cases: @@ -237,7 +237,7 @@ but before the sink has spawned the pulling thread. Mechanism -......... +^^^^^^^^^ The sink determines that the upstream elements support pull based scheduling by calling gst_pad_check_pull_range(). @@ -276,3 +276,4 @@ GST_FLOW_NOT_NEGOTIATED. Because of the low-latency requirements, changing caps in an activate pull-mode pipeline is not supported, as it might require e.g. the sound card to reconfigure its hardware buffers, and start capsnego again. + diff --git a/docs/design/part-overview.txt b/docs/design/part-overview.txt index 36c2269f2..d45e8b07a 100644 --- a/docs/design/part-overview.txt +++ b/docs/design/part-overview.txt @@ -1,32 +1,32 @@ Overview -------- - This part gives an overview of the design of GStreamer with references to - the more detailed explanations of the different topics. +This part gives an overview of the design of GStreamer with references to +the more detailed explanations of the different topics. - This document is intented for people that want to have a global overview of - the inner workings of GStreamer. +This document is intented for people that want to have a global overview of +the inner workings of GStreamer. Introduction ------------- +~~~~~~~~~~~~ - GStreamer is a set of libraries and plugins that can be used to implement various - multimedia applications ranging from desktop players, audio/video recorders, - multimedia servers, transcoders, etc. +GStreamer is a set of libraries and plugins that can be used to implement various +multimedia applications ranging from desktop players, audio/video recorders, +multimedia servers, transcoders, etc. - Applications are built by constructing a pipeline composed of elements. An element - is an object that performs some action on a multimedia stream such as: +Applications are built by constructing a pipeline composed of elements. An element +is an object that performs some action on a multimedia stream such as: - - read a file - - decode or encode between formats - - capture from a hardware device - - render to a hardware device - - mix or multiplex multiple streams +- read a file +- decode or encode between formats +- capture from a hardware device +- render to a hardware device +- mix or multiplex multiple streams - Elements have input and output pads called sink and source pads in GStreamer. An - application links elements together on pads to construct a pipeline. Below is - an example of an ogg/vorbis playback pipeline. +Elements have input and output pads called sink and source pads in GStreamer. An +application links elements together on pads to construct a pipeline. Below is +an example of an ogg/vorbis playback pipeline. +-----------------------------------------------------------+ | ----------> downstream -------------------> | @@ -40,54 +40,54 @@ Introduction | <---------< upstream <-------------------< | +-----------------------------------------------------------+ - The filesrc element reads data from a file on disk. The oggdemux element parses - the data and sends the compressed audio data to the vorbisdec element. The - vorbisdec element decodes the compressed data and sends it to the alsasink - element. The alsasink element sends the samples to the audio card for playback. +The filesrc element reads data from a file on disk. The oggdemux element parses +the data and sends the compressed audio data to the vorbisdec element. The +vorbisdec element decodes the compressed data and sends it to the alsasink +element. The alsasink element sends the samples to the audio card for playback. - Downstream and upstream are the terms used to describe the direction in the - Pipeline. From source to sink is called "downstream" and "upstream" is - from sink to source. Dataflow always happens downstream. +Downstream and upstream are the terms used to describe the direction in the +Pipeline. From source to sink is called "downstream" and "upstream" is +from sink to source. Dataflow always happens downstream. - The task of the application is to construct a pipeline as above using existing - elements. This is further explained in the pipeline building topic. +The task of the application is to construct a pipeline as above using existing +elements. This is further explained in the pipeline building topic. - The application does not have to manage any of the complexities of the - actual dataflow/decoding/conversions/synchronsiation etc. but only calls high - level functions on the pipeline object such as PLAY/PAUSE/STOP. +The application does not have to manage any of the complexities of the +actual dataflow/decoding/conversions/synchronsiation etc. but only calls high +level functions on the pipeline object such as PLAY/PAUSE/STOP. - The application also receives messages and notifications from the pipeline such - as metadata, warning, error and EOS messages. - - If the application needs more control over the graph it is possible to directly - access the elements and pads in the pipeline. +The application also receives messages and notifications from the pipeline such +as metadata, warning, error and EOS messages. + +If the application needs more control over the graph it is possible to directly +access the elements and pads in the pipeline. Design overview ---------------- +~~~~~~~~~~~~~~~ - GStreamer design goals include: +GStreamer design goals include: - - Process large amounts of data quickly - - Allow fully multithreaded processing - - Ability to deal with multiple formats - - Synchronize different dataflows - - Ability to deal with multiple devices +- Process large amounts of data quickly +- Allow fully multithreaded processing +- Ability to deal with multiple formats +- Synchronize different dataflows +- Ability to deal with multiple devices + +The capabilities presented to the application depends on the number of elements +installed on the system and their functionality. + +The GStreamer core is designed to be media agnostic but provides many features +to elements to describe media formats. - The capabilities presented to the application depends on the number of elements - installed on the system and their functionality. - - The GStreamer core is designed to be media agnostic but provides many features - to elements to describe media formats. - Elements --------- +~~~~~~~~ - The smallest building blocks in a pipeline are elements. An element provides a - number of pads which can be source or sinkpads. Sourcepads provide data and - sinkpads consume data. Below is an example of an ogg demuxer element that has - one pad that takes (sinks) data and two source pads that produce data. +The smallest building blocks in a pipeline are elements. An element provides a +number of pads which can be source or sinkpads. Sourcepads provide data and +sinkpads consume data. Below is an example of an ogg demuxer element that has +one pad that takes (sinks) data and two source pads that produce data. +-----------+ | oggdemux | @@ -95,57 +95,57 @@ Elements sink src1 +-----------+ - An element can be in four different states: NULL, READY, PAUSED, PLAYING. In the - NULL and READY state, the element is not processing any data. In the PLAYING state - it is processing data. The intermediate PAUSED state is used to preroll data in - the pipeline. A state change can be performed with gst_element_set_state(). - - An element always goes through all the intermediate state changes. This means that - when en element is in the READY state and is put to PLAYING, it will first go - through the intermediate PAUSED state. - - An element state change to PAUSED will activate the pads of the element. First the - source pads are activated, then the sinkpads. When the pads are activated, the - pad activate function is called. Some pads will start a thread (GstTask) or some - other mechanism to start producing or consuming data. - - The PAUSED state is special as it is used to preroll data in the pipeline. The purpose - is to fill all connected elements in the pipeline with data so that the subsequent - PLAYING state change happens very quickly. Some elements will therefore not complete - the state change to PAUSED before they have received enough data. Sink elements are - required to only complete the state change to PAUSED after receiving the first data. - - Normally the state changes of elements are coordinated by the pipeline as explained - in [part-states.txt]. - - Different categories of elements exist: - - - source elements, these are elements that do not consume data but only provide data - for the pipeline. - - sink elements, these are elements that do not produce data but renders data to - an output device. - - transform elements, these elements transform an input stream in a certain format - into a stream of another format. Encoder/decoder/converters are examples. - - demuxer elements, these elements parse a stream and produce several output streams. - - mixer/muxer elements, combine several input streams into one output stream. - - Other categories of elements can be constructed (see part-klass.txt). +An element can be in four different states: NULL, READY, PAUSED, PLAYING. In the +NULL and READY state, the element is not processing any data. In the PLAYING state +it is processing data. The intermediate PAUSED state is used to preroll data in +the pipeline. A state change can be performed with gst_element_set_state(). + +An element always goes through all the intermediate state changes. This means that +when en element is in the READY state and is put to PLAYING, it will first go +through the intermediate PAUSED state. + +An element state change to PAUSED will activate the pads of the element. First the +source pads are activated, then the sinkpads. When the pads are activated, the +pad activate function is called. Some pads will start a thread (GstTask) or some +other mechanism to start producing or consuming data. + +The PAUSED state is special as it is used to preroll data in the pipeline. The purpose +is to fill all connected elements in the pipeline with data so that the subsequent +PLAYING state change happens very quickly. Some elements will therefore not complete +the state change to PAUSED before they have received enough data. Sink elements are +required to only complete the state change to PAUSED after receiving the first data. + +Normally the state changes of elements are coordinated by the pipeline as explained +in [part-states.txt]. + +Different categories of elements exist: + +- source elements, these are elements that do not consume data but only provide data + for the pipeline. +- sink elements, these are elements that do not produce data but renders data to + an output device. +- transform elements, these elements transform an input stream in a certain format + into a stream of another format. Encoder/decoder/converters are examples. +- demuxer elements, these elements parse a stream and produce several output streams. +- mixer/muxer elements, combine several input streams into one output stream. + +Other categories of elements can be constructed (see part-klass.txt). Bins ----- +~~~~ - A bin is an element subclass and acts as a container for other elements so that multiple - elements can be combined into one element. +A bin is an element subclass and acts as a container for other elements so that multiple +elements can be combined into one element. - A bin coordinates its children's state changes as explained later. It also distributes - events and various other functionality to elements. +A bin coordinates its children's state changes as explained later. It also distributes +events and various other functionality to elements. - A bin can have its own source and sinkpads by ghostpadding one or more of its children's - pads to itself. +A bin can have its own source and sinkpads by ghostpadding one or more of its children's +pads to itself. - Below is a picture of a bin with two elements. The sinkpad of one element is ghostpadded - to the bin. +Below is a picture of a bin with two elements. The sinkpad of one element is ghostpadded +to the bin. +---------------------------+ | bin | @@ -157,131 +157,131 @@ Bins Pipeline --------- +~~~~~~~~ - A pipeline is a special bin subclass that provides the following features to its - children: +A pipeline is a special bin subclass that provides the following features to its +children: + + - Select and manage a global clock for all its children. + - Manage running_time based on the selected clock. Running_time is the elapsed + time the pipeline spent in the PLAYING state and is used for + synchronisation. + - Manage latency in the pipeline. + - Provide means for elements to comunicate with the application by the GstBus. + - Manage the global state of the elements such as Errors and end-of-stream. + +Normally the application creates one pipeline that will manage all the elements +in the application. - - Select and manage a global clock for all its children. - - Manage running_time based on the selected clock. Running_time is the elapsed - time the pipeline spent in the PLAYING state and is used for - synchronisation. - - Manage latency in the pipeline. - - Provide means for elements to comunicate with the application by the GstBus. - - Manage the global state of the elements such as Errors and end-of-stream. - - Normally the application creates one pipeline that will manage all the elements - in the application. - Dataflow and buffers --------------------- - - GStreamer supports two possible types of dataflow, the push and pull model. In the - push model, an upstream element sends data to a downstream element by calling a - method on a sinkpad. In the pull model, a downstream element requests data from - an upstream element by calling a method on a source pad. - - The most common dataflow is the push model. The pull model can be used in specific - circumstances by demuxer elements. The pull model can also be used by low latency - audio applications. - - The data passed between pads is encapsulated in Buffers. The buffer contains a - pointer to the actual data and also metadata describing the data. This metadata - includes: - - - timestamp of the data, this is the time instance at which the data was captured - or the time at which the data should be played back. - - offset of the data: a media specific offset, this could be samples for audio or - frames for video. - - the duration of the data in time. - - the media type of the data described with caps, these are key/value pairs that - describe the media type in a unique way. - - additional flags describing special properties of the data such as - discontinuities or delta units. - - When an element whishes to send a buffer to another element is does this using one - of the pads that is linked to a pad of the other element. In the push model, a - buffer is pushed to the peer pad with gst_pad_push(). In the pull model, a buffer - is pulled from the peer with the gst_pad_pull_range() function. - - Before an element pushes out a buffer, it should make sure that the peer element - can understand the buffer contents. It does this by querying the peer element - for the supported formats and by selecting a suitable common format. The selected - format is then attached to the buffer with gst_buffer_set_caps() before pushing - out the buffer. - - When an element pad receives a buffer, if has to check if it understands the media - type of the buffer before starting processing it. The GStreamer core does this - automatically and will call the gst_pad_set_caps() function of the element before - sending the buffer to the element. - - Both gst_pad_push() and gst_pad_pull_range() have a return value indicating whether - the operation succeeded. An error code means that no more data should be sent - to that pad. A source element that initiates the data flow in a thread typically - pauses the producing thread when this happens. - - A buffer can be created with gst_buffer_new() or by requesting a usable buffer - from the peer pad using gst_pad_alloc_buffer(). Using the second method, it is - possible for the peer element to suggest the element to produce data in another - format by attaching another media type caps to the buffer. +~~~~~~~~~~~~~~~~~~~~ - The process of selecting a media type and attaching it to the buffers is called - caps negotiation. +GStreamer supports two possible types of dataflow, the push and pull model. In the +push model, an upstream element sends data to a downstream element by calling a +method on a sinkpad. In the pull model, a downstream element requests data from +an upstream element by calling a method on a source pad. +The most common dataflow is the push model. The pull model can be used in specific +circumstances by demuxer elements. The pull model can also be used by low latency +audio applications. -Caps ----- +The data passed between pads is encapsulated in Buffers. The buffer contains a +pointer to the actual data and also metadata describing the data. This metadata +includes: - A media type (Caps) is described using a generic list of key/value pairs. The key is - a string and the value can be a single/list/range of int/float/string. + - timestamp of the data, this is the time instance at which the data was captured + or the time at which the data should be played back. + - offset of the data: a media specific offset, this could be samples for audio or + frames for video. + - the duration of the data in time. + - the media type of the data described with caps, these are key/value pairs that + describe the media type in a unique way. + - additional flags describing special properties of the data such as + discontinuities or delta units. + +When an element whishes to send a buffer to another element is does this using one +of the pads that is linked to a pad of the other element. In the push model, a +buffer is pushed to the peer pad with gst_pad_push(). In the pull model, a buffer +is pulled from the peer with the gst_pad_pull_range() function. + +Before an element pushes out a buffer, it should make sure that the peer element +can understand the buffer contents. It does this by querying the peer element +for the supported formats and by selecting a suitable common format. The selected +format is then attached to the buffer with gst_buffer_set_caps() before pushing +out the buffer. + +When an element pad receives a buffer, if has to check if it understands the media +type of the buffer before starting processing it. The GStreamer core does this +automatically and will call the gst_pad_set_caps() function of the element before +sending the buffer to the element. + +Both gst_pad_push() and gst_pad_pull_range() have a return value indicating whether +the operation succeeded. An error code means that no more data should be sent +to that pad. A source element that initiates the data flow in a thread typically +pauses the producing thread when this happens. + +A buffer can be created with gst_buffer_new() or by requesting a usable buffer +from the peer pad using gst_pad_alloc_buffer(). Using the second method, it is +possible for the peer element to suggest the element to produce data in another +format by attaching another media type caps to the buffer. + +The process of selecting a media type and attaching it to the buffers is called +caps negotiation. - Caps that have no ranges/list or other variable parts are said to be fixed and - can be used to put on a buffer. - Caps with variables in them are used to describe possible media types that can be - handled by a pad. +Caps +~~~~ + +A media type (Caps) is described using a generic list of key/value pairs. The key is +a string and the value can be a single/list/range of int/float/string. + +Caps that have no ranges/list or other variable parts are said to be fixed and +can be used to put on a buffer. + +Caps with variables in them are used to describe possible media types that can be +handled by a pad. Dataflow and events -------------------- +~~~~~~~~~~~~~~~~~~~ - Parallel to the dataflow is a flow of events. Unlike the buffers, events can pass - both upstream and downstream. Some events only travel upstream others only downstream. +Parallel to the dataflow is a flow of events. Unlike the buffers, events can pass +both upstream and downstream. Some events only travel upstream others only downstream. - the events are used to denote special conditions in the dataflow such as EOS or - to inform plugins of special events such as flushing or seeking. +the events are used to denote special conditions in the dataflow such as EOS or +to inform plugins of special events such as flushing or seeking. - Some events must be serialized with the buffer flow, others don't. Serialized - events are inserted between the buffers. Non serialized events jump in front - of any buffers current being processed. +Some events must be serialized with the buffer flow, others don't. Serialized +events are inserted between the buffers. Non serialized events jump in front +of any buffers current being processed. - An example of a serialized event is a TAG event that is inserted between buffers - to mark metadata for those buffers. +An example of a serialized event is a TAG event that is inserted between buffers +to mark metadata for those buffers. - An example of a non serialized event is the FLUSH event. +An example of a non serialized event is the FLUSH event. Pipeline construction ---------------------- +~~~~~~~~~~~~~~~~~~~~~ - The application starts by creating a Pipeline element using gst_pipeline_new (). - Elements are added to and removed from the pipeline with gst_bin_add() and - gst_bin_remove(). +The application starts by creating a Pipeline element using gst_pipeline_new (). +Elements are added to and removed from the pipeline with gst_bin_add() and +gst_bin_remove(). - After adding the elements, the pads of an element can be retrieved with - gst_element_get_pad(). Pads can then be linked together with gst_pad_link(). +After adding the elements, the pads of an element can be retrieved with +gst_element_get_pad(). Pads can then be linked together with gst_pad_link(). - Some elements create new pads when actual dataflow is happening in the pipeline. - With g_signal_connect() one can receive a notification when an element has created - a pad. These new pads can then be linked to other unlinked pads. +Some elements create new pads when actual dataflow is happening in the pipeline. +With g_signal_connect() one can receive a notification when an element has created +a pad. These new pads can then be linked to other unlinked pads. - Some elements cannot be linked together because they operate on different - incompatible data types. The possible datatypes a pad can provide or consume can - be retrieved with gst_pad_get_caps(). +Some elements cannot be linked together because they operate on different +incompatible data types. The possible datatypes a pad can provide or consume can +be retrieved with gst_pad_get_caps(). - Below is a simple mp3 playback pipeline that we constructed. We will use this - pipeline in further examples. +Below is a simple mp3 playback pipeline that we constructed. We will use this +pipeline in further examples. +-------------------------------------------+ | pipeline | @@ -293,195 +293,195 @@ Pipeline construction Pipeline clock --------------- +~~~~~~~~~~~~~~ - One of the important functions of the pipeline is to select a global clock - for all the elements in the pipeline. +One of the important functions of the pipeline is to select a global clock +for all the elements in the pipeline. - The purpose of the clock is to provide a stricly increasing value at the rate - of one GST_SECOND per second. Clock values are expressed in nanoseconds. - Elements use the clock time to synchronize the playback of data. +The purpose of the clock is to provide a stricly increasing value at the rate +of one GST_SECOND per second. Clock values are expressed in nanoseconds. +Elements use the clock time to synchronize the playback of data. - Before the pipeline is set to PLAYING, the pipeline asks each element if they can - provide a clock. The clock is selected in the following order: +Before the pipeline is set to PLAYING, the pipeline asks each element if they can +provide a clock. The clock is selected in the following order: - - If the application selected a clock, use that one. - - If a source element provides a clock, use that clock. - - Select a clock from any other element that provides a clock, start with the - sinks. - - If no element provides a clock a default system clock is used for the pipeline. +- If the application selected a clock, use that one. +- If a source element provides a clock, use that clock. +- Select a clock from any other element that provides a clock, start with the + sinks. +- If no element provides a clock a default system clock is used for the pipeline. - In a typical playback pipeline this algorithm will select the clock provided by - a sink element such as an audio sink. +In a typical playback pipeline this algorithm will select the clock provided by +a sink element such as an audio sink. - In capture pipelines, this will typically select the clock of the data producer, which - in most cases can not control the rate at which it produces data. +In capture pipelines, this will typically select the clock of the data producer, which +in most cases can not control the rate at which it produces data. Pipeline states ---------------- +~~~~~~~~~~~~~~~ - When all the pads are linked and signals have been connected, the pipeline can - be put in the PAUSED state to start dataflow. +When all the pads are linked and signals have been connected, the pipeline can +be put in the PAUSED state to start dataflow. - When a bin (and hence a pipeline) performs a state change, it will change the state - of all its children. The pipeline will change the state of its children from the - sink elements to the source elements, this to make sure that no upstream element - produces data to an element that is not yet ready to accept it. +When a bin (and hence a pipeline) performs a state change, it will change the state +of all its children. The pipeline will change the state of its children from the +sink elements to the source elements, this to make sure that no upstream element +produces data to an element that is not yet ready to accept it. - In the mp3 playback pipeline, the state of the elements is changed in the order - alsasink, mp3dec, filesrc. +In the mp3 playback pipeline, the state of the elements is changed in the order +alsasink, mp3dec, filesrc. - All intermediate states are traversed for each element resulting in the following - chain of state changes: +All intermediate states are traversed for each element resulting in the following +chain of state changes: - alsasink to READY: the audio device is probed - mp3dec to READY: nothing happens. - filesrc to READY: the file is probed - alsasink to PAUSED: the audio device is opened. alsasink is a sink and returns - ASYNC because it did not receive data yet. - mp3dec to PAUSED: the decoding library is initialized - filesrc to PAUSED: the file is opened and a thread is started to push data to - mp3dec + alsasink to READY: the audio device is probed + mp3dec to READY: nothing happens. + filesrc to READY: the file is probed + alsasink to PAUSED: the audio device is opened. alsasink is a sink and returns + ASYNC because it did not receive data yet. + mp3dec to PAUSED: the decoding library is initialized + filesrc to PAUSED: the file is opened and a thread is started to push data to + mp3dec - At this point data flows from filesrc to mp3dec and alsasink. Since mp3dec is PAUSED, - it accepts the data from filesrc on the sinkpad and starts decoding the compressed - data to raw audio samples. +At this point data flows from filesrc to mp3dec and alsasink. Since mp3dec is PAUSED, +it accepts the data from filesrc on the sinkpad and starts decoding the compressed +data to raw audio samples. - The mp3 decoder figures out the samplerate, the number of channels and other audio - properties of the raw audio samples, puts the decoded samples into a Buffer, - attaches the media type caps to the buffer and pushes this buffer to the next - element. +The mp3 decoder figures out the samplerate, the number of channels and other audio +properties of the raw audio samples, puts the decoded samples into a Buffer, +attaches the media type caps to the buffer and pushes this buffer to the next +element. - Alsasink then receives the buffer, inspects the caps and reconfigures itself to process - the buffer. Since it received the first buffer of samples, it completes the state change - to the PAUSED state. At this point the pipeline is prerolled and all elements have - samples. Alsasink is now also capable of providing a clock to the pipeline. +Alsasink then receives the buffer, inspects the caps and reconfigures itself to process +the buffer. Since it received the first buffer of samples, it completes the state change +to the PAUSED state. At this point the pipeline is prerolled and all elements have +samples. Alsasink is now also capable of providing a clock to the pipeline. - Since alsasink is now in the PAUSED state it blocks while receiving the first buffer. This - effectively blocks both mp3dec and filesrc in their gst_pad_push(). +Since alsasink is now in the PAUSED state it blocks while receiving the first buffer. This +effectively blocks both mp3dec and filesrc in their gst_pad_push(). - Since all elements now return SUCCESS from the gst_element_get_state() function, - the pipeline can be put in the PLAYING state. +Since all elements now return SUCCESS from the gst_element_get_state() function, +the pipeline can be put in the PLAYING state. - Before going to PLAYING, the pipeline select a clock and samples the current time of - the clock. This is the base_time. It then distributes this time to all elements. - Elements can then synchronize against the clock using the buffer running_time + - base_time (See also part-synchronisation.txt). +Before going to PLAYING, the pipeline select a clock and samples the current time of +the clock. This is the base_time. It then distributes this time to all elements. +Elements can then synchronize against the clock using the buffer running_time + +base_time (See also part-synchronisation.txt). - The following chain of state changes then takes place: +The following chain of state changes then takes place: + + alsasink to PLAYING: the samples are played to the audio device + mp3dec to PLAYING: nothing happens + filesrc to PLAYING: nothing happens - alsasink to PLAYING: the samples are played to the audio device - mp3dec to PLAYING: nothing happens - filesrc to PLAYING: nothing happens - Pipeline status ---------------- +~~~~~~~~~~~~~~~ - The pipeline informs the application of any special events that occur in the - pipeline with the bus. The bus is an object that the pipeline provides and that - can be retrieved with gst_pipeline_get_bus(). +The pipeline informs the application of any special events that occur in the +pipeline with the bus. The bus is an object that the pipeline provides and that +can be retrieved with gst_pipeline_get_bus(). - The bus can be polled or added to the glib mainloop. +The bus can be polled or added to the glib mainloop. - The bus is distributed to all elements added to the pipeline. The elements use the bus - to post messages on. Various message types exist such as ERRORS, WARNINGS, EOS, - STATE_CHANGED, etc.. +The bus is distributed to all elements added to the pipeline. The elements use the bus +to post messages on. Various message types exist such as ERRORS, WARNINGS, EOS, +STATE_CHANGED, etc.. - The pipeline handles EOS messages received from elements in a special way. It will - only forward the message to the application when all sink elements have posted an - EOS message. +The pipeline handles EOS messages received from elements in a special way. It will +only forward the message to the application when all sink elements have posted an +EOS message. - Other methods for obtaining the pipeline status include the Query functionality that - can be performed with gst_element_query() on the pipeline. This type of query - is useful for obtaining information about the current position and total time of - the pipeline. It can also be used to query for the supported seeking formats and - ranges. +Other methods for obtaining the pipeline status include the Query functionality that +can be performed with gst_element_query() on the pipeline. This type of query +is useful for obtaining information about the current position and total time of +the pipeline. It can also be used to query for the supported seeking formats and +ranges. Pipeline EOS ------------- +~~~~~~~~~~~~ - When the source filter encounters the end of the stream, it sends an EOS event to - the peer element. This event will then travel downstream to all of the connected - elements to inform them of the EOS. The element is not supposed to accept any more - data after receiving an EOS event on a sinkpad. +When the source filter encounters the end of the stream, it sends an EOS event to +the peer element. This event will then travel downstream to all of the connected +elements to inform them of the EOS. The element is not supposed to accept any more +data after receiving an EOS event on a sinkpad. - The element providing the streaming thread stops sending data after sending the - EOS event. +The element providing the streaming thread stops sending data after sending the +EOS event. - The EOS event will eventually arrive in the sink element. The sink will then post - an EOS message on the bus to inform the pipeline that a particular stream has - finished. When all sinks have reported EOS, the pipeline forwards the EOS message - to the application. The EOS message is only forwarded to the application in the - PLAYING state. +The EOS event will eventually arrive in the sink element. The sink will then post +an EOS message on the bus to inform the pipeline that a particular stream has +finished. When all sinks have reported EOS, the pipeline forwards the EOS message +to the application. The EOS message is only forwarded to the application in the +PLAYING state. - When in EOS, the pipeline remains in the PLAYING state, it is the applications - responsability to PAUSE or READY the pipeline. The application can also issue - a seek, for example. +When in EOS, the pipeline remains in the PLAYING state, it is the applications +responsability to PAUSE or READY the pipeline. The application can also issue +a seek, for example. Pipeline READY --------------- +~~~~~~~~~~~~~~ + +When a running pipeline is set from the PLAYING to READY state, the following +actions occur in the pipeline: + + alsasink to PAUSED: alsasink blocks and completes the state change on the + next sample. If the element was EOS, it does not wait for + a sample to complete the state change. + mp3dec to PAUSED: nothing + filesrc to PAUSED: nothing + +Going to the intermediate PAUSED state will block all elements in the _push() +functions. This happens because the sink element blocks on the first buffer +it receives. + +Some elements might be performing blocking operations in the PLAYING state that +must be unblocked when they go into the PAUSED state. This makes sure that the +state change happens very fast. + +In the next PAUSED to READY state change the pipeline has to shut down and all +streaming threads must stop sending data. This happens in the following sequence: + + alsasink to READY: alsasink unblocks from the _chain() function and returns a + WRONG_STATE return value to the peer element. The sinkpad is + deactivated and becomes unusable for sending more data. + mp3dec to READY: the pads are deactivated and the state change completes when + mp3dec leaves its _chain() function. + filesrc to READY: the pads are deactivated and the thread is paused. + +The upstream elements finish their chain() function because the downstream element +returned an error code (WRONG_STATE) from the _push() functions. These error codes +are eventually returned to the element that started the streaming thread (filesrc), +which pauses the thread and completes the state change. + +This sequence of events ensure that all elements are unblocked and all streaming +threads stopped. - When a running pipeline is set from the PLAYING to READY state, the following - actions occur in the pipeline: - - alsasink to PAUSED: alsasink blocks and completes the state change on the - next sample. If the element was EOS, it does not wait for - a sample to complete the state change. - mp3dec to PAUSED: nothing - filesrc to PAUSED: nothing - - Going to the intermediate PAUSED state will block all elements in the _push() - functions. This happens because the sink element blocks on the first buffer - it receives. - - Some elements might be performing blocking operations in the PLAYING state that - must be unblocked when they go into the PAUSED state. This makes sure that the - state change happens very fast. - - In the next PAUSED to READY state change the pipeline has to shut down and all - streaming threads must stop sending data. This happens in the following sequence: - - alsasink to READY: alsasink unblocks from the _chain() function and returns a - WRONG_STATE return value to the peer element. The sinkpad is - deactivated and becomes unusable for sending more data. - mp3dec to READY: the pads are deactivated and the state change completes when - mp3dec leaves its _chain() function. - filesrc to READY: the pads are deactivated and the thread is paused. - - The upstream elements finish their chain() function because the downstream element - returned an error code (WRONG_STATE) from the _push() functions. These error codes - are eventually returned to the element that started the streaming thread (filesrc), - which pauses the thread and completes the state change. - - This sequence of events ensure that all elements are unblocked and all streaming - threads stopped. - Pipeline seeking ----------------- +~~~~~~~~~~~~~~~~ - Seeking in the pipeline requires a very specific order of operations to make - sure that the elements remain synchronized and that the seek is performed with - a minimal amount of latency. +Seeking in the pipeline requires a very specific order of operations to make +sure that the elements remain synchronized and that the seek is performed with +a minimal amount of latency. - An application issues a seek event on the pipeline using gst_element_send_event() - on the pipeline element. The event can be a seek event in any of the formats - supported by the elements. +An application issues a seek event on the pipeline using gst_element_send_event() +on the pipeline element. The event can be a seek event in any of the formats +supported by the elements. - The pipeline first pauses the pipeline to speed up the seek operations. - - The pipeline then issues the seek event to all sink elements. The sink then forwards - the seek event upstream until some element can perform the seek operation, which is - typically the source or demuxer element. All intermediate elements can transform the - requested seek offset to another format, this way a decoder element can transform a - seek to a frame number to a timestamp, for example. - - When the seek event reaches an element that will perform the seek operation, that - element performs the following steps. +The pipeline first pauses the pipeline to speed up the seek operations. + +The pipeline then issues the seek event to all sink elements. The sink then forwards +the seek event upstream until some element can perform the seek operation, which is +typically the source or demuxer element. All intermediate elements can transform the +requested seek offset to another format, this way a decoder element can transform a +seek to a frame number to a timestamp, for example. + +When the seek event reaches an element that will perform the seek operation, that +element performs the following steps. 1) send a FLUSH_START event to all downstream and upstream peer elements. 2) make sure the streaming thread is not running. The streaming thread will @@ -491,31 +491,31 @@ Pipeline seeking 5) send NEWSEGMENT event to inform all elements of the new position and to complete the seek. - In step 1) all dowstream elements have to return from any blocking operations - and have to refuse any further buffers or events different from a FLUSH done. +In step 1) all dowstream elements have to return from any blocking operations +and have to refuse any further buffers or events different from a FLUSH done. - The first step ensures that the streaming thread eventually unblocks and that - step 2) can be performed. At this point, dataflow is completely stopped in the - pipeline. +The first step ensures that the streaming thread eventually unblocks and that +step 2) can be performed. At this point, dataflow is completely stopped in the +pipeline. - In step 3) the element performs the seek to the requested position. +In step 3) the element performs the seek to the requested position. - In step 4) all peer elements are allowed to accept data again and streaming - can continue from the new position. A FLUSH done event is sent to all the peer - elements so that they accept new data again and restart their streaming threads. +In step 4) all peer elements are allowed to accept data again and streaming +can continue from the new position. A FLUSH done event is sent to all the peer +elements so that they accept new data again and restart their streaming threads. - Step 5) informs all elements of the new position in the stream. After that the - event function returns back to the application. and the streaming threads start - to produce new data. +Step 5) informs all elements of the new position in the stream. After that the +event function returns back to the application. and the streaming threads start +to produce new data. - Since the pipeline is still PAUSED, this will preroll the next media sample in the - sinks. The application can wait for this preroll to complete by performing a - _get_state() on the pipeline. +Since the pipeline is still PAUSED, this will preroll the next media sample in the +sinks. The application can wait for this preroll to complete by performing a +_get_state() on the pipeline. - The last step in the seek operation is then to adjust the stream time of the pipeline - to 0 and to set the pipeline back to PLAYING. +The last step in the seek operation is then to adjust the stream time of the pipeline +to 0 and to set the pipeline back to PLAYING. - The sequence of events in our mp3 playback example. +The sequence of events in our mp3 playback example. | a) seek on pipeline | b) PAUSE pipeline @@ -537,4 +537,4 @@ Pipeline seeking | e) update stream time to 0 | f) PLAY pipeline - + diff --git a/docs/design/part-preroll.txt b/docs/design/part-preroll.txt index ab8d3e560..264492d0b 100644 --- a/docs/design/part-preroll.txt +++ b/docs/design/part-preroll.txt @@ -19,7 +19,7 @@ sinks (see part-element-sink.txt) Committing the state --------------------- +~~~~~~~~~~~~~~~~~~~~ When going to PAUSED and PLAYING a buffer should be queued in the pad. We also make this requirement for going to PLAYING since a flush event in the PAUSED @@ -39,7 +39,7 @@ blocking wait. Unlocking the preroll ---------------------- +~~~~~~~~~~~~~~~~~~~~~ The following conditions unlock the preroll: diff --git a/docs/design/part-push-pull.txt b/docs/design/part-push-pull.txt index d5cc9f6ec..29e415d3f 100644 --- a/docs/design/part-push-pull.txt +++ b/docs/design/part-push-pull.txt @@ -23,7 +23,7 @@ srcpads implementing a getrange function will be the exception. state changes -------------- +~~~~~~~~~~~~~ The GstBin sets the state of all the sink elements. These are the elements without source pads. diff --git a/docs/design/part-qos.txt b/docs/design/part-qos.txt index 531612e74..5cb7a87e8 100644 --- a/docs/design/part-qos.txt +++ b/docs/design/part-qos.txt @@ -18,7 +18,7 @@ made: Sources of quality problems ---------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~ - High CPU load - Network problems @@ -26,7 +26,7 @@ Sources of quality problems QoS event ---------- +~~~~~~~~~ The QoS event is generated by an element that synchronizes against the clock. It travels upstream and contains the following fields: @@ -51,7 +51,7 @@ operations. QoS message ------------ +~~~~~~~~~~~ A QOS message is posted on the bus whenever an element decides to: @@ -122,7 +122,7 @@ relevant elements or baseclasses. Collecting statistics ---------------------- +~~~~~~~~~~~~~~~~~~~~~ A buffer with timestamp B1 arrives in the sink at time T1. The buffer timestamp is then synchronized against the clock which yields a jitter J1 @@ -179,7 +179,7 @@ An element that is not receiving enough data is said to be starved. Element measurements --------------------- +~~~~~~~~~~~~~~~~~~~~ In addition to the measurements of the datarate of the upstream element, a typical element must also measure its own performance. Global pipeline @@ -389,3 +389,4 @@ Live sources will automatically drop data when it takes too long to process the that the element pushes out. Live sources should post a QoS message when data is dropped. + diff --git a/docs/design/part-query.txt b/docs/design/part-query.txt index bce78564b..4973c21cc 100644 --- a/docs/design/part-query.txt +++ b/docs/design/part-query.txt @@ -1,12 +1,16 @@ -DRAFT Query ------------ +Query +----- + +NOTE: this is implemented as proposed Purpose +~~~~~~~ - Queries are used to get information about the stream. - A query is started on a specific pad and travels up or downstream. +Queries are used to get information about the stream. +A query is started on a specific pad and travels up or downstream. Types of queries +~~~~~~~~~~~~~~~~ - get length of stream - get position in stream @@ -17,6 +21,7 @@ Types of queries - query internal links. Current implementation +~~~~~~~~~~~~~~~~~~~~~~ The current implementation of query requires pads to implement the following functions: @@ -36,12 +41,14 @@ Current implementation Requirements +~~~~~~~~~~~~ - multiple return values, grouped together when they make sense. - one pad function to perform the query - extensible queries. Proposition +~~~~~~~~~~~ - define GstQuery extending GstMiniObject and containing a GstStructure (see GstMessage) - define standard query types (see proposed types) @@ -54,6 +61,7 @@ Proposition query is not supported. Proposed types +~~~~~~~~~~~~~~ - GST_QUERY_SEEKING: diff --git a/docs/design/part-relations.txt b/docs/design/part-relations.txt index a84e4477c..eb38296cf 100644 --- a/docs/design/part-relations.txt +++ b/docs/design/part-relations.txt @@ -5,487 +5,487 @@ This document describes the relations between objects that exist in GStreamer. It will also describe the way of handling the relation wrt locking and refcounting. -1) parent-child relation - - +---------+ +-------+ - | parent | | child | - *--->| *----->| | - | F1|<-----* 1| - +---------+ +-------+ +parent-child relation +~~~~~~~~~~~~~~~~~~~~~ + + +---------+ +-------+ + | parent | | child | + *--->| *----->| | + | F1|<-----* 1| + +---------+ +-------+ + + - properties + + - parent has references to multiple children + - child has reference to parent + - reference fields protected with LOCK + - the reference held by each child to the parent is + NOT reflected in the refcount of the parent. + - the parent removes the floating flag of the child when taking + ownership. + - the application has valid reference to parent + - creation/destruction requires two unnested locks and 1 refcount. + + - usage in GStreamer + + GstBin -> GstElement + GstElement -> GstRealPad + + - lifecycle + + a) object creation + + The application creates two object and holds a pointer + to them. The objects are initially FLOATING with a refcount + of 1. + + +---------+ +-------+ + *--->| parent | *--->| child | + | * | | | + | F1| | * F1| + +---------+ +-------+ + + b) establishing the parent-child relationship + + The application then calls a method on the parent object to take + ownership of the child object. The parent performs the following + actions: + + result = _set_parent (child, parent); + if (result) { + LOCK (parent); + ref_pointer = child; + + .. update other data structures .. + UNLOCK (parent); + } + else { + .. child had parent .. + } + + The _set_parent() method performs the following actions: + + LOCK (child); + if (child->parent != NULL) { + UNLOCK (child); + return FALSE; + } + if (IS_FLOATING (child)) { + UNSET (child, FLOATING); + } + else { + _ref (child); + } + child->parent = parent; + UNLOCK (child); + _signal (PARENT_SET, child, parent); + return TRUE; + + The function atomically checks if the child has no parent yet + and will set the parent if not. It will also sink the child, meaning + all floating references to the child are invalid now as it takes + over the refcount of the object. - - properties + Visually: - - parent has references to multiple children - - child has reference to parent - - reference fields protected with LOCK - - the reference held by each child to the parent is - NOT reflected in the refcount of the parent. - - the parent removes the floating flag of the child when taking - ownership. - - the application has valid reference to parent - - creation/destruction requires two unnested locks and 1 refcount. + after _set_parent() returns TRUE: - - usage in GStreamer + +---------+ +-------+ + *---->| parent | *-//->| child | + | * | | | + | F1|<-------------* 1| + +---------+ +-------+ - GstBin -> GstElement - GstElement -> GstRealPad + after parent updates ref_pointer to child. - - lifecycle + +---------+ +-------+ + *---->| parent | *-//->| child | + | *--------->| | + | F1|<---------* 1| + +---------+ +-------+ + + - only one parent is able to _sink the same object because the + _set_parent() method is atomic. + - since only one parent is able to _set_parent() the object, only + one will add a reference to the object. + - since the parent can hold multiple references to children, we don't + need to lock the parent when locking the child. Many threads can + call _set_parent() on the children with the same parent, the parent + can then add all those to its lists. + + Note: that the signal is emited before the parent has added the + element to its internal data structures. This is not a problem + since the parent usually has his own signal to inform the app that + the child was reffed. One possible solution would be to update the + internal structure first and then perform a rollback if the _set_parent() + failed. This is not a good solution as iterators might grab the + 'half-added' child too soon. - a) object creation - - The application creates two object and holds a pointer - to them. The objects are initially FLOATING with a refcount - of 1. - - +---------+ +-------+ - *--->| parent | *--->| child | - | * | | | - | F1| | * F1| - +---------+ +-------+ - - b) establishing the parent-child relationship - - The application then calls a method on the parent object to take - ownership of the child object. The parent performs the following - actions: - - result = _set_parent (child, parent); - if (result) { - LOCK (parent); - ref_pointer = child; - - .. update other data structures .. - UNLOCK (parent); - } - else { - .. child had parent .. - } - - The _set_parent() method performs the following actions: - - LOCK (child); - if (child->parent != NULL) { - UNLOCK (child); - return FALSE; - } - if (IS_FLOATING (child)) { - UNSET (child, FLOATING); - } - else { - _ref (child); - } - child->parent = parent; - UNLOCK (child); - _signal (PARENT_SET, child, parent); - return TRUE; - - The function atomically checks if the child has no parent yet - and will set the parent if not. It will also sink the child, meaning - all floating references to the child are invalid now as it takes - over the refcount of the object. - - Visually: - - after _set_parent() returns TRUE: - - +---------+ +-------+ - *---->| parent | *-//->| child | - | * | | | - | F1|<-------------* 1| - +---------+ +-------+ - - after parent updates ref_pointer to child. - - +---------+ +-------+ - *---->| parent | *-//->| child | - | *--------->| | - | F1|<---------* 1| - +---------+ +-------+ + c) using the parent-child relationship + + - since the initial floating reference to the child object became + invalid after giving it to the parent, any reference to a child + has at least a refcount > 1. + + - this means that unreffing a child object cannot decrease the refcount + to 0. In fact, only the parent can destroy and dispose the child + object. + + - given a reference to the child object, the parent pointer is only + valid when holding the child LOCK. Indeed, after unlocking the child + LOCK, the parent can unparent the child or the parent could even become + disposed. To avoid the parent dispose problem, when obtaining the + parent pointer, if should be reffed before releasing the child LOCK. + + 1) getting a reference to the parent. - - only one parent is able to _sink the same object because the - _set_parent() method is atomic. - - since only one parent is able to _set_parent() the object, only - one will add a reference to the object. - - since the parent can hold multiple references to children, we don't - need to lock the parent when locking the child. Many threads can - call _set_parent() on the children with the same parent, the parent - can then add all those to its lists. - - Note: that the signal is emited before the parent has added the - element to its internal data structures. This is not a problem - since the parent usually has his own signal to inform the app that - the child was reffed. One possible solution would be to update the - internal structure first and then perform a rollback if the _set_parent() - failed. This is not a good solution as iterators might grab the - 'half-added' child too soon. - - c) using the parent-child relationship - - - since the initial floating reference to the child object became - invalid after giving it to the parent, any reference to a child - has at least a refcount > 1. - - - this means that unreffing a child object cannot decrease the refcount - to 0. In fact, only the parent can destroy and dispose the child - object. - - - given a reference to the child object, the parent pointer is only - valid when holding the child LOCK. Indeed, after unlocking the child - LOCK, the parent can unparent the child or the parent could even become - disposed. To avoid the parent dispose problem, when obtaining the - parent pointer, if should be reffed before releasing the child LOCK. - - I) getting a reference to the parent. - - - a referece is held to the child, so it cannot be disposed. - - LOCK (child); - parent = _ref (child->parent); - UNLOCK (child); - - .. use parent .. - - _unref (parent); - - II) getting a reference to a child - - - a reference to a child can be obtained by reffing it before - adding it to the parent or by querying the parent. - - - when requesting a child from the parent, a reference is held to - the parent so it cannot be disposed. The parent will use its - internal data structures to locate the child element and will - return a reference to it with an incremented refcount. The - requester should _unref() the child after usage. - - - d) destroying the parent-child relationship - - - only the parent can actively destroy the parent-child relationship - this typically happens when a method is called on the parent to release - ownership of the child. + - a referece is held to the child, so it cannot be disposed. + + LOCK (child); + parent = _ref (child->parent); + UNLOCK (child); - - a child shall never remove itself from the parent. + .. use parent .. - - since calling a method on the parent with the child as an argument - requires the caller to obtain a valid reference to the child, the child - refcount is at least > 1. + _unref (parent); - - the parent will perform the folowing actions: + 2) getting a reference to a child + + - a reference to a child can be obtained by reffing it before + adding it to the parent or by querying the parent. + + - when requesting a child from the parent, a reference is held to + the parent so it cannot be disposed. The parent will use its + internal data structures to locate the child element and will + return a reference to it with an incremented refcount. The + requester should _unref() the child after usage. + + + d) destroying the parent-child relationship - LOCK (parent); - if (ref_pointer == child) { - ref_pointer = NULL; + - only the parent can actively destroy the parent-child relationship + this typically happens when a method is called on the parent to release + ownership of the child. - .. update other data structures .. - UNLOCK (parent); + - a child shall never remove itself from the parent. - _unparent (child); - } - else { - UNLOCK (parent); - .. not our child .. - } + - since calling a method on the parent with the child as an argument + requires the caller to obtain a valid reference to the child, the child + refcount is at least > 1. - The _unparent() method performs the following actions: - - LOCK (child); - if (child->parent != NULL) { - child->parent = NULL; - UNLOCK (child); - _signal (PARENT_UNSET, child, parent); - - _unref (child); - } - else { - UNLOCK (child); - } - - Since the _unparent() method unrefs the child object, it is possible that - the child pointer is invalid after this function. If the parent wants to - perform other actions on the child (such as signal emmision) it should - _ref() the child first. - - -2) single-reffed relation - - +---------+ +---------+ - *--->| object1 | *--->| object2 | - | *--------->| | - | 1| | 2| - +---------+ +---------+ - - - properties - - - one object has a reference to another - - reference field protected with LOCK - - the reference held by the object is reflected in the - refcount of the other object. - - typically the other object can be shared among multiple - other objects where each ref is counted for in the - refcount. - - no object has ownership of the other. - - either shared state or copy-on-write. - - creation/destruction requires one lock and one refcount. - - - usage - - GstRealPad -> GstCaps - GstBuffer -> GstCaps - GstEvent -> GstCaps - GstEvent -> GstObject - GstMessage -> GstCaps - GstMessage -> GstObject - - - lifecycle - - a) Two objects exist unlinked. - - +---------+ +---------+ - *--->| object1 | *--->| object2 | - | * | | | - | 1| | 1| - +---------+ +---------+ + - the parent will perform the folowing actions: - b) establishing the single-reffed relationship + LOCK (parent); + if (ref_pointer == child) { + ref_pointer = NULL; - The second object is attached to the first one using a method - on the first object. The second object is reffed and a pointer - is updated in the first object using the following algorithm: + .. update other data structures .. + UNLOCK (parent); - LOCK (object1); - if (object1->pointer) - _unref (object1->pointer); - object1->pointer = _ref (object2); - UNLOCK (object1); + _unparent (child); + } + else { + UNLOCK (parent); + .. not our child .. + } - After releasing the lock on the first object is is not sure that - object2 is still reffed from object1. + The _unparent() method performs the following actions: + + LOCK (child); + if (child->parent != NULL) { + child->parent = NULL; + UNLOCK (child); + _signal (PARENT_UNSET, child, parent); - +---------+ +---------+ - *--->| object1 | *--->| object2 | - | *--------->| | - | 1| | 2| - +---------+ +---------+ + _unref (child); + } + else { + UNLOCK (child); + } + + Since the _unparent() method unrefs the child object, it is possible that + the child pointer is invalid after this function. If the parent wants to + perform other actions on the child (such as signal emmision) it should + _ref() the child first. + + +single-reffed relation +~~~~~~~~~~~~~~~~~~~~~~ + + +---------+ +---------+ + *--->| object1 | *--->| object2 | + | *--------->| | + | 1| | 2| + +---------+ +---------+ + + - properties + + - one object has a reference to another + - reference field protected with LOCK + - the reference held by the object is reflected in the + refcount of the other object. + - typically the other object can be shared among multiple + other objects where each ref is counted for in the + refcount. + - no object has ownership of the other. + - either shared state or copy-on-write. + - creation/destruction requires one lock and one refcount. + + - usage + + GstRealPad -> GstCaps + GstBuffer -> GstCaps + GstEvent -> GstCaps + GstEvent -> GstObject + GstMessage -> GstCaps + GstMessage -> GstObject + + - lifecycle + + a) Two objects exist unlinked. + + +---------+ +---------+ + *--->| object1 | *--->| object2 | + | * | | | + | 1| | 1| + +---------+ +---------+ - c) using the single-reffed relationship + b) establishing the single-reffed relationship - The only way to access object2 is by holding a ref to it or by - getting the reference from object1. - Reading the object pointed to by object1 can be done like this: + The second object is attached to the first one using a method + on the first object. The second object is reffed and a pointer + is updated in the first object using the following algorithm: - LOCK (object1); - object2 = object1->pointer; - _ref (object2); - UNLOCK (object1); + LOCK (object1); + if (object1->pointer) + _unref (object1->pointer); + object1->pointer = _ref (object2); + UNLOCK (object1); - .. use object2 ... - _unref (object2); + After releasing the lock on the first object is is not sure that + object2 is still reffed from object1. - Depending on the type of the object, modifications can be done either - with copy-on-write or directly into the object. + +---------+ +---------+ + *--->| object1 | *--->| object2 | + | *--------->| | + | 1| | 2| + +---------+ +---------+ - Copy on write can practically only be done like this: + c) using the single-reffed relationship - LOCK (object1); - object2 = object1->pointer; - object2 = _copy_on_write (object2); - ... make modifications to object2 ... - UNLOCK (object1); + The only way to access object2 is by holding a ref to it or by + getting the reference from object1. + Reading the object pointed to by object1 can be done like this: - Releasing the lock has only a very small window where the copy_on_write - actually does not perform a copy: + LOCK (object1); + object2 = object1->pointer; + _ref (object2); + UNLOCK (object1); - LOCK (object1); - object2 = object1->pointer; - _ref (object2); - UNLOCK (object1); + .. use object2 ... + _unref (object2); - .. object2 now has at least 2 refcounts making the next - copy-on-write make a real copy, unless some other thread - writes another object2 to object1 here ... + Depending on the type of the object, modifications can be done either + with copy-on-write or directly into the object. - object2 = _copy_on_write (object2); + Copy on write can practically only be done like this: - .. make modifications to object2 ... + LOCK (object1); + object2 = object1->pointer; + object2 = _copy_on_write (object2); + ... make modifications to object2 ... + UNLOCK (object1); - LOCK (object1); - if (object1->pointer != object2) { - if (object1->pointer) - _unref (object1->pointer); - object1->pointer = gst_object_ref (object2); - } - UNLOCK (object1); + Releasing the lock has only a very small window where the copy_on_write + actually does not perform a copy: - d) destroying the single-reffed relationship - - The folowing algorithm removes the single-reffed link between - object1 and object2. + LOCK (object1); + object2 = object1->pointer; + _ref (object2); + UNLOCK (object1); - LOCK (object1); - _unref (object1->pointer); - object1->pointer = NULL; - UNLOCK (object1); - - Which yields the following initial state again: + .. object2 now has at least 2 refcounts making the next + copy-on-write make a real copy, unless some other thread + writes another object2 to object1 here ... - +---------+ +---------+ - *--->| object1 | *--->| object2 | - | * | | | - | 1| | 1| - +---------+ +---------+ + object2 = _copy_on_write (object2); + .. make modifications to object2 ... -3) unreffed relation - - +---------+ +---------+ - *--->| object1 | *--->| object2 | - | *--------->| | - | 1|<---------* 1| - +---------+ +---------+ - - - properties - - - two objects have references to eachother - - both objects can only have 1 reference to another object. - - reference fields protected with LOCK - - the references held by each object are NOT reflected in the - refcount of the other object. - - no object has ownership of the other. - - typically each object is owned by a different parent. - - creation/destruction requires two nested locks and no refcounts. - - - usage - - - This type of link is used when the link is less important than - the existance of the objects, If one of the objects is disposed, so - is the link. + LOCK (object1); + if (object1->pointer != object2) { + if (object1->pointer) + _unref (object1->pointer); + object1->pointer = gst_object_ref (object2); + } + UNLOCK (object1); + + d) destroying the single-reffed relationship + + The folowing algorithm removes the single-reffed link between + object1 and object2. + + LOCK (object1); + _unref (object1->pointer); + object1->pointer = NULL; + UNLOCK (object1); - GstRealPad <-> GstRealPad (srcpad lock taken first) + Which yields the following initial state again: + + +---------+ +---------+ + *--->| object1 | *--->| object2 | + | * | | | + | 1| | 1| + +---------+ +---------+ + + +unreffed relation +~~~~~~~~~~~~~~~~~ + + +---------+ +---------+ + *--->| object1 | *--->| object2 | + | *--------->| | + | 1|<---------* 1| + +---------+ +---------+ + + - properties + + - two objects have references to eachother + - both objects can only have 1 reference to another object. + - reference fields protected with LOCK + - the references held by each object are NOT reflected in the + refcount of the other object. + - no object has ownership of the other. + - typically each object is owned by a different parent. + - creation/destruction requires two nested locks and no refcounts. + + - usage + + - This type of link is used when the link is less important than + the existance of the objects, If one of the objects is disposed, so + is the link. + + GstRealPad <-> GstRealPad (srcpad lock taken first) - - lifecycle + - lifecycle - a) Two objects exist unlinked. + a) Two objects exist unlinked. - +---------+ +---------+ - *--->| object1 | *--->| object2 | - | * | | | - | 1| | * 1| - +---------+ +---------+ + +---------+ +---------+ + *--->| object1 | *--->| object2 | + | * | | | + | 1| | * 1| + +---------+ +---------+ - b) establishing the unreffed relationship - - Since we need to take two locks, the order in which these locks are - taken is very important or we might cause deadlocks. This lock order - must be defined for all unreffed relations. In these examples we always - lock object1 first and then object2. - - LOCK (object1); - LOCK (object2); - object2->refpointer = object1; - object1->refpointer = object2; - UNLOCK (object2); - UNLOCK (object1); + b) establishing the unreffed relationship + + Since we need to take two locks, the order in which these locks are + taken is very important or we might cause deadlocks. This lock order + must be defined for all unreffed relations. In these examples we always + lock object1 first and then object2. - c) using the unreffed relationship + LOCK (object1); + LOCK (object2); + object2->refpointer = object1; + object1->refpointer = object2; + UNLOCK (object2); + UNLOCK (object1); - Reading requires taking one of the locks and reading the corresponing - object. Again we need to ref the object before releasing the lock. + c) using the unreffed relationship - LOCK (object1); - object2 = _ref (object1->refpointer); - UNLOCK (object1); + Reading requires taking one of the locks and reading the corresponing + object. Again we need to ref the object before releasing the lock. - .. use object2 .. - _unref (object2); - - d) destroying the unreffed relationship - - Because of the lock order we need to be careful when destroying this - Relation. - - When only a reference to object1 is held: + LOCK (object1); + object2 = _ref (object1->refpointer); + UNLOCK (object1); - LOCK (object1); - LOCK (object2); + .. use object2 .. + _unref (object2); + + d) destroying the unreffed relationship + + Because of the lock order we need to be careful when destroying this + Relation. + + When only a reference to object1 is held: + + LOCK (object1); + LOCK (object2); + object1->refpointer->refpointer = NULL; + object1->refpointer = NULL; + UNLOCK (object2); + UNLOCK (object1); + + When only a reference to object2 is held we need to get a handle to the + other object fist so that we can lock it first. There is a window where + we need to release all locks and the relation could be invalid. To solve + this we check the relation after grabbing both locks and retry if the + relation changed. + + retry: + LOCK (object2); + object1 = _ref (object2->refpointer); + UNLOCK (object2); + .. things can change here .. + LOCK (object1); + LOCK (object2); + if (object1 == object2->refpointer) { + /* relation unchanged */ object1->refpointer->refpointer = NULL; object1->refpointer = NULL; + } + else { + /* relation changed.. retry */ UNLOCK (object2); UNLOCK (object1); + _unref (object1); + goto retry; + } + UNLOCK (object2); + UNLOCK (object1); + _unref (object1); + + When references are held to both objects. Note that it is not possible to + get references to both objects with the locks released since when the + references are taken and the locks are released, a concurrent update might + have changed the link, making the references not point to linked objects. + + LOCK (object1); + LOCK (object2); + if (object1->refpointer == object2) { + object2->refpointer = NULL; + object1->refpointer = NULL; + } + else { + .. objects are not linked .. + } + UNLOCK (object2); + UNLOCK (object1); + + +double-reffed relation +~~~~~~~~~~~~~~~~~~~~~~ + + +---------+ +---------+ + *--->| object1 | *--->| object2 | + | *--------->| | + | 2|<---------* 2| + +---------+ +---------+ + + - properties + + - two objects have references to eachother + - reference fields protected with LOCK + - the references held by each object are reflected in the + refcount of the other object. + - no object has ownership of the other. + - typically each object is owned by a different parent. + - creation/destruction requires two locks and two refcounts. + + - usage + + Not used in GStreamer. - When only a reference to object2 is held we need to get a handle to the - other object fist so that we can lock it first. There is a window where - we need to release all locks and the relation could be invalid. To solve - this we check the relation after grabbing both locks and retry if the - relation changed. - - retry: - LOCK (object2); - object1 = _ref (object2->refpointer); - UNLOCK (object2); - .. things can change here .. - LOCK (object1); - LOCK (object2); - if (object1 == object2->refpointer) { - /* relation unchanged */ - object1->refpointer->refpointer = NULL; - object1->refpointer = NULL; - } - else { - /* relation changed.. retry */ - UNLOCK (object2); - UNLOCK (object1); - _unref (object1); - goto retry; - } - UNLOCK (object2); - UNLOCK (object1); - _unref (object1); - - When references are held to both objects. Note that it is not possible to - get references to both objects with the locks released since when the - references are taken and the locks are released, a concurrent update might - have changed the link, making the references not point to linked objects. - - LOCK (object1); - LOCK (object2); - if (object1->refpointer == object2) { - object2->refpointer = NULL; - object1->refpointer = NULL; - } - else { - .. objects are not linked .. - } - UNLOCK (object2); - UNLOCK (object1); - - -4) double-reffed relation - - +---------+ +---------+ - *--->| object1 | *--->| object2 | - | *--------->| | - | 2|<---------* 2| - +---------+ +---------+ - - - properties - - - two objects have references to eachother - - reference fields protected with LOCK - - the references held by each object are reflected in the - refcount of the other object. - - no object has ownership of the other. - - typically each object is owned by a different parent. - - creation/destruction requires two locks and two refcounts. - - - usage - - Not used in GStreamer. - - - lifecycle - - - - + - lifecycle diff --git a/docs/design/part-scheduling.txt b/docs/design/part-scheduling.txt index 6c609cba7..3b8a00928 100644 --- a/docs/design/part-scheduling.txt +++ b/docs/design/part-scheduling.txt @@ -5,7 +5,7 @@ The scheduling in GStreamer is based on pads actively pushing (producing) data o pad pulling in data (consuming) from other pads. Pushing -------- +~~~~~~~ A pad can produce data and push it to the next pad. A pad that behaves this way exposes a loop function that will be called repeadedly until it returns false. @@ -20,7 +20,7 @@ This method of producing data is called the streaming mode since the producer produces a constant stream of data. Pulling -------- +~~~~~~~ Pads that operate in pulling mode can only pull data from a pad that exposes the pull_range function. In this case, the sink pad exposes a loop function that will be @@ -31,7 +31,7 @@ push function to push the result to the peer sinkpad. Deciding the scheduling mode ----------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When tha pad is activated, the _activate() function is called. The pad can then choose to activate itself in push or pull mode depending on upstream @@ -41,162 +41,163 @@ The GStreamer core will by default activate pads in push mode when there is no activate function for the pad. The chain function ------------------- +~~~~~~~~~~~~~~~~~~ The chain function will be called when a upstream element perform a _push() on the pad. The upstream element can be another chain based element or a pushing source. The getrange function ---------------------- +~~~~~~~~~~~~~~~~~~~~~ The getrange function is called when a peer pad perform a _pull_range() on the pad. This downstream pad can be a pulling element or another _pull_range() based element. Plug-in techniques ------------------- +~~~~~~~~~~~~~~~~~~ Multi-sink elements -------------------- - - Elements with multiple sinks can either expose a loop function on each of the pads to - actively pull_range data or they can expose a chain function on each pad. +^^^^^^^^^^^^^^^^^^^ - Implementing a chain function is usually easy and allows for all possible scheduling - methods. +Elements with multiple sinks can either expose a loop function on each of the pads to +actively pull_range data or they can expose a chain function on each pad. - Pad select - ---------- +Implementing a chain function is usually easy and allows for all possible scheduling +methods. - If the chain based sink wants to wait for one of the pads to receive a buffer, just - implement the action to perform in the chain function. Be aware that the action could - be performed in different threads and possibly simultaneously so grab the STREAM_LOCK. +Pad select +---------- - Collect pads - ------------ + If the chain based sink wants to wait for one of the pads to receive a buffer, just + implement the action to perform in the chain function. Be aware that the action could + be performed in different threads and possibly simultaneously so grab the STREAM_LOCK. - If the chain based sink pads all require one buffer before the element can operate on - the data, collect all the buffers in the chain function and perform the action when - all chainpads received the buffer. +Collect pads +------------ - In this case you probably also don't want to accept more data on a pad that has a buffer - queued. This can easily be done with the following code snippet: + If the chain based sink pads all require one buffer before the element can operate on + the data, collect all the buffers in the chain function and perform the action when + all chainpads received the buffer. - static GstFlowReturn _chain (GstPad *pad, GstBuffer *buffer) - { - LOCK (mylock); - while (pad->store != NULL) { - WAIT (mycond, mylock); - } - pad->store = buffer; - SIGNAL (mycond); - UNLOCK (mylock); + In this case you probably also don't want to accept more data on a pad that has a buffer + queued. This can easily be done with the following code snippet: - return GST_FLOW_OK; + static GstFlowReturn _chain (GstPad *pad, GstBuffer *buffer) + { + LOCK (mylock); + while (pad->store != NULL) { + WAIT (mycond, mylock); } - - static void _pull (GstPad *pad, GstBuffer **buffer) - { - LOCK (mylock); - while (pad->store == NULL) { - WAIT (mycond, mylock); - } - **buffer = pad->store; - pad->store = NULL; - SIGNAL (mycond); - UNLOCK (mylock); + pad->store = buffer; + SIGNAL (mycond); + UNLOCK (mylock); + + return GST_FLOW_OK; + } + + static void _pull (GstPad *pad, GstBuffer **buffer) + { + LOCK (mylock); + while (pad->store == NULL) { + WAIT (mycond, mylock); } - + **buffer = pad->store; + pad->store = NULL; + SIGNAL (mycond); + UNLOCK (mylock); + } + Cases ------ - Inside the braces below the pads is stated what function the - pad support: - - l: exposes a loop function, so it can act as a pushing source. - g: exposes a getrange function - c: exposes a chain function - - following scheduling decisions are made based on the scheduling - methods exposed by the pads: - - (g) - (l): sinkpad will pull data from src - (l) - (c): srcpad actively pushes data to sinkpad - () - (c): srcpad will push data to sinkpad. - - () - () : not schedulable. - () - (l): not schedulable. - (g) - () : not schedulable. - (g) - (c): not schedulable. - (l) - () : not schedulable. - (l) - (l): not schedulable - - () - (g): impossible - (g) - (g): impossible. - (l) - (g): impossible - (c) - () : impossible - (c) - (g): impossible - (c) - (l): impossible - (c) - (c): impossible - - +---------+ +------------+ +-----------+ - | filesrc | | mp3decoder | | audiosink | - | src--sink src--sink | - +---------+ +------------+ +-----------+ - (l-g) (c) () (c) - - When activating the pads: - - * audiosink has a chain function and the peer pad has no - loop function, no scheduling is done. - * mp3decoder and filesrc expose an (l) - (c) connection, - a thread is created to call the srcpad loop function. - - +---------+ +------------+ +----------+ - | filesrc | | avidemuxer | | fakesink | - | src--sink src--sink | - +---------+ +------------+ +----------+ - (l-g) (l) () (c) - - * fakesink has a chain function and the peer pad has no - loop function, no scheduling is done. - * avidemuxer and filesrc expose an (g) - (l) connection, - a thread is created to call the sinkpad loop function. - - +---------+ +----------+ +------------+ +----------+ - | filesrc | | identity | | avidemuxer | | fakesink | - | src--sink src--sink src--sink | - +---------+ +----------+ +------------+ +----------+ - (l-g) (c) () (l) () (c) - - * fakesink has a chain function and the peer pad has no - loop function, no scheduling is done. - * avidemuxer and identity expose no schedulable connection so - this pipeline is not schedulable. - - +---------+ +----------+ +------------+ +----------+ - | filesrc | | identity | | avidemuxer | | fakesink | - | src--sink src--sink src--sink | - +---------+ +----------+ +------------+ +----------+ - (l-g) (c-l) (g) (l) () (c) - - * fakesink has a chain function and the peer pad has no - loop function, no scheduling is done. - * avidemuxer and identity expose an (g) - (l) connection, - a thread is created to call the sinkpad loop function. - * identity knows the srcpad is getrange based and uses the - thread from avidemux to getrange data from filesrc. - - +---------+ +----------+ +------------+ +----------+ - | filesrc | | identity | | oggdemuxer | | fakesink | - | src--sink src--sink src--sink | - +---------+ +----------+ +------------+ +----------+ - (l-g) (c) () (l-c) () (c) - - * fakesink has a chain function and the peer pad has no - loop function, no scheduling is done. - * oggdemuxer and identity expose an () - (l-c) connection, - oggdemux has to operate in chain mode. - * identity chan only work chain based and so filesrc creates - a thread to push data to identity. +~~~~~ + +Inside the braces below the pads is stated what function the +pad support: + + l: exposes a loop function, so it can act as a pushing source. + g: exposes a getrange function + c: exposes a chain function + + following scheduling decisions are made based on the scheduling + methods exposed by the pads: + + (g) - (l): sinkpad will pull data from src + (l) - (c): srcpad actively pushes data to sinkpad + () - (c): srcpad will push data to sinkpad. + + () - () : not schedulable. + () - (l): not schedulable. + (g) - () : not schedulable. + (g) - (c): not schedulable. + (l) - () : not schedulable. + (l) - (l): not schedulable + + () - (g): impossible + (g) - (g): impossible. + (l) - (g): impossible + (c) - () : impossible + (c) - (g): impossible + (c) - (l): impossible + (c) - (c): impossible + + +---------+ +------------+ +-----------+ + | filesrc | | mp3decoder | | audiosink | + | src--sink src--sink | + +---------+ +------------+ +-----------+ + (l-g) (c) () (c) + + When activating the pads: + + * audiosink has a chain function and the peer pad has no + loop function, no scheduling is done. + * mp3decoder and filesrc expose an (l) - (c) connection, + a thread is created to call the srcpad loop function. + + +---------+ +------------+ +----------+ + | filesrc | | avidemuxer | | fakesink | + | src--sink src--sink | + +---------+ +------------+ +----------+ + (l-g) (l) () (c) + + * fakesink has a chain function and the peer pad has no + loop function, no scheduling is done. + * avidemuxer and filesrc expose an (g) - (l) connection, + a thread is created to call the sinkpad loop function. + + +---------+ +----------+ +------------+ +----------+ + | filesrc | | identity | | avidemuxer | | fakesink | + | src--sink src--sink src--sink | + +---------+ +----------+ +------------+ +----------+ + (l-g) (c) () (l) () (c) + + * fakesink has a chain function and the peer pad has no + loop function, no scheduling is done. + * avidemuxer and identity expose no schedulable connection so + this pipeline is not schedulable. + + +---------+ +----------+ +------------+ +----------+ + | filesrc | | identity | | avidemuxer | | fakesink | + | src--sink src--sink src--sink | + +---------+ +----------+ +------------+ +----------+ + (l-g) (c-l) (g) (l) () (c) + + * fakesink has a chain function and the peer pad has no + loop function, no scheduling is done. + * avidemuxer and identity expose an (g) - (l) connection, + a thread is created to call the sinkpad loop function. + * identity knows the srcpad is getrange based and uses the + thread from avidemux to getrange data from filesrc. + + +---------+ +----------+ +------------+ +----------+ + | filesrc | | identity | | oggdemuxer | | fakesink | + | src--sink src--sink src--sink | + +---------+ +----------+ +------------+ +----------+ + (l-g) (c) () (l-c) () (c) + + * fakesink has a chain function and the peer pad has no + loop function, no scheduling is done. + * oggdemuxer and identity expose an () - (l-c) connection, + oggdemux has to operate in chain mode. + * identity chan only work chain based and so filesrc creates + a thread to push data to identity. diff --git a/docs/design/part-seeking.txt b/docs/design/part-seeking.txt index 31c48aab0..699a8212b 100644 --- a/docs/design/part-seeking.txt +++ b/docs/design/part-seeking.txt @@ -58,7 +58,7 @@ more important than accurately producing all frames. Seeking in push based elements ------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -66,19 +66,21 @@ Seeking in push based elements Generating seeking events -------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~ A seek event is created with gst_event_new_seek (). +Seeking variants +~~~~~~~~~~~~~~~~ The different kinds of seeking methods and their internal workings are described below. FLUSH seeking -------------- +^^^^^^^^^^^^^ This is the most common way of performing a seek in a playback application. The application issues a seek on the pipeline and the new media is immediatly @@ -86,7 +88,7 @@ played after the seek calls returns. seeking without FLUSH ---------------------- +^^^^^^^^^^^^^^^^^^^^^ This seek type is typically performed after issuing segment seeks to finish the playback of the pipeline. @@ -97,13 +99,13 @@ sinks. segment seeking with FLUSH --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^ This seek is typically performed when starting seamless looping. segment seeking without FLUSH ------------------------------ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This seek is typically performed when continuing seamless looping. diff --git a/docs/design/part-segments.txt b/docs/design/part-segments.txt index afd0eb4cc..2dfbbb068 100644 --- a/docs/design/part-segments.txt +++ b/docs/design/part-segments.txt @@ -29,7 +29,7 @@ outside of the segment and that might therefore be discarded or clipped. Use case: FLUSHING seek ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ ex. @@ -95,13 +95,13 @@ Use case: FLUSHING seek Use case: live stream ---------------------- +~~~~~~~~~~~~~~~~~~~~~ Use case: segment looping -------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~ Consider the case of a wav file with raw audio. diff --git a/docs/design/part-sparsestreams.txt b/docs/design/part-sparsestreams.txt index 4ff1bf683..4770e90f1 100644 --- a/docs/design/part-sparsestreams.txt +++ b/docs/design/part-sparsestreams.txt @@ -2,7 +2,8 @@ DRAFT Sparse Streams -------------------- Introduction ------------- +~~~~~~~~~~~~ + In 0.8, there was some support for Sparse Streams through the use of FILLER events. These were used to mark gaps between buffers so that downstream elements could know not to expect any more data for that gap. @@ -11,7 +12,8 @@ In 0.10, segment information conveyed through NEWSEGMENT events can be used for the same purpose. Use cases ---------- +~~~~~~~~~ + 1) Sub-title streams Sub-title information from muxed formats such as Matroska or MPEG consist of irregular buffers spaced far apart compared to the other streams @@ -37,7 +39,8 @@ Use cases application that uses noise-gating (to save bandwith). Details -------- +~~~~~~~ + 1) Sub-title streams The main requirement here is to avoid stalling the pipeline between sub-title packets, and is effectively updating the minimum-timestamp for that stream. diff --git a/docs/design/part-standards.txt b/docs/design/part-standards.txt index 37eaf3d66..5d17185bc 100644 --- a/docs/design/part-standards.txt +++ b/docs/design/part-standards.txt @@ -6,7 +6,7 @@ collection has many potential pitfalls as far as the pointers go. Therefore, some standards must be adhered to as far as who owns what. Strings -------- +~~~~~~~ Arguments passed into a function are owned by the caller, and the function will make a copy of the string for its own internal use. The string should @@ -21,7 +21,7 @@ original and should be freed after usage by the caller. Objects -------- +~~~~~~~ Objects passed into a function are owned by the caller, any additional reference held to the object after leaving the function should increase the @@ -40,7 +40,7 @@ called should _free() or _unref() the object after usage. Iterators ---------- +~~~~~~~~~ When retrieving multiple objects from an object an iterator should be used. The iterator allows you to access the objects one after another while making diff --git a/docs/design/part-states.txt b/docs/design/part-states.txt index 345a8a820..ef1bbe5bc 100644 --- a/docs/design/part-states.txt +++ b/docs/design/part-states.txt @@ -1,5 +1,5 @@ States -====== +------ Both elements and pads can be in different states. The states of the pads are linked to the state of the element so the design of the states is mainly @@ -10,7 +10,7 @@ is initially instantiated, it is in the NULL state. State definitions ------------------ +~~~~~~~~~~~~~~~~~ - NULL: This is the initial state of an element. - READY: The element should be prepared to go to PAUSED. @@ -24,7 +24,7 @@ a downwards state change. State transitions ------------------ +~~~~~~~~~~~~~~~~~ the following state changes are possible: @@ -89,7 +89,7 @@ the following state changes are possible: State variables ---------------- +~~~~~~~~~~~~~~~ An element has 4 state variables that are protected with the object LOCK: @@ -111,7 +111,7 @@ _set_state(), called the STATE_LOCK. Setting state on elements -------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~ The state of an element can be changed with _element_set_state(). When chaning the state of an element all intermediate states will also be set on the element @@ -147,7 +147,7 @@ is immediatly returned to the caller. Getting state of elements -------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~ The _get_state() function takes 3 arguments, two pointers that will hold the current and pending state and one GstClockTime that holds a timeout value. The @@ -184,7 +184,7 @@ function returns a GstElementStateReturn. States in GstBin ----------------- +~~~~~~~~~~~~~~~~ A GstBin manages the state of its children. It does this by propagating the state changes performed on it to all of its children. The _set_state() function on a @@ -221,15 +221,15 @@ current state fields when it receives state messages from the children. Implementing states in elements -------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ READY ------ +^^^^^ upward state change -------------------- +~~~~~~~~~~~~~~~~~~~ Upward state changes always return ASYNC either if the STATE_PENDING is reached or not. @@ -262,7 +262,7 @@ Bin: to STATE_PENDING downward state change ----------------------- +~~~~~~~~~~~~~~~~~~~~~ Downward state changes only return ASYNC if the final state is ASYNC. This is to make sure that it's not needed to wait for an element to @@ -296,7 +296,7 @@ Bin: Locking overview (element) --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ * Element commiting SUCCESS @@ -376,8 +376,8 @@ Locking overview (element) | ... STREAM_UNLOCK -********************************************* -********************************************* +Remarks +~~~~~~~ set_state cannot be called from multiple threads at the same time. The STATE_LOCK prevents this. diff --git a/docs/design/part-stream-status.txt b/docs/design/part-stream-status.txt index a435a2291..66175fb00 100644 --- a/docs/design/part-stream-status.txt +++ b/docs/design/part-stream-status.txt @@ -26,7 +26,7 @@ We allow for the following scenarios: Use cases ---------- +~~~~~~~~~ * boost the priority of the udp receiver streaming thread @@ -56,7 +56,7 @@ Use cases Messages --------- +~~~~~~~~ The existing STREAM_STATUS message will be further defined and implemented in (selected) elements. The following fields will be contained in the message: @@ -101,7 +101,7 @@ Messages Events ------- +~~~~~~ diff --git a/docs/design/part-streams.txt b/docs/design/part-streams.txt index abc9fae0f..0a4fbb5a3 100644 --- a/docs/design/part-streams.txt +++ b/docs/design/part-streams.txt @@ -6,7 +6,7 @@ Streams Stream objects --------------- +~~~~~~~~~~~~~~ The following objects are to be expected in the streaming thread: @@ -21,7 +21,7 @@ and live sources. Typical stream --------------- +~~~~~~~~~~~~~~ A typical stream starts with a newsegment event that marks the buffer timestamp range. After that buffers are send one after the diff --git a/docs/design/part-synchronisation.txt b/docs/design/part-synchronisation.txt index 54e250508..2c3185b54 100644 --- a/docs/design/part-synchronisation.txt +++ b/docs/design/part-synchronisation.txt @@ -12,7 +12,7 @@ Synchronisation in a GstPipeline is achieved using the following 3 components: A GstClock ----------- +~~~~~~~~~~ This object provides a counter that represents the current time in nanoseconds. This value is called the absolute_time. @@ -33,7 +33,7 @@ A GstClock always counts time upwards and does not necessarily start at 0. Running time ------------- +~~~~~~~~~~~~ After a pipeline selected a clock it will maintain the running_time based on the selected clock. This running_time represents the total time spent in the PLAYING @@ -66,7 +66,7 @@ This value is monotonically increasing at the rate of the clock. Timestamps ----------- +~~~~~~~~~~ The GstBuffer timestamps and the preceeding NEW_SEGMENT event (See part-streams.txt) define a transformation of the buffer timestamps to @@ -112,7 +112,7 @@ NS.stop and NS.accum == 0). Synchronisation ---------------- +~~~~~~~~~~~~~~~ As we have seen, we can get a running_time: @@ -161,7 +161,7 @@ synchronized buffers have the same timestamps. Stream time ------------ +~~~~~~~~~~~ The stream time is also known as the position in the stream and is a value between 0 and the total duration of the media file. diff --git a/docs/design/part-trickmodes.txt b/docs/design/part-trickmodes.txt index 8e14dddf5..e188ee234 100644 --- a/docs/design/part-trickmodes.txt +++ b/docs/design/part-trickmodes.txt @@ -1,5 +1,5 @@ Trickmodes -========== +---------- GStreamer provides API for performing various trickmode playback. This includes: @@ -18,7 +18,7 @@ Seeking can both be done in a playback pipeline and a transcoding pipeline. General seeking overview ------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~ Consider a typical playback pipeline: @@ -35,56 +35,60 @@ position 0 and stopping at the total duration of the file. When performing a seek, the following steps have to be taken by the application: - 1) Create a seek event: +Create a seek event +^^^^^^^^^^^^^^^^^^^ - The seek event contains: +The seek event contains: - - various flags describing: - - where to seek to (KEY_UNIT) - - how accurate the seek should be (ACCURATE) - - how to perform the seek (FLUSH) - - what to do when the stop position is reached (SEGMENT). - - extra playback options (SKIP) - - a format to seek in, this can be time, bytes, units (frames, samples), ... - - a playback rate, 1.0 is normal playback speed, positive values bigger than 1.0 - mean fast playback. negative values mean reverse playback. A playback speed of - 0.0 is not allowed (but is equivalent to PAUSING the pipeline). - - a start position, this value has to be between 0 and the total duration of the - file. It can also be relative to the previously configured start value. - - a stop position, this value has to be between 0 and the total duration. It can - also be relative to the previously configured stop value. +- various flags describing: + - where to seek to (KEY_UNIT) + - how accurate the seek should be (ACCURATE) + - how to perform the seek (FLUSH) + - what to do when the stop position is reached (SEGMENT). + - extra playback options (SKIP) +- a format to seek in, this can be time, bytes, units (frames, samples), ... +- a playback rate, 1.0 is normal playback speed, positive values bigger than 1.0 + mean fast playback. negative values mean reverse playback. A playback speed of + 0.0 is not allowed (but is equivalent to PAUSING the pipeline). +- a start position, this value has to be between 0 and the total duration of the + file. It can also be relative to the previously configured start value. +- a stop position, this value has to be between 0 and the total duration. It can + also be relative to the previously configured stop value. - See also gst_event_new_seek(). +See also gst_event_new_seek(). - 2) Send the seek event to the pipeline with gst_element_send_event() +Send the seek event +^^^^^^^^^^^^^^^^^^^ - By default the pipeline will send the event to all sink elements. - By default an element will forward the event upstream on all sinkpads. - Elements can modify the format of the seek event. The most common format is - GST_FORMAT_TIME. +Send the new seek event to the pipeline with gst_element_send_event(). - One element will actually perform the seek, this is usually the demuxer or - source element. For more information on how to perform the different seek - types see part-seeking.txt. +By default the pipeline will send the event to all sink elements. +By default an element will forward the event upstream on all sinkpads. +Elements can modify the format of the seek event. The most common format is +GST_FORMAT_TIME. - For client side trickmode a NEW_SEGMENT event will be sent downstream with - the new rate and start/stop positions. All elements prepare themselves to - handle the rate (see below). The applied rate of the NEW_SEGMENT event will - be set to 1.0 to indicate that no rate adjustment has been done. +One element will actually perform the seek, this is usually the demuxer or +source element. For more information on how to perform the different seek +types see part-seeking.txt. - for server side trick mode a NEW_SEGMENT event is sent downstream with a - rate of 1.0 and the start/stop positions. The elements will configure themselves - for normal playback speed since the server will perform the rate conversions. - The applied rate will be set to the rate that will be applied by the server. This - is done to insure that the position reporting performed in the sink is aware - of the trick mode. +For client side trickmode a NEW_SEGMENT event will be sent downstream with +the new rate and start/stop positions. All elements prepare themselves to +handle the rate (see below). The applied rate of the NEW_SEGMENT event will +be set to 1.0 to indicate that no rate adjustment has been done. - When the seek succeeds, the _send_event() function will return TRUE. +for server side trick mode a NEW_SEGMENT event is sent downstream with a +rate of 1.0 and the start/stop positions. The elements will configure themselves +for normal playback speed since the server will perform the rate conversions. +The applied rate will be set to the rate that will be applied by the server. This +is done to insure that the position reporting performed in the sink is aware +of the trick mode. + +When the seek succeeds, the _send_event() function will return TRUE. Server side trickmode ---------------------- +~~~~~~~~~~~~~~~~~~~~~ The source element operates in push mode. It can reopen a server connection requesting a new byte or time position and a new playback speed. The capabilities can be queried @@ -131,7 +135,7 @@ playback speed or direction. client side forward trickmodes ------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The seek happens as stated above. a NEW_SEGMENT event is sent downstream with a rate different from 1.0. Plugins receiving the NEW_SEGMENT can decide to perform the @@ -154,7 +158,7 @@ case, the demuxer would scale the timestamps and would set an applied rate of S. client side backwards trickmode -------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For backwards playback the following rules apply: @@ -202,7 +206,8 @@ For plugins the following rules apply: In SKIP mode, the same algorithm as for forward SKIP mode can be used. -Notes: +Notes +~~~~~ - The clock/running_time keeps running forward. - backwards playback potentially uses a lot of memory as frames and undecoded |