1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
|
# Probes
Probes are callbacks that can be installed by the application and will notify
the application about the states of the dataflow.
## Requirements
Applications should be able to monitor and control the dataflow on pads.
We identify the following types:
- be notified when the pad is/becomes idle and make sure the pad stays
idle. This is essential to be able to implement dynamic relinking of
elements without breaking the dataflow.
- be notified when data, events or queries are pushed or sent on a
pad. It should also be possible to inspect and modify the data.
- be able to drop, pass and block on data based on the result of the
callback.
- be able to drop, pass data on blocking pads based on methods
performed by the application
thread.
## Overview
The function `gst_pad_add_probe()` is used to add a probe to a pad. It accepts a
probe type mask and a callback.
``` c
gulong gst_pad_add_probe (GstPad *pad,
GstPadProbeType mask,
GstPadProbeCallback callback,
gpointer user_data,
GDestroyNotify destroy_data);
```
The function returns a gulong that uniquely identifies the probe and that can
be used to remove the probe with `gst_pad_remove_probe()`:
``` c
void gst_pad_remove_probe (GstPad *pad, gulong id);
```
The mask parameter is a bitwise or of the following flags:
``` c
typedef enum
{
GST_PAD_PROBE_TYPE_INVALID = 0,
/* flags to control blocking */
GST_PAD_PROBE_TYPE_IDLE = (1 << 0),
GST_PAD_PROBE_TYPE_BLOCK = (1 << 1),
/* flags to select datatypes */
GST_PAD_PROBE_TYPE_BUFFER = (1 << 4),
GST_PAD_PROBE_TYPE_BUFFER_LIST = (1 << 5),
GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM = (1 << 6),
GST_PAD_PROBE_TYPE_EVENT_UPSTREAM = (1 << 7),
GST_PAD_PROBE_TYPE_EVENT_FLUSH = (1 << 8),
GST_PAD_PROBE_TYPE_QUERY_DOWNSTREAM = (1 << 9),
GST_PAD_PROBE_TYPE_QUERY_UPSTREAM = (1 << 10),
/* flags to select scheduling mode */
GST_PAD_PROBE_TYPE_PUSH = (1 << 12),
GST_PAD_PROBE_TYPE_PULL = (1 << 13),
} GstPadProbeType;
```
When adding a probe with the IDLE or BLOCK flag, the probe will become a
blocking probe (see below). Otherwise the probe will be a DATA probe.
The datatype and scheduling selector flags are used to select what kind of
datatypes and scheduling modes should be allowed in the callback.
The blocking flags must match the triggered probe exactly.
The probe callback is defined as:
``` c
GstPadProbeReturn (*GstPadProbeCallback) (GstPad *pad, GstPadProbeInfo *info,
gpointer user_data);
```
A probe info structure is passed as an argument and its type is guaranteed
to match the mask that was used to register the callback. The data item in the
info contains type specific data, which is usually the data item that is blocked
or `NULL` when no data item is present.
The probe can return any of the following return values:
``` c
typedef enum
{
GST_PAD_PROBE_DROP,
GST_PAD_PROBE_OK,
GST_PAD_PROBE_REMOVE,
GST_PAD_PROBE_PASS,
GST_PAD_PROBE_HANDLED
} GstPadProbeReturn;
```
`GST_PAD_PROBE_OK` is the normal return value. `_DROP` will drop the item that is
currently being probed. `GST_PAD_PROBE_REMOVE`: remove the currently executing probe from the
list of probes.
`GST_PAD_PROBE_PASS` is relevant for blocking probes and will temporarily unblock the
pad and let the item through, it will then block again on the next item.
## Blocking probes
Blocking probes are probes with `BLOCK` or `IDLE` flags set. They will always
block the dataflow and trigger the callback according to the following rules:
When the `IDLE` flag is set, the probe callback is called as soon as no data is
flowing over the pad. If at the time of probe registration, the pad is idle,
the callback will be called immediately from the current thread. Otherwise,
the callback will be called as soon as the pad becomes idle in the streaming
thread.
The `IDLE` probe is useful to perform dynamic linking, allowing for a sufficinet
wait for a safe moment when an unlink/link operation can be done. Since the probe is a
blocking probe, it will also make sure that the pad stays idle until the probe
is removed.
When the `BLOCK` flag is set, the probe callback will be called when new data
arrives on the pad and right before the pad goes into the blocking state. This
callback is thus only called when there is new data on the pad.
The blocking probe is removed with `gst_pad_remove_probe()` or when the probe
callback return `GST_PAD_PROBE_REMOVE`. In both cases, and if this was the last
blocking probe on the pad, the pad is unblocked and dataflow can continue.
## Non-Blocking probes
Non-blocking probes or DATA probes are probes triggered when data is flowing
over the pad. They are called after the blocking probes are run and always with
data.
## Push dataflow
Push probes have the `GST_PAD_PROBE_TYPE_PUSH` flag set in the
callbacks.
In push based scheduling, the blocking probe is called first with the
data item. Then the data probes are called before the peer pad chain or
event function is called.
The data probes are called before the peer pad is checked. This allows
for linking the pad in either the BLOCK or DATA probes on the pad.
Before the peerpad chain or event function is called, the peer pad block
and data probes are called.
Finally, the `IDLE` probe is called on the pad after the data was sent to
the peer pad.
The push dataflow probe behavior is the same for buffers and
bidirectional events.
```
pad peerpad
| |
gst_pad_push() / | |
gst_pad_push_event() | |
-------------------->O |
O |
flushing? O |
FLUSHING O |
< - - - - - - O |
O-> do BLOCK probes |
O |
O-> do DATA probes |
no peer? O |
NOT_LINKED O |
< - - - - - - O |
O gst_pad_chain() / |
O gst_pad_send_event() |
O------------------------------>O
O flushing? O
O FLUSHING O
O< - - - - - - - - - - - - - - -O
O O-> do BLOCK probes
O O
O O-> do DATA probes
O O
O O---> chainfunc /
O O eventfunc
O< - - - - - - - - - - - - - - -O
O |
O-> do IDLE probes |
O |
< - - - - - - O |
| |
```
## Pull dataflow
Pull probes have the `GST_PAD_PROBE_TYPE_PULL` flag set in the
callbacks.
The `gst_pad_pull_range()` call will first trigger the `BLOCK` probes
without a `DATA` item. This allows the pad to be linked before the peer
pad is resolved. It also allows the callback to set a data item in the
probe info.
After the blocking probe and the getrange function is called on the peer
pad and there is a data item, the DATA probes are called.
When control returns to the sinkpad, the `IDLE` callbacks are called. The
`IDLE` callback is called without a data item so that it will also be
called when there was an error.
If there is a valid `DATA` item, the `DATA` probes are called for the item.
```
srcpad sinkpad
| |
| | gst_pad_pull_range()
| O<---------------------
| O
| O flushing?
| O FLUSHING
| O - - - - - - - - - - >
| do BLOCK probes <-O
| O no peer?
| O NOT_LINKED
| O - - - - - - - - - - >
| gst_pad_get_range() O
O<------------------------------O
O O
O flushing? O
O FLUSHING O
O- - - - - - - - - - - - - - - >O
do BLOCK probes <-O O
O O
getrangefunc <---O O
O flow error? O
O- - - - - - - - - - - - - - - >O
O O
do DATA probes <-O O
O- - - - - - - - - - - - - - - >O
| O
| do IDLE probes <-O
| O flow error?
| O - - - - - - - - - - >
| O
| do DATA probes <-O
| O - - - - - - - - - - >
| |
```
## Queries
Query probes have the `GST_PAD_PROBE_TYPE_QUERY_*` flag set in the
callbacks.
```
pad peerpad
| |
gst_pad_peer_query() | |
-------------------->O |
O |
O-> do BLOCK probes |
O |
O-> do QUERY | PUSH probes |
no peer? O |
FALSE O |
< - - - - - - O |
O gst_pad_query() |
O------------------------------>O
O O-> do BLOCK probes
O O
O O-> do QUERY | PUSH probes
O O
O O---> queryfunc
O error O
<- - - - - - - - - - - - - - - - - - - - - - -O
O O
O O-> do QUERY | PULL probes
O< - - - - - - - - - - - - - - -O
O |
O-> do QUERY | PULL probes |
O |
< - - - - - - O |
| |
```
For queries, the `PUSH` `ProbeType` is set when the query is traveling to
the object that will answer the query and the `PULL` type is set when the
query contains the answer.
## Use-cases
### Prerolling a partial pipeline
```
.---------. .---------. .----------.
| filesrc | | demuxer | .-----. | decoder1 |
| src -> sink src1 ->|queue|-> sink src
'---------' | | '-----' '----------' X
| | .----------.
| | .-----. | decoder2 |
| src2 ->|queue|-> sink src
'---------' '-----' '----------' X
```
The purpose is to create the pipeline dynamically up to the decoders but
not yet connect them to a sink and without losing any data.
To do this, the source pads of the decoders is blocked so that no events
or buffers can escape and we don’t interrupt the stream.
When all of the dynamic pads are created (no-more-pads emitted by the
branching point, ie, the demuxer or the queues filled) and the pads are
blocked (blocked callback received) the pipeline is completely
prerolled.
It should then be possible to perform the following actions on the
prerolled pipeline:
- query duration/position
- perform a flushing seek to preroll a new position
- connect other elements and unblock the blocked pads.
### dynamically switching an element in a PLAYING pipeline
```
.----------. .----------. .----------.
| element1 | | element2 | | element3 |
... src -> sink src -> sink ...
'----------' '----------' '----------'
.----------.
| element4 |
sink src
'----------'
```
The purpose is to replace element2 with element4 in the `PLAYING`
pipeline.
1) block element1 src pad.
2) inside the block callback nothing is flowing between
element1 and element2 and nothing will flow until unblocked.
3) unlink element1 and element2
4) optional step: make sure data is flushed out of element2:
4a) pad event probe on element2 src
4b) send `EOS` to element2, this makes sure that element2 flushes out the last bits of data it holds.
4c) wait for `EOS` to appear in the probe, drop the `EOS`.
4d) remove the `EOS` pad event probe.
5) unlink element2 and element3
5a) optionally element2 can now be set to `NULL` and/or removed from the pipeline.
6) link element4 and element3
7) link element1 and element4
8) make sure element4 is in the same state as the rest of the elements. The
element should at least be `PAUSED`.
9) unblock element1 src
The same flow can be used to replace an element in a `PAUSED` pipeline. Of
course in a `PAUSED` pipeline there might not be dataflow so the block
might not immediately happen.
|
