1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
|
[[!toc 3]]
# CameraBin 2
Current camerabin is a single element and has some complex code with input/output-selectors for controlling the flow of buffers. Making it more modularized and with less -selectors would simplify the code a lot. We hope caps negotiation will be simplified, too. And more tricky features could be implemented inside still/video capturing modules(bins). We could also switch a whole sub-bins to ones with a different implementations. The multi-pad source design enables optimizations for hardware that can stream simultaneously on all pads.
Even though the new camerabin will provide the same basic functions as the current one (and have quite similar API), it will be built from ground up again with a different design. We should make it camerabin2 and take it in parallel with camerabin. This way we avoid major regressions and api changes hassle to applications using current camerabin. When they feel ready, the switch to camerabin2 wouldn't be hard. Also, we can keep bugfixing on old camerabin and keeping everything using it working.
## Modules
[[CameraBin2|CameraBin2]] will have 4 internal modules (bins), it should be possible to use any of them outside of camerabin2 (applications, like Empathy and Cheese could make use of those):
[[!img CameraBin2.png].
### CameraSrc
* It's a bin with 3 output pads for parallel flow (video recording, image capture and viewfinder)
* Need to do some baseclass, and there should be an adapter element that would use current video source elements with a single srcpad internally (v4l2src, videotestsrc) and provide 3 output pads as [[CameraBin2|CameraBin2]] expects.
* Might implement gstreamer's Photography interface
The embedded world has complex camera drivers that offer more functionality that classic v4l2. E.g. they might have parallel outputs at the same time. The idea here is to make a more complex camerasource api. This could be implemented as a linux-camera-source (as a bin), reusing what we have in CameraBin, or be implemented on top of a different api providing these features as a monolithic block (e.g. gst-omx based camera-source).
This should be done as a baseclass (subclassing a [[GstBin|GstBin]]), with the following features:
* 3 static source pads: image, viewfinder, video (as [[GstGhostpads|GstGhostpads]])
* Previewing using 2 sometimes source pads (image_preview, video_preview) or Messages
* Do we want to proxy interfaces or application uses them directly from the elements? (photography, colorbalance, ...)
This would be the potential class hierarchy:
[[!format txt """
GstBin
GstBaseCameraSrc
GstSimpleV4l2CameraSrc
GstOmxCameraSrc
...
"""]]
If the actual camera plugin is already implemented extending e.g. [[GstPushSrc|GstPushSrc]], then one would still subclass a new element from [[GstBaseCameraSrc|GstBaseCameraSrc]] and insert the [[GstPushSrc|GstPushSrc]] based camera element there. In the most simple case the wrapper for a powerful hardware based camera source proxies gobject properties and links the internal element pads to the bins ghostpads. We would definitely want to have a [[GstSimpleV4l2CameraSrc|GstSimpleV4l2CameraSrc]] as a reference implementation and also providing legacy/fallback support.
### Image Capture Bin
Received the image buffers from the camera source, encodes them and saves to disk. Must have queues internally to allow further captures to be taken while others are encoding.
* Handles eos/tags between buffers/image captures
* Possibility of multiple implementations:
* A sequential encoding implementation using queues
* Parallel encoding (pool of pipelines)
* Other ones for trickier cases (e.g. panorama pictures)
* If we allow multiple implementations: How to guarantee a basic common API here? Do we need to guarantee it?
* Need to do tagging to multiple encoders. e.g. We use jifmux for jpg, what are we using for png?
### Video Recording Bin
Responsible for recording video and audio. The video buffers are generated from the camera source, audio is recorded from a internal audio source element (that might be muted by the application).
* There's an audio source element in this bin (be careful with the clock selection/distribution)
* Newsegment/timestamp handling on recordings was tricky/buggy in current camerabin
* Encodebin could work here
### Viewfinder Bin
* Contains the videosink and any image processing filters/analyzers the application might set
* How useful is it outside of camerabin? It's simply a branch of elements. (Hot-swapping of effects?)
## Use Cases
* Basic picture capture: User sets capture resolution to camerabin2, that forwards it to the camerasrc. User also sends the capture filename to camerabin2, which forwards it to the image capture branch. User then sends the capture signal and camerabin forwards it to the camerasrc. Camerasrc pushes a buffer through its imagesrc and it gets to the image encoding branch. The image encoding branch encodes, puts tags into the image and sends an EOS to the filesink so that it finishes the file.
* Basic video capture: User sets capture resolution and fps to camerabin2 and it forwards it to the camerasrc. User sets the capture filename, which is passed to the video recording bin. When the user sends the capture start signal, camerasrc starts pushing buffers through its videosrc pad. The audiosrc inside the videobin also starts recording the audio in sync with the video buffers. The viewfinder must still keep running during this. Recording continues until the user sends the capture stop signal. At this point, eos is pushed on the video recording bin and the file is finalized. It should be possible to pause and continue the recording. It should also be possible to mute audio recording.
* Panorama: Panorama pictures are not planned to be supported natively on camerabin2, it could, however, be implemented as a image capture module and replace the default module whenever the user would like to take panorama pics. Currently an external application has to take care of the pictures merging. From camerabin2, an application needs to be able to control camera parameters (AWB, AF and AE) so that subsequent captures use the same values. This is possible by using the 'manual' mode for these properties in the Photography interface. It is also possible to set it to 'manual' after the first capture so that the 3A values are automatically set up for the first capture and maintained as that for the other captures.
* Continuous shooting: In this mode, camerabin2 has little to do. This is handled by the camera source module, by setting it to 'continuous shooting' mode it will use faster algorithms to do 3A, thus reducing shot to shot time. The rest of camerabin2 will work as usual.
* <add your favorite use case here>
## Other features
* Digital zooming (with Photography interface or gstreamer elements)
* Night mode adjustment (Use lower framerates to allow more exposure time and have a 'lighter' image)
* Querying available capture formats
* Autopick framerate
* Highest if not specified
* rounding (when user selected fps isn't available)
* colorspace/scalers to guarantee compatibility between elements
* Hot-swapping of effects elements.
## Intentionally not handled
* Cameras that generate non-raw input
* Hot swapping of inputs/camera. This can be done by setting it to NULL and changing.
## Open Questions
* How to do effects hot swapping? cheese and empathy want it.
* Do we want a explicit prepare-for-capture step? To be set when the user finishes setting up its preferences?
* Use sometimes pads or messages for previewing on the source?
* How to map [[CameraBin2|CameraBin2]] flags to the internal bins flags?
* Detection of video sources? (Improve autovideosrc? This would be nice for Cheese)
* Effects performance
* Camerabin can do effects on all branches with a single element. Camerabin2 would need one effect element per branch.
* Proper benchmarking needed here.
* Providing effects on the source bin of camerabin2 if it supports it would help here.
## Links
* Rob Clark's camerabin branch: [[http://gitorious.org/robclark-gstreamer/gst-plugins-bad|http://gitorious.org/robclark-gstreamer/gst-plugins-bad]]
* Refactored camerabin to this new design
* Log of camerabin2 meeting on 07/12/2010 (dd/mm/yyyy): [[http://people.collabora.co.uk/~thiagoss/camerabin2_20101207.log|http://people.collabora.co.uk/~thiagoss/camerabin2_20101207.log]]
# CameraBin
## TODO
At some point we'd like to move CameraBin to gst-plugins-good. CameraBin already has unit tests and documentation. It is currently in use on Nokia N900 and is considered for cheese. This page lists things that should be done before declaring it stable.
## dataflow
In current CameraBin we have a link from videobin back to viewfinder to so what we record. When we do the camera-source baseclass, we should move the input-selector to the [[GstSimpleV4l2CameraSrc|GstSimpleV4l2CameraSrc]] implementation or even get rid of the selectors and provide paralell dataflow if needed. Keeping the viewfinder active is important for the cases where one has analyzers in e.g. the viewfinder-slot to get continuous analysis results in videorecording or continuous shooting.
Open items here:
* in still capture mode we might want to set the queue in the still image camera source output to "leaky" and block its src pad. We would only unblock the on shutter press for the seleccted amount of images (burstmode). We could also use videorate or even a specialized videogate element here. What the element should do:
* only let n frames through (n=0...)
* drop frames to produce a certain output rate
## focusing
We need more focus mode. Right now the API can start AF and get a signal when AF is done. There are also camera subsystems providing continuous focusing, where we would just lock the AF on half press, adjust frame and shot.
## continuous shooting
We would set the queue to leaky and block the capture pad. In contrast to single capture mode, we would keep capture-setting frozen and let a frame through, each time the capture button is pressed.
## bracketing
Lock settings, take various shots with some settings changed, merge single images with some filter. We can use the GstController for scheduling the parameter changes. Some issues with that though:
* the camerabin pipeline is always running - we need a newsegment on the image capture pipeline, so that timestamps start with zero
* the camerasrc is a live-src, thus it is hard to align the control-changes with frame times ( see [[bug 610338|https://bugzilla.gnome.org/show_bug.cgi?id=610338]] and [[bug 616173|https://bugzilla.gnome.org/show_bug.cgi?id=616173]]).
* v4l2 has no api yet to do timestamped CID changes
## recording timestamps
Try pushing a newsegment to videobin, instead of rebasing timestamps with a buffer-probe.
## Open Items
### effect preview
When adding effects to imagebin or videobin, we might want to get a preview on viewfinder for those too. We either need to apply certain effect in the source already or add them twice.
### isp internal effects and effect selection
Right now camerabin offers slots for setting effects on each of the imagebin, viewfinderbin and videobin. Some ISPs (image signal processor) come with internal effects (e.g. videostabilisation). Effect settings should then be exposed as gobject properties on the camera source, so that a camera ui can set a gstreamer plugin as an effect and link the ui to the gobject properties or activate the isp internal effect and connect the ui to the properties of that.
The effect parameters should not be added to [[GstPhotography|GstPhotography]] to avoid it growing unbounded. [[GstPhotography|GstPhotography]] is for capture settings. This is needed for post processing effects.
### highspeed capture
We might need a videorate element infront of the viewfinder bin to reduce the framerate there.
---
TODO: add more from [[http://live.gnome.org/Cheese/ThreeZero?action=AttachFile&do=get&target=cheese-threezero-meeting-log.txt|http://live.gnome.org/Cheese/ThreeZero?action=AttachFile&do=get&target=cheese-threezero-meeting-log.txt]]
|
