WIPVideo

Version 32 (Saúl Ibarra Corretgé, 04/22/2014 02:41 pm)

1 1 Saúl Ibarra Corretgé
h1. WIPVideo
2 1 Saúl Ibarra Corretgé
3 5 Saúl Ibarra Corretgé
Notes while video is a work in progress. Repository: http://devel.ag-projects.com/cgi-bin/darcsweb.cgi?r=saul/python-sipsimple-video;a=summary
4 1 Saúl Ibarra Corretgé
5 1 Saúl Ibarra Corretgé
h2. Dependencies
6 1 Saúl Ibarra Corretgé
7 1 Saúl Ibarra Corretgé
The following dependencies are required to build PJSIP with video support (including H264)
8 1 Saúl Ibarra Corretgé
9 1 Saúl Ibarra Corretgé
* SDL 2
10 1 Saúl Ibarra Corretgé
* ffmpeg (libavformat, libswscale, libavcodec, libavutil)
11 1 Saúl Ibarra Corretgé
* libx264
12 1 Saúl Ibarra Corretgé
13 1 Saúl Ibarra Corretgé
Versions I have tried:
14 1 Saúl Ibarra Corretgé
15 1 Saúl Ibarra Corretgé
* SDL (2.0.0-7655)
16 1 Saúl Ibarra Corretgé
* ffmpeg (2.0 release)
17 1 Saúl Ibarra Corretgé
* libx264 (snapshot-20130806-2245-stable)
18 1 Saúl Ibarra Corretgé
19 1 Saúl Ibarra Corretgé
h2. Patches
20 1 Saúl Ibarra Corretgé
21 11 Saúl Ibarra Corretgé
-If the above versions are used, PJSIP needs to be patched with the attached patch (avcodec.diff) or it won't compile. This does not occur when compiling it against the latest library versions on Debian unstable.- No longer needed.
22 3 Saúl Ibarra Corretgé
23 20 Saúl Ibarra Corretgé
h2. Installing dependencies (Debian / Ubuntu systems)
24 20 Saúl Ibarra Corretgé
25 20 Saúl Ibarra Corretgé
The situation here is a bit sad. Both Debian and Ubuntu ship with libav instead of FFmpeg, but libraries are called the same. PJSIP had to be patched in order to properly work with libav, and the patch as not yet been included upstream.
26 20 Saúl Ibarra Corretgé
27 20 Saúl Ibarra Corretgé
On Debian, when the Debian-Multimedia repositories are used (quite common) you get FFmpeg and not libav. Oh the joy!
28 20 Saúl Ibarra Corretgé
29 20 Saúl Ibarra Corretgé
Installing dependencies on Debian:
30 20 Saúl Ibarra Corretgé
31 20 Saúl Ibarra Corretgé
<pre>
32 32 Saúl Ibarra Corretgé
apt-get install libv4l-dev libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libsdl2-dev libx264-dev libavcodec-extra
33 20 Saúl Ibarra Corretgé
</pre>
34 20 Saúl Ibarra Corretgé
35 20 Saúl Ibarra Corretgé
If using the Debian-Multimedia repositories, do not install libavcodec-extra.
36 20 Saúl Ibarra Corretgé
37 20 Saúl Ibarra Corretgé
Installing dependencies on Ubuntu:
38 20 Saúl Ibarra Corretgé
39 20 Saúl Ibarra Corretgé
<pre>
40 32 Saúl Ibarra Corretgé
apt-get install libv4l-dev libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libsdl2-dev libx264-dev libavcodec-extra53
41 20 Saúl Ibarra Corretgé
</pre>
42 20 Saúl Ibarra Corretgé
43 21 Saúl Ibarra Corretgé
*Note on H.264 support*: In order to have H.264 support, FFmpeg (or libav) need to be compiled with support for it. The standard packages don't, hence the need for installing the libavcodec-extra packages.
44 20 Saúl Ibarra Corretgé
45 3 Saúl Ibarra Corretgé
h2. Compiling dependencies
46 3 Saúl Ibarra Corretgé
47 3 Saúl Ibarra Corretgé
All dependencies will be compiled to a directory in the user's HOME directory:
48 3 Saúl Ibarra Corretgé
49 3 Saúl Ibarra Corretgé
<pre>
50 3 Saúl Ibarra Corretgé
export MY_VIDEO_LIBS=$HOME/work/ag-projects/video/local
51 3 Saúl Ibarra Corretgé
</pre>
52 3 Saúl Ibarra Corretgé
53 3 Saúl Ibarra Corretgé
h3. libx264
54 3 Saúl Ibarra Corretgé
55 3 Saúl Ibarra Corretgé
<pre>
56 3 Saúl Ibarra Corretgé
./configure --enable-shared --disable-avs --disable-swscale --disable-lavf --disable-ffms --disable-gpac --prefix=$MY_VIDEO_LIBS
57 3 Saúl Ibarra Corretgé
make
58 3 Saúl Ibarra Corretgé
make install
59 8 Saúl Ibarra Corretgé
60 8 Saúl Ibarra Corretgé
# If a 32bit build is wanted, then run this configure instead:
61 8 Saúl Ibarra Corretgé
./configure --host=i386-apple-darwin --enable-shared --disable-avs --disable-swscale --disable-lavf --disable-ffms --disable-gpac --prefix=$MY_VIDEO_LIBS
62 3 Saúl Ibarra Corretgé
</pre>
63 3 Saúl Ibarra Corretgé
64 3 Saúl Ibarra Corretgé
h3. ffmpeg
65 3 Saúl Ibarra Corretgé
66 3 Saúl Ibarra Corretgé
<pre>
67 3 Saúl Ibarra Corretgé
# Some exports
68 3 Saúl Ibarra Corretgé
export PKG_CONFIG_PATH=$MY_VIDEO_LIBS/lib/pkgconfig
69 1 Saúl Ibarra Corretgé
70 7 Saúl Ibarra Corretgé
./configure --enable-shared --disable-static --enable-memalign-hack --enable-gpl --enable-libx264 --prefix=$MY_VIDEO_LIBS --extra-cflags="`pkg-config --cflags x264`" --extra-ldflags="`pkg-config --libs x264`"
71 3 Saúl Ibarra Corretgé
make
72 3 Saúl Ibarra Corretgé
make install
73 9 Saúl Ibarra Corretgé
74 9 Saúl Ibarra Corretgé
# If a 32bit build is wanted do:
75 9 Saúl Ibarra Corretgé
./configure --enable-shared --disable-static --enable-memalign-hack --enable-gpl --enable-libx264 --prefix=$MY_VIDEO_LIBS --extra-cflags="`pkg-config --cflags x264`" --extra-ldflags="`pkg-config --libs x264`" --cc="gcc -m32" --disable-asm
76 9 Saúl Ibarra Corretgé
# TODO: I WANT MY ASM
77 3 Saúl Ibarra Corretgé
</pre>
78 3 Saúl Ibarra Corretgé
79 3 Saúl Ibarra Corretgé
h3. SDL
80 3 Saúl Ibarra Corretgé
81 27 Saúl Ibarra Corretgé
82 31 Adrian Georgescu
*Note for Cocoa*: There is a bug in SDL, when a Cocoa window is closed, it brings up the next window in the z stack. There is no obvious solution for this, see SDL_cocoawindow.m
83 26 Adrian Georgescu
84 3 Saúl Ibarra Corretgé
<pre>
85 3 Saúl Ibarra Corretgé
./configure --disable-audio --prefix=$MY_VIDEO_LIBS
86 3 Saúl Ibarra Corretgé
make
87 3 Saúl Ibarra Corretgé
make install
88 10 Saúl Ibarra Corretgé
89 10 Saúl Ibarra Corretgé
# If a 32bit build is wanted:
90 10 Saúl Ibarra Corretgé
CFLAGS="-arch i386" CXXFLAGS="-arch i386" LDFLAGS="-arch i386" ./configure --disable-audio --prefix=$MY_VIDEO_LIBS
91 3 Saúl Ibarra Corretgé
</pre>
92 4 Saúl Ibarra Corretgé
93 4 Saúl Ibarra Corretgé
h2. Compiling PJSIP (pjsua, for testing)
94 4 Saúl Ibarra Corretgé
95 4 Saúl Ibarra Corretgé
<pre>
96 4 Saúl Ibarra Corretgé
svn co http://svn.pjsip.org/repos/pjproject/trunk pjsip
97 4 Saúl Ibarra Corretgé
cd pjsip
98 4 Saúl Ibarra Corretgé
echo "#define PJMEDIA_HAS_VIDEO 1" > pjlib/include/pj/config_site.h
99 4 Saúl Ibarra Corretgé
./configure --with-sdl=$MY_VIDEO_LIBS --with-ffmpeg=$MY_VIDEO_LIBS
100 22 Saúl Ibarra Corretgé
# if a 32bit build is wanted:
101 22 Saúl Ibarra Corretgé
# CFLAGS="-arch i386" CXXFLAGS="-arch i386" LDFLAGS="-arch i386" ./configure --with-sdl=$MY_VIDEO_LIBS --with-ffmpeg=$MY_VIDEO_LIBS
102 4 Saúl Ibarra Corretgé
make dep
103 4 Saúl Ibarra Corretgé
make
104 4 Saúl Ibarra Corretgé
# pjsua will be located in pjsip-apps/bin/
105 4 Saúl Ibarra Corretgé
</pre>
106 12 Saúl Ibarra Corretgé
107 12 Saúl Ibarra Corretgé
h2. Proposed API
108 12 Saúl Ibarra Corretgé
109 13 Saúl Ibarra Corretgé
API for video components is based on 2 different types of video capable entities:
110 13 Saúl Ibarra Corretgé
111 13 Saúl Ibarra Corretgé
* VideoProducer: a source for video data, for example a video camera or a remote video stream
112 13 Saúl Ibarra Corretgé
* VideoConsumer: a sink or destination for video data, for example a video rendering window
113 13 Saúl Ibarra Corretgé
114 13 Saúl Ibarra Corretgé
h3. Data flow
115 13 Saúl Ibarra Corretgé
116 13 Saúl Ibarra Corretgé
Data flow works in _pull_ fashion, that is, a producer doesn't start to produce data until there is a consumer which will consume it.
117 13 Saúl Ibarra Corretgé
118 13 Saúl Ibarra Corretgé
h3. VideoProducer
119 13 Saúl Ibarra Corretgé
120 13 Saúl Ibarra Corretgé
Produces video data. 
121 13 Saúl Ibarra Corretgé
122 13 Saúl Ibarra Corretgé
Internal API:
123 13 Saúl Ibarra Corretgé
124 13 Saúl Ibarra Corretgé
* _add_consumer: attach a consumer, called by the consumer
125 13 Saúl Ibarra Corretgé
* _remove_consumer: detach a consumer from a producer, called by the consumer
126 13 Saúl Ibarra Corretgé
127 18 Saúl Ibarra Corretgé
128 1 Saúl Ibarra Corretgé
Public API:
129 1 Saúl Ibarra Corretgé
130 18 Saúl Ibarra Corretgé
* start: start producing video as soon as a consumer is attached
131 18 Saúl Ibarra Corretgé
* stop: immediately stop producing data
132 29 Saúl Ibarra Corretgé
* close: remove all consumers and stop producing video data (also deallocate all C structures)
133 14 Saúl Ibarra Corretgé
* producer_port: pointer to the pjmedia_port object
134 14 Saúl Ibarra Corretgé
135 14 Saúl Ibarra Corretgé
h3. VideoConsumer
136 14 Saúl Ibarra Corretgé
137 14 Saúl Ibarra Corretgé
Consumes video data.
138 14 Saúl Ibarra Corretgé
139 14 Saúl Ibarra Corretgé
Public API:
140 14 Saúl Ibarra Corretgé
141 24 Saúl Ibarra Corretgé
* producer: (r/w property) attach this consumer to a producer, in order to render the video data generated by the producer. If set to None, it's detached
142 14 Saúl Ibarra Corretgé
* consumer_port: pointer to the pjmedia_port object
143 30 Saúl Ibarra Corretgé
* close: detach from producer and free all resources (also deallocate all C structures)
144 15 Saúl Ibarra Corretgé
145 15 Saúl Ibarra Corretgé
h3. Producer and consumer objects
146 15 Saúl Ibarra Corretgé
147 15 Saúl Ibarra Corretgé
* VideoDevice: Producer, acquires video from a user camera.
148 15 Saúl Ibarra Corretgé
* VideoWindow: Consumer, renders video in an SDL window. Extra methods: show/hide. Properties: native_handle, size.
149 15 Saúl Ibarra Corretgé
150 15 Saúl Ibarra Corretgé
* LocalVideoStream: Consumer, takes video from a VideoDevice and sends it to the remote party.
151 15 Saúl Ibarra Corretgé
* RemoteVideoStream: Producer, produces video sent by the remote party.
152 15 Saúl Ibarra Corretgé
153 15 Saúl Ibarra Corretgé
These are just theoretical objects, won't be implemented in the first go.
154 15 Saúl Ibarra Corretgé
155 17 Saúl Ibarra Corretgé
* VideoFileWriter: Consumer, saves incoming video data to a video file.
156 15 Saúl Ibarra Corretgé
* VideoFilePlayer: Producer, produces video data out of a video file.
157 15 Saúl Ibarra Corretgé
158 15 Saúl Ibarra Corretgé
* VideoMixer: Producer/Consumer, consumes video from multiple sources and produces aggregated video data.
159 16 Saúl Ibarra Corretgé
160 16 Saúl Ibarra Corretgé
NOTE: pjsip does have a AVI file player, which also seems to support audio.
161 23 Saúl Ibarra Corretgé
162 23 Saúl Ibarra Corretgé
h2. H264
163 23 Saúl Ibarra Corretgé
164 23 Saúl Ibarra Corretgé
Information about H264 profiles:
165 23 Saúl Ibarra Corretgé
166 23 Saúl Ibarra Corretgé
* http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
167 23 Saúl Ibarra Corretgé
* https://supportforums.cisco.com/blog/149561/video-telepresence-sip-h264-profile-level-id
168 28 Saúl Ibarra Corretgé
169 28 Saúl Ibarra Corretgé
h2. OpenH264 implementation
170 28 Saúl Ibarra Corretgé
171 28 Saúl Ibarra Corretgé
PJSIP has an initial version of a wrapper for Cisco's OpenH264 implementation (http://www.openh264.org/). http://trac.pjsip.org/repos/changeset?reponame=&old=4815%40%2F&new=4815%40%2F
172 28 Saúl Ibarra Corretgé
173 28 Saúl Ibarra Corretgé
OpenH264 seems to implement SVC, which is better than AVC.