WIPVideo

Version 42 (Saúl Ibarra Corretgé, 02/13/2015 10:12 am)

1 40 Saúl Ibarra Corretgé
h1. Video Support
2 1 Saúl Ibarra Corretgé
3 1 Saúl Ibarra Corretgé
4 1 Saúl Ibarra Corretgé
h2. Dependencies
5 1 Saúl Ibarra Corretgé
6 1 Saúl Ibarra Corretgé
The following dependencies are required to build PJSIP with video support (including H264)
7 1 Saúl Ibarra Corretgé
8 1 Saúl Ibarra Corretgé
* ffmpeg (libavformat, libswscale, libavcodec, libavutil)
9 1 Saúl Ibarra Corretgé
* libx264
10 1 Saúl Ibarra Corretgé
11 1 Saúl Ibarra Corretgé
Versions I have tried:
12 1 Saúl Ibarra Corretgé
13 41 Saúl Ibarra Corretgé
* ffmpeg 2.5.3
14 41 Saúl Ibarra Corretgé
* libx264 (snapshot-20141218-2245)
15 1 Saúl Ibarra Corretgé
16 20 Saúl Ibarra Corretgé
h2. Installing dependencies (Debian / Ubuntu systems)
17 20 Saúl Ibarra Corretgé
18 20 Saúl Ibarra Corretgé
The situation here is a bit sad. Both Debian and Ubuntu ship with libav instead of FFmpeg, but libraries are called the same. PJSIP had to be patched in order to properly work with libav, and the patch as not yet been included upstream.
19 1 Saúl Ibarra Corretgé
20 20 Saúl Ibarra Corretgé
On Debian, when the Debian-Multimedia repositories are used (quite common) you get FFmpeg and not libav. Oh the joy!
21 20 Saúl Ibarra Corretgé
22 20 Saúl Ibarra Corretgé
Installing dependencies on Debian:
23 20 Saúl Ibarra Corretgé
24 20 Saúl Ibarra Corretgé
<pre>
25 34 Saúl Ibarra Corretgé
apt-get install libv4l-dev libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libswresample-dev libx264-dev libavcodec-extra
26 1 Saúl Ibarra Corretgé
</pre>
27 20 Saúl Ibarra Corretgé
28 20 Saúl Ibarra Corretgé
If using the Debian-Multimedia repositories, do not install libavcodec-extra.
29 20 Saúl Ibarra Corretgé
30 20 Saúl Ibarra Corretgé
Installing dependencies on Ubuntu:
31 20 Saúl Ibarra Corretgé
32 20 Saúl Ibarra Corretgé
<pre>
33 39 Saúl Ibarra Corretgé
apt-get install libv4l-dev libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libx264-dev libavcodec-extra
34 20 Saúl Ibarra Corretgé
</pre>
35 1 Saúl Ibarra Corretgé
36 20 Saúl Ibarra Corretgé
*Note on H.264 support*: In order to have H.264 support, FFmpeg (or libav) need to be compiled with support for it. The standard packages don't, hence the need for installing the libavcodec-extra packages.
37 21 Saúl Ibarra Corretgé
38 41 Saúl Ibarra Corretgé
h2. Manually compiling dependencies (for OSX)
39 1 Saúl Ibarra Corretgé
40 3 Saúl Ibarra Corretgé
All dependencies will be compiled to a directory in the user's HOME directory:
41 1 Saúl Ibarra Corretgé
42 3 Saúl Ibarra Corretgé
<pre>
43 41 Saúl Ibarra Corretgé
export SIPSIMPLE_FFMPEG_PATH=$HOME/work/ag-projects/video/local
44 3 Saúl Ibarra Corretgé
</pre>
45 3 Saúl Ibarra Corretgé
46 41 Saúl Ibarra Corretgé
NOTE: yasm is required in order to enable asm optimizations. It does not come preinstalled on OSX, so it has to be manually installed. (brew install yasn will do)
47 36 Saúl Ibarra Corretgé
48 3 Saúl Ibarra Corretgé
h3. libx264
49 37 Saúl Ibarra Corretgé
50 3 Saúl Ibarra Corretgé
51 1 Saúl Ibarra Corretgé
<pre>
52 41 Saúl Ibarra Corretgé
./configure --enable-shared --disable-avs --disable-lavf --disable-ffms --disable-gpac --prefix=$SIPSIMPLE_FFMPEG_PATH
53 1 Saúl Ibarra Corretgé
make
54 8 Saúl Ibarra Corretgé
make install
55 37 Saúl Ibarra Corretgé
</pre>
56 1 Saúl Ibarra Corretgé
57 3 Saúl Ibarra Corretgé
h3. ffmpeg
58 3 Saúl Ibarra Corretgé
59 35 Saúl Ibarra Corretgé
<pre>
60 35 Saúl Ibarra Corretgé
# Some exports
61 41 Saúl Ibarra Corretgé
export PKG_CONFIG_PATH=$SIPSIMPLE_FFMPEG_PATH/lib/pkgconfig
62 35 Saúl Ibarra Corretgé
63 41 Saúl Ibarra Corretgé
./configure --enable-shared --disable-static --enable-memalign-hack --enable-gpl --enable-libx264 --prefix=$SIPSIMPLE_FFMPEG_PATH --extra-cflags="`pkg-config --cflags x264`" --extra-ldflags="`pkg-config --libs x264`"
64 4 Saúl Ibarra Corretgé
make
65 4 Saúl Ibarra Corretgé
make install
66 4 Saúl Ibarra Corretgé
</pre>
67 12 Saúl Ibarra Corretgé
68 41 Saúl Ibarra Corretgé
h2. API
69 34 Saúl Ibarra Corretgé
70 13 Saúl Ibarra Corretgé
API for video components is based on 2 different types of video capable entities:
71 13 Saúl Ibarra Corretgé
72 13 Saúl Ibarra Corretgé
* VideoProducer: a source for video data, for example a video camera or a remote video stream
73 13 Saúl Ibarra Corretgé
* VideoConsumer: a sink or destination for video data, for example a video rendering window
74 13 Saúl Ibarra Corretgé
75 13 Saúl Ibarra Corretgé
h3. Data flow
76 13 Saúl Ibarra Corretgé
77 13 Saúl Ibarra Corretgé
Data flow works in _pull_ fashion, that is, a producer doesn't start to produce data until there is a consumer which will consume it.
78 13 Saúl Ibarra Corretgé
79 13 Saúl Ibarra Corretgé
h3. VideoProducer
80 13 Saúl Ibarra Corretgé
81 13 Saúl Ibarra Corretgé
Produces video data. 
82 13 Saúl Ibarra Corretgé
83 18 Saúl Ibarra Corretgé
Public API:
84 1 Saúl Ibarra Corretgé
85 1 Saúl Ibarra Corretgé
* start: start producing video as soon as a consumer is attached
86 18 Saúl Ibarra Corretgé
* stop: immediately stop producing data
87 18 Saúl Ibarra Corretgé
* close: remove all consumers and stop producing video data (also deallocate all C structures)
88 29 Saúl Ibarra Corretgé
* producer_port: pointer to the pjmedia_port object
89 14 Saúl Ibarra Corretgé
90 14 Saúl Ibarra Corretgé
h3. VideoConsumer
91 14 Saúl Ibarra Corretgé
92 14 Saúl Ibarra Corretgé
Consumes video data.
93 14 Saúl Ibarra Corretgé
94 14 Saúl Ibarra Corretgé
Public API:
95 14 Saúl Ibarra Corretgé
96 1 Saúl Ibarra Corretgé
* producer: (r/w property) attach this consumer to a producer, in order to render the video data generated by the producer. If set to None, it's detached
97 1 Saúl Ibarra Corretgé
* consumer_port: pointer to the pjmedia_port object
98 14 Saúl Ibarra Corretgé
* close: detach from producer and free all resources (also deallocate all C structures)
99 1 Saúl Ibarra Corretgé
100 24 Saúl Ibarra Corretgé
h3. Producer and consumer objects
101 14 Saúl Ibarra Corretgé
102 42 Saúl Ibarra Corretgé
* VideoCamera: Producer, acquires video from a user camera.
103 42 Saúl Ibarra Corretgé
* FrameBufferVideoRenderer: Consumer, calls the user supplied callback with a video frame at a time.
104 15 Saúl Ibarra Corretgé
105 42 Saúl Ibarra Corretgé
* LocalVideoStream: Consumer, takes video from a VideoCamera and sends it to the remote party.
106 15 Saúl Ibarra Corretgé
* RemoteVideoStream: Producer, produces video sent by the remote party.
107 15 Saúl Ibarra Corretgé
108 42 Saúl Ibarra Corretgé
These are just theoretical objects, they are not currently implemented.
109 15 Saúl Ibarra Corretgé
110 15 Saúl Ibarra Corretgé
* VideoFileWriter: Consumer, saves incoming video data to a video file.
111 17 Saúl Ibarra Corretgé
* VideoFilePlayer: Producer, produces video data out of a video file.
112 15 Saúl Ibarra Corretgé
113 15 Saúl Ibarra Corretgé
* VideoMixer: Producer/Consumer, consumes video from multiple sources and produces aggregated video data.
114 15 Saúl Ibarra Corretgé
115 42 Saúl Ibarra Corretgé
NOTE: pjsip does have a AVI file player, which also seems to support audio (this could be used to stream a movie, for example), it doesn't have a writer though.
116 34 Saúl Ibarra Corretgé
117 23 Saúl Ibarra Corretgé
h2. H264
118 23 Saúl Ibarra Corretgé
119 23 Saúl Ibarra Corretgé
Information about H264 profiles:
120 23 Saúl Ibarra Corretgé
121 23 Saúl Ibarra Corretgé
* http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
122 23 Saúl Ibarra Corretgé
* https://supportforums.cisco.com/blog/149561/video-telepresence-sip-h264-profile-level-id
123 28 Saúl Ibarra Corretgé
124 28 Saúl Ibarra Corretgé
h2. OpenH264 implementation
125 28 Saúl Ibarra Corretgé
126 28 Saúl Ibarra Corretgé
PJSIP has an initial version of a wrapper for Cisco's OpenH264 implementation (http://www.openh264.org/). http://trac.pjsip.org/repos/changeset?reponame=&old=4815%40%2F&new=4815%40%2F
127 28 Saúl Ibarra Corretgé
128 41 Saúl Ibarra Corretgé
OpenH264 seems to implement SVC, which is better than AVC. In practice it didn't outperform libx264 and it only implements the constrained baseline profile, so it was discarded.