SIP SIMPLE Client SDK

<abbr title="Design*, depth=1">TOC</abbr>

= Video blueprint =

Design and implement Ticket [ticket:18]

Goals

The goal is to implement the support for video session is the SIPSIMPLE middleware. For the video encoding/decoding VLC library shall be used, which supports H.264 codec (first codec to be supported).

The video stream should behave like the audio stream at the transport level: SRTP and ICE should be usable. It must comply with the IMediaStream interface, the same way AudioStream does.

Integration

In the picture below the integration of the video support into SIPSIMPLE is shown divided into it's different components:

(IMAGE)

• PJSIP: a new transport is needed. Instead of creating a whole new transport, which would require to implement SRTP and ICE again, a transport adapter will be implemented. A transport adapter sits between a real transport and a pjmedia_stream. To the stream, this adapter will look like a media transport, and to existing media transport, this adapter will look like a stream. The benefit of this approach is we can use the same adapter for both kind of media transports, that is the UDP and ICE media transport. This is exactly the approach that was taken for SRTP. This transport adapter will be responsible for encoding/decoding the video information.

• Middleware: * VideoTransport: the VideoTransport will use RTPTransport to carry the video data and will be responsible for building the SDP for the video stream. A new option will be added to RTPTransport so that it starts the video transport adapter instead of the regular transport when needed. * VideoStream: implements IMediaStream interface. Will export a plugable mechanism so that the application layer can access the video data and display it in a window for example, similar to ExternalVNCViewer on MSRP streams.

• Application: the application will receive the video data from the stream and 'paint' it on a window.

Video acquisition Roadmap

In order to achive the goal, the following subtasks need to be done: * Implement a wrapper class around the VLC Python bindings. http://wiki.videolan.org/Python_bindings * Create videostream.py implementing IMediaStream interface. * Use the VLC wrapper class to encode/decode the stream. * Bundle the VLC Python wrapper together with SIPSIMPLE so that it get's compiled and installed when installing SIPSIMPLE. * Create sip_video_session.py script (or modify sip_audio_session_script) so that it uses video capabilities.

At the time of mangling the SDPs, two approaches can be taken in order to add video support: * Modify PJSIP's PJMEDIA component to encode/decode video streams and handle them the same as audio streams from the upper layers. * Just use PJMEDIA to do the SDP negotiation and audio part. Video would be external to the PJMEDIA layer.

For this case the second approach seems to be a better one. When receiving an incoming call or making a call, the SDPs will be mangled so that they include H.264 stream information i.e. {{{
...
m=video 6001 RTP/AVP 98
a=rtpmap:98 H264/90000
...
}}}

So in the case of an incoming call, an VLC player instance will be opened for playing the stream at the IP and port specified by the SDP. For an outgoing call, an VLC will stream H.264 video taken from a selected video source (webcam, still picture, ...) from the sipsimpleclient listen IP and will randomly select a port (in a given range) which doesn't interfere with the audio ports.

This way audio and video components are sepparate, but it's not necesary to modify the underlying media handling component: pjmedia.

Components

Image(videostream-components.png)

Classes

Image(videostream-classes.png)

Code mockup

There are 3 files attached containing a simple code mockup: * videostream.py: Class representing the actual video stream. Contains necessary functions to initialize/stop the stream using VLC as the engine. * vlcw.py: Wrapper class around the VLC python bindings. Provide some higher level functions to be used by the VideoStream class. * sip_video_session.py: Script to test videostream from a console interface.