DesignRTT

Version 3 (Adrian Georgescu, 06/23/2009 10:14 am)

1 3 Adrian Georgescu
= Real Time Text =
2 1 Adrian Georgescu
3 1 Adrian Georgescu
[[TOC(Design*, depth=1)]]
4 1 Adrian Georgescu
5 1 Adrian Georgescu
== Introduction ==
6 1 Adrian Georgescu
7 1 Adrian Georgescu
[[Image(http://nlnet.nl/image/logo.gif, align=right, nolink)]]
8 1 Adrian Georgescu
9 1 Adrian Georgescu
[[Image(htdocs:R3TF_logo_web.jpg, align=right, nolink)]]
10 1 Adrian Georgescu
11 1 Adrian Georgescu
This document provides a proposal for how to integrate the T.140 Real-Time Text protocol over RTP, as described in [http://tools.ietf.org/html/rfc4103 RFC 4103], into the SIP SIMPLE client and consequently into [http://www.pjsip.org PJSIP].  This is a joint effort between [http://www.realtimetext.org R3TF], [http://nlnet.nl NL NET fundation] and [http://ag-projects.com AG Projects] with the goal of enabling Real Time Text as generic purpose communication tool.
12 1 Adrian Georgescu
13 1 Adrian Georgescu
== Use cases ==
14 1 Adrian Georgescu
15 1 Adrian Georgescu
 * One can call a device directly using real-time text. The called device can be any ToIP client, it is possible with or without voice included
16 1 Adrian Georgescu
 * One can decide to switch from IM mode to a more talking mode and press an activate real-time text button. This is similar with switching to voice.
17 1 Adrian Georgescu
 * There is an hybrid form of real-time to be used as well. Real-time text preview. See: http://tools.ietf.org/html/draft-hellstrom-textpreview-05 What you get is you see immediately what the other side is typing. And when he or she presses enter, it is sent en-bloc IM style thus combining best of both worlds
18 1 Adrian Georgescu
 * Remote captioning, one big screen where one can modify the size of the text and receive the text interpreter text on it. The send window is almost non present in that mode (or only pulled up if one need to talk with the interpreter on one-on –one interpreter sessions).
19 1 Adrian Georgescu
20 1 Adrian Georgescu
> This document does not include any information on how to integrate RTT into the GUI, as this is not relevant at the PyPJUA level, the GUI is a separate project.
21 1 Adrian Georgescu
22 1 Adrian Georgescu
== Existing implementations ==
23 1 Adrian Georgescu
24 1 Adrian Georgescu
There are two known open source implementations of this protocol:
25 1 Adrian Georgescu
26 1 Adrian Georgescu
 * The [http://sourceforge.net/projects/rtp-text-t140/ RTP text/t140 Library].
27 1 Adrian Georgescu
   This library is used in the [http://sourceforge.net/projects/tipcon1/ SIPCon1] client and is written in Java.
28 1 Adrian Georgescu
   Because it is written in Java and PJSIP is written in C, this library cannot be reused.
29 1 Adrian Georgescu
 * [http://www.asterisk.org/ Asterisk] supports relaying T.140 over RTP, but is not an endpoint.
30 1 Adrian Georgescu
   Perhaps some code can be recycled from it, in particular the redundancy support (or RED).
31 1 Adrian Georgescu
32 1 Adrian Georgescu
== Relation to PyPJUA and PJSIP ==
33 1 Adrian Georgescu
34 1 Adrian Georgescu
Implementing T.140 over RTP can be subdivided into several tasks:
35 1 Adrian Georgescu
36 1 Adrian Georgescu
 * RTT codec development as described in [http://tools.ietf.org/html/rfc4103 RFC 4103] to transmit and receive text over RTP (written in C)
37 1 Adrian Georgescu
 * SDP negotiation for RTT session development (written in Python)
38 1 Adrian Georgescu
 * Handling events generated by the RTT codec in the Graphical User Interface (not addressed by this document)
39 1 Adrian Georgescu
40 1 Adrian Georgescu
Sending and receiving of [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__RTP.htm RTP] and [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__RTCP.htm RTCP] packets is already implemented in PJSIP and will be reused. The actual RTT codec must be written in C similar to an audio stream implementation and pushed to PJSIP project repository. Decoding and encoding of SDP in SIP SIMPLE client is done in PyPjUA layer, so a dedicated Python object must be developed to handle it in SIP SIMPLE client.
41 1 Adrian Georgescu
42 1 Adrian Georgescu
== Audio stream implementation in PJSIP ==
43 1 Adrian Georgescu
44 1 Adrian Georgescu
[[Image(http://www.pjsip.org/images/diagram.jpg, nolink)]]
45 1 Adrian Georgescu
46 1 Adrian Georgescu
To determine how T.140 over RTP should be implemented in PJSIP, we first need to examine how the API to create and manage audio streams works in PJSIP, as RTT should be implemented in a similar fashion.
47 1 Adrian Georgescu
48 1 Adrian Georgescu
We consider the API from the point of view of PyPJUA, which in the diagram above is in the same position as [http://www.pjsip.org/pjsip/docs/html/group__PJSUA__LIB.htm PJSUA]. Currently audio is the only type of media stream supported by PJSIP. First, the local SDP is generated, either when initating a new {{{INVITE}}} or when an incoming {{{INVITE}}} including SDP is received.
49 1 Adrian Georgescu
This can be done by PJSIP using the [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__ENDPT.htm#ga49360a690c292dddd722b39c2f8c8bb pjmedia_endpt_create_sdp()] function, which creates SDP with exactly one audio stream.
50 1 Adrian Georgescu
Once SDP negotiation has been completed, which is done in the [http://www.pjsip.org/pjsip/docs/html/group__PJSIP__INV.htm pjsip_inv] module using the [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__SDP__NEG.htm SDP negotation framework], PyPJUA is handed two [http://www.pjsip.org/pjmedia/docs/html/structpjmedia__sdp__session.htm pjmedia_sdp_session] structs, which represent the local and remote SDP.
51 1 Adrian Georgescu
Using this, PyPJUA should perform the following steps:
52 1 Adrian Georgescu
 * Call the [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__SESSION.htm#ge0400f365c07b6f141f202defe92f2a6 pjmedia_stream_info_from_sdp()] function, giving both SDP structures as parameters.
53 1 Adrian Georgescu
   This will initalize a [http://www.pjsip.org/pjmedia/docs/html/structpjmedia__stream__info.htm pjmedia_stream_info] structure.
54 1 Adrian Georgescu
 * Use this [http://www.pjsip.org/pjmedia/docs/html/structpjmedia__stream__info.htm pjmedia_stream_info] structure to create an opaque {{{pjmedia_stream}}} struct by means of the [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__STRM.htm#g67575c8e7b15e325b98ebaa89639b550 pjmedia_stream_create()] function.
55 1 Adrian Georgescu
   This opaque struct actually represents an audio stream and contains within it the following:
56 1 Adrian Georgescu
   * An instance of the audio codec used.
57 1 Adrian Georgescu
   * An [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__JBUF.htm adaptive jitter buffer]
58 1 Adrian Georgescu
   * Two instances of [http://www.pjsip.org/pjmedia/docs/html/structpjmedia__rtp__session.htm pjmedia_rtp_session], representing the RTP streams in both directions.
59 1 Adrian Georgescu
   * One instance of [http://www.pjsip.org/pjmedia/docs/html/structpjmedia__rtcp__session.htm pjmdia_rtcp_session].
60 1 Adrian Georgescu
   * A reference to the media transport that was created earlier to carry the RTP and RTCP streams.
61 1 Adrian Georgescu
     This could be a simple [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__TRANSPORT__UDP.htm UDP transport] or an [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__TRANSPORT__ICE.htm ICE transport], possibly wrapped by a [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__TRANSPORT__SRTP.htm SRTP transport].
62 1 Adrian Georgescu
 * Start the stream using the [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__STRM.htm#g93d59e3be009de86a3823303784d31a2 pjmedia_stream_start()] function and fetch the port to be connected to the sound cart or conference brige using the [http://www.pjsip.org/pjmedia/docs/html/group__PJMED__STRM.htm#ge3cb31df5aa921ef3085d5eb539af063 pjmedia_stream_get_port()] function.
63 1 Adrian Georgescu
64 1 Adrian Georgescu
== RTT codec development ==
65 1 Adrian Georgescu
66 1 Adrian Georgescu
An object would need to be implemented that is similar to the {{{pjmedia_stream}}} object, but for a RTT stream, for example by the name of {{{pjmedia_rtt_stream}}}.
67 1 Adrian Georgescu
Since RTT only has one codec, the encoding and decoding can be integrated into this object.
68 1 Adrian Georgescu
First, the SDP needs to be generated in some way to include RTT, if the user requests this.
69 1 Adrian Georgescu
This will be done by PyPJUA.
70 1 Adrian Georgescu
The sequence of events to create and use a {{{pjemdia_rtt_stream}}} after SDP negotiation is complete would look like this:
71 1 Adrian Georgescu
72 1 Adrian Georgescu
 * For the audio stream, the [http://www.pjsip.org/pjmedia/docs/html/structpjmedia__stream__info.htm pjmedia_stream_info] structure will be initialized by the [http://www.pjsip.org/pjmedia/docs/html/group__PJMEDIA__SESSION.htm#ge0400f365c07b6f141f202defe92f2a6 pjmedia_stream_info_from_sdp()] function.
73 1 Adrian Georgescu
   This function may need some adjusting as currently it does only checking on audio codecs.
74 1 Adrian Georgescu
 * A {{{pjmedia_rtt_stream}}} object is created using a function like {{{pjmedia_rtt_stream_create}}}, passing as arguments the {{{pjmedia_stream_info}}} structure and a previously created {{{pjmedia_transport}}} object.
75 1 Adrian Georgescu
   This object will manage the following:
76 1 Adrian Georgescu
   * An instance of the T.140 encoder and decoder, which probably consists of some text buffers for RED.
77 1 Adrian Georgescu
   * Two instances of [http://www.pjsip.org/pjmedia/docs/html/structpjmedia__rtp__session.htm pjmedia_rtp_session], representing the RTP streams in both directions.
78 1 Adrian Georgescu
   * One instance of [http://www.pjsip.org/pjmedia/docs/html/structpjmedia__rtcp__session.htm pjmdia_rtcp_session].
79 1 Adrian Georgescu
   * A reference to the media transport that was created earlier to carry the RTP and RTCP streams.
80 1 Adrian Georgescu
 * The stream should be started through some function, e.g. {{{pjmedia_rtt_stream_start}}}.
81 1 Adrian Georgescu
   This will set a timer that performs transmission every interval.
82 1 Adrian Georgescu
 * The application can transmit text by appending characters to the internal transmission buffer by calling some function, like {{{pjmedia_rtt_stream_send}}}.
83 1 Adrian Georgescu
 * The application could receive text either by reading from the internal reception buffer by calling a function, like {{{pjmedia_rtt_stream_receive}}}, or have a previously set callback called whenever there is new text.
84 1 Adrian Georgescu
   The former has the disadvantage that the application will have to do active polling on the object and possibly cause buffer overruns if it doesn't poll, the latter has the disadvantage that because of the short intervals the callback may be called quite frequently, which may prove to be inefficient, particularly in a Python environment.
85 1 Adrian Georgescu
86 1 Adrian Georgescu
== SDP negotiation for RTT session ==
87 1 Adrian Georgescu
88 1 Adrian Georgescu
In PyPJUA a {{{RTTStream}}} class will be created, which represents a T.140 stream. It should have a method to send text and an event will be associated with it which will be called on the event handler whenever there is incoming text. Internally, it should also govern the SDP generation.
89 1 Adrian Georgescu
90 1 Adrian Georgescu
== Handling events generated by the RTT codec ==
91 1 Adrian Georgescu
92 1 Adrian Georgescu
The GUI must be notified every time there is a new character received from RTT codec or every time the user types in a character in the GUI for an established session.