Accounting guide » History » Revision 27
Revision 26 (Tijmen de Mes, 11/26/2012 12:36 PM) → Revision 27/30 (Tijmen de Mes, 11/26/2012 12:37 PM)
h1. Accounting guide
!>http://projects.ag-projects.com/attachments/3757/OpenSIPS-accounting.png!
The platform generates usage information in the form of rated Call Detail Records (CDR). It is the responsibility of the Operator to import the rated CDRs into its own billing system and to generate printable invoices out of them.
The CDRs can be easily searched and grouped by various criteria and exported into comma separated value files. Accounting can be done postpaid (default option) or prepaid (enabled per SIP account). Postpaid accounts can have a monthly quota assigned to prevent fraud.
h2. CDR Generation
Call detail records (CDRs) are generated from the SIP sessions established through the platform. A CDR contains relevant information about a SIP session from both signaling and data planes. The CDR is generated by combining several Radius requests from OpenSIPS and MediaProxy as follows:
|*Radius request*|*When*|*Component*|*Description*|
|FAILED|First INVITE with negative code|OpenSIPS|Zero second (failed) session|
|START|First INVITE with positive code|OpenSIPS|Start of new session|
|STOP|BYE|OpenSIPS|Stop of existing session|
|UPDATE|RTP stop|MediaProxy|Stop or timeout of existing RTP stream|
h3. Radius logs
Radius requests generated by the platform are logged by the Freeradius server in the following places:
|*Directory*|*Log type*|
|/var/log/freeradius/radacct/IP/detail-YYYYMM|Radius tickets contents|
|/var/log/freeradius/radacct/radius.log|Radius errors|
The most relevant attribute of a CDR is the session duration. The cost for the session depends on this session duration. The sessions duration is determined by the Stop time and Start time of the SIP session. The Start time is considered the moment when the answer for the first INVITE has entered the SIP Proxy. Stop time is determined either by the presence of BYE message or in case BYE is missing from the last moment RTP has ben relayed though MediaProxy.
bq. It is not possible to determine the stop time of a session in the following situation:
# First INVITE is answered with a positive code
# No BYE is received
# No RTP data has been relayed
h3. Media timeout
If the RTP media has timed out (both parties stopped sending RTP) while no BYE has been received by the SIP Proxy:
# MediaProxy will set in the Radius UPDATE the MediaInfo Radius attribute to *timeout*
# MediaProxy will trigger the sending of a a BYE message via the dialog module of OpenSIPS
In this situation, the actual duration of the SIP session contains the period of the RTP stream plus the value of the timeout. The value of the timeout is determined by the connection tracking logic in the Linux kernel. At this moment there is no mechanism to store into the Radius database the value of this timeout, which is typically 180 seconds.
h3. CDR tables
CDRs are stored in monthly tables in radius database. The tables are named radacctYYYYMM. Tables are automatically created by the Radius server at the beginning of each month.
h3. CDR fields
|*Field*|*Type*|*Description*|Notes|
|RadAcctId|bigint(21)|MySQL unque index auto-incremented||
|AcctSessionId|varchar(255)|SIP call-id header||
|AcctUniqueId|varchar(255)|Radius internal session id||
|UserName|varchar(64)|SIP Username|Matched against billing_customers.subscriber|
|Realm|varchar(64)|SIP domain|Matched against billing_customers.domain|
|NASIPAddress|varchar(15)|SIP Proxy IP||
|NASPortId|varchar(50)|SIP Proxy port||
|NASPortType|varchar(255)|not used||
|AcctStartTime|datetime|Session start time||
|AcctStopTime|datetime|Session stop time||
|AcctSessionTime|int(12)|Session duration used for rating||
|AcctAuthentic|varchar(32)|not used||
|ConnectInfo_start|varchar(32)|not used||
|ConnectInfo_stop|varchar(32)|not used||
|AcctInputOctets|bigint(12)|Relayed RTP through MediaProxy||
|AcctOutputOctets|bigint(12)|Relayed RTP through MediaProxy||
|CalledStationId|varchar(50)|SIP To header, the original dialed number||
|CallingStationId|varchar(50)|SIP From header||
|AcctTerminateCause|varchar(32)|SIP response code||
|ServiceType|varchar(32)|Sip-Session||
|ENUMtld|varchar(64)|ENUM top level domain used for lookup||
|FramedIPAddress|varchar(15)|not used||
|AcctStartDelay|int(12)|not used||
|AcctStopDelay|int(12)|not used||
|SipMethod|varchar(50)|SIP method||
|SipResponseCode|smallint(5) unsigned|Response code for INVITE||
|SipToTag|varchar(128)|SIP to tag||
|SipFromTag|varchar(128)|SIP from tag||
|SipTranslatedRequestURI|varchar(255)|Final network address after DNS lookup||
|SipUserAgents|varchar(255)|SIP user agents/servers reported by MediaProxy||
|SipApplicationType|varchar(255)|Media type (e.g. audio)||
|SipCodecs|varchar(255)|RTP codecs report by MediaProxy||
|SipRPID|varchar(255)|PSTN caller id||
|SipRPIDHeader|varchar(255)|not used||
|SourceIP|varchar(255)|Source IP of the session|Matched against billing_customers.gateway|
|SourcePort|varchar(255)|Source port of the session||
|CanonicalURI|varchar(255)|Final logical destination used for rating|Matched against destinations table|
|DelayTime|varchar(5)|not used||
|Timestamp|bigint(20)|Packet timestamp||
|DestinationId|varchar(15)|Normalized E.164 prefix|Calculated during normalization|
|Rate|text|Description of rate calculation|Calculated during normalization|
|Price|double(20,4)|Cost of the session|Calculated during normalization|
|Normalized|enum('0','1')|If the CDR has been normalized||
|BillingId|varchar(255)|Reseller id used by trusted peers and SIP domains|Calculated during normalization|
|MediaInfo|varchar(32)|Mediaproxy information||
|RTPStatistics|text|RTP statistics reported by some devices||
|FromHeader|varchar(128)|not used||
|UserAgent|varchar(128)|SIP User Agent and Server headers||
|Contact|varchar(128)|not used||
h2. CDR Normalization
A CDR is considered closed when the stop time has been determined. Once the CDR is closed, a normalization process cleans up the CDR fields and calculates the price of the session. The price and the way its was calculated is saved together with the CDR in the same table. It is possible to re-normalize the sessions at a later time, for instance when the tariffs need to be changed for the past period.
Normalization is performed periodically:
# Whenever CDRTool web search is used
# By the cron job /var/www/CDRTool/script/normalize.php
Once normalized, the CDRs are no longed modified, the normalization process marks each CDR that has been processed.
|*Operation*|*When*|*Platform component*|*Description*|
|NORMALIZE|After CDR was closed|CDRTool|Rating is applied|
h3. Un-normalized sessions
Some SIP sessions can remain in the CDR database in an un-normalized state. They show in CDRTool as "in progress". Such sessions have a start date but no end date. Such sessions cannot be normalized automatically by the CDR normalization process because there is no indication when they have actually stopped. A SIP session is considered stopped when either a BYE message has been received by the SIP Proxy or when the RTP media has stopped flowing between the end-points, which is detected by the MediaProxy. If the RTP media has not started yet and no BYE has arrived after the initial INVITE/200 OK/ACK, there is no physical indication for when the session has started or when it ended. Due to the nature of the SIP protocol and in particular the fact that the RTP media is carried separately from the SIP signaling, it is always possible to have SIP sessions that from a call completion perspective have started but never ended. Such situations occur outside the control of the operator responsible for the SIP service. You may chose to apply your own policy for such sessions like deleting or archiving them.
h2. CDR Search
CDRs can be searched and displayed in CDRTool. Access to CDRTool can be provided to resellers or end-users of the platform.
!cdrtool-call-search.png!
Last placed and received calls are also available for every SIP account using SOAP/XML functions:
* SipPort->getCalls()
h2. CDR Export
CDRs can be exported from CDRTool in comma separated values (CSV) format or by selecting the raw content of the MySQL radacctYYYYMM tables. Such operation can take time to complete and is advisable to use always a MySQL slave server for these queries.
h2. Rating Logic
CDRTool provides on-the-fly rating of CDRs generated by the platform.
A Call Detail Record (CDR) is one record from the radius radacct table. The CDR contains all information related to a session, its duration, the calling and called party and media information. The rating engine calculates the price of the session. The calculation is done once and its results are saved in the CDR table for later use. It is possible to re-calculated the prices at a later time for example when having to change the tariffs.
Based on exceptions, different rates may be applied per:
# Caller party (SIP account)
# Caller domain
# Source IP address
Rating is applied only after the call has ended. CDRTool considers that a call has ended when there is a stop time. In case of missing BYEs, CDRTool relies on the fact that MediaProxy will update the CDR with the proper stop time information based on the last moment the media stream passed through the media proxy.
The rates are linked with profiles corresponding with different time of the day, day of the week or holidays. For rating calls, which span multiple profiles, the right rate is selected and applied for the call duration within each profile. Each customer may be assigned its own dedicated rating plans destination id and names. Chained profiles are possible to enable exception based rating. Multiple customers may share a common rate list, while some destinations may be rated differently, only the differences must be provisioned. Multiple time zones are supported for multiple billing parties hosted on the same platform.
Different customers can have different rating plans. A rating plan is a unique combination of holidays, day of week, time of day, destination ids, and associated costs.
The following steps are performed to rate a CDR:
# Determination of the billing party
# Determination of the destination id
# Determination of the costs
The steps are described in detail below.
h3. Determination of the billing party
To be able to calculate the Price for a call the rating engine must determine whose rating plan to use.
The rating engine does this by performing a match against entries in the billing_customers table for the the BillingPartyId field of the the CDR (radacct.UserName for radius based datasources) in the following order:
# SIP account user@domain
# SIP domain of the SIP account
# Source IP of the session
# Default (when none of the above matches)
The first match is considered to be the billing party for which the rating plan is determined.
The rating plan is further derived from the profiles associated with the entry found in the billing_customers.
bq. The billing_customers table field that matches the Source IP is called gateway. "gateway" or "trusted peer" terms are used interchangeably in this document. They both relate to the source IP address that generated the SIP session.
h3. Determination of the destination id
The rating engine identifies the 'destination id', which has associated rates depending on day of week and time of day.
The 'destination id' is for example a country prefix like '31' for the Netherlands. This 'destination id' has prices associated with it so all calls to the Netherlands will have prices associated with prefix '31'.
The 'destination id' is derived from the logical destination the SIP session has been routed to. In SIP headers and Radius records there are multiple places that contain information related to the destination. Some of them are generated by the SIP User-Agents (hence cannot be trusted and must not be used for accounting purposes) and others are generated by the SIP Proxy configured by the operator and are suitable for accounting purposes.
The rating engine considers the destination to be the first non-empty CDR field in this order:
# CanonicalURI (the destination after all lookups inside the SIP Proxy)
# SipTranslatedRequestURI (the Request URI as presented by the SIP UA)
# CalledStationId (the content of the To header, used as a last resort)
The CanonicalURI is the preferred because is reliable information generated by the operator based on the logic configured in the SIP Proxy and the subscriber cannot control it. OpenSIPS must be configured to send the CanonicalURI Radius attribute when creating the radius accounting START record. Instructions for this are available in INSTALL.txt file.
The 'destination id' is then calculated based on the longest match of this destination field in the billing_destinations table.
The longest match for the chosen destination field is performed by the E164_class, which by default uses E164_Europe that defines an European numbering plan. It assumes the destinations start with a zero for a national call and with double zero for an international call. See cdr_generic.php for the actual logic.
If your dialing plan is different, you must use other provided class like E164_US or create a custom class and point to it in global.inc for each datasource as follows:
'E164_class'=>'MyE164Class',
For example the pre-defined E164_US class from cdr_generic.php matches the American dialing plan.
h3. Determination of the costs
The following steps are taken to determine the cost for the calls based on the 'destination id' and the billing party determined at the previous steps.
# Lookup the billing profiles in cdrtool.billing_customers table in the following order: subscriber,domain,gateway (based on $this->dayofweek), as explained in "Determination of the billing party" above:
#* profile_name1 matches week days [1-5]
#* profile_name1_alt matches week days [1-5] if no rates for profile_name1 are found
#* profile_name2 matches week-ends [6-0]
#* profile_name2_alt matches week-ends [6-0] if no rates for profile_name2 are found
#* profile_name2 matches also holidays from billing_holidays table
The week starts with 0 (Sunday) and ends with 6 (Saturday). This step determines which rates should be applied based on the day of the week when the call started.
# Using the profile_name found, lookup the rate_name based on $this->hourofday in cdrtool.billing_profiles table
If no rate_name is found for the given profiles a second set of profiles are used, profile_name1_alt and profile_name2_alt.
#* the day may be split in maximum 4 periods
#* the days starts with hour 0 and ends with hour 24
#* rate_name1 defines the first interval after hour 0
#* rate_name2 defines the first interval after rate_name1
#* rate_name3 defines the first interval after rate_name2
#* rate_name4 defines the first interval after rate_name3
When the hour matches an interval use the rate_nameX found to lookup the rate in billing_rates, if no record is found use the rate called 'default'. This step determines which rate should be applied for the time of day when the call started.
# Lookup in the cdrtool.billing_rates table the record having same name found at point 2 having billing_rates.destination = 'destination id' and billing_rates.application = application type found in the steps above.
#* return an array with all the rating values and the duration rated
No rate will be returned if no 'destination id' is found. Make sure each possible destination has a 'corresponding id' and name in the destinations table.
This step determines the costs within the current time span associated with the time of day and destination id. If the call duration exceeds this time span (that is a new interval for which another rate applies is reached), step 4 is performed.
# If the duration rated at point 3 is less than total call duration, apply point 3 again for the remaining call duration in the next profile. A maximum of 10 spans (different rates depending of time of day, day of the week) can be calculated using this mechanism.
After 10 spans, the engine bails out to avoid loops caused by invalid tables provisioning.
# Calculate the total call Price based on its duration and connection fees.
In global.inc there are several variables that affect how the price is calculated. These settings are global per CDRTool installation but some can be overwritten with per customer values in the billing_customers table.
<pre>
$RatingEngine=array(
"priceDenominator" => 10000, // Rates units (global setting)
"priceDecimalDigits" => 4, // Decimal information (global setting)
"minimumDurationCharged" => 0, // Rate a minimum of X seconds (per customer)
"minimumDuration" => 0, // Minimum duration to rate, if call duration is shorter the price is zero (per customer)
"durationPeriodRated" => 60 // Rate is per 60 seconds (global setting)
"trafficSizeRated" => 1024, // Default we rate per 1 MB (global setting)
"reportMissingRates" => 0 // Send emails to administrator in case of missing rates
);
</pre>
Pricing formula:
<pre class="wiki">
if min_duration then
minimumDurationCharged = min_duration
else if minimumDurationCharged set in global inc
use minimumDurationCharged from global.inc
else
minimumDurationCharged = call duration
if increment then
durationForRating = round to the next increment
else
durationForRating = call duration
if durationForRating >= minimumDurationCharged then
Price = connectCost/priceDenominator+
durationRate*durationForRating/durationPeriodRated/priceDenominator
else
Price = 0
</pre>
ENUM discounts
The rating engine can apply a discount associated with the ENUM top level domain that returned the final destination.
Price = Price - Price * ENUM discount / 100
To apply ENUM based discounts, the ENUM TLD must be saved with each CDR and the TLDs with their corespondent discounts must be provisioned in the Rating tables section. See ENUM TLD discounts section for more information.
Purchasing price
A second price called 'Price in' is calculated using the same formula but based on connectCostIn and durationRateIn values. It can be used to match the purchasing price and calculate the margin between purchasing and selling prices. The information about both prices is stored in the RateInfo field of the CDR. The values for connectCostIn and durationRateIn must be provisioned in the billing_rates and billing_rates_history tables using the web interface or by importing csv files.
6. Save the calculated Price, billing party and 'destination id' for each call in the CDR table. Having the price stored in the database, it is possible to build statistics to display consolidated revenues per country code, network or subscriber.
h3. Rating files
To generate billable CDRs from the SIP traffic that goes through the platform you must provision the rating plans in CDRTool software that is responsible for CDR mediation and rating.
Before provisioning CDRTool, is important that you understand how the rating engine works. The modus operandi of the rating engine is described at:
https://mdns.sipthor.net/CDRTool/doc/RATING.txt
After you read and understand the document, you will need to generate the following files:
# destinations.csv
# customers.csv
# billing_profiles.csv
# billing_rates.csv
The file formats are described in 'Importing and exporting of rating files' section of RATING.txt document.
Samples files can be found in the CDRTool software archive available at:
http://download.ag-projects.com/CDRTool/sample_rating_files/
The files must then be uploaded in CDRTool rating engine, once the software is operational on your platform.
The operator is responsable for creating the files. The initial provisioning is performed by AG Projects.
The content of the rating tables can be edited in the web interface provided by CDRTool application.
h3. Importing and exporting of rating files
There are different data files needed for rating. The data files are imported into their corresponding MySQL tables. The files must be uploaded to /var/spool/cdrtool directory. To load the files into the database run the following command:
/var/www/CDRTool/scripts/importRatingTables.php
The import script knows to import the files only once so you may dump several files there and safely run the import script from cron. The import script detects whether each file have been imported by building a unique key out of the filename and the hash of the file content. So you may use the same filenames as long as the content differs and viceversa. If the import file has changed any records, the rating engine is automatically instructed to reload the changes.
Sample csv files are found in the setup directory. The CSV field order is described in setup/*.csv sample files. The first element on each line specifies the operation will be performed with the current record. The operation can be 2 (update/insert), 1 (insert) or 3 (delete).
The updates are performed based on a unique key present in each table:
<pre>
billing_customers - cust_idx (gateway,domain,subscriber)
destinations - cust_dest_idx (gateway,domain,subscriber,dest_id)
billing_profiles - profile_idx (name)
billing_rates - rate_idx (name,destination,application)
billing_rates_history - rate_idx (name,destination,application,startDate,endDate)
</pre>
The content of the rating tables can be exported in the Rating tables page.
The import script detects the type of file to import based on its filename.
The filename must comply with the following naming convention:
# Must start with the name of the table without the billing_
# May optionally contain extra characters after the name
# Must end with .csv extension
Examples:
* rates.csv or rates20061201.cvs will be loaded into the rates table
* profiles.csv or profiles20061201.cvs will be loaded into the profiles table
* destinations200601.csv will be loaded in the destinations table
* ratesHistory200801.csv will be loaded in the rates_history table
Do not use 'billing_' prefix in front of the file name.
It is advisable to name the files in a consistent manner like tableYYYYMMDD.csv
The results of the import operation is logged in the database and can be viewed in the Log section of the web interface and the syslog.
h3. Renormalizing CDRs
Sometime is useful to be able to change the rates for calls that have been already normalized and rated, for example after changing the rating tables you wish to apply the changes for the previous month for a customer.
To re-rate the CDRs do the following:
# Change the current rates by using cvs files/WEB interface or add rates valid for specific dates/destinations in the rates_history table
# Re-normalize the calls to be re-rated by either selecting ReNormalize check-box in the search screen or by changing the Normalized field in the CDR MySQL table (e.g. radacct): radacct):</li>
Examples:
# Re-rate calls for this month (2004-12) SIP domain example.com:
<pre>
UPDATE radacct set Normalized = '0' where Realm = 'example.com'
and AcctStartTime >= '2004-12-01'
</pre>
# Re-rate calls for SIP subscriber " sip01@example.com":mailto:sip01@example.com:
<pre>UPDATE radacct set Normalized = '0' where UserName = 'sip01@example.com'</pre>
# Apply rating again using command:
<pre>/var/www/CDRTool/scripts/normalize.php</pre>
Notes
Renormalization process can take long time during which your database (radacct table) will be intermitently locked. Perform this operation only during low traffic periods.
It is advisable to re-rate only the CDRs for destinations that have different rates. To do this, select a filter in the CDR search screen, if the selection is right re-run the query by selecting Re-normalize button.
After renormalization, the monthly usage information used by the quota system will be out of date. At the next run of the quotaCheck script, a full table scan will be performed. See QuotaSystem.txt for more information about quota.
Holidays must be added as individual days YYYY-MM-DD in table billing_holidays. The profile applied for holidays is the same as for week-ends. Holidays are global and cannot be specified per customer.
Renormalization process does not affect the balance of prepaid users. Prepaid is a real time un-reversible process, it goes in one direction. The prepaid balance is changed only by placing a call or adding credit to it. There are several reasons for this:
* The balance before and after each CDR is not known to be able to roll it back at a later time
* Re-rating correctly is mathematically not possible for prepaid users that have calls in progress
* If the prices are higher than previously debited and end up with a negative balance, the software cannot force the user to pay more retroactively
Re-normalization for the purpose of re-rating is useful only for postpaid accounts where you send an invoice at the end of the month and your can change things back and forth. If you need to perform manual credit/debit operations to some prepaid users because of faulty pricing, you can edit in CDRTool in the prepaid table the balance by using + or -.
h2. Rating Assignment
To asign a particular rating plan to a SIP account, SIP domain of Trusted Peer:
h3. SOAP/XML functions
* RatingPort->setEntityProfiles()
* RatingPort->deleteEntityProfiles()
* RatingPort->getEntityProfiles()
h3. Graphical clients
CDRTool->Rating->Customers
SIP settings page->Settings tab->Billing Profiles
h2. Prepaid Accounts
"!http://callcontrol.ag-projects.com/raw-attachment/wiki/WikiStart/CallControl.png(http://callcontrol.ag-projects.com/raw-attachment/wiki/WikiStart/CallControl.png)!":http://callcontrol.ag-projects.com/raw-attachment/wiki/WikiStart/CallControl.png
Prepaid functionality can be enabled by setting the SIP account prepaid attribute to 'true'. As a result, a new record is created in CDRTool application, in the prepaid table. This record is used to maintain the balance of the prepaid account. A SIP account marked as prepaid can make PSTN calls within the limits of the balance associated with the CDRTool prepaid account.
Every credit and debit operation is logged into the prepaid_history table, which can be queried using SOAP functions or can be accessed using CDRTool web pages.
h3. SOAP/XML functions
* SipPort->addBalance()
* SipPort->getPrepaidStatus()
* SipPort->getCreditHistory()
h3. Graphical client
CDRTool->Rating->Prepaid
!cdr-prepaid-accounts.png!
CDRTool->Rating->Prepaid history
!cdr-prepaid-history.png!
h3. Voice prompts
Voice prompts can be played when the balance is not sufficient to make a new session. This feature must be enabled in the " SIP Proxy configuration":http://msp-documentation.ag-projects.com/wiki/ConfigurationGuide#SIPProxy:
<pre class="wiki">
define(`USE_FAILURE_PROMPTS', `1')
# Voice message prompts for prepaid failure conditions.
define(`PREPAID_NO_CREDIT_PROMPT', `sip:800301@MEDIA_SERVER')
define(`PREPAID_ERROR_PROMPT', `sip:800399@MEDIA_SERVER')
</pre>
h2. Prepaid Cards
You may generate prepaid cards with designated values. The cards are randomly generated, identified by a combination of ID and NUMBER combination that can be safely sold to end-users or resellers in the form of scratch cards.
h3. SOAP/XML functions
Each prepaid card has a value, the value is transfered to the SIP account when the correct ID and NUMBER are supplied to the function:
* SipPort->addBalanceFromVoucher()
h3. Graphical client
The Batches with prepaid cards can be exported in comma separated values format to external systems, for example to a printer that makes scratch cards.
@CDRTool->Rating->Prepaid cards@
!cdr-prepaid-cards.png!