mod_opus

About

Opus codec implementation in Freeswitch.

Use the Opus libraries from the FreeSWITCH repository; the Debian repo has versions that are too old to work with newer builds of FS. Add the repo using the instructions on the FS 1.6 Video installation page.

Configuration

Configuration in autoload_configs/opus.conf.xml.

<param name="asymmetric-sample-rates" value="1"/>

Enable negotiation of sample rates. If set, FS will use maxplaybackrate and sprop-maxcapturerate from the incoming offer, and configure the encoder according to RFC 7587.

If an entity receives an fmtp: maxplaybackrate=R1,sprop-maxcapturerate=R2 and sends an fmtp with maxplaybackrate=R3,sprop-maxcapturerate=R4 then it should start the encoder at sample rate: min(R1, R4) and the decoder at sample rate: min(R3, R2).

Note though that until v1.6.11 this setting also impacted the decision of what bitrate to use when initialising the encoder. See 'negotiate-bitrate' below.

When set to 0, FS will ignore maxplaybackrate and sprop-maxcapturerate from the offer.

This is not an FTMP param.

<param name="use-vbr" value="1"/>

Enable variable bitrate. Opus VBR is disabled by default in Freeswitch (after FS 1.6.2) .

This is an FMTP param: https://tools.ietf.org/html/rfc7587#page-11 (cbr)

<param name="use-dtx" value="0"/>

Disable DTX . DTX is enabled by default (after FS 1.6.2)

This is an FMTP param: https://tools.ietf.org/html/rfc7587#page-11 (usedtx)

<param name="complexity" value="10"/>

Configures the encoder's computational complexity.

The supported range is 0-10 inclusive with 10 representing the highest complexity.

This is not an FMTP param.

<param name="maxaveragebitrate" value="0"/>

maxaveragebitrate: the maximum average codec bitrate (values: 6000 to 510000 in bps) 0 is not considered and is recommended by default.

• This will set the bitrate on the local encoder and instruct the remote encoder trough specific "FMTP" attibute in the SDP.

Example: if you receive "maxaveragebitrate=20000" from SDP and you have set "maxaveragebitrate=24000" in this configuration

the lowest will prevail in this case "20000" is set on the encoder and the corresponding fmtp attribute will be set

to instruct the remote encoder to do the same.

About maxaveragebitrate FMTP param : https://tools.ietf.org/html/rfc7587#page-10

<param name="maxplaybackrate" value="0"/>

maxplaybackrate: the maximum codec internal frequency (values: 8000, 48000 in Hz) . Defaults to 48000 hz .

• This will set the sampling rate of the local encoder and instruct the remote encoder trough specific "FMTP" attibute in the SDP.

About maxplaybackrate FMTP param : https://tools.ietf.org/html/rfc7587#page-9

<param name="sprop-maxcapturerate" value="0"/>

sprop-maxcapturerate: sample rate of the encoded stream we send to tell to the remote party . This param is optional and it can be any Opus rate: 8000, 12000, 16000, 24000 or 48000 hz.

About sprop-maxcapturerate FMTP param: https://tools.ietf.org/html/rfc7587#page-9

<param name="packet-loss-percent" value="15"/>

Initial packet loss percentage passed to the Opus encoder, for FEC (Forward Error Correction).

This is not an FMTP param.

<param name="keep-fec-enabled" value="1"/>

This option will keep FEC (Forward Error Correction) enabled on the encoder by slightly adjusting the bitrate, because at specific packet loss percentage and bitrate FEC will be turned off by the Opus encoder. This is a Freeswitch specific feature.

More about Opus FEC when encoding : https://tools.ietf.org/html/rfc7587#page-6

This is not an FMTP param.

<param name="use-jb-lookahead" value="1"/> 

This option will enable FEC decoding . With this option enabled the Opus decoder will look for the next packet in the Jitter Buffer if the current packet is missing and decode FEC from it if found, hence countering packet loss. The Jitter Buffer must be enabled.

More about Opus FEC when decoding: https://tools.ietf.org/html/rfc7587#page-6

This is not an FMTP param.

 <param name="advertise-useinbandfec" value="1"/>

This option will put useinbandfec=1 inside sent FMTP. Use this to let the other party know it's okay to send packets with FEC. This is a FMTP param: https://tools.ietf.org/html/rfc7587#page-11 . Should be enabled by default , but there are situations where this should be turned off.

<param name="adjust-bitrate" value="1"/>

When set to 1, this option allows the RTP estimators to provide indications whether the bitrate should be dynamically increased or decreased according to packet loss (as detected in the received RTCP info).

This is a FreeSWITCH specific feature (available from v1.6.11).

This is not an FMTP param.

<param name="bitrate-negotiation" value="1"/>

When set to 1, will tell FreeSWITCH to evaluate the remote party's maxaveragebitrate when deciding how to initialise the opus encoder.

This is a FreeSWITCH specific feature (available from v1.6.12).

This is not an FMTP param.

More info:

FreeSWITCH And The Opus Audio Codec)

Comments:

other optional params for opus.conf.xml (as of d18c964ba703df54efc56a313ff605cc4c9a5c98 ): sprop-maxcapturerate: (int) maximum input sampling rate to tell to the remote receiver (values: 8000, 12000, 16000, 24000, 48000 in Hz) 0 is not consideredeg: <param name="sprop-maxcapturerate" value="8000"/> asymmetric-sample-rates (boolean)eg: <param name="asymmetric-sample-rates" value="1"/>Feature "asymmetric-sample-rates" implements the following interpretation of: https://tools.ietf.org/html/rfc7587 :If an entity receives an fmtp: maxplaybackrate=R1,sprop-maxcapturerate=R2 and sends an fmtp with:maxplaybackrate=R3,sprop-maxcapturerate=R4then it should start the encoder at sample rate: min(R1, R4) and the decoder at sample rate: min(R3, R2)*/ keep-fec-enabled (boolean)eg:<param name="keep-fec-enabled" value="1"/>Feature "keep-fec-enabled" tries to FEC enabled on the Opus encoder, depending on the packet loss received through RTCP .It increases the bitrate slightly , only to make more room for FEC. Usefull for certain packet loss patterns (lots of small gaps) . packet-loss-percent (int)eg:<param name="packet-loss-percent" value="40"/>Start the encoder with this value that will be taken into consideration when the encoder is coding FEC information inside the voice payload. Posted by dragos_oancea at Jul 13, 2015 19:40
use-dtx (boolean)value 1: enable DTX value 0: disable DTX (default )eg:<param name="use-dtx" value="0"/>https://tools.ietf.org/html/rfc6716#section-2.1.9 Posted by dragos_oancea at Jul 15, 2015 19:03
I am going to leave this example configuration here with explanations . This is for transcoding OPUS@8000h to PCMA or PCMU or any other codec with 8000 hz sample rate. You should use this only if you do this kind of transcoding , for other things (like WebRTC) you will probably want to use the 48000 hz version (OPUS - which in default in FS's default config files).Moreover this config is good if you have client apps on congested WiFI networks as it addresses audio quality. The client apps have to send RTCP with proper info about current packet loss. opus.conf.xml :<!-- for opus@8000h (NARROWBAND)--><configuration name="opus.conf"> <settings><!-- "use-vbr" setting - set OPUS in VBR mode - consume less bandwidth . VBR is OPUS's native mode --><param name="use-vbr" value="1"/> <!-- ""use-dtx" setting - do not use DTX . perhaps the remote client/user app does not support it --><param name="use-dtx" value="0"/> <!-- "complexity" setting - this is a codec specific setting . we assume you have a powerfull machine to do transcoding and complexity can be 10 --> <param name="complexity" value="10"/> <!-- "maxaveragebitrate" setting - this bitrate is chosen so that FEC is present in the payload whenever the Opus codec decides it can add it. There is a threshold bitrate for FEC activation in OPUS : 12400 . Lower than this bitrate , there will be no FEC --><param name="maxaveragebitrate" value="14400"/> <!-- "maxplaybackrate" setting - sample rate of encoder . this will tell the remote party that it's pointless to send us higher rates, because we will downsample anyway, so better send 8000 to spare us some CPU–><param name="maxplaybackrate" value="8000"/> <!-- "packet-loss-percent" setting - is the initial packet loss that we pass to the encoder. This value in conjunction with the maxaveragebitrate value will make the codec decide if it will use FEC or not –> <param name="packet-loss-percent" value="15"/><!-- "keep-fec-enabled" setting - will enforce the encoder to use FEC regardless of the maxaveragebitrate supplied to it. In fact it will recalculate a new bitrate based on the packetloss percentage taken from RTCP arriving to us from the user app if the packet loss is higher than 10%. You might expect that this bitrate would be lower than the maxaveragebitrate, but in reality the actual bitrate will vary with packet loss during the call - but only to keep FEC enabled. If there's no more packet loss during the call then the bitrate will go to the supplied maxaveragebitrate (the one set from remote fmtp or local config file). ###This works in today's networks . For example a congested WiFi router or core router will drop a 60 bytes packet (40 bytes IP/UDP/RTP header + 20 bytes voice payload ) as easy as an 70 bytes packet , so if you increase the bitrate with 10 bytes per packet (it's the price to pay of having FEC enabled - but the good side is that only certain packets will contain it ) will not matter that much for that congested router, it will continue to forward some packets and drop some packets in the same manner. It's a packet-loss pattern (many small gaps). The usage of FEC will improve the audio quality because there will be redundant information in the packets eventually arriving at the decoder after the congestion adventure.This is just an example, don't take these values above exactly as they are (60, 70 , 10) . Only 40 bytes for header overhead is a correct value and it's the only fixed value in calculating network bitrate . "maxaveragebitrate" is an average , and the actual network bandwidth used might be a little bit higher or a little bit smaller , depending on the content being encoded. The voice payload size varies (because Opus is VBR) and so does the FEC payload inside the voice payload. ###--><param name="keep-fec-enabled" value="1"/> <!-- "use-jb-lookahead" setting - you must enable the jitter buffer for FEC decoding to work . This setting has impact on decoding received FEC - it will improve the audio quality greatly under packet loss conditions . We decode incoming FEC by looking in the jitter buffer, checking if we have the next packet (only if the current packet is missing due to packet loss ) and then checking if it has FEC info in it. If it has FEC, then first we decode the FEC and then we decode the main payload. This is n+1 FEC . --> <param name="use-jb-lookahead" value="1"/> <!-- Tell the remote party it's okay to send us voice payloads with FEC . If we don't have a jitter buffer activated we will just ignore the voice payloads with FEC, and play only the main part of the voice payload --> <param name="advertise-useinbandfec" value="1"/> </settings></configuration> Notes:From FS 1.6.11 there is another setting called "adjust-bitrate" that you can put in opus.conf.xml - this will attempt to vary the bitrate during the call as well and it will work in all modes ."adjust-bitrate" would work in the whole bitrate range .The "keep-fec-enable" feature only has 25 magic bitrates between 12400 and 15600 and they correspond to 1-25% percentage loss. Of course this will work for packet loss higher than 25 %, it will just return 15600.FEC only works in NARROWBAND and WIDEBAND. in OpusBecause OPUS is such a great codec, it will decode any kind of Opus voice packet it will receive regardless of sample rate and bitrate. One again thanks Claude Lamblin for her codec insight and the Freeswitch developers to merge all the Opus stuff , and especially thanks Anthony for changing the jitter buffer to accommodate Opus . And thanks Giacomo for asking the right questions. This development was all done back in 2015 .For more clarification look into the code or email me. Posted by dragos_oancea at Oct 05, 2016 14:43