Codecs and media negotiation

Summary

  • Call codecs are negotiated by SDP exchange using the offer/answer model (RFC 3264): one side offers a list of formats (offer), the other answers with those selected from it (answer).

  • The platform tries not to transcode: when passing the offer to the other leg, formats known to the platform from the incoming SDP remain first in their order, and additional codecs are added after them (except --* and the esg offer into the platform — see opts.payloads_audio).

  • The composition and order of additional codecs is set by the payloads_audio_offer role parameter. The base codecs PCMU/8000, PCMA/8000, and telephone-event/8000 are always offered unless explicitly excluded (except the esg offer into the platform, the provider leg with opts.payloads_audio, and audio_formats of recording servers).

  • For a provider the opts.payloads_audio codec list is optional, and if set — exact: only the specified codecs in the specified order are offered on the provider leg. Applied only when servicing media through the bgmg media gateway.

  • The answer sent by the platform contains a set of codecs or one codec depending on the allow_ac_set parameter.

  • Video in SIP calls is generally passed through by the platform without decoding.

Terms

Leg

A call segment between two nodes. Each signaling microservice on the call path services two legs — incoming and outgoing. A call of an internal subscriber to a provider (subscriber → sg → b2b → esg → provider) has 4 signaling legs. There can be as many media legs as there are media gateways in the processing chain — from 0 to 3, depending on tasks and settings.

Format (fmt, payload type, PT)

The number under which the codec is specified in the SDP m= line. Numbers 0-95 are static and assigned to codecs (0 — PCMU, 8 — PCMA, 18 — G729, etc.), 96-127 are dynamic and linked to a codec by the a=rtpmap attribute.

Termination

The connection point of a leg stream to the media gateway. Each media leg has its own termination with its own local and remote SDP.

Transcoding

Re-encoding audio from one codec to another by the media gateway. Depending on the codec combination it may require additional resources: for example, repacketization between PCMU and PCMA is table-based and almost free, while encoding to G729 or opus is noticeably more expensive.

Supported codecs

Codec names in parameters are specified in Name/Freq format; case does not matter.

Codec Default PT Note

PCMU/8000

0

G.711 μ-law. Base

PCMA/8000

8

G.711 A-law. Base

GSM/8000

3

G722/8000

9

Wideband

CN/8000

13

Comfort noise. Not used as the main codec

G729/8000

18

speex/8000, speex/16000, speex/32000

97, 98, 99

Dynamic

telephone-event/8000

101

DTMF signals per RFC 2833/4733. Base. Not used as the main codec

G726-16/8000, G726-24/8000, G726-32/8000, G726-40/8000

102–105

Dynamic

iLBC/8000

106

Dynamic

opus/48000/2

107

Dynamic

Video: H263/90000 (34), H263-1998/90000, H264/90000, VP8/90000, VP9/90000 (dynamic).

Unknown audio formats are discarded when SDP is processed for a media gateway with transcoding; unknown names in parameters are ignored. Unknown video formats are not discarded: video is passed through, see Video.

Base codecs

PCMU/8000, PCMA/8000, and telephone-event/8000 are always offered, even if not specified in payloads_audio_offer (except the esg offer into the platform, the provider leg with opts.payloads_audio, and audio_formats of recording servers). Thus different roles of one platform instance with different settings (for example, ivr3 and b2b2) always have a common codec and do not answer each other with the final unsuccessful 415 Unsupported Media Type response.

A base codec can be explicitly excluded: -PCMU/8000 — do not add PCMU, -* — do not add any unspecified base codec.

Parameters

payloads_audio_offer, payloads_video_offer (roles)

Parameters of the b2b, conf, esg, ivr, prompt, sg roles. Applied when a role instance forms an offer: creates a new outgoing call or passes an incoming call offer to the other leg. sg applies the parameter only to the SDP of the OPTIONS answer.

Element Meaning

Name/Freq

The codec is offered; its position in SDP follows its position in the list

-Name/Freq

The codec is not added (usually a base one)

-*

No unspecified base codec is added

--Name/Freq

The codec is not added and is excluded from the incoming call SDP passed to the other leg

--*

Like -*; additionally, when passing the incoming call to the other leg, only the codecs specified in the list enter the offer, in its order (telephone-event/8000 and CN/8000 — at the end); other codecs of the incoming SDP are excluded

Final list: first the specified codecs in list order, then unspecified and non-excluded base ones. CN/8000 and telephone-event/8000 are placed at the end. If the parameter is not set or empty, only base codecs are offered.

For video there are no base codecs: the list defines composition and order.

opts.payloads_audio (provider)

The provider opts.payloads_audio field.

  • Media is not serviced (the provider media field is off, bgmg is not used): codecs are not applied to provider calls, SDP passes through esg unchanged. The field must be empty.

  • Media is serviced through bgmg, the field is set: only the specified codecs in the specified order are offered on the provider leg, and the media gateway transcodes if necessary; telephone-event/8000 is added at the end unless excluded by the -telephone-event/8000 or -* element. The list applies only to the provider leg. A call, re-INVITE, or INVITE with Replaces from the provider is accepted if its SDP contains a codec from the list, otherwise esg answers 488 Not Acceptable Here (re-INVITE is rejected without dropping the call). The answer to the provider contains the list codecs from its offer in list order; with opts.allow_ac_set = false — one: the one already used by the other leg, otherwise the first in the list. If the INVITE arrived without SDP, the inward offer is built from the list. The offer into the platform contains the provider codecs from the list in provider order (telephone-event/8000 — if not excluded), so transcoding to devices is performed by the b2b media gateway, not bgmg. The list codecs must be supported by the mg media gateways (codecs parameter) through which b2b calls are serviced. Internal legs are not restricted by the list: if the inward offer contains no list codecs, bgmg transcodes.

  • Media is serviced through bgmg, the field is not set: esg passes SDP unchanged, the media gateway passes media without transcoding — this reduces bgmg load. If the INVITE, re-INVITE, or INVITE with Replaces contains no main codec known to the platform, esg answers 488 Not Acceptable Here (re-INVITE is rejected without dropping the call). The payloads_audio_offer of the esg instance is applied only in two cases:

    • the provider reinvite field has value 0: re-INVITEs are not forwarded, esg answers them itself, and the media gateway can transcode. Taking payloads_audio_offer into account, INVITEs to the provider and INVITEs to the new target on provider-side transfer (REFER), which esg services, are built; the provider offer into the platform is passed with its codecs in its order (formats unknown to the platform are discarded). The esg answer to re-INVITE and INVITE with Replaces contains only the offer codecs; if there are no common ones — 488 Not Acceptable Here, and the re-INVITE is rejected without dropping the call;

    • the INVITE arrived without SDP: esg itself forms the offer to the other leg according to payloads_audio_offer.

The provider list order is intentionally not adjusted to the incoming SDP order: a list of several codecs may express a preference (for example, G729/8000 to save bandwidth, PCMA/8000 as a fallback). Therefore transcoding is possible: the internal device offered PCMA, and the provider chose the first G729 in the list. If transcoding on the media gateway is undesirable, put first the codec used by internal devices (for example, ["PCMA/8000", "G729/8000"]), or specify only it.

allow_ac_set

The b2b role parameter (default true) and the provider opts.allow_ac_set field (default false, applied with bgmg). Determines how many codecs the platform specifies in its answer to the offer:

  • true — a subset of the offer codecs known to the platform; first are the codecs selected by the opposite (called) side in its answer, if they are in the offer (the esg answer to a provider with a list — in list order);

  • false — one codec: the first in offer order that is also in the opposite side answer; if there are none — the first main codec of the offer. telephone-event/8000 and CN/8000 from the offer are added to it (the esg answer to a provider with a list — see opts.payloads_audio).

How SDP is built

New outgoing call

The role forms the offer itself, without incoming SDP: an outgoing call from an IVR script, a conference participant call, service calls. The list is built according to the role instance payloads_audio_offer.

Table 1. Examples (outgoing call from IVR, m=audio line)
payloads_audio_offer Formats in SDP

not set

0 8 101

["PCMA/8000", "PCMU/8000", "G722/8000", "CN/8000"]

8 0 9 13 101

["PCMA/8000", "PCMU/8000"]

8 0 101

["G722/8000", "PCMA/8000"]

9 8 0 101

["G722/8000", "PCMA/8000", "-PCMU/8000"]

9 8 101

["G722/8000", "PCMA/8000", "-*"]

9 8

["PCMA/8000", "telephone-event/8000", "G722/8000", "--*"]

8 9 101 — telephone-event/8000 at the end regardless of list position; in a new call --* acts as -*

["PCMA/8000", "-*"]

8 — only PCMA, without DTMF per RFC 2833 (see DTMF without telephone-event)

For calls to SIPREC and streamed recording servers the codec composition can be set separately — by the server audio_formats field in the domain settings siprec_options, mixed_call_rec_options, streamed_call_rec_options. If the field is set, exactly the specified codecs are offered in list order (telephone-event/8000 and CN/8000 — at the end if specified), and payloads_audio_offer is not applied; if not set — the offer is built according to the b2b instance payloads_audio_offer.

Passing the offer to the other leg

The role receives an INVITE with SDP and passes the offer to the other leg (b2b; esg — see opts.payloads_audio). Order:

  1. formats of the incoming SDP known to the platform — in their order (except excluded --);

  2. then missing codecs according to payloads_audio_offer — in final list order;

  3. CN/8000 and telephone-event/8000 — at the end.

Incoming formats remain first because the called side usually selects the first suitable format. If it selects any format offered by the calling side, the media gateway does not need to transcode: streams in this format are passed between legs unchanged.

Table 2. Examples (incoming SDP: m=audio …​ 8 0 101)
payloads_audio_offer of the b2b instance Formats in the offer to the other leg

["G729/8000"]

8 0 18 101

["PCMU/8000", "PCMA/8000"]

8 0 101 — incoming format order does not change

["PCMU/8000", "PCMA/8000", "telephone-event/8000", "--*"]

0 8 101 — with --* the offer is built in list order

["PCMA/8000", "G722/8000", "-PCMU/8000"]

8 0 9 101 — - does not exclude incoming PCMU

["G722/8000", "PCMA/8000", "--PCMU/8000"]

8 9 101

["PCMA/8000", "-*"]

8 0 101

["G722/8000", "--*"]

9 — the media gateway transcodes PCMA ↔ G722

Answer

The answer contains only formats from the offer it answers: static — by number, dynamic — by codec name. The count is per allow_ac_set.

Table 3. Example (leg A offered 8 0 101; b2b passed to leg B the offer 8 0 18 101, adding G729 per payloads_audio_offer)
Leg B answer b2b answer to leg A with allow_ac_set = true with allow_ac_set = false

8 101 (PCMA)

8 0 101

8 101

0 8 101

0 8 101

8 101

18 101 (G729, absent from A offer)

0 8 101

8 101

If the codec selected by leg B is in the A offer, there is no transcoding. Otherwise the media gateway transcodes.

Dynamic format numbers

In the answer dynamic formats are indicated under the numbers from the offer it answers (RFC 3264, section 6.1). In the offer passed to the other leg, the dynamic format numbers of the incoming SDP are preserved; added dynamic codecs receive free numbers not occupied in the incoming SDP. If the sides declared different numbers for one codec, the media gateway matches formats by codec name: each side receives a stream with numbers from its own SDP.

Example
Incoming offer:
m=audio 30000 RTP/AVP 8 96 101
a=rtpmap:96 opus/48000/2
a=rtpmap:101 telephone-event/8000

Offer to the other leg (payloads_audio_offer = ["G722/8000", "speex/16000"]):
m=audio 40000 RTP/AVP 8 96 9 98 0 101
a=rtpmap:8 PCMA/8000
a=rtpmap:96 opus/48000/2
a=rtpmap:9 G722/8000
a=rtpmap:98 speex/16000
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000

Call scenarios

Outgoing from IVR via provider

IVR → b2b → esg → provider.

  1. IVR forms a new offer according to its payloads_audio_offer.

  2. b2b passes it to the esg leg, adding codecs per its own payloads_audio_offer.

  3. esg on the provider leg: if the provider has opts.payloads_audio set (and bgmg is used) — only its codecs in its order; otherwise SDP is passed unchanged, and with reinvite = 0 — supplemented per esg payloads_audio_offer (see opts.payloads_audio).

  4. The provider answers; the answer on each leg contains codecs from that leg offer, and the media gateway connects the streams, transcoding if necessary.

codecs_ivr_provider

Call between subscribers

Device → sg → b2b → sg → device (sg may be the same instance or different ones).

sg using bgmg does not change codecs, passing the INVITE through, and works without transcoding. b2b passes the calling device offer to the called one, adding codecs per payloads_audio_offer; incoming formats remain first. If the called device selects any of the calling formats, there is no transcoding.

codecs_device_device

From device to provider

Device → sg → b2b → esg → provider. Example: the device does not support telephone-event/8000 (sends DTMF via SIP INFO), and the provider has opts.payloads_audio not set.

b2b supplements the offer with base telephone-event/8000, and esg passes SDP to the provider unchanged (with reinvite = 0 — supplements per esg payloads_audio_offer). The b2b answer to the device contains only the codecs of its offer, without telephone-event/8000. Provider DTMF signals per RFC 2833 are passed by b2b to the device via SIP INFO and back (see DTMF without telephone-event).

codecs_device_provider

Incoming from provider

Provider → esg → b2b → …​

  • The provider has opts.payloads_audio set (bgmg): esg accepts the call only if the provider SDP contains a codec from the list (otherwise — 488 Not Acceptable Here), and answers with the intersection codecs: a set or one per opts.allow_ac_set. The offer further into the platform contains the provider codecs from the list in provider order, and b2b supplements it per its own payloads_audio_offer.

  • Not set: esg passes the provider offer into the platform unchanged (with reinvite = 0 — without formats unknown to the platform); b2b supplements it per its own payloads_audio_offer. The answer to the provider contains codecs from its offer.

Different roles of one platform instance

Roles of one platform instance may be configured differently, for example ivr3 with ["G722/8000"] and b2b2 with ["G729/8000"]. Thanks to the base codecs they always have a common codec, and the call does not end with the final unsuccessful 415 response. When excluding base codecs (-PCMU/8000, -*), it is also worth making sure all instance roles keep a common codec.

sg and esg when using bgmg

  • sg does not change codecs: the INVITE passes through sg with its SDP, and there is no transcoding on sg.

  • esg acts per the provider opts.payloads_audio: if the list is set, it is applied to the provider leg SDP, the provider codecs from the list are passed inward, and transcoding is performed if necessary; if not set — the provider offer is passed inward with its codecs, and the esg instance payloads_audio_offer is applied to offers to the provider with reinvite = 0 and to INVITEs without SDP.

DTMF without telephone-event

DTMF signals are transmitted in RTP per RFC 2833 (format telephone-event/8000) or by SIP INFO messages (application/dtmf-relay). b2b repacketizes them between legs:

  • SIP INFO from one leg is passed to the other as RFC 2833 if the other leg SDP contains telephone-event/8000, otherwise — as SIP INFO;

  • an RFC 2833 event from one leg is passed to the other as SIP INFO if the other leg SDP does not contain telephone-event/8000.

Therefore excluding telephone-event/8000 (-telephone-event/8000, -*) does not deprive the call of DTMF if the device or provider accepts SIP INFO.

RFC 3264 and allow_ac_set

RFC 3264 allows exchanging codec sets in both the offer and the answer:

  • the offer lists formats in preference order (section 5.1); the offering side must be ready to receive any of them;

  • the answer must contain at least one format from the offer; it is recommended to answer with a subset of the offer formats, in the same relative order, with the same dynamic format numbers (section 6.1);

  • the answer may also include formats not present in the offer, but they are not used in the current session until they appear in a new offer;

  • during a session the sides can switch between negotiated formats without a new SDP exchange.

In practice most devices and providers answer with one codec from the offer. There are implementations that incorrectly handle an answer with a set of codecs: for example, a device plays silence when the stream arrives in a negotiated but not the first codec of the answer. To maintain operability of the platform with such devices and providers, the platform supports answering with one codec: allow_ac_set = false (for a provider this is the default). The parameter is set for the b2b role instance and for the provider; it is not set for an individual subscriber account.

Media gateway: tubes

The media gateway receives local and remote SDP for the termination of each media leg. local arrives as a template: addresses, ports, and specific attributes are substituted by the media gateway itself.

Between terminations of opposite legs, tubes are built — paths from each format of one leg to the formats of the other, in both directions. An RTP packet finds its entry by format number (PT) and passes through the tube into the opposite stream: without re-encoding if the format matches (format number from the receiving side SDP), or with re-encoding. Tubes are built so that transcoding is minimal: matching formats are connected directly.

Therefore a side changing codec within the negotiated set, as allowed by RFC 3264, is serviced without a new SDP exchange: packets of the other format simply go through another tube.

codecs_tubes

Video

The platform does not record video in SIP calls, so as a rule it passes it between legs without decoding: video formats, including those unknown to the platform, are passed as is. Video transcoding is off by default (use_video_transcoding parameter). payloads_video_offer sets the composition and order of video formats in formed offers; there are no base video codecs.

Currently video and web conferencing services are mainly used for video communication, including the meet service on board the platform (Video conferencing), so development emphasis has shifted there.

FAQ

Why are codecs in the INVITE in a different order than in payloads_audio_offer?

In the offer passed to the other leg, incoming SDP formats remain first in their order; the configuration list sets the order of added codecs. For a new call (IVR, conference) the order matches the configuration, with CN/8000 and telephone-event/8000 at the end.

How to offer only PCMA?

["PCMA/8000", "-*"]. When passing the offer to the other leg incoming formats are preserved; to leave only PCMA there too, use ["PCMA/8000", "--*"].

How to remove telephone-event?

-telephone-event/8000 in payloads_audio_offer or in the provider opts.payloads_audio. DTMF is then transmitted by SIP INFO, see DTMF without telephone-event.

Why was the call rejected with final unsuccessful 415 or 488?

The sides have no common codec: for example, base codecs were excluded and the offer has no codec from the list. esg answers 488 Not Acceptable Here if the provider SDP has no codec from opts.payloads_audio or the offer has no codec known to the platform.

Where to see the call SDP?

In trn logs and on the call sequence diagram (/api/monitor/v1/trace).