Call recording
General
The platform can record calls.
A necessary condition is media traffic processing by the server. Recording cannot be performed if the configuration sets call gatewaying without media processing. For example, the call arrived through the corresponding SG or ESG instance, or it is a dialog with IVR and media processing is disabled in the B2B instance servicing it for such calls.
If recording is enabled, it starts immediately after the called subscriber answers and lasts until dialog completion. Each dialog is recorded independently of the others.
If several recordings were formed during the dialog due to media gateway migration, the final recording is merged.
The recording contains only the audio of the initiator and the called subscriber. The server-generated hold melody does not enter the recording; silence is recorded instead.
By default (unless configured otherwise) the recording is stored in an MP3 file in stereo mode. The left channel carries the initiator stream, and the right channel carries the called subscriber stream.
It is possible to enable recording temporarily in an already ongoing dialog (via the API).
Recording rules
Recording is enabled or disabled according to the recording rules applied to each new dialog.
When both subscribers are established (the called subscriber answered), the most priority recording rule suitable for this dialog is searched in their common domain based on the numbers of its participants.
If the subscribers belong to different domains, recording rules are executed in both based on the given numbers of both participants. If neither domain requested recording, no recording is performed. If at least one domain requested recording, recording is performed and mixed, but stored only in that domain. If both domains requested recording, recording is enabled, mixed, and copied to storages of both domains.
A recording rule enables the necessary recording modes, which can operate simultaneously. For local recording the rule can also specify the domain storage where the recording will be placed (see Selecting the domain storage).
Recording modes
Local recording by platform means
Local recording is the standard mode. The media gateway, simultaneously with media traffic exchange between subscriber devices, saves each channel to a local file.
After dialog completion the mixer retrieves the necessary files and merges them into one. Then the files are distributed to storages configured in the domains.
Such recordings are present in the call journal and available for listening from platform applications (according to role policy) or via the API.
External recording
Simultaneously with regular recording or instead of it, each dialog can be recorded to an external server by making special call(s). A mode of several simultaneous calls to several external servers can be configured. Recording by several different modes simultaneously can be configured.
With external recording, the formation, storage, and administration of the recording archive, as well as providing recordings to users, are moved outside the platform — to the supersystem.
External recording via RFC-7866 (siprec)
Recording is performed by a special call to an external recording server via RFC-7866. Responsibility for recording and storage, as well as providing recordings to users in this case, lies with the external system.
Recording is enabled by the recording rule 'siprec' field and requires configuring connection to recording servers, as well as some other options.
More about configuring this mode: 'siprec' settings
External recording by two calls (streamed_call_rec)
Recording is performed by two SIP calls to an external recording server, where each call relays the channel of one participant of the recorded dialog. Responsibility for recording and storage, as well as providing recordings to users in this case, lies with the external system.
Recording is enabled by the recording rule 'streamed_call_rec' field and requires configuring connection to recording servers, as well as some other options.
More about configuring this mode: 'streamed_call_rec' settings
External recording by one call (mixed_call_rec)
Recording is performed by a SIP call to an external recording server relaying a mixed channel with streams of both participants of the recorded dialog. Responsibility for recording and storage, as well as providing recordings to users in this case, lies with the external system.
Recording is enabled by the recording rule 'mixed_call_rec' field and requires configuring connection to recording servers, as well as some other options.
More about configuring this mode: 'mixed_call_rec' settings
Transcription
The platform can transcribe recorded conversations.
Only dialog recordings made by the system itself (local recording) can be transcribed.
Transcription is enabled or disabled by transcription rules. These rules, like recording rules, are applied to each dialog in the domains of its participants. Only dialogs for which at least one domain requested transcription are transcribed.
Transcription requires configured access to a text recognition service. A docker image with a recognition system can be supplied with the system, and a container started. Accesses to it must be configured in the master domain settings.
Transcription consumes a lot of resources. One server (container) can transcribe up to 6 dialogs in parallel. Transcript preparation time may be equivalent to the dialog duration itself. A queue of tasks to the service is formed from dialogs requiring transcription.
Transcription results in dialog format are placed in the corresponding field in the call journal.
The transcription quality of the built-in service is far from ideal, especially places with terms or national language specifics.
Microservices
Recording and its processing are provided by the following microservices:
-
B2B — organizes the dialog, applies recording rules, initiates recording, and places the completion event for the mixer. When activating special recording modes with external server calls — initiates these calls to external servers directly (point-to-point).
-
MGC — relays commands from the control layer to the media gateway servicing the context.
-
MG — provides RTP traffic exchange, performs local recording of each stream separately, and stores local recordings for 6 hours.
-
MIXER — retrieves recordings from the media gateway, mixes them according to configuration settings, and places the recording in the shared directory (NFS or local). There may be several mixer instances, and they compete for tasks.
-
RECMOVER — moves the recording to storages of the domains whose subscribers participated in the dialog and whose rules enabled recording. There is one active recmover instance per site.
-
MWARE — initiates recording transcription and transcript placement.
Recording movement path
Process description
-
The B2B microservice services a specific conversation and manages its media context. With recording rules configured, a command to save RTP traffic is passed to the media gateway. If external recording (SIPREC) is enabled by rules, then simultaneously with the conversation start, the configured SRS is called per RFC7866 with relaying of streams of both subscribers participating in the conversation.
-
The active current media gateway — the MG microservice — writes locally to the rectemp directory (server configuration). Channel recordings are stored there for no more than 6 hours.
-
Mixer initialization for recording preparation is performed by the mixer.task event through the broker (BROKER microservice) — generated either from B2B or from CallStore (if the B2B service crashed).
-
Mixers — microservices, broker consumers — compete for tasks without exceeding the parallel load limits set in the configuration.
-
The mixer creates a task and puts it in its working folder (
cachefolderparameter). -
The mixer asynchronous handler (queue) processes tasks and retrieves per-channel recordings for the conversation from the task from all media gateways (several media gateways if there was migration during the conversation). Places them in its working folder and deletes them from the media gateway rectemp folder.
-
The mixer brings recordings to a common format, aligns them, fills gaps with silence, mixes and concatenates them (if several) — all according to timestamps in the RTP channel recordings. As a result a resulting file (mp3 or wav) appears in the working directory of this microservice. This is performed in a lower-priority process.
-
Continuing the process, the mixer applies the analytics service — calculates the percentage of ownership and interruption from the recording.
-
The mixer corrects the task and passes it to another queue — placement into the mixer storage. While the file is in the working directory of the specific mixer.
-
The mixer asynchronous handler (queue) processes placement tasks — moves files to the recpath directory (configuration: mixer.recstorageid → server.recstorepaths.Key → aliases.Key, corresponding to a docker volume). This directory may be NFS. It must contain the signal file rshare.sign. While the file is absent, the mixer considers the directory not ready and not mounted into storage. If the directory is not connected to NFS, the recording at this stage is stored on the single specific server in this directory. But the signal file rshare.sign must also be there. The file path contains no domain name, only date-time by minutes. The mixer logs the processing stages.
-
After moving the file to the specified storage, the mixer deletes the task and artifacts from the working directory and generates events to the broker (callevents.call_rec_links, recmover.task).
-
The RecMover microservice is a broker consumer and, receiving a task, services it through a queue. The file placed by the mixer is downloaded from the local recpath folder, which may or may not be an NFS client. If the file is absent there (the system is multi-server, NFS is not connected), a request is made to the mixers of the current site. The file is either moved to the server with the mixer or copied (recmover mode configuration parameter). Depending on whether the call was intra-domain or cross-domain, the recording is sent according to these domains' settings to the corresponding storages (see Selecting the domain storage). The recpath directory must also contain rshare.sign. If the directory is an NFS client, all operations are performed locally and effectively reside in the same storage in adjacent subdirectories. In TRACE mode recmover prints a task execution trace. Errors, including S3 access, are there. On failure the task is stored in recmover for some time and periodically retries processing (see Behavior on storage unavailability). At the end the callevents.records_moved event is generated, as well as asr.task — a task to the broker for conversation transcription.
-
The transcription service runs within mware. According to transcription rule settings — downloads the conversation from the recording domain, unpacks it into channels, accesses the external speech recognition service, and composes the result into text dialog format. At the end it generates the callevents.call_asr event.
Recording storage
Locally made recordings pass through several directories and are ultimately placed in storages of the dialog participant domains. The permanent storage location of the recording is the domain storage. All previous directories are intermediate.
| Stage | Directory | Configuration |
|---|---|---|
Channel recordings on MG |
RECORD (rectemp), up to 6 hours |
mg role parameter; at installation — a docker volume on each server |
Mixer working directory |
By default |
mix.cachefolder → |
Shared mixer result storage |
RECSTORE (recpath) |
mix.recstorageid → |
Domain storage |
S3, a mounted NFS partition, or the default partition in RECSTORE |
storage entity, recording rule recordrule.storagekey, domain setting settings.rec_storagekey_default, recmover configuration parameter recmover.default_recstorageid |
Channel recording directory
During the dialog recording is performed by the MG microservice into the local RECORD folder. By default the configuration specifies for this a folder attached as a docker volume to the platform docker container. During system installation on each server the installer asks for the host location of this folder.
It is recommended that this folder be placed on a local disk without the risk of forming a write access queue. Although writing data to the file is asynchronous, opening the file itself is synchronous and strictly time-limited. With a timeout of several seconds — the conversation will not happen.
Mixer directories
The mixer uses two directories.
Working directory is set by the cachefolder parameter. By default it is the node working directory in the /var/lib/era partition of the container.
It holds tasks, downloaded channel recordings, and mixing results before they are placed into shared storage.
Shared storage is set by the recstorageid parameter. Since the system is multi-server and absolute paths on servers may differ, recording directories are described in two levels: path aliases in the aliases section, and keyed sections in the server recstorepaths parameter.
mix.recstorageid ──► servers[].recstorepaths[key] ──► aliases.paths[alias] ──► path in container
The mixer places results in shared storage only if the rshare.sign signal file is present in the directory. The file is placed by the installer during installation. When preparing a new directory it must be copied manually.
|
Domain storages
Domain storages are storage entities with typed connection parameters. Of the file types, two are applicable for storing conversation recordings:
-
s3— external S3 object storage; -
nfs— a directory whose path is defined uniformly for all containers on all servers. As a rule — a mounted network partition, but the directory may also be local.
The fs and fsync types are not intended for storing conversation recordings.
Selecting the domain storage
Recmover places the recording in each domain that requested recording as follows:
-
The storage whose code (
instance) is specified in the storagekey field of the recording rule applied by the domain and enabling recording is used. -
If the field is not filled, the recording directory code from the domain settings.rec_storagekey_default parameter is used, and if that is not set — the default partition.
-
If a storage with such code is absent in the domain at that moment or has an unsuitable type, the recording is placed in the default partition, i.e., the partition where the mixer placed the recording, but in the domain subdirectory
domain/{DOMAIN}.
The default partition is determined as follows:
-
starting from version 1.11.1 — by the recmover default_recstorageid parameter, following the chain
recmover.default_recstorageid→servers[].recstorepaths→aliases.paths; -
if
default_recstorageidis not set or empty (and in all versions before 1.11.1) — the same partition where the mixer placed the recording (the mixerrecstorageid).
From the shared mixer storage the recording is deleted after moving, or remains there, depending on the recmover role mode parameter.
Default scheme
When installed via the installer with default parameters, the platform does not use network storages:
-
in the container the
/var/lib/era_files/recpathdirectory is mounted as a volume to the host directory (by default/opt/era/era_recpath); -
mixer instances in configurations created by the configuration wizard point to this partition;
-
recording rules do not specify storage, so recmover places domain recordings in the same partition, in the domain subdirectory.
In a single-server configuration all recordings end up on the server disk.
In a multi-server configuration the recording is placed locally on the server where the active recmover instance ran; if necessary recmover first retrieves the file from the server where the mixer ran. If the active recmover ran on different servers at different times, the archive is distributed among them. When accessing a file, all servers with active recmover instances are iterated.
The scheme requires no additional configuration. Its limitations are local disk volume and the fact that the recording archive is distributed across servers and depends on their preservation.
To extend or change the partition in this scheme, use the instruction Connecting a new partition for recording storage.
Using a mounted partition
For centralized storage the recpath directory can be connected to network storage — NFS, virtiofs, and similar. Then all servers see a single directory, and the archive does not depend on individual server disks.
If at the same time the shared mixer storage and domain storages remain in the same mounted partition (as in the default scheme), each recording passes through the slow interface multiple times:
-
the mixer writes the file to the network partition;
-
recmover reads it from the network partition and writes a copy to the domain storage in the same network partition;
-
recmover deletes the source file in the network partition.
Metadata operations on network and virtualized file systems are significantly more expensive than on a local disk. In addition, short-lived mixer files and the constantly growing domain archive are mixed in one partition.
| In versions before 1.11.1 periodic pauses in mixer operation could be observed with such a scheme: scheduled cleaning of empty directories in the shared mixer storage also covered the growing domain recording archive. |
Local partition for mixer results
The target scheme when using network storage:
-
the mixer places results on a local server partition;
-
domain storages are located in the network partition, or in external S3 (see Recommended solution: archive in S3).
In this case only one operation passes through the network interface — writing the final file to the domain storage. Deleting the source file and servicing the shared mixer storage are performed on the local disk.
Below describes the transition to this scheme for a configuration where the mixer and domain storages already use one mounted partition. The recording archive remains in the previous place, and links to previously made recordings continue to work.
Conventional values are used in the examples:
| Value | Purpose |
|---|---|
|
Previous partition key and path alias to the mounted partition. Not changed. |
|
New partition key and path alias to the local mixer directory. |
|
Code of the |
Step 1. Preserve domain recording placement
Before switching the mixer, it must be ensured that recmover continues to place domain recordings in the previous partition. Otherwise domain recordings for which storage is not defined will end up in the default partition, i.e., in the new local mixer directory.
Option A, since version 1.11.1. Set the default_recstorageid parameter of recmover instances to the previous key:
{
"roletype": "recmover",
"default_recstorageid": "main"
}
Domain configuration is not required in this case.
Option B. In each domain where recording is done, create an nfs type storage referencing the previous partition key:
{
"type": "nfs",
"instance": "recnfs",
"params": {
"configkey": "rec:main",
"prefix": "domain"
}
}
where configkey has the form rec:<KEY> (<KEY> is the previous mixer recstorageid), and prefix is the subdirectory where recmover placed domain recordings.
Then specify "storagekey": "recnfs" in the domain recording rules enabling recording (rec = 1), or set recnfs in the domain rec_storagekey_default parameter.
Step 2. Prepare the local directory
On each server with mixer instances select a directory on a local disk mounted to the container as a volume — for example, a subdirectory of siteshare if it is located on the host local disk. A directory inside the container layer is not the best choice.
Create the directory and place rshare.sign in it:
mkdir -p <HOST_SITESHARE>/mixrec
touch <HOST_SITESHARE>/mixrec/rshare.sign
Verify that the selected directory is truly not located on a network partition. The actual siteshare path is set by the sitesharepath server parameter.
|
Step 3. Modify the configuration
aliases.paths{ "alias": "mixrec", "value": "/var/lib/era_files/siteshare/mixrec" }
recstorepaths of each server with a mixer (the previous key is preserved)"recstorepaths": [
{ "key": "main", "value": "alias://paths/recpath" },
{ "key": "mixlocal", "value": "alias://paths/mixrec" }
]
{
"roletype": "mix",
"recstorageid": "mixlocal"
}
Activate the configuration.
Step 4. Verify
-
Make a test recorded call.
-
Verify that the file briefly appears in the local mixer directory and disappears after recmover processing.
-
Verify that the recording appeared in the previous partition next to previously made recordings and plays from the interface.
-
Check availability of previously made recordings.
Behavior on storage unavailability
Mixer. If shared storage is unavailable or lacks the rshare.sign signal file, the mixer does not place recordings but continues to take channel recordings from MG and mix them. Results accumulate in the working directory (cachefolder).
Since channel recordings are stored on MG for no more than 6 hours, this behavior allows preserving recordings for a longer period — until storage restoration.
The working directory size grows; there may be several mixer instances, and each accumulates its own part of recordings.
When choosing the partition for cachefolder, consider the expected storage unavailability time. For estimation: with mp3 format at 24 kbit/s, a minute of recording takes about 180 KB, an hour of conversations — about 11 MB. By default the working directory is located in the /var/lib/era partition of the container.
|
Recmover. When storage is unavailable, recmover saves tasks and periodically retries: up to 100 thousand recordings within 24 hours. After this time the task is deleted. There is one active recmover instance per site.
Monitoring. If storages are unavailable or access to them takes significantly longer than necessary, the mixer and recmover place warnings in system health monitoring.
Mounted partitions. When the NFS server is unavailable, processes accessing the mounted partition freeze until its restoration.
| In versions before 1.10.7 the blocking scope on NFS server unavailability could be wider: work with the file system as a whole became dependent on the network partition, which could block platform operation. |
Requirements for network file storage
If the recording archive is placed in network file storage, it becomes a critical component:
-
storage must be fault-tolerant (ideally an HA solution);
-
storage must be administered separately from the platform: backup, volume control, cleaning of outdated recordings.
A local partition for the mixer reduces but does not eliminate dependence on the network partition: recmover and archive access still access it.
Recommended solution: archive in S3
The most reliable and recommended way to store the recording archive is external S3 storage.
-
In the domain create an storage of type
s3:{ "type": "s3", "instance": "records_s3", "params": { "endpoint": "https://s3.example.ru", "bucket": "call-records", "prefix": "domain1/", "keyId": "...", "secretKey": "..." } } -
The storage code is specified in the
storagekeyfield of domain recording rules, or in the domainrec_storagekey_defaultparameter.
In this case:
-
the recording archive is moved outside the platform and administered separately, as it should be;
-
archive access does not depend on mounted partitions in server file systems;
-
each domain can use its own storage, bucket, and retention policies.
The shared mixer storage in such a scheme remains an intermediate buffer and is placed on the local disk of servers.
Final recommendations
| Situation | Recommendation |
|---|---|
Single-server installation, moderate recording volume |
Default scheme. Control disk space. |
Centralized archive required |
S3 storage in domains; shared mixer storage — on a local partition. |
NFS or another mounted partition is used |
Separate directories: mixer on a local partition, domain storages on the network. Provide HA and separate storage administration. |
Recording and archive are maintained by an external system |
External recording, for example SIPREC. |
Access to recordings for listening
Listening to recordings by platform means (via the API or from platform applications) is possible only if the recording was made in local mode.
Access to recordings is determined by role policy.
For file access a temporary link with a complex unique name is always formed, publicly available for 5 minutes.
In special cases access can be configured via permanent links in an authorized connection with access to the corresponding endpoints:
-
using the xref:api:api/admin/callrecords.adoc endpoint, parameterized directly by the recording address in storage.
-
by enabling the option to save call and recording information into a separate archive collection and accessing attachments.
Conference recording
A conference is a separate user agent with its own media context. Each participant is connected to the conference media context through a dialog with its own media context within the call.
Like other media contexts, the conference context has a recording function, but it is unavailable to the user. In complex conference topologies there is no unambiguously correct participant composition for forming a single recording file. Therefore the platform does not record the conference media context, assuming recording is done in supersystems:
-
Each participant must be able to listen to exactly the part and composition of the conference in which they participated and could listen. Therefore the conversation of the subscriber with the conference can be recorded. This is controlled by standard recording rules. Given the subscriber has access to recordings of their conversations, they will also have access to the conference recording (i.e., their conversation with conference participants).
-
Any number of technical participants — recorders — can be added to the conference at any moment, and the required topology set for them. For example, these can be calls from IVR with the "Record" component. In this case placement and access to this recording are the responsibility of the script and supersystem.