The first stage is a quick statistical and heuristic analysis of the signal. In parallel, the second stage searches for characteristic frequencies in the sound, pre-set by the engineer. +\ The third stage is performed only if a person is recognized based on the results of the first two: the recorded sound is compared with a database of audio fingerprints of known robots and answering machines in order to weed out false positives.

CPD Component (Response Detector)

:icons:font :hardbreaks: :imagesdir: ../../../assets/images :toc-title: Contents :toc:

Description

Used in outgoing campaigns to detect the response of a “living” person.

Records the start of a conversation for a specified time (up to 7 seconds by default) and during this time determines the type of audio source: human, robot/IVR, tone, fax or silence.

The analysis is performed in three stages. +\

The algorithm is not demanding on system resources. The parameters of the first stage can be further trained on your own labeled records, and the fingerprint database can be built on your own templates.

Table 1. System characteristics

Index

225

Short name

cpd

Script types

IVR

Start module

era_sip_ivr_script_component_cpd

Mode

Asynchronous

Icon

225

Branching type

Branching, closing

Properties

Table 2. Properties
Specification Description

Name: Timeout, s

Code: collectTimeSec

Visibility condition: no

Default: 5

An argument containing the time in seconds. Determines how long the conversation recording will be accumulated for further analysis by the response detector.

Name: Save recording file

Code: saveRecord

Visibility condition: no Default: No

Indicator of saving the recording file sent to the response detector service. Saving is performed in the directory /var/lib/era_files/answer_detector_records/<domain>/<type>

  • domain - the domain within which the conversation took place.

  • type - the type of result that the response detector returned.

Name: Result into variable

Code: result

Visibility condition: no Default: — 

Variable where the reason for the component’s shutdown is entered.

Name: Transition, detected

Code: transferFound

Visibility condition: no

Default: — 

The component to which control is transferred after a “living” person is detected by the response detector.”

"Name: Transition, not detected Code: transferNotFound Visibility condition: no

Default: — 

The component to which control is transferred if the response detector does not detect a “living” person in the recording.”

"Name: Transition, error

Code: transferError

Visibility condition: no

Default: — 

Component to which control is transferred in case of an error.

Operating principle

The analysis is performed by the console application cpd, which is part of the platform (./era_env/priv/bin/cpd/cpd).

The component runs cpd in analysis mode and feeds it the callee’s audio via standard input (stdin). \ The application “listens” to the beginning of the conversation and answers the only question: whether a live person answered the call.

This allows the automated dialer to connect an operator only when there is a high probability of a person on the line, and not silence, fax, beeps or a robot.

The component succeeds only with the result human.

Input data and result

The cpd accepts raw PCM data without header as input: 8000 Hz, 16 bit, mono. \ Data is read in chunks as it arrives and immediately passed through the analyzer.

The result is printed to stdout in one line:

Result

Meaning

human

A living person answered. The only result in which the component succeeds.

ivr

Robot, answering machine, voice menu, music while waiting.

tone

Tone: beeps, busy signal, answering machine tone, etc.

fax

Fax.

silence

During the analysis, not a single significant replica was found.

Termination conditions

cpd exits and produces a result on the first of the following events:

  1. The current stage of analysis has made a decision.

  2. Data does not arrive in stdin for longer than --recv-timeout (default 3000 ms).

  3. cpd runs longer than --timeout (default 7000ms).

  4. Stdin is closed by the sender.

When completed by a timeout, the result is given based on the data that has accumulated. If there were no significant replicas, the result is silence.

===Steps of analysis

Stage 1. Statistical and heuristic analysis

A lightweight algorithm that works in a stream as data arrives. He identifies cues and pauses between them in the sound and, based on their duration and nature, makes a decision:

  • the person is identified after a pause following the subscriber’s short greeting;

  • robot or IVR is determined by a long continuous replica;

  • tone and fax are determined by a stable signal amplitude, fax - by a longer duration;

  • silence is determined by the expiration of a timeout without significant remarks.

The algorithm uses a set of parameters. The default values ​​are hard-coded into the program and are not optimal. They can be selected for your own records in the learning mode (see [_learn_mode_learn]). The selected values ​​are saved in the file SettingsML.json, which is transferred to cpd with the key --ml-settings-path.

Stage 2. Search for characteristic frequencies

It works in parallel with the first stage on the same data. Unlike the first stage, nothing is selected automatically here: the required characteristics are specified by the engineer. Having examined the robot’s recording in an audio editor, he notes at what frequencies and in what time interval a characteristic burst is visible, and describes this in the SettingsSpectrum.json file (see SettingsSpectrum.json). cpd checks each described interval as soon as it is fully accepted, and if there is a match, it immediately ends the analysis with the result specified for that interval.

If the file is not specified, the search for characteristic frequencies is not performed.

Stage 3. Search in the audio fingerprint database

The first two stages work simultaneously, and the analysis is completed as soon as one of them has made a decision. The third stage is performed only if this solution is human (that is, the first stage recognized the person, but the second did not find characteristic frequencies) and the fingerprint database (--fingerprints-db) is specified. The goal of this stage is to weed out robots and answering machines that are similar in nature to humans.

The entire received record is compared with a fingerprint database built in advance from template robot records (see Fingerprint database construction mode (rebuild-fingerprints)). Fingerprints are a detailed, automatically constructed description of the characteristic frequencies of a recording over time, that is, the same thing that an engineer does manually in the second stage, but without human intervention and with greater detail.

If the entry matches a pattern, the result is replaced with the type of that pattern. If there are no matches, the result remains human.

If the base is not specified, the third step is not executed.

Settings files

SettingsML.json

Parameters of the first stage. The file is generated automatically in training mode and cannot be edited manually. If no file is specified, default values ​​are used.

SettingsSpectrum.json

Descriptions of characteristic frequencies for the second stage. The file is compiled manually by an engineer and is a JSON array, each element of which describes one time interval:

Field

Type

Meaning

name

string

An arbitrary name for the convenience of the engineer.

from

number

Start of the interval in ms from the beginning of the data entering stdin.

to

number

The end of the interval in ms from the beginning of the data entering stdin. It must be greater than from and less than --timeout, otherwise the interval will not have time to be checked.

spectrum

array

List of frequency ranges {"min": Hz, "max": Hz} in which bursts are expected. The number of elements determines how many of the strongest peaks will be extracted from the interval spectrum. All of them must fall within the listed ranges.

result

string

Result if there is a match: Human, IVR, Tone, Fax or Silence (case is not important).

Intervals are counted from the moment when the component began transmitting sound to cpd, so you need to mark up in the audio editor the same recording that goes to cpd, with the same beginning.

Example: a robot whose two strongest peaks lie around 100 Hz and 200 Hz in the interval 0.5…​1.5 s and 1…​2 s from the beginning of the data.

[
  { \
    "name": "Robot1", \
    "from": 500, \
    "to": 1500, \
    "spectrum": [{"min": 195, "max": 205}, {"min": 97, "max": 103}], \
    "result": "IVR" \
  }, \
  { \
    "name": "Robot1-late", \
    "from": 1000, \
    "to": 2000, \
    "spectrum": [{"min": 195, "max": 205}, {"min": 97, "max": 103}], \
    "result": "IVR" \
  } \
] \

Fingerprint database

Binary file cpd_fingerprint.db, generated by cpd in database construction mode. The path to it is transmitted by the key --fingerprints-db. The database needs to be rebuilt after any change in the set of templates.

Placing on the platform

The SettingsML.json, SettingsSpectrum.json files and the cpd_fingerprint.db fingerprint database are located in the :SYNC/domains/<YOUR_DOMAIN>/data/cpd directory, where <YOUR_DOMAIN> is the domain name. The component transfers the paths to them to cpd every time it is launched.

To apply new settings, just replace the files in the directory: cpd reads them every time it starts, no platform restart is required.

cpd operating modes

The mode is set by the --mode key:

Mode Purpose

analyze (default)

Analysis of sound from stdin. The result is the type of sound source.

learn

Selection of stage 1 parameters based on marked records. The result is SettingsML.json.

rebuild-fingerprints

Building a database of audio fingerprints from template recordings. The result is a database file.

The analyze mode starts automatically on every call, no engineer intervention is required.

The engineer launches the rebuild-fingerprints and learn modes manually each time a set of marked records is updated: the fingerprint database needs to be rebuilt, and it is advisable to repeat the training. The resulting files are placed on the platform manually (see [_placement_on_platform]).

The --help key displays help for all parameters.

Analysis mode (analyze)

An example of how a component calls cpd for analysis:

cpd \
    --mode=analyze \
    --ml-settings-path=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/SettingsML.json \
    --spectrum-settings-path=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/SettingsSpectrum.json \
    --fingerprints-db=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/cpd_fingerprint.db \
    --timeout=7000 \
    --recv-timeout=3000 \
Parameter Default Description

--ml-settings-path

not specified

Path to SettingsML.json with stage 1 parameters. Optional. If not specified, hardwired (non-optimal) values ​​are used.

--spectrum-settings-path

not specified

Path to SettingsSpectrum.json with characteristic frequencies. Optional. If not specified, the second step is not executed.

--fingerprints-db

not specified

Path to the audio fingerprint database. Optional. If not specified, the third step is not executed.

--timeout

7000

Total work timeout in ms. After it expires, the result based on the accumulated data is displayed.

--recv-timeout

3000

Timeout for waiting for the next portion of data in stdin in ms. If there is no data longer than this time, the result is given based on the accumulated data.

--verbose

off

Print extended result (see below).

The output to stdout is one line with the result. With the key --verbose three lines are displayed:

  1. result(human, ivr, tone, fax, silence);

  2. path to the matched template from the fingerprint database, or an empty string if there was no match;

  3. degree of match with the template (a number from 0 to 1), or 0.

If you need to check the performance of cpd on a specific recording, you can use the following bash script, which cuts off a 44-byte header from a wav file (PCM, 8000 Hz, 16 bits, mono) and supplies the remaining raw PCM data to stdin cpd:

#!/bin/bash

CPD_DIR=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd \
SAMPLE_FILE=/some/path/to/sample.wav

dd if=$SAMPLE_FILE skip=44 bs=64 status=none | \
cpd \
    --mode=analyze \
    --ml-settings-path=$CPD_DIR/SettingsML.json \
    --spectrum-settings-path=$CPD_DIR/SettingsSpectrum.json \
    --fingerprints-db=$CPD_DIR/cpd_fingerprint.db \
    --timeout=7000 \
    --recv-timeout=3000 \
    --verbose \

This way you can see how cpd will process a specific record, whether the third stage worked and what template the record matched.

===Learn mode

Training is the selection of parameters of the first stage from a pre-labeled (arranged in folders) set of records, taking into account bias (priority). Bias determines which mistake is more expensive: losing a live subscriber or transferring a non-human to the operator.

How the training works:

  1. Entries from the data directory are loaded into memory. Only the first 10 seconds of each are used.

  2. The first iteration is performed on the default values. The existing SettingsML.json is not readable in training mode: each launch starts from scratch.

  3. At each subsequent iteration, a new set of parameters is selected, all records are run through the first stage, and a quality score is calculated taking into account bias.

  4. The best set is saved to the output file as soon as it is found, so interrupted training does not lose the result.

Only the first stage is involved in training. The characteristic frequencies and fingerprint base are not used in it.

The selection of parameters is random, so the results of different runs may vary. If necessary, you can run the training several times and select the best result from the report.

Command line example:

cpd \
    --mode=learn \
    --learn-bias=0 \
    --learn-iterations=10000 \
    --learn-data-dir-path=/some/path/to/wavs/ \
    --learn-output-file-path=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/SettingsML.json \

Parameter

Default

Description

--learn-bias

0

Priority from −9 to +9. −9 — allow loss of live answers, +9 — allow bad calls to operators, 0 — neutral.

--learn-iterations

10000

Number of iterations. 1 - only displays current statistics at default values, 10000 - real training.

--learn-data-dir-path

—

Path to the directory with marked data (see Training data). Required.

--learn-output-file-path

not specified

Path to output SettingsML.json. If not specified, the result is only printed to the console. The existing file is overwritten.

The output file is written only if a set of parameters better than the default values is found. If there is no improvement (including with --learn-iterations=1), the file is not created.

As training progresses, the number of records of each class, each improvement found, and the final report are displayed in the console: the achieved accuracy, the proportion of lost people and bad calls to operators, the average detection time and the error matrix (row is the true class by folder name, column is the result of the analyzer).

The finished SettingsML.json should be placed in the :SYNC/domains/<YOUR_DOMAIN>/data/cpd catalog, replacing the previous one.

Fingerprint database construction mode (rebuild-fingerprints)

Builds a database of audio fingerprints from a catalog of template recordings. Templates are recordings of robots, answering machines, voice menus and other signals that need to be confidently filtered out at the third stage of analysis.

Command line example:

cpd \
    --mode=rebuild-fingerprints \
    --templates-dir=/some/path/to/templates/ \
    --fingerprints-db=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/cpd_fingerprint.db \

Parameter

Default

Description

--templates-dir

—

Path to the directory with template entries (see Templates for the fingerprint database). Required.

--fingerprints-db

—

Path to the database file to be created. Required. The entire existing file is overwritten.

Requirements for template records are described in the [_templates_for_fingerprint_database] section.

Upon completion, the build time in milliseconds is displayed in the console.

The finished cpd_fingerprint.db database should be placed in the :SYNC/domains/<YOUR_DOMAIN>/data/cpd catalog, replacing the previous one.

Preparing records

Records are needed in two cases: for training the parameters of the first stage and for building a fingerprint database. Requirements for format and markup are general; requirements for the content of records vary and are described separately.

File format

Format: *.wav, PCM, 8000 Hz, 16 bit, mono.

The recordings must be the same as those received in real calls, without further processing (normalization, noise reduction, compression). The best source is recordings of actual campaign calls made by the platform. If the record is received from another source, it must be converted to the required format, for example:

ffmpeg -i source.wav -ar 8000 -ac 1 -c:a pcm_s16le result.wav \

Markup rules

The files are arranged in subfolders, the name of the subfolder specifies the recording class. The case of the name is not important, the nesting is arbitrary: only the name of the folder directly containing the file matters. The names of the files themselves are arbitrary.

Folder What is put

Human

Live responses from people: a person picks up the phone and answers with a greeting.

IVR

Robots, answering machines, voice menus, music while waiting, voice messages from the telecom operator.

Tone

Tones: beeps, busy signal, answering machine tone before recording, etc.

Fax

Fax.

Silence

Silence or background noise with no speech or signals.

The class is determined by what is heard in the first seconds of the recording, that is, by the result that cpd should produce for this call. If the recording starts with a robot and a human responds a few seconds later, it’s IVR. If a person responds later than the analysis lasts (7 seconds by default), it is better not to use such an entry at all.

Each entry must belong to exactly one class. Records in which the class is not obvious (strong noise, interruption, signal overlap) are not included in the set: one incorrectly labeled record spoils the result more than its absence.

Training data

The training selects parameters based on what the beginning of a call actually looks like, so the recordings should reproduce exactly the stream that cpd receives from the component:

  • recording begins from the moment the call is answered (whoever or whatever answers), that is, from the moment the component begins transmitting audio to the cpd. The beginning is not cut off to the first line and is not supplemented with silence: the analyzer takes into account the pause before the first word;

  • useful duration is the first 10 seconds, the rest is ignored. Entries shorter than 2–3 seconds are not very informative;

  • no extraneous sounds after the start of the analysis: operator speech, call transfer signals, etc. should be cut off;

  • examples of all five classes are needed, primarily Human and IVR. The more diverse the recordings within a class are (different people, different robots, different quality of communication), the better the selected parameters are transferred to real calls;

  • class sizes must be comparable. If there are ten times more Human records than IVR, the quality estimate will be skewed.

Files from folders with names that do not match the classes do not participate in training.

Templates for the fingerprint database

Template is a reference recording of a specific robot, answering machine or voice menu. The template is used to build fingerprints with which the real call is compared, so the template should contain only what is heard in a real call from this source:

  • the recording contains the robot’s replica itself from its beginning: greeting, background music, voice menu. Silence before the cue does not interfere, but is not necessary;

  • nothing should be included in the template except the source: no beeps before the connection, no human speech, no sounds after the end of the replica. All unnecessary is cut off in the audio editor;

  • template duration - several seconds, sufficient to cover that part of the replica that manages to sound during the analysis (by default 7 seconds). Very short or almost silent recordings are not included in the search;

  • one template is enough for each robot. If the robot has several different greetings, a separate file is needed for each;

  • it makes sense to put into the database only records whose matches should change the result: robots and answering machines. A match with a pattern like Human will not change the result.

The template type is determined by the name of the subfolder using the same markup rules. Files from subfolders with different names receive the type IVR.