The first stage is a quick statistical and heuristic analysis of the signal. In parallel, the second stage searches for characteristic frequencies in the sound, pre-set by the engineer. +\ The third stage is performed only if a person is recognized based on the results of the first two: the recorded sound is compared with a database of audio fingerprints of known robots and answering machines in order to weed out false positives.
CPD Component (Response Detector)
:icons:font :hardbreaks: :imagesdir: ../../../assets/images :toc-title: Contents :toc:
Description
Used in outgoing campaigns to detect the response of a “living” person.
Records the start of a conversation for a specified time (up to 7 seconds by default) and during this time determines the type of audio source: human, robot/IVR, tone, fax or silence.
The analysis is performed in three stages. +\
The algorithm is not demanding on system resources. The parameters of the first stage can be further trained on your own labeled records, and the fingerprint database can be built on your own templates.
Index |
|
Short name |
|
Script types |
|
Start module |
|
Mode |
Asynchronous |
Icon |
|
Branching type |
Branching, closing |
Properties
| Specification | Description |
|---|---|
Name: Timeout, s Code: Visibility condition: no Default: 5 |
An argument containing the time in seconds. Determines how long the conversation recording will be accumulated for further analysis by the response detector. |
Name: Save recording file Code: Visibility condition: no Default: No |
Indicator of saving the recording file sent to the response detector service. Saving is performed in the directory
|
Name: Result into variable Code: Visibility condition: no Default: — |
Variable where the reason for the component’s shutdown is entered. |
Name: Transition, detected Code: Visibility condition: no Default: — |
The component to which control is transferred after a “living” person is detected by the response detector.” "Name: Transition, not detected
Code: Default: — |
The component to which control is transferred if the response detector does not detect a “living” person in the recording.” "Name: Transition, error Code: Visibility condition: no Default: — |
Component to which control is transferred in case of an error. |
Operating principle
The analysis is performed by the console application cpd, which is part of the platform (./era_env/priv/bin/cpd/cpd).
The component runs cpd in analysis mode and feeds it the callee’s audio via standard input (stdin). \
The application “listens” to the beginning of the conversation and answers the only question: whether a live person answered the call.
This allows the automated dialer to connect an operator only when there is a high probability of a person on the line, and not silence, fax, beeps or a robot.
The component succeeds only with the result human.
Input data and result
The cpd accepts raw PCM data without header as input: 8000 Hz, 16 bit, mono. \
Data is read in chunks as it arrives and immediately passed through the analyzer.
The result is printed to stdout in one line:
Result |
Meaning |
|
A living person answered. The only result in which the component succeeds. |
|
Robot, answering machine, voice menu, music while waiting. |
|
Tone: beeps, busy signal, answering machine tone, etc. |
|
Fax. |
|
During the analysis, not a single significant replica was found. |
Termination conditions
cpd exits and produces a result on the first of the following events:
-
The current stage of analysis has made a decision.
-
Data does not arrive in stdin for longer than
--recv-timeout(default 3000 ms). -
cpdruns longer than--timeout(default 7000ms). -
Stdin is closed by the sender.
When completed by a timeout, the result is given based on the data that has accumulated. If there were no significant replicas, the result is silence.
===Steps of analysis
Stage 1. Statistical and heuristic analysis
A lightweight algorithm that works in a stream as data arrives. He identifies cues and pauses between them in the sound and, based on their duration and nature, makes a decision:
-
the person is identified after a pause following the subscriber’s short greeting;
-
robot or IVR is determined by a long continuous replica;
-
tone and fax are determined by a stable signal amplitude, fax - by a longer duration;
-
silence is determined by the expiration of a timeout without significant remarks.
The algorithm uses a set of parameters. The default values are hard-coded into the program and are not optimal. They can be selected for your own records in the learning mode (see [_learn_mode_learn]). The selected values are saved in the file SettingsML.json, which is transferred to cpd with the key --ml-settings-path.
Stage 2. Search for characteristic frequencies
It works in parallel with the first stage on the same data. Unlike the first stage, nothing is selected automatically here: the required characteristics are specified by the engineer. Having examined the robot’s recording in an audio editor, he notes at what frequencies and in what time interval a characteristic burst is visible, and describes this in the SettingsSpectrum.json file (see SettingsSpectrum.json). cpd checks each described interval as soon as it is fully accepted, and if there is a match, it immediately ends the analysis with the result specified for that interval.
If the file is not specified, the search for characteristic frequencies is not performed.
Stage 3. Search in the audio fingerprint database
The first two stages work simultaneously, and the analysis is completed as soon as one of them has made a decision. The third stage is performed only if this solution is human (that is, the first stage recognized the person, but the second did not find characteristic frequencies) and the fingerprint database (--fingerprints-db) is specified. The goal of this stage is to weed out robots and answering machines that are similar in nature to humans.
The entire received record is compared with a fingerprint database built in advance from template robot records (see Fingerprint database construction mode (rebuild-fingerprints)). Fingerprints are a detailed, automatically constructed description of the characteristic frequencies of a recording over time, that is, the same thing that an engineer does manually in the second stage, but without human intervention and with greater detail.
If the entry matches a pattern, the result is replaced with the type of that pattern. If there are no matches, the result remains human.
If the base is not specified, the third step is not executed.
Settings files
SettingsML.json
Parameters of the first stage. The file is generated automatically in training mode and cannot be edited manually. If no file is specified, default values are used.
SettingsSpectrum.json
Descriptions of characteristic frequencies for the second stage. The file is compiled manually by an engineer and is a JSON array, each element of which describes one time interval:
Field |
Type |
Meaning |
|
string |
An arbitrary name for the convenience of the engineer. |
|
number |
Start of the interval in ms from the beginning of the data entering stdin. |
|
number |
The end of the interval in ms from the beginning of the data entering stdin. It must be greater than |
|
array |
List of frequency ranges {"min": Hz, "max": Hz} in which bursts are expected. The number of elements determines how many of the strongest peaks will be extracted from the interval spectrum. All of them must fall within the listed ranges. |
|
string |
Result if there is a match: |
Intervals are counted from the moment when the component began transmitting sound to cpd, so you need to mark up in the audio editor the same recording that goes to cpd, with the same beginning.
Example: a robot whose two strongest peaks lie around 100 Hz and 200 Hz in the interval 0.5…1.5 s and 1…2 s from the beginning of the data.
[
{ \
"name": "Robot1", \
"from": 500, \
"to": 1500, \
"spectrum": [{"min": 195, "max": 205}, {"min": 97, "max": 103}], \
"result": "IVR" \
}, \
{ \
"name": "Robot1-late", \
"from": 1000, \
"to": 2000, \
"spectrum": [{"min": 195, "max": 205}, {"min": 97, "max": 103}], \
"result": "IVR" \
} \
] \
Fingerprint database
Binary file cpd_fingerprint.db, generated by cpd in database construction mode. The path to it is transmitted by the key --fingerprints-db. The database needs to be rebuilt after any change in the set of templates.
Placing on the platform
The SettingsML.json, SettingsSpectrum.json files and the cpd_fingerprint.db fingerprint database are located in the :SYNC/domains/<YOUR_DOMAIN>/data/cpd directory, where <YOUR_DOMAIN> is the domain name. The component transfers the paths to them to cpd every time it is launched.
To apply new settings, just replace the files in the directory: cpd reads them every time it starts, no platform restart is required.
cpd operating modes
The mode is set by the --mode key:
| Mode | Purpose |
|---|---|
|
Analysis of sound from stdin. The result is the type of sound source. |
|
Selection of stage 1 parameters based on marked records. The result is |
|
Building a database of audio fingerprints from template recordings. The result is a database file. |
The analyze mode starts automatically on every call, no engineer intervention is required.
The engineer launches the rebuild-fingerprints and learn modes manually each time a set of marked records is updated: the fingerprint database needs to be rebuilt, and it is advisable to repeat the training. The resulting files are placed on the platform manually (see [_placement_on_platform]).
The --help key displays help for all parameters.
Analysis mode (analyze)
An example of how a component calls cpd for analysis:
cpd \
--mode=analyze \
--ml-settings-path=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/SettingsML.json \
--spectrum-settings-path=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/SettingsSpectrum.json \
--fingerprints-db=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/cpd_fingerprint.db \
--timeout=7000 \
--recv-timeout=3000 \
| Parameter | Default | Description |
|---|---|---|
|
not specified |
Path to |
|
not specified |
Path to |
|
not specified |
Path to the audio fingerprint database. Optional. If not specified, the third step is not executed. |
|
7000 |
Total work timeout in ms. After it expires, the result based on the accumulated data is displayed. |
|
3000 |
Timeout for waiting for the next portion of data in stdin in ms. If there is no data longer than this time, the result is given based on the accumulated data. |
|
off |
Print extended result (see below). |
The output to stdout is one line with the result. With the key --verbose three lines are displayed:
-
result(
human,ivr,tone,fax,silence); -
path to the matched template from the fingerprint database, or an empty string if there was no match;
-
degree of match with the template (a number from 0 to 1), or
0.
If you need to check the performance of cpd on a specific recording, you can use the following bash script, which cuts off a 44-byte header from a wav file (PCM, 8000 Hz, 16 bits, mono) and supplies the remaining raw PCM data to stdin cpd:
#!/bin/bash
CPD_DIR=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd \
SAMPLE_FILE=/some/path/to/sample.wav
dd if=$SAMPLE_FILE skip=44 bs=64 status=none | \
cpd \
--mode=analyze \
--ml-settings-path=$CPD_DIR/SettingsML.json \
--spectrum-settings-path=$CPD_DIR/SettingsSpectrum.json \
--fingerprints-db=$CPD_DIR/cpd_fingerprint.db \
--timeout=7000 \
--recv-timeout=3000 \
--verbose \
This way you can see how cpd will process a specific record, whether the third stage worked and what template the record matched.
===Learn mode
Training is the selection of parameters of the first stage from a pre-labeled (arranged in folders) set of records, taking into account bias (priority). Bias determines which mistake is more expensive: losing a live subscriber or transferring a non-human to the operator.
How the training works:
-
Entries from the data directory are loaded into memory. Only the first 10 seconds of each are used.
-
The first iteration is performed on the default values. The existing
SettingsML.jsonis not readable in training mode: each launch starts from scratch. -
At each subsequent iteration, a new set of parameters is selected, all records are run through the first stage, and a quality score is calculated taking into account bias.
-
The best set is saved to the output file as soon as it is found, so interrupted training does not lose the result.
Only the first stage is involved in training. The characteristic frequencies and fingerprint base are not used in it.
The selection of parameters is random, so the results of different runs may vary. If necessary, you can run the training several times and select the best result from the report.
Command line example:
cpd \
--mode=learn \
--learn-bias=0 \
--learn-iterations=10000 \
--learn-data-dir-path=/some/path/to/wavs/ \
--learn-output-file-path=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/SettingsML.json \
Parameter |
Default |
Description |
|
0 |
Priority from −9 to +9. −9 — allow loss of live answers, +9 — allow bad calls to operators, 0 — neutral. |
|
10000 |
Number of iterations. 1 - only displays current statistics at default values, 10000 - real training. |
|
— |
Path to the directory with marked data (see Training data). Required. |
|
not specified |
Path to output |
The output file is written only if a set of parameters better than the default values is found. If there is no improvement (including with --learn-iterations=1), the file is not created.
As training progresses, the number of records of each class, each improvement found, and the final report are displayed in the console: the achieved accuracy, the proportion of lost people and bad calls to operators, the average detection time and the error matrix (row is the true class by folder name, column is the result of the analyzer).
The finished SettingsML.json should be placed in the :SYNC/domains/<YOUR_DOMAIN>/data/cpd catalog, replacing the previous one.
Fingerprint database construction mode (rebuild-fingerprints)
Builds a database of audio fingerprints from a catalog of template recordings. Templates are recordings of robots, answering machines, voice menus and other signals that need to be confidently filtered out at the third stage of analysis.
Command line example:
cpd \
--mode=rebuild-fingerprints \
--templates-dir=/some/path/to/templates/ \
--fingerprints-db=/var/lib/era_files/syncroot/domains/pbx.incoplax.com/data/cpd/cpd_fingerprint.db \
Parameter |
Default |
Description |
|
— |
Path to the directory with template entries (see Templates for the fingerprint database). Required. |
|
— |
Path to the database file to be created. Required. The entire existing file is overwritten. |
Requirements for template records are described in the [_templates_for_fingerprint_database] section.
Upon completion, the build time in milliseconds is displayed in the console.
The finished cpd_fingerprint.db database should be placed in the :SYNC/domains/<YOUR_DOMAIN>/data/cpd catalog, replacing the previous one.
Preparing records
Records are needed in two cases: for training the parameters of the first stage and for building a fingerprint database. Requirements for format and markup are general; requirements for the content of records vary and are described separately.
File format
Format: *.wav, PCM, 8000 Hz, 16 bit, mono.
The recordings must be the same as those received in real calls, without further processing (normalization, noise reduction, compression). The best source is recordings of actual campaign calls made by the platform. If the record is received from another source, it must be converted to the required format, for example:
ffmpeg -i source.wav -ar 8000 -ac 1 -c:a pcm_s16le result.wav \
Markup rules
The files are arranged in subfolders, the name of the subfolder specifies the recording class. The case of the name is not important, the nesting is arbitrary: only the name of the folder directly containing the file matters. The names of the files themselves are arbitrary.
| Folder | What is put |
|---|---|
|
Live responses from people: a person picks up the phone and answers with a greeting. |
|
Robots, answering machines, voice menus, music while waiting, voice messages from the telecom operator. |
|
Tones: beeps, busy signal, answering machine tone before recording, etc. |
|
Fax. |
|
Silence or background noise with no speech or signals. |
The class is determined by what is heard in the first seconds of the recording, that is, by the result that cpd should produce for this call. If the recording starts with a robot and a human responds a few seconds later, it’s IVR. If a person responds later than the analysis lasts (7 seconds by default), it is better not to use such an entry at all.
Each entry must belong to exactly one class. Records in which the class is not obvious (strong noise, interruption, signal overlap) are not included in the set: one incorrectly labeled record spoils the result more than its absence.
Training data
The training selects parameters based on what the beginning of a call actually looks like, so the recordings should reproduce exactly the stream that cpd receives from the component:
-
recording begins from the moment the call is answered (whoever or whatever answers), that is, from the moment the component begins transmitting audio to the
cpd. The beginning is not cut off to the first line and is not supplemented with silence: the analyzer takes into account the pause before the first word; -
useful duration is the first 10 seconds, the rest is ignored. Entries shorter than 2–3 seconds are not very informative;
-
no extraneous sounds after the start of the analysis: operator speech, call transfer signals, etc. should be cut off;
-
examples of all five classes are needed, primarily
HumanandIVR. The more diverse the recordings within a class are (different people, different robots, different quality of communication), the better the selected parameters are transferred to real calls; -
class sizes must be comparable. If there are ten times more
Humanrecords thanIVR, the quality estimate will be skewed.
Files from folders with names that do not match the classes do not participate in training.
Templates for the fingerprint database
Template is a reference recording of a specific robot, answering machine or voice menu. The template is used to build fingerprints with which the real call is compared, so the template should contain only what is heard in a real call from this source:
-
the recording contains the robot’s replica itself from its beginning: greeting, background music, voice menu. Silence before the cue does not interfere, but is not necessary;
-
nothing should be included in the template except the source: no beeps before the connection, no human speech, no sounds after the end of the replica. All unnecessary is cut off in the audio editor;
-
template duration - several seconds, sufficient to cover that part of the replica that manages to sound during the analysis (by default 7 seconds). Very short or almost silent recordings are not included in the search;
-
one template is enough for each robot. If the robot has several different greetings, a separate file is needed for each;
-
it makes sense to put into the database only records whose matches should change the result: robots and answering machines. A match with a pattern like
Humanwill not change the result.
The template type is determined by the name of the subfolder using the same markup rules. Files from subfolders with different names receive the type IVR.