Implementation details and operation specifics of the conversation recovery service

1. SIP node monitoring

1.1. Monitors all SIP nodes (serving SIP signaling).

1.2. The RPCI microservice additionally regularly pings nodes once every 5 seconds and informs the others when loss is detected.

1.3. If the service catches, or RPCI reports to its node, that the ping with a SIP node is gone — it records the loss and starts a one-second timer.

1.4. Waits for a second to see which other SIP nodes are also lost.

1.5. Gives the accumulated set as a nodedown event to the recovery service.

2. Dialog tracking

2.1. The service processes callevents: dlg_start, dlg_end, dlg_dead, dlg_down, and dlg_binding.

2.2. On receiving dlg_start, the service trims the data and saves it in its distributed storage (mnesia). It sets a reserve timer for two hours of maximum conversation duration, after which it deletes the dialog data from storage.

2.3. The dlg_start event, among other things, contains a_hops, b_hops — addresses and ports of the intermediate nodes from b2b to the user agent participating in SIP signaling servicing of the legs, as well as tags, URIs, contacts, usernames, domains — what is needed to recover the dialog.

2.4. On receiving dlg_end, the service deletes the dialog data from storage.

2.5. On receiving dlg_dead or dlg_down, the service restarts the timer for 5 seconds so as not to be left with extra data. When the timer expires it clears the data from storage.

2.6. On receiving dlg_binding, the service updates the saved information in the context needed to recover sessions (CCS and CC, needed to recover queues).

2.7. If the recovery service node restarts or a standby instance starts, it reads all dialogs from storage and deletes expired ones, setting timers for the remaining time on current ones.

2.8. As soon as a nodedown event about SIP node loss arrives, the service searches among all current dialogs in its storage for those affected (intersection with hops).

2.9. For each affected dialog it starts an asynchronous recovery process, passing it the context — dialog information and the list of affected nodes. It saves the recovery process identifier in the record, and leaves the record itself in storage; it is deleted in the standard way when callevents are processed.

3. Preparing for recovery

3.1. If there is a prompt (monitor, whisper) among the hops, the task is canceled. Connections to conversations are not recovered.

3.2. If the dialog was already sent for recovery earlier, the new recovery process synchronizes with the previous handler. If it is alive, it tries to stop it (abort), then waits for 5 minutes (an extreme case). At the end of its work, as well as right before starting processing, the previous process checks the abort command in the message queue. When it finds it, it returns its starting context and terminates.

3.3. The new process that initiated synchronization, receiving the response from the previous handler, acts depending on the response (canceled, successfully recovered, failed with an error). Or it continues by merging its own and the previous contexts (merges the lists of affected nodes, since the second task may contain additional nodes whose loss was not detected earlier).

3.4. If there is no previous handler or it has completed, recovery starts with its own starting context.

3.5. Before starting recovery it waits for paths to appear for calling both subscribers. It checks regularly for a minute: registration through a live available SG, a connection with the provider, nodes and active microservices of all required types — conferences, IVRs, b2b, sg, esg — all according to the context of the dialog being recovered.

3.6. As soon as path availability is confirmed, the handler checks whether it is being canceled by a newer task. If an abort command is already in the queue, it terminates and returns its starting context.

3.7. In normal mode the handler determines the dialog recovery mode and starts it.

4. Recovery

4.1. A total of 7 different recovery procedure types are supported.

4.2. In any case the new dialog will be from A to B (according to the dialog being recovered).

4.3. The most trivial — a one-step replaces with a replaces refer — for the case when only the b2b node is lost.

4.4. The most complex — a parallel intercom call of both subscribers from an IVR script with silence, and a subsequent double replaces by the call manager — for the case when all nodes are completely lost. This mode exists but is effectively unused.

4.5. The medium mode — an intercom call of subscriber B, then a replaces call of subscriber A and a replaces refer for A to substitute the new B leg.

4.6. All 7 modes are as follows in order of increasing complexity:

4.6.1. one-step replaces into leg A with a replaces refer to leg B,

4.6.2. one-step intercom to number A with a replaces refer to leg B,

4.6.3. two-step replaces: first replaces into leg B from ivr with silence, second replaces into leg A with a replaces refer to the new leg B.

4.6.4. two-step with intercom: first intercom to number B from a dummy ivr, second replaces into leg A with a replaces refer to the new leg B.

4.6.5. two-step double intercom: first intercom to number B from a dummy ivr, second intercom to number A with a replaces refer to the new leg B.

4.6.6. sequential calling of A and B followed by a call-manager double replaces of the new legs. Initially intercom or replaces — depends on whether the leg has lost hops or not.

4.6.7. parallel calling of A and B followed by a call-manager double replaces. Initial intercom or replaces — depends on whether the leg has lost hops or not.

4.7. The call is always first made into the leg that has lost hops.

4.8. An attempt is always first made to take the call by replaces; if the operation fails, another more complex operation from 4.6 with intercom is selected (optional mode, disabled by default).

4.9. The refer is always a replaces refer, made to the leg reliably known not to be lost. It is either already recovered by a dummy ivr, or initially had no lost hops (except possibly the central b2b).

4.10. If the call of the second leg fails, the first recovered leg is terminated (by the call manager bye from b2b into the leg). The terminated call has live signaling, since it was created in the new conditions already after recovery started.

4.11. When the new dialog is successfully created, termination of the previous dialog on b2b is initiated, as well as of the legs on user agent A and user agent B. This matters for the server if b2b survived but both legs dropped, and it matters for phones when the legs dropped and b2b can no longer cancel the previous call so that the phone resets the previous hanging dialog. For this a BYE is sent directly (for now only from sg toward the phone/softphone).

4.12. The dialog is deleted from storage either on completion on b2b or deferred on the dlg_down event.

4.13. In the log you can find many 481 responses to BYE and re-INVITE sent by the phone.

5. Effects

5.1. Media traffic exchange continues until one of the phones switches to the new dialog. The whole time paths are awaited, traffic exchange continues. The main condition — the media gateway serving the dialog remains alive.

5.2. If b2b survived and only sg restarted, the conversation does not necessarily need recovery, but it is nevertheless recreated (since cases of using a border media gateway, connecting devices through standby sg instances, and some others are not distinguished).

5.3. When recovering such a conversation, in some cases a hold melody may be briefly audible, since the phone switches to hold during an intercom call. But the probability is low, since replaces is done first and it succeeds in the vast majority of cases.

5.4. If something happens in signaling while the path is awaited and recovery is starting, the dialog will break down, and then its waiting process can recover it by intercoms if such a mode is enabled.

5.5. If the handset is hung up while the path is awaited — the system will reconnect the subscribers by intercoms, since the task is already being processed. This also requires enabling the intercom call mode.

5.6. Task activation occurs no earlier than a second later, but in general may be after the pang_to_disconnect time set in the config + 2-3 seconds — the case of connectivity loss between servers and subsequent long waiting for recovery or TCP connection completion.

5.7. If b2b crashed and it has standby instances, recovery starts in 1-2 seconds.

5.8. If sg crashed and does not come up but there are standby instances (the case when the phone is configured for several outbound proxies and Virtual IP is not used), then about 15 seconds are first counted before sending the re-registration command (NOTIFY event=reload-account), and in special cases the command may be sent within a minute (for example, the DB and mdc moved at the same time). Then when the phone re-registers, recovery starts in 1-2 seconds. The phone must support this event type and process it on arrival, including from other servers (without filtering). The Virtual IP case is handled in the standard way 1-2 seconds after loss detection.

5.9. If sg crashed and came back up, recovery starts 1-2 seconds after its load.

5.10. Recovery is attempted once per dialog — in reaction to each aggregated event about loss of SIP nodes having coupling with the nodes serving the dialog.

5.11. Cross-site nodes are not tracked. If one leg goes to another site, recovery requires losing the node on the site where b2b runs.

5.12. No ghost calls are recovered. These include service operations initiated by the call manager.

5.13. No unestablished calls are recovered. That is, calls that did not receive a final answer.

5.14. If a call is on hold, when recovered by replaces it preserves hold mode by phone means. When recovered by intercom it does not, since this is a new dialog.

5.15. For intercom recovery to work without the subscriber interacting with the device, intercom priority must be set in the phone.

5.16. Configured by callstore.sip_restore_enabled (default false), sg.reregister (default false) — the latter is useful only if the multiple outbound proxy configuration mode is used.

5.17. The WebRTC softphone does not drop the conversation on sg loss but, preserving the conversation and media, starts an attempt to reconnect to the same or an alternative address. As soon as registration is restored, a new dialog arises substituting the previous one. If the media gateway remained active, sound loss is within half a second.

5.18. During recovery, binding is made to the same session identifier that the lost conversation had. This makes it possible to restore the servicing context in the call center.

5.19. Enabling the recovery service for a single-server system makes little sense, since it protects only against the risk of node or whole system restart. The first is possible either manually, with insufficient RAM, or with critical system errors — all unlikely. The second requires too long (2-3 minutes) waiting by subscribers for the system to load to a ready state to expect they will not hang up. Thus for a single-server system the functionality is more demonstrative and test-related.

5.20. In the conversation journal the recovered conversation will be a separate record. A conversation recording will be formed for each of the records (provided the server with the previous conversation media gateway is available).

6. External subscribers

6.1. If esg did not crash, recovery occurs by a call with substitution, which the surviving esg processes as usual.

6.2. The provider account configures the recovery mode (opts.sip_restore_mode). It applies only if the esg instance crashed. Default is replace. There are also regular and disabled.

6.3. To recover the leg where esg crashed, the service waits for successful registration/punch-through. That is, esg must report that the connection is available.

6.4. In replace mode external calls are recovered by sending the uplink information about the substituted leg. These headers pass through the new incarnation of esg, although this esg forms a full new call. The uplink is expected to be able to process replaces (invites with the replaces header).

6.5. In regular mode a new call is made without passing leg substitution information.

6.6. In disabled mode the recovery procedure is canceled.

6.7. Intercom is not applied to uplinks.

7. Conferences

7.1. When part of the b2b through which conference call servicing goes is lost, a substitute call initiation is performed on a common basis. The subscriber calls the conference by substitution. For the conference this is a normal substitution, and for the subscriber it is a normal recovery case.

7.2. If the active conference microservice crashes, all subscribers call the conference again and effectively create a new one with the same number. Almost unnoticeable to all subscribers.

8. IVR

8.1. On b2b loss — a substitute call initiation is performed on a common basis. The subscriber calls the live ivr by substitution. For IVR this is a normal substitution, as if the subscriber transferred another subscriber to it. And for the subscriber it is a normal recovery case. Servicing in IVR was not suspended.

8.2. If the active IVR microservice crashed — the subscriber calls IVR again by the original number. Effectively a new script handler is created from the very beginning from the feature code. Therefore with serious switches it is important to transfer to another feature code number — then recovery will proceed from the beginning of the script by the last dialed feature code. If the script uses nested scripts — it starts from the beginning of the first script — the feature code to which the last call was made.

8.3. The script can save its full state, including information about the nested script stack, to storage (the microservice with the store role, which is multi-active-passive — in any number of sharding groups simultaneously).

8.4. Recovery of the IVR script leg occurs with passing information about the recovery fact. And the new script can find the state in storage and fully load it.

8.5. Execution starts by sending the last answer and then from the beginning of the last saved component.

8.6. At the moment state saving is handled by the PAUSE component with the corresponding option enabled. Disabled by default.

8.7. How much state saving will slow the system down remains to be established. If not overdone — seemingly not much. But if a saving pause is added after each component, load and execution time will obviously increase by an order of magnitude if the script does algorithmic work without pauses. And of course the script has related blocks that make no sense to separate. Preparing a script for recovery is creative and responsible work.

8.8. IVR is recovered only while the media session is alive. The post-branch is not saved; before the 200 OK answer saving is also pointless, because an unestablished dialog is not recovered by the service.

9. Development prospects

9.1. The recovery process can generate intensive traffic and CPS. Task launches should be distributed over time and priorities of different dialogs managed based on their types and possibly other attributes. For example, on one hand, IVR recovery may be more priority in terms of call center servicing than internal calls. On the other hand it is less priority because IVR servicing is effectively just waiting — if the media gateway and playback of long segments or hold melody are preserved, recovery speed is not critical. There appear to be quite many such examples and cases.

9.2. Save in the dialog context information about the border gateway usage type (sg, bgmg, IP address) to exclude recovering the conversation entirely provided that SIP requests through the new gateway incarnation continue and media traffic exchange does not stop.

9.3. A call waiting in the call center queue should recover its position based on the session identifier, replacing its previous record there.

9.4. In call center reports recovered conversations may duplicate conversation records and may distort part of the statistics.

9.5. A time cutoff setting would be useful, excluding recovery task launch after an excessively long interval — for example, no available path for a long time, or after a system restart or update.