Site availability and data integrity

Overview

A site is a set of computing resources with a wide communication channel between them and high mutual availability, preventing disintegration into several disconnected parts except when individual servers are lost, and providing full functionality regardless of the availability of other parts of the system.

Static

The impossibility of a site disintegrating into several disconnected groups of servers is due to the need to ensure data integrity. To guarantee this requirement, a quorum model is used: data-handling microservices (made redundant in active-passive mode) are active only on servers that belong to a connected group of more than half of the site’s active servers. The quorum mode is enabled in the configuration parameters of each site with the check_quorum parameter (disabled by default). The quorum case for half of the servers can be resolved with an elementary arbiter: an external IP address checked by a simple ping. This is valid when communication between servers and communication with the outside world for each server are provided by the same communication channel, so that a server failure or its disconnection from the network looks the same both to other system servers and to external devices.

If there is only a risk of an individual server disconnecting, there is no need to use quorum mode. First, an individual server in a multi-server site deployment does not have full functionality, and with high probability no business process can complete successfully because it needs to interact with many other microservices. The probability that all of them are located on this server is non-zero only if the server is deliberately configured to provide full functionality (for example, a two-server site). Second, an individual server is connected by the same channel both to external served devices and to other servers of the site. Third, the loss of communication of an individual server with other servers of the same datacenter while it remains operational is indistinguishable from a server failure. If none of these conditions holds or is acceptable, quorum mode should be used. In that case, active-passive microservices on the disconnected server leave service.

A special case is site redundancy by placing computing resources simultaneously in two datacenters with a wide communication channel between them and low ping. This configuration is intended to ensure site availability when there is a risk of losing an entire datacenter (power, connectivity, or other reasons).

Placing a site in two datacenters

The site must be configured so that:

  • if one datacenter is unavailable, the remaining part of the system provides full functionality to ensure service availability.

  • if the direct (third) communication channel between datacenters fails, one and only one of the groups provides full functionality to ensure data integrity.

To achieve this, the site servers are split in half so that each group contains a complete set of microservices:

  • Instances of microservices operating in active-active mode are present in both groups.

  • Instances of microservices operating in active-passive mode are present in each group with different priorities that balance the load between datacenters. Those serving domain data in each group are capable of processing the full domain tree.

  • Database instances are present in both groups and configured in master-replica mode.

  • File storages are available to both groups and automatically restore integrity after both groups recover and become available after a failure.

Depending on how communication between datacenters is provided — via a common external channel (1) or a separate direct channel (2) — configuration can be done in different ways.

1. Case of using an external channel between datacenters.

Static
Static

Failure of datacenter A’s computing resources and loss of communication with it are indistinguishable for external systems and for servers of the other datacenter B. Thus, the group of servers A left without communication will not receive requests from outside and will not pose a threat to data integrity.

To ensure the operability of the remaining part of system B, it is necessary and sufficient that it considers itself in quorum.

Static

Option 1. Split the servers strictly in half, enable quorum mode, and register any external always-available address as the arbiter.

Then, if group A is unavailable, quorum mode in group B records that exactly half of the servers are available and checks the arbiter’s availability. If the arbiter is available as expected, group B decides that quorum exists and continues to operate. Group A is of no interest if it is inactive. If it is active without external communication, it also records that exactly half of the servers are available and performs an unsuccessful arbiter check (since server and arbiter unavailability are equivalent due to the shared external channel); as a result, without quorum, the data-handling microservices are unloaded and group B enters the waiting-for-quorum state.

Option 2. Do not use quorum mode, relying on the guaranteed unavailability of group A in both kinds of problems: communication failure and computing resource failure.

In this case, it is not mandatory that servers be distributed strictly in half across datacenters. It is only important that:

  • each group is able to withstand the entire site load,

  • further splitting of the remaining group of servers into mutually unavailable parts is impossible, except when individual servers disconnect.

2. Case of separate external and direct channels between datacenters.

Static

In corporate networks, datacenters are usually initially configured at the hardware level to ensure redundancy of distributed services:

  • one datacenter is designated leader (A), the other follower (B);

  • the cross-channel between datacenters is an order of magnitude wider than their external channels;

  • a VPN cross-channel between datacenters is maintained through external interfaces;

  • routers exchange heartbeat messages simultaneously through the cross-channel and through VPN over WAN;

  • when the external communication channel fails, routers automatically change the route for outgoing requests to external addresses, activating redirection of outgoing traffic to the other datacenter’s router through the cross-channel;

  • the router in datacenter B is configured to:

    • automatically disable traffic routing when there is no heartbeat from the cross-channel and a heartbeat from VPN over WAN is simultaneously present;

    • automatically enable traffic routing when there is no heartbeat from either direction;

    • automatically enable traffic routing when there is a heartbeat from both directions;

    • automatically enable traffic routing and additionally provide routing for outgoing requests arriving through the cross-channel when there is a heartbeat from the cross-channel and no heartbeat from VPN over WAN;

If these conditions are met, there is no integrity threat due to the asymmetric hardware configuration. Because the hardware routing service is automatically disabled, a situation where both datacenters contain two integrity points becomes impossible. And because outgoing traffic is automatically redirected through the other datacenter’s external channel, a situation where business processes use microservice instances without external communication becomes impossible.

When configuring the system in such an infrastructure, it is sufficient to ensure only one condition: split the servers in half so that each group can provide full functionality when the other group is absent.

Further consideration concerns datacenters that do not have the above hardware settings and are symmetric.

2.1. Case of datacenter computing resource failure.

Static

This case is equivalent to case 1 and presents no other difficulties.

2.2. Case of external communication channel failure at one datacenter (A).

Static

There is no threat to quorum, and the arbiter is not polled by any server.

All microservices of the internal perimeter remain active in groups on both datacenters and available to other system microservices.

2.2.1. External-perimeter microservices in group A remain without communication with the outside world.

  • ws (webserver). Operates in active-active mode. Existing WebSocket connections break; new ones are established by clients to an available web server in group B. By default, ws does not make outgoing requests unless configured otherwise in scripts.

  • sg (sip gate). Operates in active-active mode. New registrations and registration renewals are made to an available instance in group B. Existing calls passing through group A’s sg lose control capability. If group A’s media gateway is used for traffic, audio is lost and the conversation is forcibly ended by the subscribers.

  • mg (media gate). Operates in active-active mode. It is reserved for call servicing by other system microservices; thus, without special settings, on average half of the calls will land on group A’s media gateways and prevent subscribers from exchanging traffic. The MG problem is considered separately: 3.2. Media gateway problem.

  • esg (external sip gate). Operates in active-active mode, but each provider account is served by one and only one esg instance. Priority settings for selecting esg for each provider account are set directly in the properties of the entity describing it (provider). A different esg is selected based on unavailability. In the case of no external communication channel at datacenter A, accounts bound by priority to group A’s esg instances continue to be served there without the ability to exchange data with the provider. Solving the issue of providing PSTN communication for subscribers is considered separately: 3.1. Provider problem.

  • bgmg (border media gate). Active on servers near esg and not used without the corresponding esg being active.

2.2.2. Internal-perimeter microservices with processes accessing external systems outside the external communication channel continue to work, receiving unsatisfactory responses (3.3. Problem of access to external systems).

  • script (all types of scripts). Operate in active-active mode. Each script is served on one instance, and the instance is selected by other system microservices. Components accessing web resources, mail servers, and external DBs in scripts executed by group A instances will fail. In particular, ASR and TTS when using Yandex SpeechKit. The hashring problem is considered separately: 3.4. Problem of selecting a microservice instance.

  • im (instant messaging). Provides text message exchange with messenger servers. The service operates in active-passive mode. The group A instance will receive a communication error when attempting to exchange data. Message exchange with messengers stops. The problem of distributed services with a leader is considered separately: 3.5. Problem of distributed applications and services with a leader.

  • email. Provides mail exchange with mail servers. The service operates in active-passive mode. The group A instance will receive a communication error when attempting to exchange data. Mail exchange with mail servers stops. The problem of distributed services with a leader is considered separately: 3.5. Problem of distributed applications and services with a leader.

  • mware (middleware). The service for generating letsencrypt certificates and accessing Telegram for administration needs. The group A instance will receive a communication error when attempting to exchange data. The problem of distributed services with a leader is considered separately: 3.5. Problem of distributed applications and services with a leader.

Static

In this case, external communication channels in both datacenters are active.

There is no communication between system servers over network interface 1 (local addresses), and the system cannot function in the standard way because microservices address each other by local addresses. Two disconnected groups arise, each operational, providing full functionality, and ready to serve client requests and business processes. Each contains exactly half of the site’s servers.

According to the arbiter poll, both groups are ready to continue independent operation.

To ensure data integrity, one and only one of the groups must be taken out of service: either A or B. Then the case reduces to cases 1 and 2.1 (failure of one datacenter’s computing resources or its availability to all).

The determination is coordinated: both groups make the same decision.

To determine which group has priority, network interface 2 and the external channel are used. Servers of group A poll servers of group B at their external addresses specified in the configuration. Conversely, servers of group B poll servers of group A. If mutual availability is detected, priority is given to the group based on a function of the servers in each group (the servers and their parameters in the two disconnected groups are guaranteed to differ). If the other group is unreachable at its external address, the group considers itself the only one and, given arbiter availability and availability of half of the site’s servers, continues to operate.

The group priority function uses the server configuration parameter quorum_leadership_key and selects the group with the lexicographically smallest value among its servers. If the parameter is absent, the server name in the configuration is used as the default. If keys match for different servers, their names are compared as a secondary criterion.

Mutual polling between groups is performed over HTTP/HTTPS by sending an HTTP request to a public URL (with a static key) to the web servers of the current site located in the unreachable zone. The external access address for servers (WAN) is set by the server configuration parameter wan_url. If it is not set, the request is made to the static (non-virtual) server addresses specified by the 'ifaces' parameter for those web servers whose HTTP access is allowed.

Thus, if a site splits into two equal islands, there is a combination of configuration settings in which both islands can remain active. This should be avoided when such a risk exists.

2.4. Failure of several entities.

If both the external channel of one datacenter and the internal channel between them fail simultaneously, the case is equivalent to failure of one datacenter’s computing resources. The external arbiter solves the problem because the disconnected group is unavailable to external parties and cannot poll the external arbiter.

2.4.1. Channels and computing resources of one datacenter

If computing resources in a datacenter and one of its channels fail simultaneously, the case reduces to a simple failure of computing resources in the datacenter.

2.4.2. Several servers in different groups

If several servers scattered across different groups are lost, operability depends on the configuration and is equivalent to the usual case where all site servers are in one datacenter. There is a case when all servers serving a certain type of microservice according to the configuration have failed. Then all business processes involving it are impossible on the site.

2.4.3. One datacenter and a server in the other datacenter

If one datacenter is unavailable or fails, quorum mode is used to ensure integrity. Each of the two datacenters has the minimum necessary number of servers for quorum: 50%.

With this layout, the site does not allow the loss or removal from service of any additional server in the group remaining in quorum.

In such an event, quorum is lost and the group enters the waiting-for-quorum mode, which makes the service unavailable (except for the case 2.3. Case of communication channel failure between datacenters.).

Given the randomness of the lost server selection, besides the lack of quorum, there is another problem. The site configuration must be prepared for the loss of three arbitrary servers. This means that:

  • active-active microservices must have 4 or more instances, at least 2 in each group.

  • active-passive microservices must have four instances instead of two, at least 2 in each group.

  • DBMS instances must be located not on site servers but in highly available virtual environments, or must be managed externally to ensure monitoring of 4 replicas.

2.4.4. Weighted models in quorum calculation

(TODO) Extended weighted models can potentially be used in quorum calculation, allowing groups with fewer than half of the total site servers to operate. This capability is provided by the concept of arbitration servers and a predefined boundary between groups passing between datacenters. In this case:

  • The requirement to distribute servers strictly in half can be removed, up to distributing site servers across more than two datacenters.

  • Low-performance servers outside the datacenter that do not affect quorum can be connected to the site. For example, this may be useful for preferentially using media gateways located in remote branches for servicing calls within the remote branch.

3. Consideration of identified problems.

3.1. Provider problem

3.1.1. Outgoing calls

Access to the provider is made redundant by creating several accounts. They are configured with oppositely directed priorities for binding to esg instances.

In routing, several identical rules are created that differ only in bindings to different accounts (and possibly different priorities). This ensures sequential application of routing rules when the account state check fails, initiated during each attempt to apply a call routing rule to the provider.

To ensure that accounts and esg servers are evenly loaded in the normal state, the rules can be given the same priority. In this case, random ordering is applied.

3.1.2. Incoming calls.

Options:

  • If one of the servers is unavailable to the provider, the INVITE request does not receive a 100 Trying confirmation. On this basis, the provider’s equipment promptly sends the request through another account (internal redirect). The provider must support call redirection on no-answer within a short interval significantly shorter than the transaction lifetime (32 seconds).

  • The provider’s equipment sends incoming calls to an external number to two accounts at once (a group number with parallel calling, or variants with a small delay). When one answers, a CANCEL is sent to the other. The system can be configured to receive calls in a script with an instant answer (and instant rejection of the duplicate call) and subsequent transfer through the routing table, to eliminate duplicates in statistics.

  • Requests are sent to both servers simultaneously (forking); when one answers, the other receives CANCEL. Similarly to the previous case, call reception can be configured in a script with an instant answer.

  • (TODO) Two esg instances register simultaneously under the same account (forking). The provider must support this mode. Incoming calls are sent simultaneously to two devices registered under one account. The system’s task is to: a) allow configuring simultaneous registration under one account, b) reject a duplicate incoming call when its counterpart is detected on another instance based on a configured criterion, using a transaction.

3.2. Media gateway problem

The media gateway is selected by the active mgc controller. Several active media groups can be used simultaneously, each consisting of a controller and several media gateways statically bound to it in the configuration. Then the media group for call servicing is selected randomly and sequentially.

3.2.1. Configuring permanent use of the border media gateway bgmg, which is always located directly on the same server as the border SIP server (esg, sg) that initiates the corresponding call leg. Then the unavailability of the external channel for the main media gateway used in call servicing is not significant. However, in this case, failure of the border server makes it impossible to continue traffic exchange for existing calls.

3.2.2. When the controller selects a media gateway, it automatically takes into account the availability of the arbiter to it. The filter is applied only to media contexts managed by the b2b microservice. The filter is not applied to media contexts managed by the ivr and conf microservices (since the possibility of their contact with external subscribers is excluded, they are always associated with b2b media gateways, and the internal channel is not disrupted. The problem of selecting a microservice instance is considered separately: 3.4. Problem of selecting a microservice instance.

3.2.3. (TODO) Distributing media groups across different sites together with their media gateways. When organizing a random list, preferential use of the media group located in the same datacenter as the call initiator server (sg, esg, ivr, conf).

To solve it, regular polling of the arbiter is used by each server where microservices from the list requiring external access run. Based on the arbiter poll result, the server is assigned offline or online state. The arbiter in this case is used far beyond determining quorum for the connected half of servers. To reduce the number of requests, a 20-second cache is used.

Offline state tracking mode is enabled in the configuration parameters of each site with the check_offline parameter (disabled by default). If the arbiter is unavailable, the server is considered offline.

Among the microservices with this problem, some operate in active-active mode (for example, script) and some in active-passive mode (for example, email). Different schemes are used for them.

3.3.1. Active-active.

When selecting a node to execute a script, an offline filter is applied based on information about arbiter availability for servers with script and ivrscript microservices. A cache with a 20-second refresh interval is used.

Similar behavior applies when selecting media gateways (3.2. Media gateway problem).

3.3.2. Active-passive

When there is no communication with the arbiter, the microservice in group A enters a mode waiting for it to appear, without participating in the leadership competition. Thus the microservice instance in group B is activated regardless of the leadership priority configured. Recovery is also based on information about the arbiter becoming reachable. The check interval is 10 seconds. Cached data is not used but is refreshed.

3.4. Problem of selecting a microservice instance

When several microservice instances exist, an external channel availability filter is applied every time an executor instance is selected.

Regular (20-second interval) initiation of arbiter availability checks from the server hosting the microservice instance, and caching of the information for later use in selection (in particular, for mediagate on the controlling media controller). The arbiter in this case is used far beyond determining quorum for the connected half of servers; it is polled regularly from each server (where microservices requiring an external-communication filter run).

To eliminate the risk of depending on the arbiter’s own operability, one or more highly available external servers are registered in the site configuration. The site configuration option odd_referee is a string "IP1, IP2, IP3". IP addresses or FQDNs can be used. If the 'odd_referee' parameter is not set in the configuration, an island of half the servers is considered by the platform to be in quorum. If the parameter is set and empty, then conversely, it is not considered in quorum. If one or more arbiter addresses are present, the system performs a normal ping to them; if at least one request is answered, the arbiter is considered available and quorum is recorded.

3.5. Problem of distributed applications and services with a leader

Leader selection for distributed applications is based on priorities set in the site role configuration. When a failure occurs, the next microservice instance by priority on the site becomes the new leader (failover). When a higher-priority instance recovers, it forcibly takes leadership (takeover).

Leader selection for dynamic distributed services is performed by a capture transaction and is random. The failover operation is accompanied by the next iteration of random leadership capture by one of the remaining instances. The takeover operation is not used.

Instances running on servers without access to the external arbiter are excluded from the leadership competition. They are not activated on load; during operation, 3 consecutive failed attempts 2 seconds apart are made before a decision. Site option check_offline.

To eliminate the risk of depending on the arbiter’s own operability, several highly available external servers are registered in the site configuration. The site configuration option odd_referee is a string "IP1, IP2, IP3".

3.6. PostgreSQL problem

All PostgreSQL instances have streaming replicas and are connected to a replication controller (mware microservice operating in active-passive mode). The replication controller is active only in the group that is in quorum and remains leader after mutual polling over WAN. If the DB instance in the active group is a replica, this is detected by the active replication controller. 30 seconds after detecting master unavailability, the controller converts the replica to master. The conversion operation is point-in-time and completes in milliseconds. From then on, microservices interacting with the DB (mdc, dms, possibly script) can connect to the DB.

After communication between groups is restored, the controller detects two instances in master state and initiates conversion of the older master to a replica by creating a full backup. The operation is long and, depending on data size, can take tens of minutes. A backup is left nearby (by default in /tmp, cleared on each container/server restart).