Split-brain problem and quorum monitoring

Problem statement

The case in question concerns a multi-server cluster but is possible already in a two-server installation. Some services manage and store data. For this they use a distributed object storage or a connection to the postgresql master instance (with streaming replication to the recovery instance). Data integrity must be preserved.

If there is a risk of losing connectivity between servers while isolated operability is preserved, then there is a risk of splitting the integrity point. The platform is divided into microservices and has several different integrity points.

Suppose two islands have formed — some subscribers are connected to the first island, some to the second. The platform has no central node. Therefore each island quickly strives to assemble into a fully functional system on the servers available in the island. The only limitation is the configuration of the specific platform instance. Thus, standby instances of microservices that have an integrity point activate quickly.

In the limit, two systems may form, each with its own subscriber group, and each ready to serve. However:

  • duplicated initiation of outgoing interactions is possible;

  • some subscribers are unavailable on each island;

  • data modification transactions differ and violate integrity.

The negative consequences become especially obvious when connectivity between the islands is restored:

  • Data merging is generally impossible, so modifications of one island are lost in favor of the other’s data.

  • If postgresql is installed on the same machine, then instead of active streaming replication two master instances are found. The streaming replication recovery process involves a full backup and, depending on database sizes and disk speeds, may take hours. All this time the replica is effectively absent, and the system is not ready for the loss of the DB master server.

This problem is conventionally called Split-brain.

Solution overview

The concept of quorum serves to solve this problem.

If an island contains fewer than 50% + 1 servers, its services capable of violating integrity should be forcibly shut down:

  • facade services through which subscribers connect and register;

  • initiator services that can generate new records;

  • especially the postgresql replication controller, so that it does not turn recovery into master and does not lead to a further multi-hour streaming replication recovery operation;

  • any services with an integrity point.

Whichever islands the system splits into, there are never two islands of 50% + 1 servers.

A special case is when the system can split into two equal halves. This is the case

  • two-server installations with separated servers and indirect connection, or with possible spontaneous server maintenance resulting in one server being disconnected from the switch.

  • installations with an even number of servers distributed across two data centers.

In both cases the goal is to ensure a server on one half when the other half fails. But the side effect is possible split-brain.

Obviously, quorum in its usual sense will not allow either half to be active.

The solution may be adding another node to the cluster. It can be:

  • another server, even one not loaded with configuration;

  • an arbitrary arbiter on the external network whose ping means the public network is preserved.

The arbiter case deserves more attention since it is simpler, less resource-demanding, and at the same time creates another branch. The island that is not connected to the arbiter loses the ability to be in quorum. The island connected to the arbiter gets a chance to be active.

Consider the case when the internal network between islands is lost but both islands can ping the arbiter over the external network. In this case the split-brain problem remains real, but through the external network the islands are reachable to each other — through the arbiter’s network. To get permission to activate in this case, the island tries to contact the servers of the other island that are unreachable over the internal network and hold elections. Permission is granted if the other island has recognized itself as a follower or the other island is unreachable.

Elections are held based on special ordinal values specified in the server configuration, or, if absent, based on lexicographic sorting of the configuration server names. The one containing the minimum value is recognized as leader. To communicate over the external network, a special URL can be set for all servers in the configuration. Servers access the web servers (microservices) located on these servers via these URLs.

Obviously, the failure of any server in a configuration with more than two servers can also provoke split-brain. Therefore quorum monitoring should be configured for these cases too.

Configuring quorum monitoring

Quorum is configured in the configuration.

  • Parameter of the general or sites section: 'check_quorum' — enabling quorum monitoring.

  • Parameter of the general or sites section: 'odd_referee' — specifying an external arbiter for even-server configurations.

  • Server parameter: 'wan_url' — address.

  • Server parameter: 'quorum_leadership_key'.

  • Additionally, a web server should be added on each server — whether with a dedicated limited purpose or general availability.

By default quorum monitoring is disabled in automatically generated configurations.

Operating mechanism

Quorum monitoring is performed in server nodes. Other nodes, as needed, poll and subscribe to information about quorum presence in the island involving the current server.

Quorum is monitored by:

Since the domain center in a distributed system is reserved in active-passive mode and monitors quorum, other microservices that work with the domain tree but are reserved in active-active mode also depend on it, in particular:

  • product layer supervisors (msvc),

  • and consequently all product microservices.

Since all microservices working with distributed object storage are reserved in active-active mode, the integrity of these storages is not violated.

Since mware in a distributed system is reserved in active-passive mode and monitors quorum, the following depend on quorum:

  • the postgresql streaming replication controller (turns the DB recovery instance into master and restores streaming replication),

  • the backup creation controller,

  • the domain hot redundancy controller,

  • the scheduled service script launch controller,

  • etc.

Thus, a couple of seconds after quorum loss on an island, no operations occur, no one connects or registers, and no requests to external systems and providers are made.