Chapter 5. Quality attribute management methodologies
Scaling
Scaling depends on the load pattern and mainly involves moving part of the system’s functional blocks to separate physical or virtual servers. To solve this problem, the system was initially designed as a set of functional roles, each with its own responsibilities.
Each functional role can be located on a separate server. Under excessive load, it has a methodology for splitting into groups by responsibility.
At the same time, the architecture of the «Incoplax» communication platform classifies roles by the availability strategy: active-active and active-passive (see Step 6).
The entire system can be assembled from several independent parts, each with full functionality and separate physical resources (sites).
At the limit, the system looks like this:
-
Many sites (groups of highly available servers) connected by channels with lower requirements for stability and bandwidth.
-
Each site contains many servers.
-
Each server contains one or more roles.
-
Each role is located on several servers within each site.
-
Each role interacts with other roles within the site for services.
-
Each role interacts with its mirror on other sites for data coordination.
On one server
The consideration starts with the system installed on a single server. All roles are active. Let it be a 4-core server.
The system is not protected against machine failure; when a role fails, it is restarted, current activities are lost, and data is preserved. Conventionally, one server can handle the following workloads at full load:
-
either 6,000 SIP messages per second on SIP roles;
-
or 20,000 media packets per second (200 simultaneous switches * 2 trunks * 50 packets per second) on MG;
-
or 5,000 routing rule lookups per second (~5 routes, ~10 vectors per route);
-
or 40,000 registration operations per second per core;
-
…
One server can hold many domains, many registrations, and many calls, but all together are limited.
Load calculations
Each subscriber sends a registration on average once every 5 minutes: UA - Gate - B2B. 12 SIP messages.
10,000 subscribers - 400 SIP messages per registration operation per second - 8% of the limit.
Each established dialog with audio exchange is ~0.5% of the limit.
Each additional CPS is (30 + 15*(number of forks) + 24*(number of reinvites)) SIP messages + (5 + 3*(number of reinvites)) MEGACO messages. In the simple case, this is 0.75% + 0.5% + 0.2% + one routing operation of the server’s capacity limit.
Conventionally, 10,000 subscribers on one server can exist with up to 150 simultaneous conversations and CPS up to 50 (each at maximum), or conventionally 100/20 or 50/30.
State subscriptions, group calls, and multi-fork calls also add significant load.
Load increases slightly with complex cross-domain routing with many rules and with redirects.
Finding bottlenecks
-
As simultaneous calls increase, the MGC and MG roles must be allocated to separate servers. When simultaneous calls exceed 200, the number of MG role instances can and should be multiplied.
-
When simultaneous calls exceed 2,000, MGC must be multiplied and media groups allocated. Media groups can also be multiplied to increase availability and resilience during failures.
-
As CPS increases, B2B and SG must be allocated to separate servers and multiplied.
-
With a large number of SGs, it makes sense to introduce the REDIRECT role.
-
As CPS increases further, in some cases it makes sense to split STORE into groups.
-
When the number of users reaches 50,000 or CPS reaches 1,000 (routing, authorization, registration, subscriptions), users must be split into domains. The recommended maximum number of users per domain is up to 30 thousand.
-
With massive use of subscriptions, StateStore must be split into groups bound to domains, and B2B and SG must be multiplied and allocated to separate servers.
-
As load on several domains increases, DC, REGISTRAR, and StateStore must be split into server groups bound to domains.
-
When the number of DC, Registrar, StateStore, Store, MGC, and RPC servers reaches 40, sites must be allocated.
Scaling directions
Infrastructure:
-
adding servers;
-
adding sites.
Logic:
-
adding active-active roles and allocating them to new servers;
-
splitting active-passive roles into groups with non-overlapping responsibility (by domains, by hash ring).
Data:
-
adding new domains to the tree, moving parts of organizational structures to other domains;
-
moving databases to separate servers for each domain;
-
allocating a dedicated server for each role with a PostgreSQL/ClickHouse data replica (dc, dms);
-
allocating a separate S3 storage for each domain.
Increasing availability
The system is built on functional roles forming an active-active hash ring and active-passive standby groups. Roles provide services to each other and to external users.
In the architecture context, each role independently ensures its own availability.
Each role can be used on several physical nodes.
When a node is active, role activity is guaranteed by the configuration supervisor, whose activity is ensured by OS daemons/services.
Increasing system availability is related to increasing the availability of individual roles, communication channels, and physical units, as well as reducing the dependence of some units on others. Role availability is increased by duplicating on several servers:
-
Hot standby (active-passive).
-
Parallel availability (active-active).
-
A mix in which active-passive groups are split by responsibility (reg, dc, store) or by load (mgc, mg) into several simultaneously operating groups.
-
Fast application of configuration changes.
-
The external network perimeter of the cluster contains only the WS, ESG, SG, REDIRECT, and possibly RPCO roles, each with mechanisms protecting against intrusion, account password guessing, and performance loss from (D)DoS attacks through built-in border filters.
Increasing system availability in connection with storages (external to the platform) — S3, PostgreSQL, ClickHouse — is directly ensured by the availability of the storages themselves and by increasing their availability to the system. Thus, in the system configuration, each storage connection can be duplicated with an arbitrary number of alternatives. In this case, if the connection via the first connection string is unavailable, the system iterates through the remaining options and stops at the next available one. Domain-level settings for connecting to S3 and ClickHouse can also be duplicated with an arbitrary number.
The main case is ensuring communication processing; when storages for statistics and call recordings are unavailable, the system’s main work does not stop.
Increasing role availability
ACTIVE-ACTIVE roles (B2B, MG, WS, …)
-
Adding servers using the corresponding roles to the configuration (count 1, 3, 15, …).
ACTIVE-PASSIVE roles (REGISTRAR, MDC, SDC, MGC, STATESTORE, …)
-
Adding servers using the corresponding roles in standby mode to the configuration (count 1, 2, 3).
-
For roles split by domains: splitting the standby group into several standby groups by domain responsibility, with partial or complete allocation to separate servers and formation of non-overlapping active-passive groups (registrar, mdc, sdc, statestore, …).
-
For the system object storage role (store): splitting into several groups with allocation to separate servers; requests are distributed automatically via a hash ring by keys.
-
For the media controller role (mgc): splitting into several groups with allocation to separate servers; requests are distributed automatically by random group selection when creating a media context. Each group must receive its associated MGs, and it is rational to use the same number of MGs in all MGC groups on the site.
Increasing service availability
Media processing
-
adding new servers to the configuration, on which new MGC-MG groups with new numbers are deployed.
External provider connections
-
duplicating connections on different ESG role servers, while configuring sequential or priority-competing routes.
SIP signaling
-
multiplying SG, adding duplicated Outbound proxy server addresses to SIP devices.
-
multiplying SG, DNS routing to a backup server at the same address.
-
using REDIRECT.
Increasing availability during physical failures
No communication between cities
-
allocating sites with full internal functionality that do not require constant communication with other sites.
-
binding domains to sites, ensuring caching and storage of domain information on the site.
(D)DoS attacks on a server
-
multiplying SG, ESG, and WS with different external addresses.
-
using white/black lists on SG and ESG.
-
SG and ESG have a built-in network filter by sender addresses of failed authentications.
Site failure
-
binding a domain to several sites (backup or parallel); data is replicated automatically.
-
configuring phones to iterate over Outbound proxies, including another site.
-
Previous chapter: Product scheme
-
More about site redundancy: Site availability and data integrity