Step 4. Changing the configuration

Suppose changes need to be made to the configuration, for example adding a server, redistributing roles, or changing their parameters. It would seem simple: distribute the new configuration file to all servers and restart the system. Then ServerShell will do everything as during startup (step 3). You might not even need to restart, relying on ServerShell regularly monitoring file system changes. However, this approach has three significant drawbacks:

  1. With many servers, there is a lot of work, especially considering possible difficulties accessing remote sites.

  2. If some servers are unavailable, the operation must be postponed; otherwise there is a risk of significant and harmful desynchronization of individual system clusters.

  3. If the configuration file contains an error, such a configuration must not be applied. This applies to both static and dynamic errors: not every change can be applied immediately.

Therefore, there is an API for working with configuration files. This is not a domain-level API but a system-level API, so only master domain administrators can use it.

Case: an administrator in the master domain uploads and applies a new configuration file with changed settings, and as a result of its application the system acquires a new coherent state.

This case is handled by a set of roles jointly responsible for configuration: MIC (Master Infrastructure Configurator) and IC (Site Infrastructure Configurator).

seq cfg uploadFirst, by analogy with step 1, the configuration file uploaded by the administrator reaches MIC through WS. There it is stored and awaits further administrator actions. The administrator checks the uploaded file; validation is performed by the same MIC role. Then, if everything is successful, the administrator applies the file with the API request /api/admin/v1/cfg/activate. MIC initiates a complex process of synchronizing the configuration file across all sites, servers, and nodes. This process reaches every running role so that it can apply the new settings hot.

Configuration application is a tree-branching process.

  1. MIC sends a command to synchronize the configuration file to all sites: the IC roles.

  2. IC sends a command to synchronize the configuration to all servers: the ServerShell service process in the root node.

  3. ServerShell calculates the difference and applies the changes, unloading unnecessary nodes and starting the necessary ones. Node names may also have changed; they are generated automatically based on the roles running in them. It sends the configuration application command to the other nodes of its server, specifically to the BasedRole service process in each node.

  4. BasedRole calculates the difference and applies the changes. Node names depend on the set of roles, so the set could not have changed; therefore BasedRole sends a command to apply the new parameters to all roles running on the current worker node.

  5. Each role independently analyzes the changes and applies its new startup parameters, either hot or by restarting.

Each node of the configuration synchronization process sends a command to all its subscribers and subscribes to its parent node. Thus, the responsibility of a child node is to discover the parent node through the configuration and subscribe (see figure).

config_subscription_scheme_uml

MIC and IC are roles like any other, so their basic behavior is subordinate to their local ServerShell and BasedRole. Their target functionality places them at the top of the configuration management hierarchy (see the figure above). These are two views of the relationship between configuration services, two levels of connection: from the subscription and configuration update process perspective, MIC is the head, followed by MIC → IC → ServerShell → BasedRole; from the system startup perspective, ServerShell is the head, followed by ServerShell → BasedRole → Role, including MIC and IC. Drawing an adequate flat, compact diagram reflecting both views at once is non-trivial, but representing this model abstractly is more than realistic:

ServerShell starts BasedRole, BasedRole starts MIC among other roles, and MIC begins working and providing configuration management functionality. Similarly for IC on its server. If the configuration changes, MIC notifies all available ICs, each IC notifies all available ServerShells on the site (including its own), ServerShell manages worker nodes and notifies BasedRole (including on the MIC node). After that, the MIC role can change its location and behavior. While IC is not running as a role, all ServerShells make unsuccessful attempts to connect to it based on the current configuration. These attempts are asynchronous and do not interfere with the target process of starting worker nodes and the roles in them.

mic_ic_servershell_sample

MIC is a special role: it must be present on exactly one site. The site where MIC is located is called the master site. Conversely, if a site is the master site, MIC must be present on it. Other sites are called worker sites. If a site (whether worker or master) has the full set of necessary roles, it is called communication-capable. Thus, a communication-capable site is a sign of the site’s product-level completeness. Any worker site must be communication-capable.

The diagram shows the configuration change subscription hierarchy using a binary-branching tree as an example (two sites, two servers each, two roles each):

cfg_roles

BasedRole components are at the bottom layer of the diagram, but we remember that they in turn manage worker nodes with functional roles. Thus, one of the BasedRole instances shown manages the startup of the node with the MIC role, and two more manage the startup of nodes with IC roles. This model should be carefully considered before moving on.

Disabled servers and sites

If any system node is unavailable at the time the configuration is applied, this does not prevent the configuration from being applied. All available nodes switch to the new configuration, and nodes that were unavailable when connecting to the cluster connect based on the previous configuration file to the parent node (in particular, a connected site connects to MIC from the previous configuration, and a connected server connects to IC from the previous configuration). The parent node immediately provides the new configuration, and the newly connected node receives the new settings for itself and the rest of the infrastructure. The same happens if a new site or server is connected to the system.

The first configuration

What configuration exists initially? The configuration change subscription tree must exist and be active.

When installing the program on each new server, the parameters specify nothing other than the address of the server hosting the MIC role (as a node name). Thus, the server with the MIC role starts first so that subsequent servers can connect to it. A basic configuration of three roles automatically appears on it: MIC, DC, WS. Each newly installed server acts as an analogue of IC in the sense that it is a direct subscriber of MIC to configuration changes. Only when it discovers itself in the configuration does it start fully and reorient to the IC of the current site.

When the current configuration already contains sites and running IC roles, the connection point for new servers can be either MIC or any available IC instance.

Difficulties when changing the configuration

There are some minor difficulties with reconfiguration, which is why not every configuration can be applied to the current one. For example:

  • It is impossible to move MIC to another server in one step. However, this can be done in two steps within the master site.

  • There are difficulties moving MIC to another site.

  • There are difficulties changing server IP addresses.

  • There are a number of roles that, like MIC, can be moved to other servers only in two steps to avoid data loss.

  • There are difficulties changing site names.

  • There are difficulties changing role identifiers.

  • There are difficulties moving roles to which domain entities are explicitly bound.

  • It is impossible to change the master domain name.

"Apply in two steps" means: first create an intermediate transition configuration based on the current and target configurations, apply it, wait until the necessary servers finish applying it (all roles start and reach steady state; this process may be slow due to data synchronization), and only then apply the target configuration.

To avoid these difficulties, think through the future state of the system before starting implementation. It is recommended to prepare a project design document that approves the master domain, master site, names of all sites, fixes the IP address range for the servers of the future system, and models the distribution of roles across servers. Expanding the system is easy; shrinking it is harder; moving and changing some configuration parameters is not just difficult but sometimes impossible without fully reconnecting servers one by one. However, even in that case, entities in domains are preserved and can be loaded from DomainDB.

Table 1. Terms used
Term Definition

MIC

!

IC

!

Master site

!

Worker site

!

Communication-capable site

!

Review questions
  1. What prevents configuration changes from being made directly on servers?

  2. In which domains is configuration management possible?

  3. Which role is responsible for configuration management?

  4. Which roles participate in applying the configuration?

  5. Which services/roles are present in both the configuration management tree and the system startup tree?

  6. Under what condition can disabled servers connect to the system with the new configuration?

  7. At what moment does the first (basic) configuration appear?