Middleware (mware)

Description

Provides data transfer functions between internal services within a site, as well as operational correctness control.

Split into microservices by domains. Provides automatic synchronization between sites serving the domain.
Made redundant in Active-Passive mode. Scaled by splitting the domain tree.

Services and responsibilities:
- Administrator bot in messengers: system health notifications and commands (more).
- Support for sipuser accounts configured to work without registration.
- Call control via API (callmanager) (Calls, Conferences).
- Device control via API (devicemanager) (/api/devicemanager/v1/…​)
- Service for publishing role applications.
- Service for running service scripts on a schedule. (Service tasks).
- Critical section service (script component "Mutex" etc.).
- PostgreSQL streaming replication controller. (More about setup below, the 'pg_controller' parameter)
- Backup creation controller. (More about setup below, the 'backup_controller' parameter).
- Domain hot standby controller.
- ACME LetsEncrypt SSL certificate auto-generation service (more about setup).
- Global name monitoring.
- MEGACO connection state monitoring between MGC and MG.
- Health monitoring of the FSYNC file synchronization service.
- Service for distributing telephony provider accounts across ESG microservice instances.
- User password recovery service (more).
- Service for sending invitations to new users (more).
- Self-registration service (more).
- Service for registration during OAuth authorization (more).
- Syslog event collection and SNMP trap sending service (more about setup).
- Common product-layer microservice log collected from standard output messages.
- Conversation recording transcription service (more about setup).
- UDP packet mirroring test service.
- Service for registering callback requests (script component "Callback request" etc.).
- Service providing email_to_fax functionality.
- Service for tracking subscribers called via Hunt numbers and personal queues.
- Cache of external numbers whose states must be tracked (more about setup).
- CTI event provider service, including a supervisor of adapter servers for supporting various protocols.
- Availability monitoring of sg microservices with automatic removal of related WebSocket registrations (WebRTC softphones) and TLS registrations when they are lost.

Table 1. System characteristics

Code

mware

Operation mode

Service

Redundancy mode

Active-Passive

Site types

Any

Layer

Service

Placement

Internal

State preservation and recovery on restart

Yes

Application

era_mware

Parameters

Table 2. Parameters
Name Type Default Description

name

str

required

Name. May consist of Latin letters and digits and must start with a letter.

roletype

str

required

Role type. Possible values: "mware".

ext

json

empty

Additional role options. Contains a JSON object or list.

enabled

bool

empty

Role activity flag. When set to false, the role does not participate in validation and is not started.

limit_of_cores

int

empty

Limit on the number of CPU logical cores used. Default: 0 - unlimited.

logging_default_level

int

empty

Default logging level set specifically for the role instance (microservice type). More

allow_divergence

bool

false

Flag allowing parameter differences in roles of the same group. Default: false - not allowed.

group

int

required

Group number.
As part of horizontal scaling, the role can be split into several groups on a site, in each of which only one instance is active and the rest are standby.
Responsibility is divided between groups by domain sets. Together all groups serve the full domain set. All roles in one group must have the same responsibility.
An integer from 1 to 9999999.

Currently, a number of master services run within the mware role. This imposes a limitation on horizontal splitting capabilities (roadmap).

order

int

required

Order of the role instance within the group.
Determines the order in which takeover occurs in Active-Passive mode.

include_domains

array<str>

empty

List of domain names served by this group.
Any domain of any level can be specified. Specifying a domain causes its subtree to be served on this group and not on other role groups, except for branches mentioned in other groups.
Together, all role groups on the site serve the full domain tree.

pg_controller

array<object>

empty

Settings for the PostgreSQL server controller in streaming replication mode (pgsql 12+).
The list contains group objects. Each group contains a list of two servers in streaming replication mode.
The mware service ensures that each group has 1 master, automatically bringing it to a 1+1 state.
If two masters are detected, one is switched to recovery mode. Initially, the first server in the list is chosen as master. Subsequently, the server that last performed the master function.
If the master instance becomes unavailable, the recovery server becomes master within one to two minutes.
If the previous master appears again, it is switched to recovery mode by stopping and taking a full data backup from the current master.
All service activity is traced to the middleware log.
Operations with DB server instance states are performed over SSH; the corresponding options must be specified in the server properties.
The recovery mode switch is accompanied by moving the data directory to the specified backup folder. Default: /tmp/pg_backups.
Switching to master occurs only 30 seconds after the current master is lost.
Switching to recovery is possible only if an active master exists.

Value settings for groups:

  • key - unique string group key. Generated automatically if absent.

  • referee - external arbitration server address. If two instances are active simultaneously, only the one that can ping this server can act (quorum).

  • servers - list of servers included in the group.

Value settings for servers:

  • ssh_host - SSH host of PostgreSQL (default: pg_host).

  • ssh_port - SSH port of the PostgreSQL server (default: 22).

  • ssh_user - SSH user of the PostgreSQL host.

  • ssh_pwd - SSH password of the PostgreSQL host.

  • pg_host - actual current PostgreSQL host address available for replicas (default: ssh_host).

  • pg_port - actual current PostgreSQL port available for replicas (default: 5432). Recommended to set the same value on all replicas.

  • pg_replica_user - actual current PostgreSQL replica user login available for replicas. Must be the same on all replicas.

  • pg_replica_pwd - actual current PostgreSQL replica user password available for replicas. Must be the same on all replicas.

  • pg_ctl_command - command (name, path, or alias) available to the SSH user for pg_ctl of the current PostgreSQL version.

  • pg_basebackup_command - command (name, path, or alias) available to the SSH user for pg_basebackup of the current PostgreSQL version.

  • pg_database - actual current PostgreSQL initial database to connect to as the replica user and save state (default: postgres).

  • pg_data_folder - PostgreSQL instance data folder. Recommended to set the same value on all replicas.

  • pg_backup_folder - PostgreSQL instance backup folder.

  • pg_log_filepath - PostgreSQL instance log file.

The specified value of this parameter is not validated by the configuration validator in the mic role but passes through a runtime filter. If incorrect settings are detected, the group is excluded. If there are no correctly configured groups, the controller is inactive and waits for configuration changes.

All system roles automatically switch to work with the new master specified in the configuration, regardless of this setting.

Maintains a dedicated log "pgctrl_*.log".

=PostgreSQL setup=

For the controller service to work correctly, PostgreSQL instances must be preconfigured.
Servers can be installed using the platform script; in that case they receive this configuration automatically.

PostgreSQL version 12 or newer.
A replication account has been created (for example, replicator), allowed 10 or more connections.
The following settings are made in the configuration files (individual parameter values may vary at the administrator’s discretion):

postgresql.conf
listen_addresses = '*'
hot_standby = on
wal_level = replica
max_wal_senders = 10
wal_keep_size = 32
pg_hba.conf
host	all	            era_replica     	0.0.0.0/0		md5

If the second server is configured in master mode, after the first start the controller automatically switches one of them to replication mode.

Additionally, some settings can be specified in master domain parameters (more…​).

General parameter example:

[
  {
    "key": "general",
    "referee": "8.8.8.8",
    "servers": [
      {
        "pg_backup_folder": "/mnt/db/backup/era",
        "pg_basebackup_command": "/usr/lib/postgresql/14/bin/pg_basebackup",
        "pg_config_folder": "/mnt/db/era",
        "pg_ctl_command": "/usr/lib/postgresql/14/bin/pg_ctl",
        "pg_data_folder": "/mnt/db/era",
        "pg_database": "postgres",
        "pg_host": "db1.local",
        "pg_log_filepath": "/mnt/db/era/postgresql-14-era.log",
        "pg_port": 5441,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "...",
        "ssh_host": "db1.local",
        "ssh_port": 2022,
        "ssh_user": "postgres",
        "ssh_pwd": "..."
      },      {
        "pg_backup_folder": "/mnt/db/backup/era",
        "pg_basebackup_command": "/usr/lib/postgresql/14/bin/pg_basebackup",
        "pg_config_folder": "/mnt/db/era",
        "pg_ctl_command": "/usr/lib/postgresql/14/bin/pg_ctl",
        "pg_data_folder": "/mnt/db/era",
        "pg_database": "postgres",
        "pg_host": "db2.local",
        "pg_log_filepath": "/mnt/db/era/postgresql-14-era.log",
        "pg_port": 5441,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "...",
        "ssh_host": "db2.local",
        "ssh_port": 2022,
        "ssh_user": "postgres",
        "ssh_pwd": "..."
      }
    ]
  },
  ...
]

Secret (ssh_pwd, pg_replica_pwd, including in servers): ciphertext when read; the plaintext value is not returned via API and is decrypted by the service when used; a plaintext value can be supplied on input (Encryption and keys).

backup_controller

object

empty

Settings for the backup creation controller.
When configuring, specify a list of weekdays (weekdays) and an hour interval (hours_interval) outside which the service is inactive. Hours are specified in UTC.
The service is activated at most once a day. When the middleware microservice restarts, it starts no earlier than after 15 minutes. Maintains a dedicated log 'bkp_*.log'.
Creates a backup of the specified PostgreSQL DBMS instances (version 12+) and the :SYNC synchronization directory.
As a result, archive files are created for each specified DBMS instance and one for the :SYNC directory with exclusions.
Archive files are uploaded via the specified SSH connections to an external storage server in the specified folder. Each archive can be sent to its own server and/or its own folder.
Storage server access settings have a parameter for determining the destination directory path (in SSH mode, this is a path relative to the user’s home folder). The path can contain templates:

  • weekday - substituted with the weekday number (1-7).

  • day - substituted with the current day of the month in '00' format (01-31).

  • month - substituted with the current month in '00' format (01-12).

  • year - substituted with the current year in '0000' format.

By specifying templates, you can store several backup instances and organize their circular replacement. By default, the user’s home directory on the storage server is used, and files are regularly overwritten.

If the backup operation fails, the entire process is interrupted and this is reflected in the system state. The next attempt is made according to the schedule no earlier than after 20 minutes.

The controller can create backups for several PostgreSQL DBMS instances. To do this, specify them in the list.
Each PostgreSQL DBMS instance can have one or more replicas (different servers in streaming replication mode).
If replicas must be specified, the instance settings object must use the servers parameter containing a list of individual settings from the instance’s general section.
In this case, failure of an operation on one replica causes a switch to the next.
Successful completion of backup creation on any replica stops work on the instance and moves to the next instance.
Failure of backup creation on all replicas aborts the entire backup operation with an error.

When creating a backup archive of a PostgreSQL DBMS instance, the 'pg_basebackup' utility is used.
The instance settings in 'pg_hba.conf' must have replication permissions.
When installing the system with the installer and choosing to install PostgreSQL on the host, these parameters are set automatically for the 'era_replica' user.
In this case, that account should be specified.

Value settings for a PostgreSQL DBMS instance:

  • ssh_host - host where the 'postgresql' instance is installed and for which access to the target DBMS instance for replication is allowed via 'pg_hba.conf'. It must also have 'zip', 'ssh', 'ssh-keygen', 'ssh-keyscan', 'sshpass', 'scp', and 'rm' available.

  • ssh_port - SSH connection port.

  • ssh_user - SSH connection user name. Mostly postgres.

  • ssh_pwd - SSH connection user password (if public-key access is configured from all containers of the current site where the 'middleware' role may be located, this parameter can be omitted).

  • pg_host - host of the target PostgreSQL DBMS instance from which replication and data backup must be performed.

  • pg_port - port of the target PostgreSQL DBMS instance (default 5432, which differs from the default port set by the system installer).

  • pg_replica_user - PostgreSQL user name for replication connection. Must be listed in 'pg_hba.conf'.

  • pg_replica_pwd - PostgreSQL user password.

  • destination_folder - temporary directory path. Created and deleted on completion. Default: '/tmp/backup_temp/pgdb/'.

  • pg_basebackup_command - command to run 'pg_basebackup' under the specified ssh_user. Default: "pg_basebackup".

  • send_mode - synchronization mode. Default: 'rsync'.

    • zip_cmd - packs into a zip archive on the host under the specified ssh_user and sends it to storage using 'scp'.

    • rsync - sends files using rsync without compression but with identity comparison. Assumes a static path without templates in storage, or setup of external automatic deletion of outdated backups.

  • zip_command - command to run 'zip' under the specified ssh_user. Default: "zip". Used only in send_mode = "zip_cmd".

  • ssh_command - command to run 'ssh' under the specified ssh_user. Default: "ssh".

  • sshpass_command - command to run 'sshpass' under the specified ssh_user. Default: "sshpass".

  • sshkeygen_command - command to run 'ssh-keygen' under the specified ssh_user. Default: "ssh-keygen".

  • sshkeyscan_command - command to run 'ssh-keyscan' under the specified ssh_user. Default: "ssh-keyscan".

  • rsync_command - command to run 'rsync' under the specified ssh_user. Default: "rsync". Used only in send_mode = "rsync".

  • scp_command - command to run 'scp' under the specified ssh_user. Default: "scp". Used only in send_mode = "zip_cmd".

  • rm_command - command to run 'rm' under the specified ssh_user. Default: "rm".

  • storage_ssh_host - storage server host for placing the created archive.

  • storage_ssh_port - storage server port.

  • storage_ssh_user - user name for SSH connection to the storage server.

  • storage_ssh_pwd - user password for SSH connection to the storage server (if public-key access is configured from all hosts specified by ssh_host, this parameter can be omitted).

  • storage_ssh_path - path to the directory on the storage server (relative to the storage_ssh_user home directory) where the created archive should be placed. Substitution templates can be used to store files created on different days and organize circular overwrite. When collecting archives from several DBMS instances and placing them on the same storage, different paths should be set to avoid overwriting one archive with another.

  • servers - list of replicas (array of objects containing individual values of the parameters above).

The archive file is in zip format. Approximate recovery algorithm:

  1. Stop the PostgreSQL instance service in the target data and configuration directory (pg_ctl -D $DataFolder stop).

  2. Copy the archive to the target machine and unpack it into the target directory.

  3. If necessary, change parameters in the instance configuration files 'postgresql.conf' and 'pg_hba.conf' (if the archive was collected from a system with different settings: port, interfaces, access, and replication parameters).

  4. Ensure that the promote_trigger_file parameter in 'postgresql.conf' is set to watch for the master.signal signal file in the target directory.

  5. Set the owner required for operation in the target system on all unpacked files in the directory (chown -R postgres:postgres $DataFolder).

  6. Start the PostgreSQL instance service in the target data and configuration directory (pg_ctl -D $DataFolder start).

  7. Promote it to master (touch master.signal).

The controller can create a backup archive of the ':SYNC' directory.
It creates a zip archive excluding the 'standardexpressions' directory and unpacked role application data (the archives themselves are packed).

  • excluded_paths - list of excluded directories as path masks (e.g., '*/roleapps/'). Exclusions can reduce archive size by removing temporary and static data. On a multi-site system, the shared section can also be excluded, leaving backup responsibility only to the central site.

  • send_mode - synchronization mode. Default: "rsync".

    • zip_cmd - packs the directory into a zip archive on the local host (container with era installed) and sends it to storage using 'scp'.

    • rsync - sends files using 'rsync' without compression but with identity comparison. Assumes a static path without templates in storage, or setup of external automatic deletion of outdated backups.

  • zip_command - command to run 'zip'. Default: "zip". Used only in send_mode = "zip_cmd".

  • ssh_command - command to run 'ssh'. Default: "ssh".

  • sshkeygen_command - command to run 'ssh-keygen'. Default: "ssh-keygen".

  • sshkeyscan_command - command to run 'ssh-keyscan'. Default: "ssh-keyscan".

  • rsync_command - command to run 'rsync'. Default: "rsync". Used only in send_mode = "rsync".

  • scp_command - command to run 'scp'. Default: "scp". Used only in send_mode = "zip_cmd".

  • rm_command - command to run 'rm'. Default: "rm". Used only in send_mode = "zip_cmd".

  • storage_ssh_host - storage server host for placing the created archive.

  • storage_ssh_port - storage server port.

  • storage_ssh_user - user name for SSH connection to the storage server.

  • storage_ssh_pwd - user password for SSH connection to the storage server (if public-key access is configured from all hosts specified by ssh_host, this parameter can be omitted).

  • storage_ssh_path - path to the directory on the storage server (relative to the storage_ssh_user home directory) where the created archive should be placed. Substitution templates can be used to store files created on different days and organize circular overwrite. When collecting archives from several DBMS instances and placing them on the same storage, different paths should be set to avoid overwriting one archive with another.

To restore, it is enough to unpack/copy to the ':SYNC' directory on any server of the site.
Secret (ssh_pwd, pg_replica_pwd, storage_ssh_pwd, including in servers; sync_folder.storage_ssh_pwd): ciphertext when read; the plaintext value is not returned via API and is decrypted by the service when used; a plaintext value can be supplied on input (Encryption and keys).

domain_group

int

group

Domain distribution group number. Instances with the same value share the domain subtree (include_domains) as one group. Defaults to the group number.
An integer from 0.

Configuration example

Configuration management is performed in an application available to master domain administrators. The application hides the full configuration content, but it is still available through the API.

The configuration contains a section describing all instances of all roles. Parameters are defined for each specific role instance.

Node example
{
  "name": "mware2",
  "roletype": "mware",

  "pg_controller": [
    {
      "key": "general",
     "servers": [
       {
        "ssh_host": "192.168.0.123",
        "ssh_port": 9022,
        "ssh_user": "postgres",
        "ssh_pwd": "123456",
        "pg_host": "192.168.0.123",
        "pg_port": 5420,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "654321",
        "pg_database": "postgres",
        "pg_ctl_command": "/usr/lib/postgresql/12/bin/pg_ctl",
        "pg_basebackup_command": "pg_basebackup",
        "pg_data_folder": "/var/lib/postgresql/12/instance01",
        "pg_backup_folder": "/var/lib/postgresql/12/backups/instance01",
        "pg_log_filepath": "/var/log/postgresql/postgresql-12-instance01.log"
       },
       {
        "ssh_host": "192.168.0.124",
        "ssh_port": 9022,
        "ssh_user": "postgres",
        "ssh_pwd": "123456",
        "pg_host": "192.168.0.124",
        "pg_port": 5420,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "654321",
        "pg_database": "postgres",
        "pg_ctl_command": "/usr/lib/postgresql/12/bin/pg_ctl",
        "pg_basebackup_command": "pg_basebackup",
        "pg_data_folder": "/var/lib/postgresql/12/instance01",
        "pg_backup_folder": "/var/lib/postgresql/12/backups/instance01",
        "pg_log_filepath": "/var/log/postgresql/postgresql-12-instance01.log"
       }
     ]
    }
  ],

  "backup_controller": {
    "hours_interval": [20, 3],
    "weekdays": [1,2,3,4,5,6,7],
    "storage_ssh_host": "storage.local",
    "storage_ssh_port": 8022,
    "storage_ssh_user": "user414",
    "storage_ssh_pwd": "pwd414",
    "storage_ssh_path": "backup/dow_%weekday",
    "sync_folder": {
      "send_mode": "rsync",
      "excluded_paths": ["**/common/fax/*", "**/some_temp_files/*"]
    },
    "pgdb": [
      {
        "servers": [
          {
            "ssh_host": "192.168.0.123",
            "pg_host": "192.168.0.123"
          },        {
            "ssh_host": "192.168.0.124",
            "pg_host": "192.168.0.124"
          }
        ],
        "ssh_port": 9022,
        "ssh_user": "postgres",
        "ssh_pwd": "123456",
        "pg_port": 5420,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "654321"
      }
    ]
  },

  "group": 11120,
  "order": 10
}

See also