Middleware (mware)

Description

Provides functions for transferring data between internal services within the site, as well as controlling correctness of operation.

Divided into microservices by domain. Provides automatic synchronization of work between sites serving a domain.
Reserved in Active-Passive mode. Scaled by domain tree partitioning.

Services:
- Telegram-System state monitoring bot (Customization details).
- Ensuring functioning of sipuser accounts configured to work without registration.
- Call management via API (callmanager) (Calls, Conferences).
- Controlling devices via API (devicemanager) (/api/devicemanager/v1/…​)
- Publishing service roleapps.
- Service to run scheduled service scripts. (Service Tasks).
- Critical sections service (component scripts "Mutex", etc.).
- PostgreSQL stream replication controller. (More details on the configuration below, parameter 'pg_controller'])
- Backup Controller (Configuration details below, parameter 'backup_controller').
- ACME LetsEncrypt automatic SSL certificate generation service (Read more about customization).
- Global Name Monitoring.
- Monitoring the status of MEGACO connections between MGC and MG.
- File synchronization service performance monitoring FSYNC.
- User password recovery service (Read more).
- Service for sending invitations to new users (Read More).
- User self-registration service (Read More).
- Service of providing registration at authorization via OAuth and ESIA (Read more).
- Service collecting syslog events and sending SNMP trap (Configuration details).
- A generic log of product layer microservices, collected from standard output messages.
- Call recording shorthand service (Read more about customization).
- UDP packet mirroring test service.
- Callback order registration service (scripts component "Callback order" etc.).
- Functionality assurance service email_to_fax.
- A service for tracking callers called through Hunt numbers and personal queues.
- Cache of external numbers to be tracked states (Read more about customization).
- CTI event provider service, including a supervisor of adapter servers to support various protocols.
- Monitoring availability of microservices sg with automatic deletion of linked websocket registrations (webrtc softphones) and tls registrations when lost.

Table 1. System Characteristics

Code

mware

Mode of operation

Service

Backup mode

Active-Passive

Types of sites

Any

Layer

Service

Placement

Internal

Saving and restoring state on reboot

Yes

appendix

era_mware

Parameters

Table 2. Parameters
Name Type Default Description

name

str

required

Name. May consist of Latin letters and numbers, must begin with a letter.

roletype

str

required

Role Type. Possible values: "mware".

iface

str

required

The alias of the server network interface on which the roles will interact internally with each other.

ext

json

empty

Additional role options. Contains a json object or list.

enabled

bool

empty

Role activity flag. When set to false, the role does not participate in validation and is not started.

group

int

required

Group Number.
As part of horizontal scaling, a role can be divided into multiple groups on a site, with only one instance active in each group and the rest reserved.
Responsibilities between groups are shared across domain sets. Together, all groups maintain a complete domain set. All roles in the same group must have the same responsibility.
Integer from 1 to 9999999.

order

int

required

The order of a role instance within a group.
Defines in what order the flow in the mode will take place Active-Passive.

include_domains

array<str>

empty

A list of domain names served by this group.
Any domain of any level can be specified. Specifying a domain causes its subdomain tree to be served on this group and not on other role groups, excluding only those branches mentioned in other groups.
Collectively, all role groups on a site serve a complete domain tree.

pg_controller

array<object>

empty

Settings for the controller of PostgreSQL servers in streaming replication mode (version pgsql 12+).
The list contains objects-groups. Each group contains a list of two servers that are in the streaming replication mode.
The mware service ensures that each group has 1 master, automatically leading to the state of 1+1.
If two masters are detected, one of them is put into recovery mode. Initially, the first server in the list is selected as the master. Subsequently, the one that performed the master function last.
If the master instance becomes unavailable, the recovery server turns into the master within a minute or two.
If the previous master reappears, it is put into recovery mode by stopping and completely backing up the data from the current master.
All service activity is traced to the middleware log.
Working with the database server instance states is performed via SSH, for which the corresponding options must be specified in the server properties.
The recovery operation is accompanied by moving the data directory to the specified backup folder. By default /tmp/pg_backups.
Switching to master mode only after 30 seconds have elapsed since the current master was lost.
You can only enter recovery mode if there is an active master.

Group value settings:

  • key - unique string key of the group. If missing, it is generated automatically.

  • referee - address of the external arbitration server. When two instances are active simultaneously, only the one to whom this server is available for pinging (quorum) can act.

  • servers - list of servers that are members of the group.

Value settings for servers:

  • ssh_host - ssh host of postgresql (default as pg_host).

  • ssh_port - ssh port of postgresql server (default 22).

  • ssh_user - ssh user of postgresql host.

  • ssh_pwd - ssh password of postgresql host.

  • pg_host - actual current postgresql host address, available for replicas (default as ssh_host).

  • pg_port - actual current postgresql port, available for replicas (default 5432). Recommended to set the same value to all replicas.

  • pg_replica_user - actual current postgresql replica user login, available for replicas. Should be the same on all replicas.

  • pg_replica_pwd - actual current postgresql replica user password, available for replicas. Should be the same on all replicas.

  • pg_ctl_command - available for ssh user command (name, path or alias) to pg_ctl of current postgresql version.

  • pg_basebackup_command - available for ssh user command (name, path or alias) to pg_basebackup of current postgresql version.

  • pg_database - actual current postgresql initial database to connect to as replica user and save state (default postgres).

  • pg_data_folder - postgresql instance data folder. Recommended to set the same value to all replicas.

  • pg_backup_folder - postgresql instance backups folder.

  • pg_log_filepath - postgresql instance log file.

  • pg_signal_filepath - recovery signal file to make master. Recommended to set the same value to all replicas.

The specified value of this parameter is not checked by the configuration validator in the mic role, but passes through the filter in the rantime. If incorrect settings are detected, the group is excluded. If there are no correctly configured groups, the controller does not act and waits for configuration changes.

All system roles are automatically switched to work with the new master specified in the configuration, regardless of this setting.

Maintains a dedicated logfile "pgctrl_*.log".

=Customization PostgreSQL=

Pre-configuration of instances is required for the controller service to work correctly PostgreSQL.
Servers can be installed using a platform script, in which case they get this configuration automatically.

PostgreSQL version at least 12.
An account (for example, replicator) has been created for replication and is allowed 10 or more connections.
The following settings are made in the configuration files (the values of individual parameters may vary at the discretion of the administrator):

postgresql.conf
listen_addresses = '*'
hot_standby = on
wal_level = replica
max_wal_senders = 10
wal_keep_segments = 32
promote_trigger_file = '/var/lib/postgresql/12/era_instance_01/master.signal'
pg_hba.conf
host	all	            era_replica     	0.0.0.0/0		md5

If the second server is configured in master mode, the controller will automatically put one of them in replication mode after the first startup.

Additionally, some settings can be specified in the master domain parameters (more..).

General view of the parameter as an example:

[
  {
    "key": "general",
    "referee": "8.8.8.8",
    "servers": [
      {
        "pg_backup_folder": "/mnt/db/backup/era",
        "pg_basebackup_command": "/usr/lib/postgresql/14/bin/pg_basebackup",
        "pg_config_folder": "/mnt/db/era",
        "pg_ctl_command": "/usr/lib/postgresql/14/bin/pg_ctl",
        "pg_data_folder": "/mnt/db/era",
        "pg_database": "postgres",
        "pg_host": "db1.local",
        "pg_log_filepath": "/mnt/db/era/postgresql-14-era.log",
        "pg_port": 5441,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "...",
        "pg_signal_filepath": "/mnt/db/era/master.signal",
        "ssh_host": "db1.local",
        "ssh_port": 2022,
        "ssh_user": "postgres",
        "ssh_pwd": "..."
      },      {
        "pg_backup_folder": "/mnt/db/backup/era",
        "pg_basebackup_command": "/usr/lib/postgresql/14/bin/pg_basebackup",
        "pg_config_folder": "/mnt/db/era",
        "pg_ctl_command": "/usr/lib/postgresql/14/bin/pg_ctl",
        "pg_data_folder": "/mnt/db/era",
        "pg_database": "postgres",
        "pg_host": "db2.local",
        "pg_log_filepath": "/mnt/db/era/postgresql-14-era.log",
        "pg_port": 5441,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "...",
        "pg_signal_filepath": "/mnt/db/era/master.signal",
        "ssh_host": "db2.local",
        "ssh_port": 2022,
        "ssh_user": "postgres",
        "ssh_pwd": "..."
      }
    ]
  },
  ...
]

backup_controller

object

empty

Settings for the archive copy controller.
When configuring it is necessary to specify the list of days of the week (weekdays) and the interval of hours of the day (hours_interval), beyond which the service is inactive. The hours are set in UTC.
The service is activated not more than once a day. When restarting the microservice, middleware starts not earlier than 15 minutes later. Maintains a dedicated logfile 'bkp_*.log'.
Creates a backup of the specified PostgreSQL DBMS (versions 12+) and synchronization directory :SYNC.
The output creates archive files for each specified DBMS instance, and one for the :SYNC directory with exceptions.
The archive files are uploaded via specified ssh connections to an external storage server in a specified folder. Each archive can be sent to its own server and/or folder.
The storage server access settings have an option to define the path to the destination directory (in ssh mode, this is the path relative to the user’s home folder). The path can contain templates:

  • weekday - weekday number (1-7).

  • day - the number of the current day of the month is substituted in the format '00' (01-31).

  • month - the number of the current month is substituted in the format '00' (01-12).

  • year - the current year is substituted in the format '0000'.

By specifying templates, storing multiple instances of archive copies is configured, as well as circularly replacing them. By default, the user’s home directory on the storage server is used and the files are regularly overwritten.

If the backup operation fails, the whole process is aborted and this is reflected in the system status. The next attempt will be made according to the schedule in 20 minutes at the earliest.

The controller can create backup archive copies for several PostgreSQL DBMS instances. To do this, they must be specified in the list.
Each PostgreSQL DBMS instance can have one or more replicas (different servers in streaming replication mode).
If replicas need to be specified in the instance configuration object, the servers parameter must be used to list the individual settings from the instance’s common section.
In this case, the failure of an operation on one of the replicas leads to a transition to the next one.
Successfully completing a backup archive on any of the replicas aborts the instance and moves on to the next instance.
Failure to complete the backup archive across all replicas aborts the entire backup operation with an error.

The PostgreSQL DBMS instance data backup archive is created using the utility 'pg_basebackup'.
It is required that replication permissions are set in the instance settings in the 'pg_hba.conf' file.
When installing the system using the installer and choosing to install PostgreSQL in the host, these settings are automatically prescribed for the user 'era_replica'.
In this case, it is his account that should be specified.

Setting values for the DBMS instance PostgreSQL:

  • ssh_host - host where the 'postgresql' instance is installed and for which access to the target DBMS instance for replication is allowed using 'pg_hba.conf'. The following should also be available there 'zip', 'ssh', 'ssh-keygen', 'ssh-keyscan', 'sshpass', 'scp', 'rm'.

  • ssh_port - dial-up port ssh.

  • ssh_user - the username for the ssh connection. Predominantly postgres.

  • ssh_pwd - user password for ssh connection (if public key access is configured from all containers of the current site, where the 'middleware' role can be placed, this parameter can be omitted).

  • pg_host - host of the target PostgreSQL DBMS instance from which replication and backup data archive creation should be performed.

  • pg_port - port of the target PostgreSQL DBMS instance (5432 by default - does not match the default port set by the system installer).

  • pg_replica_user - PostgreSQL user name to connect for replication purposes. Must be entered in 'pg_hba.conf'.

  • pg_replica_pwd - user password PostgreSQL.

  • destination_folder - path to the temporary directory. It is created and deleted when finished. The default directory is '/tmp/backup_temp/pgdb/'.

  • pg_basebackup_command - command to run 'pg_basebackup' from the specified ssh_user account. By default "pg_basebackup".

  • send_mode - synchronization mode. Default 'rsync'.

    • zip_cmd - packs to a zip archive on the host under the specified ssh_user account and sends it to the repository using the 'scp'.

    • rsync - sends files using rsync without compression, but using an identity comparison mechanism. Assumes the use of a static path without templates in the repository, or setting up external automatic deletion of obsolete archive copies.

  • zip_command - command to run 'zip' from the specified ssh_user account. Defaults to "zip". Used only in send_mode = "zip_cmd"

  • ssh_command - command to run 'ssh' from under the specified ssh_user account. By default "ssh".

  • sshpass_command - command to run 'sshpass' from under the specified ssh_user account. By default "sshpass".

  • sshkeygen_command - command to run 'ssh-keygen' from under the specified ssh_user account. By default "ssh-keygen".

  • sshkeyscan_command - command to run 'ssh-keyscan' from under the specified ssh_user account. By default "ssh-keyscan".

  • rsync_command - command to run 'rsync' from the specified ssh_user account. Defaults to "rsync". Used only in send_mode = "rsync"

  • scp_command - command to run 'scp' from the specified ssh_user account. Defaults to "scp". Used only in send_mode = "zip_cmd"

  • rm_command - command to run 'rm' from under the specified ssh_user account. By default "rm".

  • storage_ssh_host - host of the remote storage server to host the created archive.

  • storage_ssh_port - port of the remote storage server.

  • storage_ssh_user - user name to connect to the storage server by ssh.

  • storage_ssh_pwd - user password to connect to the storage server via ssh (if public key access from all hosts specified by the ssh_host parameter is configured, the parameter can be omitted).

  • storage_ssh_path - path to the directory on the storage server (relative to the home directory of storage_ssh_user) where the created archive should be placed. Substitution templates may be used to store files created on different days and organize the round-robin overwriting. When building archives from several DBMS instances and placing them on the same storage, different paths should be set to avoid overwriting one archive with another.

  • servers - replica list (an array of objects containing individual values of the above parameters).

The archive file is in zip format. Approximate recovery algorithm:

  1. Preemptively stop the PostgreSQL instance service residing in the target data and configuration directory (pg_ctl -D $DataFolder stop).

  2. Copy the archive to the target machine and unzip to the target directory.

  3. If necessary, change parameters in the instance configuration file 'postgresql.conf', 'pg_hba.conf' (if the archive was built from a system that has other settings - port, interfaces, access and replication parameters).

  4. Ensure that the promote_trigger_file parameter in the 'postgresql.conf' settings is set to track the master.signal file in the target directory.

  5. Set all unpacked files in the directory to have an owner that is required to work on the target system (chown -R postgres:postgres $DataFolder).

  6. Start the PostgreSQL instance service in the target data and configuration catalog (pg_ctl -D $DataFolder start).

  7. Convert it to master (touch master.signal).

The controller can back up the catalog archive copy ':SYNC'.
Forms a zip archive excluding the 'standardexpressions' directory, unpacked role application data (the archives themselves are packed).

  • excluded_paths - list of excluded directories in the form of path masks (note '*/roleapps/'). Exclusions can be used to reduce archive size by eliminating temporary and static data. Also, on a multisite system, you can exclude a shared partition, leaving the responsibility of backing it up to the central site.

  • send_mode - synchronization mode. Default "rsync".

    • zip_cmd - packs the catalog into a zip archive on the local host (container with era installed) and sends it to the repository using the 'scp'.

    • rsync - sends files using 'rsync' without compression, but using an identity comparison mechanism. Assumes use of a static path without templates in the repository, or setting up external automatic deletion of obsolete archive copies.

  • zip_command - command to run 'zip'. Defaults to "zip". Used only in send_mode = "zip_cmd"

  • ssh_command - command to start 'ssh'. By default "ssh".

  • sshkeygen_command - command to run 'ssh-keygen'. By default "ssh-keygen".

  • sshkeyscan_command - command to run 'ssh-keyscan'. By default "ssh-keyscan".

  • rsync_command - command to run 'rsync'. Defaults to "rsync". Used only in send_mode = "rsync"

  • scp_command - command to run 'scp'. Defaults to "scp". Used only in send_mode = "zip_cmd"

  • rm_command - command to run 'rm'. Defaults to "rm". Used only in send_mode = "zip_cmd"

  • storage_ssh_host - host of the remote storage server to host the created archive.

  • storage_ssh_port - port of the remote storage server.

  • storage_ssh_user - user name to connect to the storage server by ssh.

  • storage_ssh_pwd - user password to connect to the storage server via ssh (if public key access from all hosts specified by the ssh_host parameter is configured, the parameter can be omitted).

  • storage_ssh_path - path to the directory on the storage server (relative to the storage_ssh_user home directory) where the created archive should be placed. Substitution templates can be used to store files created on different days and organize overwriting on a round robin basis. When building archives from several DBMS instances and placing them on the same storage, different paths should be set to avoid overwriting one archive with another.

To restore, simply unzip/copy to the ':SYNC' directory on any of the site’s servers.

General view of the facility as an example:

{
  "hours_interval": [20, 3],
  "weekdays": [2,4,7],
  "sync_folder": {
    "send_mode": "rsync",
    "excluded_paths": ["**/common/temp/*"],
    "storage_ssh_host": "storage.local",
    "storage_ssh_port": 8022,
    "storage_ssh_user": "user414",
    "storage_ssh_pwd": "...",
    "storage_ssh_path": "backup/dow_%weekday"
  },
  "pgdb": [
    {
      "servers": [
        {
          "ssh_host": "192.168.0.191",
          "pg_host": "192.168.0.191"
        },        {
          "ssh_host": "192.168.0.192",
          "pg_host": "192.168.0.192"
        }
      ],
      "ssh_port": 9022,
      "ssh_user": "postgres",
      "ssh_pwd": "...",
      "pg_port": 5420,
      "pg_replica_user": "era_replica",
      "pg_replica_pwd": "...",
      "storage_ssh_host": "storage.local",
      "storage_ssh_port": 8022,
      "storage_ssh_user": "user414",
      "storage_ssh_pwd": "...",
      "storage_ssh_path": "backup/dow_%weekday"
    },
    ...
  ]
}

cti_servers

array<object>

empty

Adapter Server Settings CTI.

The list contains objects with adapter server settings.
Each such adapter server is brought up to listen on the specified port on the specified interface.
Specifies the type of adapter supported by the platform.

Facility Keys:

  • key - adapter type from the list of supported adapters.

  • iface - listener local interface. Must belong to the server where the microservice instance is executing.

  • port - listener’s port.

Supported adapters:

  • 'Stc.SmartLogger.VoIPRecorder.CtiInterface.Grpc.SlCtiProtocolService' - GRPC server for delivery of CTI events to SmartLogger (produced by MDG).

Possible interface values:

  • '0.0.0.0' - to listen to all interfaces.

  • 'local' - listen '127.0.0.1'

  • 'default' - cThe main interface of the node on which the node is deployed.

  • any other value, as well as the absence of a key, is equivalent to 'default'.

Example of a value:

[
  {

    "key": "Stc.SmartLogger.VoIPRecorder.CtiInterface.Grpc.SlCtiProtocolService",
    "iface": "default",
    "port": 9794
  }
]

mnesia_storage_mode

str

disc "Mode of operation of object storage.

Possible options:

* disc - full mode

in RAM with saving to disk on all instances of the group.

  • ram - only in RAM (with support for distributed

transactional). If all instances of the group are rebooted/shut down simultaneously

Configuration example

The configuration is managed in an application available to master domain administrators. The application hides the full content of the configuration, but it is nevertheless accessible via the API.

The configuration contains a section to describe all instances of all roles. Parameters are defined for each specific role instance.

Example node
{
  "name": "mware2",
  "roletype": "mware",
  "iface": "eth0",

  "pg_controller": [
    {
      "key": "general",
     "servers": [
       {
        "ssh_host": "192.168.0.123",
        "ssh_port": 9022,
        "ssh_user": "postgres",
        "ssh_pwd": "123456",
        "pg_host": "192.168.0.123",
        "pg_port": 5420,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "654321",
        "pg_database": "postgres",
        "pg_ctl_command": "/usr/lib/postgresql/12/bin/pg_ctl",
        "pg_basebackup_command": "pg_basebackup",
        "pg_data_folder": "/var/lib/postgresql/12/instance01",
        "pg_backup_folder": "/var/lib/postgresql/12/backups/instance01",
        "pg_log_filepath": "/var/log/postgresql/postgresql-12-instance01.log",
        "pg_signal_filepath": "/var/lib/postgresql/12/instance01/master.signal"
       },
       {
        "ssh_host": "192.168.0.124",
        "ssh_port": 9022,
        "ssh_user": "postgres",
        "ssh_pwd": "123456",
        "pg_host": "192.168.0.124",
        "pg_port": 5420,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "654321",
        "pg_database": "postgres",
        "pg_ctl_command": "/usr/lib/postgresql/12/bin/pg_ctl",
        "pg_basebackup_command": "pg_basebackup",
        "pg_data_folder": "/var/lib/postgresql/12/instance01",
        "pg_backup_folder": "/var/lib/postgresql/12/backups/instance01",
        "pg_log_filepath": "/var/log/postgresql/postgresql-12-instance01.log",
        "pg_signal_filepath": "/var/lib/postgresql/12/instance01/master.signal"
       }
     ]
    }
  ],

  "backup_controller": {
    "hours_interval": [20, 3],
    "weekdays": [1,2,3,4,5,6,7],
    "storage_ssh_host": "storage.local",
    "storage_ssh_port": 8022,
    "storage_ssh_user": "user414",
    "storage_ssh_pwd": "pwd414",
    "storage_ssh_path": "backup/dow_%weekday",
    "sync_folder": {
      "send_mode": "rsync",
      "excluded_paths": ["**/common/fax/*", "**/some_temp_files/*"]
    },
    "pgdb": [
      {
        "servers": [
          {
            "ssh_host": "192.168.0.123",
            "pg_host": "192.168.0.123"
          },        {
            "ssh_host": "192.168.0.124",
            "pg_host": "192.168.0.124"
          }
        ],
        "ssh_port": 9022,
        "ssh_user": "postgres",
        "ssh_pwd": "123456",
        "pg_port": 5420,
        "pg_replica_user": "era_replica",
        "pg_replica_pwd": "654321"
      }
    ]
  },

  "group": 11120,
  "order": 10
}

See also