System monitoring

Requests

HTTP verb Endpoint Description

GET

/api/monitor/v1/system/state

System state

GET

/api/monitor/v1/system/node

System node state

GET

/api/monitor/v1/system/keys

List of analysis strategies

GET

/api/monitor/v1/system/sites

System sites

GET

/api/monitor/v1/system/nodes

System nodes

GET

/api/monitor/v1/system/roles

System roles

GET

/api/monitor/v1/system/domains

Domain servicing on sites

GET

/api/monitor/v1/system/version

Product version

For all requests:
Domain type: master

System state

Collects information about deviations from correct system behavior.

Some parameters can be set in the master domain settings.

Request

Table 1. Request parameters
Specification Description

Name: level
Type: int

Detail level. Applies only when keys are absent. Possible values:

  • 0 – Normal. (default)

  • 1 – Heuristic.

  • 2 – Files.

  • 3 – Full.

Name: keys
Type: str

System analysis strategies. List of keys separated by ',' or ';'. Key description

Name: timeout
Type: int

Timeout for waiting for system analysis completion in ms. By default calculated from collection strategy data.

Name: full
Type: bool

Return all keys; when activated, keys with empty values are filtered out. Default false.

Example request
GET /api/monitor/v1/system/state HTTP/1.1

Response

Example response
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": {
    {
      "unavailable_sites": [
        "slave2"
      ],
      "unavailable_servers": [
        {
          "site": "central",
          "servers": [
            "192.168.0.123",
            "192.168.0.127"
          ]
        }
      ],
      "inactive_nodes": [
        {
          "site": "central",
          "nodes": [
            "lwm5ws1@192.168.0.127",
            "lwmrpco32@192.168.0.123"
          ]
        }
      ],
      "cpu_usage": [
        {
          "site": "central",
          "servers": [
            {
              "server": "192.168.0.35",
              "busy_percent": 52.49,
              "rup_load_avg1": 5.76,
              "rup_load_avg5": 5.45,
              "rup_load_avg15": 6.54,
              "cpu_count": 4
            }
          ]
        }
      ]
    }
  }
}

What to do for each key — System state warnings and actions.

Table 2. Key description
Name Description

level = 0

unavailable_sites

Unavailable sites.

unavailable_servers

Unavailable servers.

inactive_nodes

Inactive nodes.

timeout_nodes

Timeout collecting information from the node.

current_time

Current time differs on servers by more than 1 second.

disk_space

Insufficient disk space on the server.
The size on the working disk is analyzed (location of directories with rotation data - logs, raw recordings, mixed recordings, object distributed DB data). If free disk space is less than 10 GB, a 'free_mb' message is issued indicating available space.
Additionally, all container-visible partitions are analyzed. On each, free space (10 GB threshold) and the percentage of free inodes of the total in the partition file system (40% threshold) are evaluated. Data is output for partitions whose initial total size is more than 1 GB and which are not 'tempfs'. Keys 'warn_free_disk_space', 'warn_free_disk_inodes'

mem_usage

Insufficient free RAM on the server.
Available memory is analyzed. A warning is output if free memory (including Cached and Buffers) is less than 200 MB, or if allocated memory (excluding Cached and Buffers) is more than 80%.

cpu_usage

CPU load exceeded.
avg1 and avg5 metrics are analyzed. A warning is output when either exceeds a value equal to the number of cores.

message_queue

Overflow of process message queues on nodes: more than mq_len_pid_threshold messages in a process queue or more than mq_len_total_threshold in total (master domain settings; default 500 and 1000).

config_diffs

Configuration differs on nodes.

basic_roles_not_found

Basic microservices are missing on the site.

solution_roles_not_found

Microservices required for full product operation are missing.

mgc_mg_connects

Number of 'mg' media gateways connected to the 'mgc' controller does not match the number configured.

db_connects

Domain center DB connection errors by domain.

site_role_reserve

Reserved, not implemented (result always empty).

assembly_diffs

Product version information (assembly.info) differs on nodes.

unavailable_dc_domains

Unavailable domains on nodes.

node_startup

Node started less than 60 seconds ago.

role_startup

Microservice instance started less than 60 seconds ago, or a startup error occurred.

role_state

States of non-running microservices on nodes.
Application-specific warnings are also reflected here:

* roles with virtual addresses (esg, sg, b2bua, etc., webserver) - no instance of the group (role and virtual address set) is active: addresses are not raised on any server; warning on waiting instances. A waiting instance next to an active one is a normal standby.
* 'mg' - information about recorded failures of connected media gateway processes: unscheduled restart, decommissioning based on errors and timeouts of requests.
* 'mware' - information about errors during DB and :SYNC directory backup creation (if the mode is configured).
* 'domaincenter' (active domain center) - values encrypted with a non-current installation key (general.crypto_kid; another key, and with a key set - also the previous method) in storage passwords, settings secrets, integration channel tokens, and configuration secrets (master domain): total count and domains with object types (first 10). Checked on domain object loading and modification and on configuration change (including activation of a configuration with another key); the warning does not expire, it is cleared by the next check or when the domain center role moves to another server. Values are re-encrypted with the current key when the domain is loaded by the domain center (domain center restart), configurations - on activation; values of another key require its file; without a configured key values are not changed.
* 'domaincenter' (active domain center) - DMS classes with secret properties not re-encrypted with the current installation key or after the secret property composition changed (ext.secrets_mark): total count and domains with classes (first 10). Checked on domain loading, class modification, and configuration change. Cleared after class re-encryption with the reencrypt_secrets command (more).
* 'dms' - storage password cannot be decrypted: the installation key file is absent on the server (storage classes do not start, requests to them return 503). Updated on class start retries, expires 3 minutes after the key file appears.
* server key service (environment, server central node; check at start, after 30, 60, and 120 seconds, then every 5 minutes) - active configuration key file:
absent in :SYNC/secure/keys, only the server local copy is used: other servers will not receive the key, the file must be returned to :SYNC/secure/keys;
error: not found on the server (:SYNC/secure/keys, local copies): key secrets are not decrypted (DB connection is retried until the file appears); the file must be restored from backup.

distributed_apps

Non-running Active-Passive microservices on the site.

unavailable_global_names

Unavailable global service names in nodes.

roleapps

Information about user applications, installation errors or installation process.

connects

Cross-site connections between nodes are present.

certificate

Errors and warnings for TLS/SSL certificates used in microservices. Information is reflected about certificates loaded to disk and specified in the 'ws', 'sg', and 'esg' microservice parameters in the configuration. Certificate validity periods loaded into domains are also tracked.

license

License warnings and errors (missing license, alarm_notify signal, empty domain count), according to the active domain center.

ext_svcscript

Project warnings issued by the result of a special service script execution. The script code is set by the system_state_options.svc_script_code parameter in the master domain settings.

Version comparison (version_*) is performed if there is more than one server; on a single-server system there can be no differences and data is not collected.

level = 1 (including level 0)

version_diffs

Total hash of binary files (*.beam) of platform applications, their dependencies, and Erlang differs on servers.

version_files_diffs

Platform binary files (*.beam) differ on servers (size or modification time).

level = 2 (including level 0)

version_diffs

Total hash of binary files (*.beam) of platform applications, their dependencies, and Erlang differs on servers.

version_all_files_diffs

Platform files differ on servers (size or modification time). Backup and build directories (backup, _bkp*, _build, .eunit, .git, node_modules) are not counted.

System node state

Request

Table 3. Request parameters
Specification Description

Name: node
Type: str

Node for which data should be collected. Possible values:

  • all – all nodes.

  • <node_name>@<ip_addr> – node name.

Name: level
Type: int

Detail level. Applies only when keys are absent. Possible values:

  • 0 – Normal. (default)

  • 1 – Heuristic.

  • 2 – Files.

  • 3 – Full.

Name: keys
Type: str

Node analysis strategies. List of keys separated by ',' or ';'. Key description

Name: roles
Type: str

Return only the specified microservices in the node(s). List of microservices separated by ',' or ';'.

Example request
GET /api/monitor/v1/system/node?node=b2bua1@192.168.0.35 HTTP/1.1

Response

Example response
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    {
      "site": "central",
      "nodes": [
        {
          "node": "b2bua1@192.168.0.35",
          "available": true,
          "parentnode": "testserv@192.168.0.35",
          "start_utc": "2019-10-21T08:02:23.38Z",
          "now_utc": "2019-10-24T13:05:37.01Z",
          "is_open": false,
          "config_hash": 104521630,
          "globals_hash": false,
          "eaddr": "192.168.0.35",
          "ifaces": [
            "192.168.0.35",
            "172.17.0.1"
          ],
          "cpu": {
            "busy": 52.49346178074708,
            "nonbusy": 47.50653821925292,
            "avg1": 9.37109375,
            "avg5": 7.609375,
            "avg15": 6.12890625,
            "cpu_count": 4
          },
          "mem": {
            "total": 33599832064,
            "allocated": 32505409536,
            "free": [
              "!tuple!",
              "free_memory",
              1071202304
            ],
            "cached": [
              "!tuple!",
              "cached_memory",
              10312015872
            ],
            "buffers": [
              "!tuple!",
              "buffered_memory",
              1033371648
            ]
          },
          "disk_space": 31896128,
          "roleapps": false,
          "versions": {
            "assembly_date": "2018-05-31T07:26:28.380Z",
            "product": "2019.4.16-nightly",
            "product_name": "{product_name}",
            "era": "2021.4.16",
            "rtx": "v3.2.0",
            "default_solution": "{product_name}",
            "fe-autoprovision": "v.0.0.1",
            "fe-calls-statistic": "v.0.2.0",
            "fe-data-manager": "v.0.4.0",
            "fe-monitor": "v.0.2.0",
            "fe-reporter": "v.0.1.1",
            "fe-root": "v.0.0.2",
            "fe-script-editor": "v.0.3.0",
            "fe-selector": "v.0.2.1"
          },
          "connected": [
            "testserv@192.168.0.35",
            "statestore2@192.168.0.35",
            "callstorage1@192.168.0.35"
          ],
          "roles": [
            {
              "role": "environment",
              "is_running": true,
              "boot_state": {
                "result": "wait_for_boot",
                "reason": "undefined",
                "utc": "undefined"
              },
              "app_type": "basic",
              "state": "ok"
            },
            {
              "role": "trace",
              "is_running": true,
              "boot_state": {
                "result": "wait_for_boot",
                "reason": "undefined",
                "utc": "undefined"
              },
              "app_type": "basic",
              "state": "ok"
            },
            {
              "role": "b2bua",
              "is_running": true,
              "boot_state": {
                "result": "ok",
                "reason": "OK",
                "utc": "2019-10-21T08:02:25.68Z"
              },
              "app_type": "sip",
              "state": "ok"
            }
          ]
        }
      ]
    }
  ]
}
Table 4. Key description
Name Description

level=0

node

Full node name.

available

Node availability (whether the method call from the statistics collector node succeeds).

is_open

Whether the node is open.

start_utc

Node start time in UTC.

now_utc

Current time on the node in UTC.

config_hash

Configuration hash on the node.

connected

List of connected nodes.

cpu

CPU load metrics on the server with the node.

disk_space

Free disk space on the server in KB.

eaddr

Address on which the node is raised.

globals_hash

Hash of global names registered on the node.

license

License warnings and errors (active domain center node only), otherwise ok.

ifaces

Interfaces on the server.

mem

RAM load metrics on the server with the node.

parentnode

Parent node (ServerShell).

pidqueue

Processes with an overflowing message queue and total queue length when thresholds are exceeded (see message_queue).

roleapps

Information about user applications, or false if not serviced on the node.

roles

List of microservices serviced on the node.

timezone

Container time zone (set by the master domain setting) and its application status (ok, {ok,need_restart}, error).

versions

Product version on the server.

level=1 (including level=0)

sourcecode_hash

Total hash of binary files (*.beam) of platform applications, their dependencies, and Erlang on the server.

level=2 (including level=1)

sourcecode_data_beam

List of platform binary files (*.beam) on the server: path, size, modification time.

level=3 (including level=2)

sourcecode_data_all

List of platform files on the server: path, size, modification time (excluding backup and build directories).

List of analysis strategies

Request

Table 5. Request parameters
Specification Description

Name: mode
Type: str

Analysis strategy type. Possible values:

  • state – system state strategies.

  • node – node state strategies.

Name: full
Type: bool

Display system strategies. Default false.

Example request
GET /api/monitor/v1/system/keys?mode=node HTTP/1.1

Response

Example response
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    "available",
    "config_hash",
    "connected",
    "cpu",
    "disk_space",
    "eaddr",
    "globals_hash",
    "ifaces",
    "is_open",
    "mem",
    "node",
    "now_utc",
    "parentnode",
    "roleapps",
    "roles",
    "sourcecode_data_all",
    "sourcecode_hash",
    "start_utc",
    "versions"
  ]
}

System sites

Request

Example request
GET /api/monitor/v1/system/sites HTTP/1.1

Response

Example response
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    "central",
    "slave2"
  ]
}

System nodes

Distribution of nodes across servers on sites.

Request

Example request
GET /api/monitor/v1/system/nodes HTTP/1.1

Response

Example response
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    {
      "site": "central",
      "servers": [
        {
          "server": "192.168.0.35",
          "nodes": [
            "b2bua1@192.168.0.35",
            "bgmg1@192.168.0.35"
          ]
        },
        {
          "server": "192.168.0.123",
          "nodes": [
            "lwmic2@192.168.0.123",
            "lwmrpco32@192.168.0.123",
            "lwmws1lwmgate32@192.168.0.123"
          ]
        }
      ]
    }
  ]
}

System roles

System roles (including service ones) distributed across nodes with the site indicated.

Request

Table 6. Request parameters
Specification Description

Name: opts
Type: bool

Display microservice parameters. Default false.

Example request
GET /api/monitor/v1/system/roles HTTP/1.1

Response

Example response
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    {
      "site": "central",
      "nodes": [
        {
          "node": "b2bua1@192.168.0.35",
          "roles": ["b2bua"]
        },
        {
          "node": "bgmg1@192.168.0.35",
          "roles": ["mg"]
        },
        {
          "node": "callstorage1@192.168.0.35",
          "roles": ["mnesia","callstore"]
        }
      ]
    }
  ]
}

Domain servicing on sites

Request

Example request
GET /api/monitor/v1/system/domains HTTP/1.1

Response

Example response
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    {
      "domain": "ceceron.ru",
      "sites": ["central"]
    },
    {
      "domain": "ak.ceceron.ru",
      "sites": ["central"]
    },
    {
      "domain": "mmak.ceceron.ru",
      "sites": ["central"]
    },
    {
      "domain": "test.ceceron.ru",
      "sites": ["central","slave2"]
    },
    {
      "domain": "w.ceceron.ru",
      "sites": ["central"]
    }
  ]
}

Product version

Request

Example request
GET /api/monitor/v1/system/version HTTP/1.1

Response

Example response
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": {
    "assembly_date": "2019-11-26T19:51:43.332Z",
    "product": "{product_name}",
    "projects": [
      {
        "branch": "develop",
        "commitid": "ba102bee006b54cac8a3d4ef591f88fafe0be574",
        "name": "era",
        "version": "2020.11.26-nightly-build.105"
      },
      {
        "commitid": "ecb12af3336dc15c594351cf7c9e99c95d715824",
        "name": "era_assets",
        "tag": "v1.8.0",
        "version": "1.8.0"
      },
      {
        "commitid": "3d39ae33036df4ee5f8d3f1ab43bcddce477a7fe",
        "name": "rtx_assets",
        "tag": "v3.9.0",
        "version": "3.9.0"
      },
      {
        "branch": "develop",
        "commitid": "eade2bb358a0e3d8b78629f07f8e28c1e761f018",
        "name": "rtx",
        "version": "2019.11.26-nightly-build.105"
      },
      {
        "branch": "master",
        "commitid": "69ba462b699d18bda820b298c0f0e0fdb14e8ad0",
        "name": "r_assets_www",
        "version": "2019.11.26-nightly-build.105"
      },
      {
        "branch": "develop",
        "commitid": "59a61f6affa592aaa19795d2ae3e4e57ccba479b",
        "name": "app-configuration",
        "version": "2019.11.26-nightly-build.105"
      }
    ],
    "version": "2019.11.26-nightly-build.105"
  }
}
Table 7. Key description
Name Description

product

Product name

version

Product version.

assembly_date

Product assembly date.

projects

Projects included in the product.

projects.name

Project name.

projects.version

Project version.

projects.branch

Repository branch from which the product was built.

projects.tag

Repository tag from which the product was built.

projects.commitid

Repository commit identifier from which the product was built.