System monitoring

Requests

HTTP verb Endpoint Description

GET

/api/monitor/v1/system/state

System status

GET

/api/monitor/v1/system/node

System node status

GET

/api/monitor/v1/system/keys

List of analysis strategies

GET

/api/monitor/v1/system/sites

System sites

GET

/api/monitor/v1/system/nodes

Nodes of the system

GET

/api/monitor/v1/system/roles

System Roles

GET

/api/monitor/v1/system/domains

Service of domains on sites

GET

/api/monitor/v1/system/version

Product version

For all requests:
Domain type: master

System status

Collecting information on deviations from correct system behavior.

Some parameters can be set in settings of the master domain.

Request

Table 1. Request parameters
Specification Description

Name: level
Type: int

Level of detail. Applies only if there are no keys. Possible values:

  • 0 – Normal. (Default)

  • 1 – Heuristic.

  • 2 – Files.

  • 3 – Full.

Name: keys
Type: str

Strategies for analyzing the system. A list of keys separated by ',' or ';'. Key Description

Name: timeout
Type: int

Timeout to wait for system analysis in ms. By default, it is calculated using data from the collection strategies.

Name: full
Type: bool

Return all keys, keys with an empty value will be filtered out on activation. Default false.
Request example
GET /api/monitor/v1/system/state HTTP/1.1

Response

Response example
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": {
    {
      "unavailable_sites": [
        "slave2"
      ],
      "unavailable_servers": [
        {
          "site": "central",
          "servers": [
            "192.168.0.123",
            "192.168.0.127"
          ]
        }
      ],
      "inactive_nodes": [
        {
          "site": "central",
          "nodes": [
            "lwm5ws1@192.168.0.127",
            "lwmrpco32@192.168.0.123"
          ]
        }
      ],
      "cpu_usage": [
        {
          "site": "central",
          "servers": [
            {
              "server": "192.168.0.35",
              "busy_percent": 52.49,
              "rup_load_avg1": 5.76,
              "rup_load_avg5": 5.45,
              "rup_load_avg15": 6.54,
              "cpu_count": 4
            }
          ]
        }
      ]
    }
  }
}
Table 2. Key Description
Name Description

level = 0

unavailable_sites

Unavailable sites.

unavailable_servers

Unavailable servers.

inactive_nodes

Inactive nodes.

timeout_nodes

Timeout for collecting information from node.

current_time

The current time differs on different servers by more than 1 second.

disk_space

Not enough disk space on the server.
The size on the working disk is analyzed (the location of directories with rotational data - logs, raw records, mixed records, object distributed database data). If the free disk space is less than 10 GB, the 'free_mb' message with the available space is displayed.
Additionally, all partitions available to the container are analyzed. Free space on each of them is estimated (10 GB threshold) and the percentage of free inodes from the total number in the partition’s file system (40% threshold). Partitions whose original full size is greater than 1 GB and which are not 'tempfs' are output. Keys 'warn_free_disk_space', 'warn_free_disk_inodes'

mem_usage

Insufficient free RAM on the server.
The amount of available memory is analyzed. A warning is displayed if the amount of free memory (including Cached and Buffers) is less than 200 MB, or if the amount of allocated memory (excluding Cached and Buffers) is more than 200 MB. 80%.

cpu_usage

Overloading CPU.
The avg1 and avg5 indicators are analyzed. A warning is displayed if any of them exceeds the value equal to the number of cores.

config_diffs

The configuration is different on the nodes.

basic_roles_not_found

Basic microservices are missing from the site.

fsync_roles_not_found

Microservice instance not found fsync.

mgc_mg_connects

The number of 'mg' media gateways connected to the 'mgc' controller does not match the number configured in the configuration.

assembly_diffs

Product version information (assembly.info) on nodes is different.

global_sync

Global names on websites are different.

unavailable_dc_domains

Unavailable domains on nodes.

node_startup

Noda was launched less than 60 seconds ago.

role_startup

The microservice instance was started less than 60 seconds ago, or a startup error occurred.

role_state

States of non-started microservices on nodes.
Specific warnings from applications are also reflected here:

* 'mg' - information on recorded failures in the operation of connected media gateway processes: unscheduled restart, withdrawal from service based on errors and timeouts as a result of requests.
* 'mware' - information about errors in the process of creating backups of the database and :SYNC catalog (in case of mode setting in the configuration).

distributed_apps

Active-Passive microservices not running on the site.

unavailable_global_names

Unavailable global service names in nodes.

roleapps

Information about custom applications, installation errors or the installation process.

connects

Cross-site connectivity between nodes is present.

certificate

Errors and warnings on TLS/SSL certificates used in microservices. Information about certificates loaded on disk and specified in the 'ws', 'sg' and 'esg' microservices parameters in the configuration is reported. It also tracks the validity time of certificates downloaded to domains.

ext_svcscript

Project warnings issued as a result of running a special service script. The script code is specified by the system_state_options.svc_script_code parameter in settings of the domain master.

level = 1 (including level 0)

version_diffs

The total hash of the binary files (*.beam) of the platform applications their dependencies and erlang is different on the servers.

version_files_diffs

The total hash of platform binary files (*.beam) differs on servers.

level = 2 (including level 0)

version_diffs

The total hash of the binary files (*.beam) of the platform applications their dependencies and erlang is different on the servers.

version_all_files_diffs

The total hash of all platform files is different on the servers.

System node status

Request

Table 3. Request parameters
Specification Description

Name: node
Type: str

The node for which data should be collected. Possible values:

  • all – all nodes.

  • <node_name>@<ip_addr> – node name.

Name: level
Type: int

Level of detail. Applies only if there are no keys. Possible values:

  • 0 – Normal. (Default)

  • 1 – Heuristic.

  • 2 – Files.

  • 3 – Full.

Name: keys
Type: str

Strategies for analyzing nodes. List of keys separated by ',' or ';'. Key Description

Name: roles
Type: str

Return only the specified microservices in the node(s). List of microservices with ',' separator or ';'.
Request example
GET /api/monitor/v1/system/node?node=b2bua1@192.168.0.35 HTTP/1.1

Response

Response example
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    {
      "site": "central",
      "nodes": [
        {
          "node": "b2bua1@192.168.0.35",
          "available": true,
          "parentnode": "testserv@192.168.0.35",
          "start_utc": "2019-10-21T08:02:23.38Z",
          "now_utc": "2019-10-24T13:05:37.01Z",
          "is_open": false,
          "config_hash": 104521630,
          "globals_hash": false,
          "eaddr": "192.168.0.35",
          "ifaces": [
            "192.168.0.35",
            "172.17.0.1"
          ],
          "cpu": {
            "busy": 52.49346178074708,
            "nonbusy": 47.50653821925292,
            "avg1": 9.37109375,
            "avg5": 7.609375,
            "avg15": 6.12890625,
            "cpu_count": 4
          },
          "mem": {
            "total": 33599832064,
            "allocated": 32505409536,
            "free": [
              "!tuple!",
              "free_memory",
              1071202304
            ],
            "cached": [
              "!tuple!",
              "cached_memory",
              10312015872
            ],
            "buffers": [
              "!tuple!",
              "buffered_memory",
              1033371648
            ]
          },
          "disk_space": 31896128,
          "roleapps": false,
          "versions": {
            "assembly_date": "2018-05-31T07:26:28.380Z",
            "product": "2019.4.16-nightly",
            "product_name": "{product_name}",
            "era": "2021.4.16",
            "rtx": "v3.2.0",
            "default_solution": "{product_name}",
            "fe-autoprovision": "v.0.0.1",
            "fe-calls-statistic": "v.0.2.0",
            "fe-data-manager": "v.0.4.0",
            "fe-monitor": "v.0.2.0",
            "fe-reporter": "v.0.1.1",
            "fe-root": "v.0.0.2",
            "fe-script-editor": "v.0.3.0",
            "fe-selector": "v.0.2.1"
          },
          "connected": [
            "testserv@192.168.0.35",
            "statestore2@192.168.0.35",
            "callstorage1@192.168.0.35"
          ],
          "roles": [
            {
              "role": "environment",
              "is_running": true,
              "boot_state": {
                "result": "wait_for_boot",
                "reason": "undefined",
                "utc": "undefined"
              },
              "app_type": "basic",
              "state": "ok"
            },
            {
              "role": "trace",
              "is_running": true,
              "boot_state": {
                "result": "wait_for_boot",
                "reason": "undefined",
                "utc": "undefined"
              },
              "app_type": "basic",
              "state": "ok"
            },
            {
              "role": "b2bua",
              "is_running": true,
              "boot_state": {
                "result": "ok",
                "reason": "OK",
                "utc": "2019-10-21T08:02:25.68Z"
              },
              "app_type": "sip",
              "state": "ok"
            }
          ]
        }
      ]
    }
  ]
}
Table 4. Key Description
Name Description

level=0

node

The full name of the node.

available

Node accessibility (whether the method call passes from the statistical builder node).

is_open

Is the node open.

start_utc

Node start time in UTC.

now_utc

Current time on the node in UTC.

config_hash

Hash node configurations.

connected

List of connected nodes.

cpu

CPU utilization indicators on the server with node.

disk_space

Free space on the server disk in KB.

eaddr

The address on which the node is raised.

globals_hash

Hash of global names registered on node.

ifaces

Interfaces on the server.

mem

Indicators of RAM utilization on the server with node.

parentnode

Parent node (ServerShell).

roleapps

Information about user applications or false in case of no service on the node.

roles

List of microservices maintained on the node.

versions

Product version on the server.

level=1 (including level=0)

sourcecode_hash

Total hash of binary files (*.beam) of applications of the platform platform their dependencies and erlang on the server.

level=2 (including level=1)

sourcecode_data_beam

Total hash of binary files (*.beam) of the platform on the server.

level=3 (including level=2)

sourcecode_data_all

Total hash of all platform files on the server.

List of analysis strategies

Request

Table 5. Request parameters
Specification Description

Name: mode
Type: str

Type of analysis strategies. Possible values:

  • state – system state strategies.

  • node – node state strategies.

Name: full
Type: bool

Display system strategies. Default false.
Request example
GET /api/monitor/v1/system/keys?mode=node HTTP/1.1

Response

Response example
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    "available",
    "config_hash",
    "connected",
    "cpu",
    "disk_space",
    "eaddr",
    "globals_hash",
    "ifaces",
    "is_open",
    "mem",
    "node",
    "now_utc",
    "parentnode",
    "roleapps",
    "roles",
    "sourcecode_data_all",
    "sourcecode_hash",
    "start_utc",
    "versions"
  ]
}

System sites

Request

Request example
GET /api/monitor/v1/system/sites HTTP/1.1

Response

Response example
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    "central",
    "slave2"
  ]
}

Nodes of the system

Distribution of nodes to servers on sites.

Request

Request example
GET /api/monitor/v1/system/nodes HTTP/1.1

Response

Response example
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    {
      "site": "central",
      "servers": [
        {
          "server": "192.168.0.35",
          "nodes": [
            "b2bua1@192.168.0.35",
            "bgmg1@192.168.0.35"
          ]
        },
        {
          "server": "192.168.0.123",
          "nodes": [
            "lwmic2@192.168.0.123",
            "lwmrpco32@192.168.0.123",
            "lwmws1lwmgate32@192.168.0.123"
          ]
        }
      ]
    }
  ]
}

System Roles

Roles (including service roles) of the system distributed by nodes with site indication.

Request

Table 6. Request parameters
Specification Description

Name: opts
Type: bool

Display microservices parameters. Default false.
Request example
GET /api/monitor/v1/system/roles HTTP/1.1

Response

Response example
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    {
      "site": "central",
      "nodes": [
        {
          "node": "b2bua1@192.168.0.35",
          "roles": ["b2bua"]
        },
        {
          "node": "bgmg1@192.168.0.35",
          "roles": ["mg"]
        },
        {
          "node": "callstorage1@192.168.0.35",
          "roles": ["mnesia","callstore"]
        }
      ]
    }
  ]
}

Service of domains on sites

Request

Request example
GET /api/monitor/v1/system/domains HTTP/1.1

Response

Response example
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": [
    {
      "domain": "ceceron.ru",
      "sites": ["central"]
    },
    {
      "domain": "ak.ceceron.ru",
      "sites": ["central"]
    },
    {
      "domain": "mmak.ceceron.ru",
      "sites": ["central"]
    },
    {
      "domain": "test.ceceron.ru",
      "sites": ["central","slave2"]
    },
    {
      "domain": "w.ceceron.ru",
      "sites": ["central"]
    }
  ]
}

Product version

Request

Request example
GET /api/monitor/v1/system/version HTTP/1.1

Response

Response example
HTTP/1.1 200 OK

{
  "resultcode": 0,
  "resultmsg": "OK",
  "data": {
    "assembly_date": "2019-11-26T19:51:43.332Z",
    "product": "{product_name}",
    "projects": [
      {
        "branch": "develop",
        "commitid": "ba102bee006b54cac8a3d4ef591f88fafe0be574",
        "name": "era",
        "version": "2020.11.26-nightly-build.105"
      },
      {
        "commitid": "ecb12af3336dc15c594351cf7c9e99c95d715824",
        "name": "era_assets",
        "tag": "v1.8.0",
        "version": "1.8.0"
      },
      {
        "commitid": "3d39ae33036df4ee5f8d3f1ab43bcddce477a7fe",
        "name": "rtx_assets",
        "tag": "v3.9.0",
        "version": "3.9.0"
      },
      {
        "branch": "develop",
        "commitid": "eade2bb358a0e3d8b78629f07f8e28c1e761f018",
        "name": "rtx",
        "version": "2019.11.26-nightly-build.105"
      },
      {
        "branch": "master",
        "commitid": "69ba462b699d18bda820b298c0f0e0fdb14e8ad0",
        "name": "r_assets_www",
        "version": "2019.11.26-nightly-build.105"
      },
      {
        "branch": "develop",
        "commitid": "59a61f6affa592aaa19795d2ae3e4e57ccba479b",
        "name": "app-configuration",
        "version": "2019.11.26-nightly-build.105"
      }
    ],
    "version": "2019.11.26-nightly-build.105"
  }
}
Table 7. Key Description
Name Description

product

Product Name

version

Product version.

assembly_date

Product build date.

projects

Projects included in the product.

projects.name

Project Name.

projects.version

Project version.

projects.branch

A branch in the repository from which the product is built.

projects.tag

Tag in the repository from which the product was built.

projects.commitid

ID of the commit in the repository from which the product was built.
  • Personalization
  • Increase font size
  • Decrease font size