System monitoring
Requests
| HTTP verb | Endpoint | Description |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For all requests:
Domain type: master
System state
Collects information about deviations from correct system behavior.
Some parameters can be set in the master domain settings.
Request
| Specification | Description |
|---|---|
|
Detail level. Applies only when keys are absent. Possible values:
|
|
System analysis strategies. List of keys separated by ',' or ';'. Key description |
|
Timeout for waiting for system analysis completion in ms. By default calculated from collection strategy data. |
|
Return all keys; when activated, keys with empty values are filtered out. Default false. |
GET /api/monitor/v1/system/state HTTP/1.1
Response
HTTP/1.1 200 OK
{
"resultcode": 0,
"resultmsg": "OK",
"data": {
{
"unavailable_sites": [
"slave2"
],
"unavailable_servers": [
{
"site": "central",
"servers": [
"192.168.0.123",
"192.168.0.127"
]
}
],
"inactive_nodes": [
{
"site": "central",
"nodes": [
"lwm5ws1@192.168.0.127",
"lwmrpco32@192.168.0.123"
]
}
],
"cpu_usage": [
{
"site": "central",
"servers": [
{
"server": "192.168.0.35",
"busy_percent": 52.49,
"rup_load_avg1": 5.76,
"rup_load_avg5": 5.45,
"rup_load_avg15": 6.54,
"cpu_count": 4
}
]
}
]
}
}
}
What to do for each key — System state warnings and actions.
| Name | Description |
|---|---|
|
|
|
Unavailable sites. |
|
Unavailable servers. |
|
Inactive nodes. |
|
Timeout collecting information from the node. |
|
Current time differs on servers by more than 1 second. |
|
Insufficient disk space on the server. |
|
Insufficient free RAM on the server. |
|
CPU load exceeded. |
|
Overflow of process message queues on nodes: more than |
|
Configuration differs on nodes. |
|
Basic microservices are missing on the site. |
|
Microservices required for full product operation are missing. |
|
Number of 'mg' media gateways connected to the 'mgc' controller does not match the number configured. |
|
Domain center DB connection errors by domain. |
|
Reserved, not implemented (result always empty). |
|
Product version information (assembly.info) differs on nodes. |
|
Unavailable domains on nodes. |
|
Node started less than 60 seconds ago. |
|
Microservice instance started less than 60 seconds ago, or a startup error occurred. |
|
States of non-running microservices on nodes. |
|
Non-running Active-Passive microservices on the site. |
|
Unavailable global service names in nodes. |
|
Information about user applications, installation errors or installation process. |
|
Cross-site connections between nodes are present. |
|
Errors and warnings for TLS/SSL certificates used in microservices. Information is reflected about certificates loaded to disk and specified in the 'ws', 'sg', and 'esg' microservice parameters in the configuration. Certificate validity periods loaded into domains are also tracked. |
|
License warnings and errors (missing license, |
|
Project warnings issued by the result of a special service script execution. The script code is set by the |
|
|
|
|
|
Total hash of binary files (*.beam) of platform applications, their dependencies, and Erlang differs on servers. |
|
Platform binary files (*.beam) differ on servers (size or modification time). |
|
|
|
Total hash of binary files (*.beam) of platform applications, their dependencies, and Erlang differs on servers. |
|
Platform files differ on servers (size or modification time). Backup and build directories (backup, _bkp*, _build, .eunit, .git, node_modules) are not counted. |
System node state
Request
| Specification | Description |
|---|---|
|
Node for which data should be collected. Possible values:
|
|
Detail level. Applies only when keys are absent. Possible values:
|
|
Node analysis strategies. List of keys separated by ',' or ';'. Key description |
|
Return only the specified microservices in the node(s). List of microservices separated by ',' or ';'. |
GET /api/monitor/v1/system/node?node=b2bua1@192.168.0.35 HTTP/1.1
Response
HTTP/1.1 200 OK
{
"resultcode": 0,
"resultmsg": "OK",
"data": [
{
"site": "central",
"nodes": [
{
"node": "b2bua1@192.168.0.35",
"available": true,
"parentnode": "testserv@192.168.0.35",
"start_utc": "2019-10-21T08:02:23.38Z",
"now_utc": "2019-10-24T13:05:37.01Z",
"is_open": false,
"config_hash": 104521630,
"globals_hash": false,
"eaddr": "192.168.0.35",
"ifaces": [
"192.168.0.35",
"172.17.0.1"
],
"cpu": {
"busy": 52.49346178074708,
"nonbusy": 47.50653821925292,
"avg1": 9.37109375,
"avg5": 7.609375,
"avg15": 6.12890625,
"cpu_count": 4
},
"mem": {
"total": 33599832064,
"allocated": 32505409536,
"free": [
"!tuple!",
"free_memory",
1071202304
],
"cached": [
"!tuple!",
"cached_memory",
10312015872
],
"buffers": [
"!tuple!",
"buffered_memory",
1033371648
]
},
"disk_space": 31896128,
"roleapps": false,
"versions": {
"assembly_date": "2018-05-31T07:26:28.380Z",
"product": "2019.4.16-nightly",
"product_name": "{product_name}",
"era": "2021.4.16",
"rtx": "v3.2.0",
"default_solution": "{product_name}",
"fe-autoprovision": "v.0.0.1",
"fe-calls-statistic": "v.0.2.0",
"fe-data-manager": "v.0.4.0",
"fe-monitor": "v.0.2.0",
"fe-reporter": "v.0.1.1",
"fe-root": "v.0.0.2",
"fe-script-editor": "v.0.3.0",
"fe-selector": "v.0.2.1"
},
"connected": [
"testserv@192.168.0.35",
"statestore2@192.168.0.35",
"callstorage1@192.168.0.35"
],
"roles": [
{
"role": "environment",
"is_running": true,
"boot_state": {
"result": "wait_for_boot",
"reason": "undefined",
"utc": "undefined"
},
"app_type": "basic",
"state": "ok"
},
{
"role": "trace",
"is_running": true,
"boot_state": {
"result": "wait_for_boot",
"reason": "undefined",
"utc": "undefined"
},
"app_type": "basic",
"state": "ok"
},
{
"role": "b2bua",
"is_running": true,
"boot_state": {
"result": "ok",
"reason": "OK",
"utc": "2019-10-21T08:02:25.68Z"
},
"app_type": "sip",
"state": "ok"
}
]
}
]
}
]
}
| Name | Description |
|---|---|
|
|
|
Full node name. |
|
Node availability (whether the method call from the statistics collector node succeeds). |
|
Whether the node is open. |
|
Node start time in UTC. |
|
Current time on the node in UTC. |
|
Configuration hash on the node. |
|
List of connected nodes. |
|
CPU load metrics on the server with the node. |
|
Free disk space on the server in KB. |
|
Address on which the node is raised. |
|
Hash of global names registered on the node. |
|
License warnings and errors (active domain center node only), otherwise |
|
Interfaces on the server. |
|
RAM load metrics on the server with the node. |
|
Parent node (ServerShell). |
|
Processes with an overflowing message queue and total queue length when thresholds are exceeded (see |
|
Information about user applications, or false if not serviced on the node. |
|
List of microservices serviced on the node. |
|
Container time zone (set by the master domain setting) and its application status ( |
|
Product version on the server. |
|
|
|
Total hash of binary files (*.beam) of platform applications, their dependencies, and Erlang on the server. |
|
|
|
List of platform binary files (*.beam) on the server: path, size, modification time. |
|
|
|
List of platform files on the server: path, size, modification time (excluding backup and build directories). |
List of analysis strategies
Request
| Specification | Description |
|---|---|
|
Analysis strategy type. Possible values:
|
|
Display system strategies. Default false. |
GET /api/monitor/v1/system/keys?mode=node HTTP/1.1
Response
HTTP/1.1 200 OK
{
"resultcode": 0,
"resultmsg": "OK",
"data": [
"available",
"config_hash",
"connected",
"cpu",
"disk_space",
"eaddr",
"globals_hash",
"ifaces",
"is_open",
"mem",
"node",
"now_utc",
"parentnode",
"roleapps",
"roles",
"sourcecode_data_all",
"sourcecode_hash",
"start_utc",
"versions"
]
}
System nodes
Distribution of nodes across servers on sites.
Response
HTTP/1.1 200 OK
{
"resultcode": 0,
"resultmsg": "OK",
"data": [
{
"site": "central",
"servers": [
{
"server": "192.168.0.35",
"nodes": [
"b2bua1@192.168.0.35",
"bgmg1@192.168.0.35"
]
},
{
"server": "192.168.0.123",
"nodes": [
"lwmic2@192.168.0.123",
"lwmrpco32@192.168.0.123",
"lwmws1lwmgate32@192.168.0.123"
]
}
]
}
]
}
System roles
System roles (including service ones) distributed across nodes with the site indicated.
Request
| Specification | Description |
|---|---|
|
Display microservice parameters. Default false. |
GET /api/monitor/v1/system/roles HTTP/1.1
Response
HTTP/1.1 200 OK
{
"resultcode": 0,
"resultmsg": "OK",
"data": [
{
"site": "central",
"nodes": [
{
"node": "b2bua1@192.168.0.35",
"roles": ["b2bua"]
},
{
"node": "bgmg1@192.168.0.35",
"roles": ["mg"]
},
{
"node": "callstorage1@192.168.0.35",
"roles": ["mnesia","callstore"]
}
]
}
]
}
Domain servicing on sites
Response
HTTP/1.1 200 OK
{
"resultcode": 0,
"resultmsg": "OK",
"data": [
{
"domain": "ceceron.ru",
"sites": ["central"]
},
{
"domain": "ak.ceceron.ru",
"sites": ["central"]
},
{
"domain": "mmak.ceceron.ru",
"sites": ["central"]
},
{
"domain": "test.ceceron.ru",
"sites": ["central","slave2"]
},
{
"domain": "w.ceceron.ru",
"sites": ["central"]
}
]
}
Product version
Response
HTTP/1.1 200 OK
{
"resultcode": 0,
"resultmsg": "OK",
"data": {
"assembly_date": "2019-11-26T19:51:43.332Z",
"product": "{product_name}",
"projects": [
{
"branch": "develop",
"commitid": "ba102bee006b54cac8a3d4ef591f88fafe0be574",
"name": "era",
"version": "2020.11.26-nightly-build.105"
},
{
"commitid": "ecb12af3336dc15c594351cf7c9e99c95d715824",
"name": "era_assets",
"tag": "v1.8.0",
"version": "1.8.0"
},
{
"commitid": "3d39ae33036df4ee5f8d3f1ab43bcddce477a7fe",
"name": "rtx_assets",
"tag": "v3.9.0",
"version": "3.9.0"
},
{
"branch": "develop",
"commitid": "eade2bb358a0e3d8b78629f07f8e28c1e761f018",
"name": "rtx",
"version": "2019.11.26-nightly-build.105"
},
{
"branch": "master",
"commitid": "69ba462b699d18bda820b298c0f0e0fdb14e8ad0",
"name": "r_assets_www",
"version": "2019.11.26-nightly-build.105"
},
{
"branch": "develop",
"commitid": "59a61f6affa592aaa19795d2ae3e4e57ccba479b",
"name": "app-configuration",
"version": "2019.11.26-nightly-build.105"
}
],
"version": "2019.11.26-nightly-build.105"
}
}
| Name | Description |
|---|---|
|
Product name |
|
Product version. |
|
Product assembly date. |
|
Projects included in the product. |
|
Project name. |
|
Project version. |
|
Repository branch from which the product was built. |
|
Repository tag from which the product was built. |
|
Repository commit identifier from which the product was built. |