System state warnings and actions
Overview
Analysis of the system state report keys: what a warning means, typical causes, and what to do. The format and full key list are in the API description, thresholds and collection parameters — in the master domain settings.
An empty report means the system is normal. Immediately after restarting nodes and microservices, informational node_startup
and role_startup are acceptable; the rest requires administrator attention.
The report can be analyzed by the master domain AI agent: the system_state tool and the system_check prompt
of the MCP server (agent connection).
Availability
unavailable_sites— sites are unavailable from the collection site.-
Causes: network between sites, the site central node is stopped. Check connectivity between site servers and container operation. While a site is unavailable, domains served only by it do not work.
unavailable_servers— servers are unavailable.-
Causes: the server or container is stopped, network. Server nodes do not work; check the server, container (
docker ps, container log), and network. inactive_nodes— nodes from the configuration are not started or not connected.-
Causes: startup error, insufficient resources, incorrect node configuration. Look at node logs (
boot,error_logger); the node is restarted by the server central node. timeout_nodes— the node did not respond to the collector in time.-
Causes: overload or hang. Check
cpu_usage,message_queue, and node state (/system/node). node_startup— the node started less than 60 seconds ago.-
After a planned restart this is normal. If it repeats — the node is restarting: look at node crash logs.
role_startup— the microservice instance started less than 60 seconds ago or did not start.-
If it repeats or there is a startup error — microservice logs on the node.
distributed_apps— Active-Passive microservices are not running on the site.-
No active instance: check the microservice nodes (
inactive_nodes) and their logs. unavailable_global_names— global service names are unavailable on nodes.-
Services are not registered or nodes are disconnected. Check
rpcinodes and connectivity between nodes. connects— there are connections between nodes of different sites.-
Not expected in a normal configuration: check the site configuration.
Server resources
current_time— time on servers differs by more than 1 second.-
Configure time synchronization (NTP) on hosts. Affects certificate validation, logs, and data synchronization.
disk_space— low disk space.-
The working disk (logs, conversation recordings) or container partitions are below the threshold. Delete outdated logs and recordings, check retention periods, mount a partition: new partition for recordings, disk load.
mem_usage— low free RAM.-
Find the nodes with the highest consumption (
/system/node), checkmessage_queue. Add memory or redistribute microservices (sizing). cpu_usage— CPU load above the number of cores.-
Briefly under load this is acceptable. Persistently — redistribute microservices between servers, check sizing.
message_queue— long process message queues.-
Overload or a hung process. Note the node and microservice, look at their logs. If the queue grows — restart the microservice; thresholds are
mq_len_pid_thresholdand other parameters in the collection settings.
Configuration and versions
config_diffs— configuration differs on nodes.-
The configuration was not applied everywhere. Apply the configuration again, check
:SYNCdirectory synchronization. basic_roles_not_found,solution_roles_not_found— required microservices are missing.-
Basic on the site or required by the product. Add microservices to the configuration.
mgc_mg_connects— fewermgmedia gateways are connected to themgccontroller than configured.-
mgnodes are stopped or not connected: check the nodes and their logs. assembly_diffs,version_diffs,version_files_diffs,version_all_files_diffs— versions on servers differ.-
The update did not reach all servers, or a server has a patch. Bring all servers to the same version.
Domains and databases
db_connects— the domain center does not connect to the domain DB.-
Check the DB server, network, credentials, and the installation key file on the server (storage passwords are encrypted, no key).
unavailable_dc_domains— domains are unavailable on nodes.-
The domain center did not load domains: license (product and version), domain DB, load errors. Look at the domain center log and license.
Security and license
certificate— TLS certificates expired, are expiring, or are invalid.-
Replace the microservice or domain certificate: certificate connection.
license— license warnings.-
No license, expiration warning, empty domain count. Check the master domain license and upload a valid one.
Applications
role_state— states of non-started microservices and application warnings.-
Act according to the warning text:
-
roles with virtual addresses — no instance of the group is active: addresses are not raised on any server, and the role service (for example, providers on
esg) does not work. Check the virtual address mechanism on the group servers (Virtual IP configuration) and quorum; -
mg— media gateway process failures:mglogs; -
mware— DB and:SYNCbackup errors: space, access to the backup directory, backup parameters; -
domaincenter— values encrypted with a non-current installation key and DMS classes with secret properties without re-encryption: re-encryption (encryption and keys); -
dms— the storage password does not decrypt: return the installation key file to the server; -
server key service — the key is absent in
:SYNC/secure/keysor on the server: return the key file.
-
roleapps— role applications: installation in progress or installation error.-
Error — check the application package and installation log.
ext_svcscript— project service script warnings.-
Act according to their text; the script and its logic are project-specific.