Log files

Overview

Any business process served by the system as a whole is served by at least one logical role, and usually by several of them.

During operation, roles write information to log files: /var/log/era/<NODE>/<CATEGORY>/<FILE>.log, where

  • NODE – folder with the node name.

  • CATEGORY – folder with the log category name.

  • FILE – specific file of a log portion of a certain type, bound to time and limited in size.

Additionally, product microservices write information to log files: /var/log/era/<MSVC_NODE>/domains/<DOMAIN_NAME>/<MICROSERVICE_NAME>/<FILE>.log, where additionally

  • MSVC_NODE – folder with the msvc node name.

  • DOMAIN_NAME – folder with the domain name (domain entity).

  • MICROSERVICE_NAME – folder with the product microservice name.

Log files are written locally on the server in the node working directory where the role instance runs. Log files are divided by categories, so the node log folder contains category folders with files inside. Each file represents a certain logging type – reflected in its name prefix, for example trn_2019-10-07.0002.log.

Log files of one type are split at 10 MB and numbered with a postfix.

Log files are deleted automatically 2 days after closing. When the total volume in the node log directory exceeds 1 GB, the oldest files are deleted automatically. Empty category directories are also deleted automatically.

When the date changes in the server local time zone, current files are closed automatically and logging continues in files containing the new date in their name.

When the next file closes, it is automatically sent to the log storage (the logstore role) of the current site. If it is inaccessible, attempts to send the file are made periodically until the file is deleted.

Logging is suspended if less than 3 GB of free space remains on the disk with the node working directory. Logging resumes after free space appears. The check operation runs every 1 minute.

Working with log files is itself reflected in log files of the log category.

Each log file records the time and identifier of the process placing the message. When the node restarts, the last log file is reopened for writing and the initial data set is placed into it.

================================================================================
======== Log started at 2024-09-05 11:33:02.487 ================================
======== Cookie='ELIVFDFOXKYQSMLOZZEN'
================================================================================

11:33:02.487    <0.26543.6>     message 1
11:33:02.530    <0.26543.6>     message 2
...

Logging configuration

By the method of configuring output to log files, information is divided into:

  • Permanent – placed in log files without the ability to disable.

  • Debug – disabled in production.

    • In the master domain, in the "Settings" application under "System. Nodes", a logging level can be set for each node. Default 'INFO'. The level is reset automatically on restart.

    • For debugging purposes it can be enabled by developers and provided as an archive with a patch.

  • Configurable

    • In the configuration (in the corresponding role options, for example log_trn, log_media for the b2b role, log_level for the mg role);

    • In entity properties (in the corresponding entity options, for example the script logging level.

Logging layers

By the method of binding to processes and entities, information in log files is divided into:

  • infrastructure;

  • process;

  • intra-domain

    • general;

    • by processes related to entities;

    • product layer microservices.

It should be noted that

  • a number of domain messages belong to the infrastructure layer. For example, the domain data synchronization process between sites.

  • a number of processes, despite their connection to domains, belong to the process layer. For example, call processing and SIP traffic logging.

Category contents

Table 1. Log categories
Category Types and descriptions

ap

Log files of the ap role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

boot

Log files of the boot application – the role loader inside the node.

  • erl – Main boot application log. Reflects the role loading process in the node.

callstore

Log files of the callstore role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

  • cdr – Reflects building complex CDR records from call events (debug level).

configurator

Log files of the node loading service

  • cfg – Main configuration log. Within server shell reflects the loading and startup of role nodes. Within a based role reflects the role node loading process.

  • child_node – Node log for checking the ability to ping child nodes. Reflects the test node loading process. Server shell only.

  • node_monitor – Log of running role node accounting. Server shell only.

  • shell_monitor – Server shell monitoring service log, including execution of the server pre-start handler. Server shell only.

  • site_pinger – Log of pings from server shell to the ic role. Server shell only.

  • site_start_stats – Statistics about the epmd process, written at server shell startup. Server shell only.

dfsync

Log files of the dfsync role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

  • fsynclib – Log of the file synchronization library between servers.

dms

Log files of the dms role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

  • mns – Reflects the role system interaction with the mnesia object DB.

domaincenter

Log files of the mdc and sdc roles.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

  • sync – Reflects domain data synchronization between the working site and the master site on the working site side.

domains

Directory for roles logging independent intra-domain processes. Contains subdirectories by domains in which messages saved to log files existed.

For the msvc role - contains log files of product layer microservices. Logging depth is configured in the product layer administration application of the specific domain.

domstore

Log files of the domstore role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

email

Log files of the email role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

env

  • db_structure – Reflects the DB update process on PostgreSQL servers.

  • dbaccess – Reflects the PostgreSQL DB connection process.

  • env – Records errors of common functions and low-level services.

  • erl – Base library log in nodes of the broker, dfsync, dms, meet, msvc, rpci roles.

  • mns – Reflects the system interaction process with the mnesia object DB.

  • tz – Reflects application of the time zone set by master domain settings.

  • msgqwtc – Records messages of the erlang virtual machine process message queue monitoring service.

  • wtc – Records messages of the performance and inter-node connection monitoring service. Records the node ALIVE state once a minute.

error_logger

Log files of the erlang virtual machine message handler service: errors, warnings, information messages.

  • info – Records information messages from the erlang virtual machine.

  • warn – Records warnings from the erlang virtual machine.

  • err – Records error messages from the erlang virtual machine.

  • other – Records other messages from the erlang virtual machine.

fs

Log files of the fs role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

huntq

Log files of the huntq role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

  • trace_obj – Short log of object fsm actions within the role. Reflects actions on queued objects.

  • trace_rsrc – Short log of resource fsm actions within the role. Reflects actions on used resources in the queue.

  • dbg_obj – Extended log of object fsm actions within the role. Reflects actions on queued objects.

  • dbg_rsrc – Extended log of resource fsm actions within the role. Reflects actions on used resources in the queue.

ic

Log files of the ic role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

  • srv_pinger – Log of pings from server shell and other ic roles, and pings to the mc role. Contains failed ping attempts.

im

Log files of the im role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

log

Log files of the logging service.

  • log – Reflects system interaction with log files: file opening, closing, date changes, cleanup, transfer to storage.

logstore

Log files of the logstore role.

  • erl – Main role log. Reflects role loading and migration, and work with log archive files: adding, deleting.

mc

Log files of the mc role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

  • update – Log of update system fsm actions within the role and the update node.

  • pinger – Log of incoming pings from ic roles from all sites.

  • trace – Log of configuration master fsm actions. Reflects configuration actions within the role.

  • cfg_error – Log of configuration validation errors within the configuration master fsm.

  • cfg_applying – Log of configuration application within the configuration master fsm.

meet

Log files of the meet role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

megaco

Log files of the mgc role on the MEGACO protocol side.

  • trn – Records ServiceChange requests from media gateways.

mgc

Log files of the mgc role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during role operation, and regularly records the current state.

mg

Log files of the mg and bgmg roles and the rtx_mg3 application.

  • erl – Main role log. Reflects the rtx_mg3 application startup and monitoring process.

  • mg3 – Reflects the general media gateway operation process.

  • mg_module – Reflects the general media gateway operation in terms of media context servicing.

  • mg_rtp_stat – Records statistics of received and sent RTP packets when media context terminations complete.

  • proto – Records the content of all received and sent MEGACO protocol packets (on the rtx_mg3 media gateway side).

  • stat – Records a periodic snapshot of the number of objects in rtx_mg3 application memory.

middleware

Log files of the mware role.

  • erl – Main role log. Reflects the state of services supported by the mware role.

  • bkp – backup_controller service log.

  • pgctrl – pg_controller service log.

  • cpd – cpd (call progress detector) service log.

  • accdr – alertcall event log: changes of alert states and calls to subscribers.

  • svctask – Domain service task service log.

  • msvclog – Messages sent for centralized logging by product layer microservices.

  • webservice – Domain integration channel validity monitoring log.

  • cti – SmartLogger CTI adapter log (when adapter logging is enabled).

mixer

Log files of the mix role and the rtx_mixer application.

  • erl – Main role log. Reflects the rtx_mixer application startup and monitoring process.

  • subscr – Reflects the mix role subscription to internal broker message queues.

  • mix_error – Log of errors during mixing or moving completed files.

  • dwnld_files – Log of downloading files generated by rtx_mg3 for mixing.

msvc

Log files of the msvc role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

recmover

Log files of the recmover role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

registrar

Log files of the sr role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

reserver

Log files of the rsv role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

rpc

  • rpci – Log of the rpci role. Reflects role operation and records some significant events during operation.

  • rpco – Log of the rpco role. Reflects role operation and records some significant events during operation.

sc

  • update – Log of update system fsm actions within the role and the update node.

script

Log files of the script script processing application and the svc role.

  • cdr – Records events of the scriptevents class.

  • erl – Main log of the scr role and script handler service. Reflects service task loading and migration, records some significant events during operation, and records control transfer to service script components.

  • <SCRIPTTYPE>@<DOMAIN> – Records information about script and component operation, binding it to the script type and domain. Contains information messages, warnings, and execution errors.

  • svcscr – Log of the service script launch service. Reflects the loading process and records some events during operation.

selector

Log files of the sel role.

  • cdr – Records events of the selectorevents class.

  • erl – Main log of the sel role. Reflects role loading and migration, records some significant events during operation.

sip

Log files of SIP protocol handler roles.

  • callid – Records all CallId processed by the role.

  • erl – Main SIP role log. Reflects role loading and migration, records some significant events during operation.

  • cdr – Records CDR events of the callevents, confevents, providerevents classes.

  • media – Reflects message exchange on the internal interface between SIP dialog servicing and media context components.

  • nksip – SIP stack log (nksip, nkpacket, nkservice, nklib). Records transport, transaction, dialog, and authorization errors and warnings, as well as plugin and listener startup. Debug levels are enabled only by build.

  • mgct – Reflects message exchange on the control interface between SIP and mgc (media-gate-controller) roles on the SIP side.

  • rgf – Records SIP device registration failures (address, To, Contact, User-Agent, reason, Call-Id) on the b2b role.

  • sip – Records events and actions on the interface between the SIP stack and business logic, and reflects internal dialog FSM processing.

  • trn – Records the content of all received and sent SIP packets.

broker

Log files of the broker role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

  • brokerlib – Message queue broker library log.

  • mns – Reflects the role system interaction with the mnesia object DB.

srvc

  • update – Log of update system fsm actions within server shell.

statestore

Log files of the sts role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

store

Log files of the st role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

systemstate

Log files of the mware role.

  • ss – Log of regular placement of the system state monitoring poll result.

trace

Log files of the internal process tracing service.

  • trace_multi.txt – System code trace log. Records function calls, input parameters, and return values.

  • trace_multi3.txt – System code trace log. Records function calls.

update

  • stdout – Default update system fsm action log. Applied if the update system logging method is not defined within the role.

usercenter

Log files of the usr role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

voicemail

Log files of the vmail role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

  • vmctrl – Reflects work with voice message files (placement, copying between sites).

  • vmstat – Reflects the state of domain voicemail boxes and its synchronization between sites.

ws

Log files of the ws role.

  • erl – Main role log. Records web server errors and message exchange on the WebSocket interface.

  • dbg – Records excess duration of REST requests to the domain center and dms via WebSocket.

wssubscr

Log files of the wssubscr role.

  • erl – Main role log. Reflects role loading and migration, records some significant events during operation.

In the working directory of each node, a certain set of the log types above can be found depending on which roles operate in it and which processes they participate in.

Special categories

There are several log files that do not fit the described storage and event recording structure.

Table 2. Separate files
File Description

erlang.log

erlang.log.<PART>

Records console output of erlang virtual machine worker nodes.

Placed directly in the node working directory.

<PART> cycles from 0 to 9. The last file should be determined by the last write time.

erl_crash.dump

Debug dump of the virtual machine. Its presence indicates that the node crashed in the previous period. Created when the node process crashes.

Placed directly in the node working directory.