Logfiles

Table of contents

Overview
Configuring logging
Logging Layers
Content of the categories
Special categories
See also

Overview

Any business process served by the system as a whole is served by at least one logical role, and typically several of them.

During operation, roles log information to logfiles: /var/log/era/<NODE>/<CATEGORY>/<FILE>.log, where

NODE – node folder.
CATEGORY – folder with the name of the log category.
FILE – A specific log portion file of a particular type, time-bound and size-limited.

Additionally, product microservices log information to the following logs: /var/log/era/<MSVC_NODE>/domains/<MICROSERVICE_NAME>/<FILE>.log, wherein additionally

MSVC_NODE – node folder msvc.
MICROSERVICE_NAME – folder with the name of the product microservice.

Logs are written locally on the server in the working directory of the node where the role instance is executing. The log files are separated by categories, so the node’s log files folder contains category folders within which the files lie. Each file represents a specific type of logging - reflected in the prefix of its name, e.g. trn_2019-10-07.0002.log.

Log files of the same type are divided by 10 MB and numbered with a postfix.

Log files are automatically deleted after 2 days from the closing date. If the total volume of node logs accumulated in the node log directory is more than 1 GB, the oldest files are automatically deleted. Empty category catalogs are also automatically deleted.

When the date in the local time zone of the server changes, the current files are automatically closed and logging continues in the files containing the new date in the name.

When another file is closed, it is automatically sent to the logstore (role logstore) of the current site. If not accessed, attempts to send the file will be made periodically until the file is deleted.

Logging is suspended if less than 3 GB of free space remains on the node’s working directory disk. Logging resumes when free space becomes available. The check operation runs at 1 minute intervals.

The logfile operation itself is reflected in the category logs log.

Each log records the time and process ID of the process posting the message. When the node is restarted, the last log file will be reopened for writing and the starting set of data will be placed into it.

=====================================================================
======== Log started at 11:33:02.487 ================================
======== Cookie='ELIVFDFOXKYQSMLOZZEN'
=====================================================================

11:33:02.487    <0.26543.6>     message 1
11:33:02.530    <0.26543.6>     message 2
...

Configuring logging

According to the way the output to the logs is configured, the information is divided into:

Permanent - placed in logs with no possibility of shutdown.
Debugging is turned off in production.
- In the master domain, in the Settings application in the System. Nodes" each node can be set to a logging level. The default is 'INFO'. The level is reset automatically on restart.
- For debugging purposes, it can be enabled by developers and provided as an archive with the patch.
Customizable
- In the configuration (in the options of the respective roles, e.g. log_trn, log_media of role b2b, log_level of role mg);
- In entity properties (in the options of the respective entities. such as scenario logging level.

Logging Layers

According to the method of linking to processes and entities in logs, the information is divided into:

infrastructural;
process;
intra-domain
- common;
- on entity-related processes;
- product layer microservices.

It should be noted that

a number of domain-specific messages belong to the infrastructure layer. For example, the process of domain data synchronization between sites.
a number of processes, despite their connection to domains, belong to the process layer. For example, call processing and logging of SIP traffic.

Content of the categories

Table 1. Log categories
Category	Types and descriptions
`boot`	Application logs `boot` is the role loader inside the node. `erl` – The main log of the boot application. Reflects the process of loading roles into the node.
`callstore`	Role Logs `callstore`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.
`configurator`	Node loading service logs `cfg` – Main configuration log. Within server shell reflects the process of loading and starting role nodes. Within the based role reflects the process of loading a role node. `child_node` – Node log to check if child nodes can be pinged. Reflects the process of loading a test node. Only for server shell. `site_pinger` – Ping log from server shell to role ic. Only for server shell. `site_start_stats` – Statistics about the epmd process, recorded when the server shell is started. Only for server shell.
`domaincenter`	Role logs `mdc` and `sdc`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work. `sync` – Reflects the process of synchronizing domain data between the worker site and the master site on the worker site side.
`domains`	Catalog at roles logging independent intradomain processes. Contains subdirectories by domain that had messages saved to the log logs. For role `msvc` - contains logs of product layer microservices. The logging depth is configurable in the product layer administration application of a specific domain.
`domstore`	Role logs `domstore`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.
`env`	`db_structure` – Reflects the process of updating the database on the servers PostgreSQL. `dbaccess` – Reflects the database connection process PostgreSQL. `env` – Captures errors of common functions and lower-level services. `mns` – Reflects the process of system interaction with the object database mnesia. `msgqwtc` – Captures messages from the virtual machine process message queue tracking service erlang. `wtc` – Captures performance service messages and traces connections between nodes. Captures ALIVE state of a node once a minute.
`error_logger`	Logs of the erlang virtual machine message handler service: errors, warnings, informational messages. `info` – Captures informational messages from the virtual machine erlang. `warn` – Captures alerts from the virtual machine erlang. `err` – Captures error messages from the virtual machine erlang. `other` – Captures other messages from the virtual machine erlang.
`huntq`	Role Logs `huntq`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work. `trace_obj` – Abbreviated log of fsm actions of objects within a role. Reflects actions on queued objects. `trace_rsrc` – An abbreviated log of fsm resource actions within a role. Reflects actions on used resources in the queue. `dbg_obj` – Extended log of fsm actions of objects within a role. Reflects actions on queued objects. `dbg_rsrc` – Extended log of fsm resource actions within a role. Reflects actions on used resources in the queue.
`ic`	Role Logs `ic`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work. `srv_pinger` – Log of pings from server shell and other ic roles, as well as pings to role mc. Contains failed ping attempts.
`log`	Logging service logs. `log` – Reflects system interaction with logs: opening files, closing files, changing dates, clearing, transferring to storage.
`logstore`	Role Logs `logstore`. `erl` – Main role log. Reflects the process of loading and overflow of the role, reflects the work with log archive files: addition, deletion.
`mc`	Role Logs `mc`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work. `update` – The log of fsm system update actions within a role. `update_node` – The log of actions within an upgrade node. Reflects interactions with servers of older versions. `pinger` – Log of incoming pings from roles ic from all sites. `trace` – Configuration wizard fsm action log. Reflects actions on the configuration within the role. `cfg_error` – Log of errors when checking configurations within fsm configuration wizard. `cfg_applying` – Log of applying the configuration within the fsm configuration wizard.
`mgc`	Role Logs `mgc`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of the role and makes a regular record of the current state.
`mg`	Logs of roles `mg` and `bgmg` and applications `rtx_mg3`. `erl` – Main Role Log. Reflects the process of starting and monitoring the application rtx_mg3. `mg3` – Reflects the overall process of the media gateway. `mg_module` – Reflects the overall process of the media gateway as it relates to serving media contexts. `mg_rtp_stat` – Captures statistics of received and sent RTP packets when media context terminations are completed. `proto` – Captures the content of all incoming and outgoing MEGACO protocol packets (on the media gateway side) rtx_mg3). `stat` – Fixes a periodic slice on the number of objects in the application RAM rtx_mg3.
`middleware`	Role Logs `mware`. `erl` – The main log of the role. Reflects the status of services supported by the role mware. `accdr` – Captures the events of the class `alertcallevents`.
`mixer`	Logs of role `mix` and applications `rtx_mixer`. `erl` – Main Role Log. Reflects the process of starting and monitoring the application rtx_mixer. `subscr` – Reflects the process of subscribing the mix role to the broker’s internal message queues. `mix_error` – Log errors when mixing or moving finished files. `dwnld_files` – The download log of the files generated by rtx_mg3 for mixing.
`registrar`	Role Logs `sr`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.
`reserver`	Role Logs `rsv`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.
`rpc`	`rpci` – Role log rpci. Reflects the workflow of the role, capturing some significant events in the workflow. `rpco` – Role log rpco. Reflects the workflow of the role, captures some significant events in the workflow.
`sc`	`update` – The log of fsm system update actions within a role. `update_node` – The log of actions within an upgrade node. Reflects interactions with servers of older versions.
`script`	Logs of `script` script processing application and roles `svc`. `cdr` – Captures the events of the class `scriptevents`. `erl` – The main log of scr role and script handler service. Reflects the process of loading and flow of service task service, captures some significant events in the process, captures the transfer of control to service script components. `<SCRIPTTYPE>@<DOMAIN>` – Captures information about the operation of scripts and their components, linking it to the type of scripts and to the domain. Contains information messages, warnings, and runtime errors. `svcscr` – Log of the service of launching service scripts. Reflects the boot process, records some events in the process of operation.
`selector`	Role Logs `sel`. `cdr` – Captures the events of the class `selectorevents`. `erl` – The main log of the role sel. Reflects the process of loading and re-flowing a role, captures some significant events in the process.
`sip`	SIP protocol handler role logs. `callid` – Captures all the roles processed by the role `CallId`. `cdr` – Captures CDR class events `callevents`, `confevents`, `providerevents`. `media` – Reflects the messaging process on the backend between the SIP Dialog and Media Context service components. `mgct` – Reflects the messaging process on the management interface between the SIP and mgc (media-gate-controller) roles from the side SIP. `sip` – Captures events and actions at the interface between the SIP stack and the business logic, and reflects the internal processing of the FSM dialog. `trn` – Captures the contents of all packets received and sent SIP.
`broker`	Role Logs `broker`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.
`srvc`	`update` – Log of fsm actions of the update system within server shell.
`sync`	Log logs of the automatic file synchronization application `fsync`. `fsync` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work. `syncthing` – Log for debugging the operation of the role.
`statestore`	Role Logs `sts`. `cdr` – Captures CDR class events `callevents`, `confevents`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.
`store`	Role Logs `st`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.
`trace`	Internal process trace service logs. `trace_multi.txt` – System code trace log. Captures function calls. `trace_multi2.txt` – todo `trace_multi3.txt` – System code trace log. Captures function calls, input parameters and output values.
`update`	`stdout` – Default update system fsm action log. Applies if no logging method for the update system is defined within the role.
`usercenter`	Role Logs `usr`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.
`vmail`	Role Logs `vmail`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.
`ws`	Role Logs `ws`. `erl` – Main role log. Captures web server errors as well as the process of messaging on the websocket interface.
`wssubscr`	Role Logs `wssubscr`. `erl` – Main role log. Reflects the process of loading and flow of the role, captures some significant events in the process of work.

In each node’s working directory, you can find some combination of the above log types depending on what roles are active in it and what processes are involved.

Special categories

There are several logs that do not fall under the described structure of storing and capturing events.

Table 2. Individual files
File	Description
`erlang.log` `erlang.log.<PART>`	Commits the console output of the virtual machine’s worker nodes. Placed directly in the node’s working directory. `<PART>` is cyclically closed from `0` to `9`. The last file should be determined by the time of the last record.
`erl_crash.dump`	Debug dump of a virtual machine. Its presence indicates that the node collapsed in the previous period. Created when the node process crashes. Placed directly in the node’s working directory.
`error.log,` `error.log.<PART>`	Registers errors that occurred in the node. Created only by nodes executing the `lager` application. These are ws and all roles SIP. Placed in the root of the logging directory. `<PART>` is cyclically closed from `0` to `9`. The last file should be determined by the time of the last record.
`crash.log` `crash.log.<PART>`	Logs crashes of internal virtual processes within a node. Created only by nodes executing the `lager` application. These are ws and all roles SIP. Placed in the root of the logging directory. `<PART>` is cyclically closed from `0` to `9`. The last file should be determined by the time of the last record.
`console.log` `console.log.<PART>`	Registers some key internal points in the underlying processes. Created only by nodes executing the `lager` application. These are ws and all roles SIP. Placed in the root of the logging directory. `<PART>` is cyclically closed from `0` to `9`. The last file should be determined by the time of the last record.