Data Model Service (dms)

Description

Domain Data Model Service.
Works on the basis of the domain center entity class `classes` interpreting each instance as metadata of a new user/project collection of information objects.
Provides a CRUD API for the rest of the roles and services to manage instances of custom/project collections. Supports the `create`, `update`, `replace`, `read` collection, `read` collection item`, `clear` collection methods, `lookup`. Depending on the collection metadata, it supports in-memory dictionary, distributed object database, PostgreSQL server, KAFKA and columnar database operations Clickhouse. Supports single-threaded, parallel and asynchronous modes of working with collections. Can guarantee the integrity of modification/reading/notification, ensures that modifications in storage and notifications remain consistent. Allows you to use a temporary or permanent cache for collection items.

Works in conjunction with wssubscr to notify subscribers of changes (event class/type `modelevents.data_changed`). Supports subscriptions with filters and masking. Notifies with filter-adjusted commands (e.g., create instead of update). Selecting data from a collection supports complex filtering, masking, sorting, aggregation, offsetting, and limiting.

Access via HTTP(s) REST API `/rest/v1/model/…​`, where further is the name (or path) of the class. Management is also possible via Websocket-API both with authorization under user session and via token.
Access to collections and methods is determined by webserver rautes. By default, all paths and methods are available to the administrator, and the rest is determined by the roles (roles class) assigned to the user (user.opts.roles) or integration channel (integration_point.opts.roles).

Supports attachment file binding to entities that have `attachment` type properties of single or list form.
Attachments are hosted on the storage specified in the class, connections to which are configured using entities of class storage with file storage related types.

Divided into microservices by domain.
Reserved in Active-Passive mode. Scaled by domain tree partitioning.

Table 1. System Characteristics

Code

dms

Mode of operation

Storage, service

Backup mode

Active-Passive

Types of sites

Any

Layer

Business logic

Placement

Internal

Saving and restoring state on reboot

Yes, except for classes like RAM.

appendix

era_dms

Own storage

RAM, Mnesia, PostgreSQL, KAFKA, Clickhouse

Limitations

  • Available through the information and communications platform 'incoplax'.

  • The speed of operation depends on the selected collection storage type. A collection performance speed of up to 10 thousand requests per second is achievable.

  • Data between sites is not synchronized by the role. Merging via external storage is possible with the correction for possible integrity issues and the need to disable cache.

Parameters

Description of columns in the table

Table 2. Parameters
Name Type Default Description

name

str

required

Name. May consist of Latin letters and numbers, must begin with a letter.

roletype

str

required

Role Type. Possible values: "dms".

iface

str

required

The alias of the server network interface on which the roles will interact internally with each other.

ext

json

empty

Additional role options. Contains a json object or list.

enabled

bool

empty

Role activity flag. When set to false, the role does not participate in validation and is not started.

dbconnstrings

array<str>

required

Connection strings to the PostgreSQL DB domain server farm, where domain databases with product model entity store are created.
These connection strings are used for classes that have storage_mode: 'category' or 'history', which do not have specific parameters of connection to PostgreSQL database, but have 'auto' mode set (by default).

Each domain uses a pool of 10 connections to each database (unique connection string). A new connection is created only when busy. There are no connections to the database in the quiet mode.
Each domain uses the original specified string only to create its own database with the unique name 'era_model_%DOMAIN%', and for operation uses a modified connection string where it substitutes the name of its database as the database.

Connection string format: "alias://pgdb_strings/<ALIAS_NAME>", where <ALIAS_NAME> is a specific configuration alias from the section pgdb_strings.

Listing multiple connection strings in the list provides redundant access to the database server farm - if the connection through the first string fails, the connection through the second string is made, and so on.

At the central site where the MDC role is located, in the absence of this parameter, the connection strings specified in the role parameters are applied for classes with the 'auto' mode mdc.

group

int

required

Group Number.
As part of horizontal scaling, a role can be divided into multiple groups on a site, with only one instance active in each group and the rest reserved.
Responsibilities between groups are shared across domain sets. Together, all groups maintain a complete domain set. All roles in the same group must have the same responsibility.
Integer from 1 to 9999999.

order

int

required

The order of a role instance within a group.
Defines in what order the flow in the mode will take place Active-Passive.

include_domains

array<str>

empty

A list of domain names served by this group.
Any domain of any level can be specified. Specifying a domain causes its subdomain tree to be served on this group and not on other role groups, excluding only those branches mentioned in other groups.
Collectively, all role groups on a site serve a complete domain tree.

domain_group

int

undefined

The number of the domain tree service group.
Used when you want to split the service across multiple instances by domain and by sections at the same time.
To create several different groups serving the same incomplete domain tree but different sections, you need to set the same value in include_domains and the same value in the domain_group.
Thus the mechanism of subtree grouping when excluding subtrees of other groups applies the domain_group value in priority instead of group.
Integer from 1 to 9999999.

max_loading_domains

int

2

Limit the number of domains that can be loaded at the same time.
Comes in handy in systems with a large number of domains.
It is used to limit the CPU load during microservice reboot.
The parameter is applied at startup, when restarting the microservice, and when creating new domains in bulk.

max_loading_classes

int

1

Limit the number of concurrently loaded classes within a single domain.
Comes in handy in systems with a large number of domains.
It is used to limit the CPU load during microservice reboot.
The parameter is applied at startup, when restarting the microservice, and when mass creating new classes in the domain.

cache_filter_interval_sec

int

300

Temporary cache clearing interval for classes with the corresponding caching mode.
The more often a purge is performed, the less memory the cache occupies, the fewer resources a single iteration takes, but the more CPU time is spent on the operation as a whole.
Recommended values are from 30 to 600 seconds.

class_readers_count

int

10

The number of processes serving read operations in parallel within a single class, if parallel reading is allowed by the class mode.
Including sampling operations with a filter.
Minimum value - 2, maximum value - 32.
Comes in handy in systems with a large number of domains.
If the system has a large number of unloaded domains, the recommended values are 4 or lower.
If the number of domains is small (up to 100), the recommended values are higher 10.
A restart of the microservice is required to apply the value.

This is the default value for those classes where the value of opts.worker_r_count = 0.

class_writers_count

int

4

The number of processes serving parallel write operations within a single class, if parallel reading is allowed by the class mode.
The write operation involves preparing the object and checking the acceptability of values.
Minimum value - 2, maximum value - 32.
Comes in handy in systems with a large number of domains.
If the system has a large number of unloaded domains, the recommended values are 4 or lower.
If the number of domains is small (up to 100), the recommended values are higher than 4. О
A restart of the microservice is required to apply the value.

This is the default value for those classes where the value of opts.worker_w_count = 0.

stat_enabled

bool

true

Minute-by-minute statistics logging switch for classes for which the level of minute-by-minute statistics logging is not individually set.
Every minute there is a collection and logging of work statistics in the form of several lines with general performance indicators: the number of requests passed under different conditions, minimum, maximum, average time of request processing.
To increase the level of logging for a particular class, you can set the option stat_mode = 'high'. Then the statistics on minimum, maximum and average processing time will be displayed in the context of the operation performed.
To reduce the level of minute-by-minute logging of statistics by classes, it is necessary to turn off this mode and configure specific classes to the required level.

pg_max_connections_default

int

10

The default number of threads in the connection pool to the postgresql server.
The pool is defined by the connection string - each server and each database is organized with its own pool.
For individual classes, the number of threads in the connection pool can be set individually in the class options.

There is a separate pool for running server services and a pool for running client connections. Each of them is maximally expandable up to the specified number of connections.
If the connection is not used for more than 60 seconds, it is closed.

query_workers_count

int

4

Number of processes serving dmsquery operations in parallel on the server.
The execution operation involves executing an arbitrary query in the database and then preparing the result, the size of which can be arbitrary.
The minimum value is 1, the maximum value - 32.
Serves to limit the peak loads created when building custom reports that use the query in parallel dmsquery.
Should be commensurate with the design logic, the number of cores on the server where the microservice instance is executing, and the composition of other microservices on that server.
A restart of the microservice is required to apply the value.

sections

array<str>

A list of sections in which the instance participates.
The section can be specified in the entity properties class, in which case only instances with the specified section present in that field are involved in its execution.
The parameter must be identical on instances of the same group (active-passive).

mnesia_storage_mode

str

disc "Default mode of object storage operation. It is used by classes of 'runtime' type

for which the storage operation mode is not set individually.

Possible options:

* disc - full mode

in RAM with saving to disk on all instances of the group.

  • ram - only in RAM (with support for distributed

transactional). If all instances of the group are rebooted/shut down simultaneously

data is lost.

  • disc_only - on disk only (noticeably less productive

but saves the RAM)."

Configuration example

The configuration is managed in an application available to master domain administrators. The application hides the full content of the configuration, but it is nevertheless accessible via the API.

The configuration contains a section to describe all instances of all roles. Parameters are defined for each specific role instance.

Example node
{
  "name": "dms1",
  "roletype": "dms",
  "iface": "eth0",

  "group": 11310,
  "order": 1
}

See also

  • Personalization
  • Increase font size
  • Decrease font size