Class/collection (class)

Description

User class metadata.
Allows you to define a structure for the data and the rules for maintaining it. For each class in the REST-API appears endpoint /rest/v1/model/CLASSPATH and possibility to perform operations via WebSocket-API with capability=rest.

Can place data in memory, in a distributed object database, in a postgres relational database (with and without partitions for historical data), in a Kafka broker, and store data in an analytic columnar database Clickhouse.

Each collection publishes changes delivered to subscribers (subscriptions via websocket-API, authorized based on roles).

Supports multiple modes of operation: speed read, speed notification, asynchronous mode, synchronous mode.
Each of them ensures internal integrity by preventing the original sequence of requests from being altered when they are processed and notifications are sent out. The exception is the processing of different entities in modes that utilize parallel processing. +

Supports property inheritance.
Supports restricting data access for different users to methods and collections by role (role.routes and iam_all.json), as well as based on complex filters to specific data on read, modify, field visibility and the ability to modify them (class.opts.security_filter_read, class.opts.security_filter_write, class.properties..security_filter_read, class.properties..security_filter_write).
Classes in a distributed object database can be auto-deleted by lifetime with automatic renewal when saving; at the same time, the maximum possible number of records in the collection is limited due to the fact that to apply a filter to a selection, the entire collection is searched element by element.

Limitations

  • The collection is not available in the master domain.

Fields

Entity structure
{
  "id": uuid,
  "classname": str,
  "name": str,
  "description": str,
  "parent_id": uuid,
  "properties": [
    {
      "name": str,
      "data_type": str,
      "multi": bool,
      "required": bool,
      "default": any,
      "idclass": uuid,
      "items": array<str>,
      "security_filter_read": array,
      "security_filter_write": array
    }
  ]: array<obj>,
  "storage_mode": str,
  "integrity_mode": str,
  "cache_mode": str,
  "opts": {
    "title": str,
    "comment": str,
    "dms_group": object,
    "check_required_fill_defaults": bool,
    "max_limit": int,
    "max_mask": array<str>,
    "max_size": int,
    "store_changehistory_mode": str,
    "caption_property": str,
    "expires_mode": str,
    "expires_ttl_property": str,
    "expires_ts_property": str,
    "storage_instance": str,
    "filestorage_instance": str,
    "lookup_properties": array<str>,
    "partition_property": str,
    "partition_interval": str,
    "partition_count": int,
    "replication_factor": str,
    "notify_transactions": bool,
    "cache_sec": int,
    "cache_limit": int,
    "security_filter_read": array,
    "security_filter_write": array
  },
  "ext": {
    "ct": date,
    "lwt": date
  }
}

Description of the specification’s characteristics

Table 1. Fields
Specification Description

Field: id
Mode: inout
Type: uuid
Default: generated

Identifier. Can be specified at creation, otherwise generated by the system.

Field: classname
Mode: in
Type: str
Default: required

Class collection name. May contain a path through '/'. Typically in plural.

Field: name
Mode: in
Type: str
Default: empty

Displayed class name

Field: description
Mode: in
Type: str
Default: empty

Class Description

Field: parent_id
Mode: in
Type: uuid
Default: empty

Ancestor class identifier. Inherits all its fields.

Field: properties
Mode: in
Type: array<object>
Default: empty

List of class instance properties.
Each property contains:

  • name  —  Mandatory field. The English-language name of the property used in the API and when generating the database.

  • data_type  —  Mandatory field. Value Type. Value options:

    • string - string;

    • boolean - switch;

    • integer - 32-bit integer;

    • long - integer 64 bits;

    • float - fractional number;

    • increment - integer 64 bits. Only allowed for storage_mode = category, history, transactionlog. Only one increment field per class is allowed, including all inherited fields. Each entity creation and replacement operation increments the increment in the class, even if the operation fails - consequently there may be voids in the sequence. The last increment value is stored in the object store, so that it is detected when the class is loaded. However, it is possible to generate a situation where the last increment value for a class is not counted on loading, and a conflict of identical values may occur.

    • date - date;

    • time - time;

    • datetime - date and time;

    • uuid - unique identifier 128 bits;

    • enum - enumeration, requires an additional field items;

    • any - arbitrary object, stored in json format;

    • entity - reference to another data model entity, requires an additional field to be specified idclass;

    • attachment - file attachment. The value is filled automatically when the attachment is loaded via REST-API, when reading an entity it returns the file metadata as a json object. Attachments are accessed in the REST API by concatenating the entity path and property name.

  • caption  —  Defaults to an empty string. Readable property name.

  • multi  —  Defaults to false. Switches the field to list mode. Instead of a number - an array of numbers, instead of an attachment - a list of attachments, etc. In case of attachments - REST API access to a specific file by name through the property name as part of the path.

  • required  —  The default is false. Sign of mandatory filling of the property.

  • default  —  Default is null. Default value to populate the property.

  • idclass  —  Just for data_type='object'.

  • items  —  Just for data_type='enum'.

  • security_filter_read  —  Default []. Filter on access to view the field value for the specific user on whose behalf the read request is made. The format and application conditions are similar to class.opts.security_filter_read. If uses dependency on field values of a particular entity, the field is excluded when fetching from the collection. Works similarly to the 'mask' parameter when fetching from a collection.

  • security_filter_write  —  Defaults to []. Filter on access to change the value of the field for the specific user on whose behalf the query is executed. Multiplied by the filter in the field’s 'security_filter_read' parameter. Format and application conditions are similar class.opts.security_filter_write.

Field: storage_mode
Mode: in
Type: string
Default: category

Storage Mode:

  • abstract - abstract collection is used only for inheriting and supplying fields to other collections, including abstract collections.

  • ets - storage in the instance’s RAM. Test or non-critical to data loss mode. Speed ~15000 operations per second.

  • ram - storage in a distributed object database (mnesia on ram only). Does not save data to disk. Data integrity and safety is ensured as long as one of the instances of the microservices group responsible for class maintenance is available. Performance of key access operations and write operations ~10000 per second per free thread handler.

  • runtime - storage in a distributed object database (mnesia on disk+ram). Stores data to disk and under load significantly loads the disk subsystem for writing. Performance of key access operations and write operations is ~10000 per second per free thread handler.

  • category - storage in relational database postgresql without partitioning. Speed up to 10000 operations per second in parallel processing mode, ~3000 operations per second in synchronous mode.

  • history - storage in postgresql relational database with partitioning by date. Requires specifying the partition dimension (year, month, day), and a field with a value for partitioning.
    in* transactionlog - sending data to Kafka cluster. When storage with type=clickhouse with the same value is detected, instance automatically generates a receiver table from Kafka to ClickHouse (ReplicatedReplacingMergeTree), and makes read operations available.

Field: integrity_mode
Mode: in
Type: string
Default: sync_fast_read

Maintenance Mode:
* async - N(=10) read queues with distribution by identifier hashes, the read queue is not used.
* sync - one write queue, one read queue.
* sync_fast_read - N(=10) reading queues with distribution by identifier hashes, M(=10) reading queues. Sending a notification after saving to the database. Readers wait for earlier write operations to complete.
* sync_fast_notify - N(=10) reading queues with distribution by identifier hashes, M(=10) reading queues. Sending a change notification before saving to the database. Readers do not wait for earlier write operations to complete.

Field: cache_mode
Mode: in
Type: string
Default: temp

Memory cache mode. Makes sense only for types category, history, transactionlog.

  • full - the entire amount of data is cached. Applicable only for type=category. Reading is done from the cache - searching by identifier in the cache is fast, but sampling can be slower than in the original database engine if the volume is large. Not applicable for storage_mode = 'transactionlog'

  • temp - cached last used in a limited amount. It is deleted automatically when the cache overflows. Searching by identifier in the cache is fast, sampling is always performed in the original database storage.

  • none - the cache is not being used.

In 'transactionlog' storage mode without using clickhouse storage, the mode is automatically applied 'none'.
In 'transactionlog' storage mode using clickhouse storage, the 'full' mode is automatically reset to 'temp'. And 'none' mode assumes no read, replace and update accesses for at least some time after the last modification operation, because the data reaches the clickhouse asynchronously and with some delay.

Field: opts
Mode: in
Type: object
Composite field

Field: opts.title
Mode: in
Type: str
Default: empty

Arbitrary header

Field: opts.comment
Mode: in
Type: str
Default: empty

Arbitrary comment

Field: opts.section
Mode: in
Type: str
Default: empty

Section name to select responsible dms group (by configuration option sections).

Field: opts.dms_group
Mode: in
Type: object
Default: empty

Site → role group index. Default - unique (less number) dms group for domain
Not realized

Field: opts.check_required_fill_defaults
Mode: in
Type: bool
Default: true

if entities should be checked for required and filled by defaults by the server. Model can economy

Field: opts.max_limit
Mode: in
Type: int
Default: 100

How many items could be requested on read in max.

Field: opts.max_mask
Mode: in
Type: array<string>
Default: empty

Max mask of properties, that could be returned on collection read.

Field: opts.max_size
Mode: in
Type: int
Default: 10000

Max size of storage (only for 'ram' and 'runtime').
Max 1000000.
For storage_mode = 'ram', 'runtime'.

Field: opts.store_changehistory_mode
Mode: in
Type: str
Default: none

Transaction log saving mode for all changes to class entities.
Possible options:

  • none - Transaction log saving is disabled.

  • sync - Saving the transaction log is performed in synchronous mode. Work with the class is slowed down up to the completion of information placement in the transaction log.

  • async - Transaction log saving is performed in asynchronous mode with saving the sequence of changes. In case of heavy load, queues are built to place information into the transaction log.

The data model class is used as a log 'platform/log/HistoryChanges'.
It is the single and only receiver of the entire flow of changes across the classes configured for transaction logging.

The default change history class is created from a fixture descriptor with type storage_mode='history' and placement in the storage_instance='auto'.
The descriptor is applied on every domain restart if the 'ext.fixture_version' value of the detected entity is less than the value of 'ext.fixture_version'.

If necessary, the class can be adjusted by placing it in a different storage or at a different address. The main thing is the values of 'ext.fixture_version', 'opts.store_changehistory_mode', field structure, as well as leaving the type 'history' or 'transactionlog'.
In the future, you will need to keep track of fixchi changes with system updates.

Field: opts.caption_property
Mode: in
Type: str
Default: empty

The name of the property with the name to display for the entity, particularly in the change history.

Field: opts.stat_mode
Mode: in
Type: str
Default: "default"

Mode of minute-by-minute logging of class statistics.
Possible options:

  • disabled - Logging of class statistics is forcibly turned off.

  • default - Logging of class statistics at the medium level is controlled by microservice configuration setting dms

  • low - Low level of minute-by-minute logging of statistics. The record appears only if the class works with qualitative deviations from the norm.

  • medium - Average level of everymingute logging statistics. In addition to 'low', a line with stats appears if the class handled requests. Also 2 lines with performance evaluation of handler and notification threads for the last minute appear: number, minimum execution time per 1 request, maximum execution time per one request, average execution time per one request.

  • high - High level of minute-by-minute statistics logging. Information on handler threads is displayed multiline by each operation (collection reading, entity reading, key search, modification, replacement, deletion, cleaning)

Field: opts.worker_r_count
Mode: in
Type: str
Default: 0

Number of parallel threads for processing read operations.
A value of 0 applies the specified default settings (dms' class_readers_count).
Only makes sense for classes running in modes with parallel processing (all but 'sync').
Values from 0 to 64.

Field: opts.worker_w_count
Mode: in
Type: str
Default: 0

Number of parallel threads processing write operations.
A value of 0 applies the specified default settings (dms' class_writers_count).
Only makes sense for classes running in modes with parallel processing (all but 'sync').
Values from 0 to 64.

Field: opts.expires_mode
Mode: in
Type: str
Default: "none"

when to auto set (reset) timestamp in ts property
Only for storage_mode = runtime.
Values:

  • none - the auto delete mode is not used.

  • modify - The timestamp is set automatically at the time of any entity modification.

  • create - The timestamp is set automatically only when the entity is created.

  • custom - The timestamp can only be set manually using the REST.

Field: opts.expires_ttl_property
Mode: in
Type: str
Default: empty

Property of int to setup auto ttl (time-to-live) in seconds.
Only for storage_mode = runtime when opts.expires_mode != none.

Field: opts.expires_ts_property
Mode: in
Type: str
Default: empty

Property of long to setup ts in milliseconds from 1970 (unix time * 1000).
Only for storage_mode = runtime when opts.expires_mode != none.

Field: opts.aggr_cache_ttl_ms
Mode: in
Type: int
Default: 1000

Time to store in the cache the results of aggregate read requests.
Applies to storage_mode = runtime and ram.

Data aggregation queries in the listed repositories are performed by full scanning, so they consume resources in voluminous collections.
If regular simultaneous polling of the same aggregated data is possible, for example, getting the number of filtered items from user application dashboards, it makes sense to use a cache of the results of data aggregation calculation.
A value of 0 disables the cache.

Field: opts.storage_instance
Mode: in
Type: str or object
Default: auto

InstanceKey or #{Site ⇒ InstanceKey}.
Only for storage_mode = category, history, transactionlog.

If there is no configured storage with code 'auto', the class is automatically bound to the model base ('era_model_DOMAIN') created through the main connection to the Postgres domain database).

Field: opts.filestorage_instance
Mode: in
Type: str or object
Default: "auto"

InstanceKey or #{Site ⇒ InstanceKey} (for attachments), when InstanceKey defines storage of type s3, fs, nfs, fsync.
Used only when properties of data_type = attachment found.
The actual path to attachments is formed the same regardless of the storage type:

  • Class-categories, single nesting  —  …​/Domain/ClassName/Id12/Id34/Id56/IdRest/PropertyName

  • Class-categories, multiple nesting  —  …​/Domain/ClassName/Id12/Id34/Id56/IdRest/PropertyName/FileName

  • Class-histories, single attachment  —  …​/Domain/ClassName/PartDate/Id12/Id34/Id56/IdRest/PropertyName

  • Class-histories, multiple nesting  —  …​/Domain/ClassName/PartDate/Id12/Id34/Id56/IdRest/PropertyName/FileName

If there is no configured storage with code 'auto', the class is automatically bound to be hosted on system file servers.

Field: opts.lookup_properties
Mode: in
Type: array<str>
Default: empty

List of property names for lookup operation (indexes, separate fields etc).
Only for storage_mode = category, history, transactionlog.

Field: opts.partition_property
Mode: in
Type: str
Default: empty

Property of type=datetime to make partitioned history storage.
Optional for storage_mode = history (pg+partition) - uses date to build new partition table in postgresql database.
Required for storage_mode = transactionlog (kafka+clickhouse) - uses date as clickhouse primary key prefix.

Field: opts.partition_interval
Mode: in
Type: str
Default: month

Size of partition in partitioned history storage.
For storage_mode = history (pg+partition) - defines the size of one partition and, equally, the conditions for creating a new partition when placing data.
For storage_mode = transactionlog (kafka+clickhouse) - uses the date as the primary sort key when placing data in the ClickHouse.
Possible values:

  • year - partition for the year

  • month - monthly partition

  • day - party of the day

Field: opts.partition_count
Mode: in
Type: int
Default: 2

How many partitions does topic have (1-10).
Only for storage_mode = transactionlog (kafka).

Field: opts.replication_factor
Mode: in
Type: int
Default: 2

How many replicas does topic have (1-4).
Only for storage_mode = transactionlog (kafka).

Field: opts.notify_transactions
Mode: in
Type: bool
Default: false

If should forcely notify. Note, than storage_mode = transactionlog is not notified by default.
Only for storage_mode = transactionlog.

Field: opts.replace_without_read
Mode: in
Type: bool
Default: false

Cancels the read operation of the entity before replacement. This, in turn, is done to a) check the replacement rights, b) generate a notification with the update or create operation, depending on the presence of the entity before replacement.

In 'transactionlog' storage mode without using clickhouse storage, it is not used because the replace operation is not available.

Field: opts.cache_sec
Mode: in
Type: int
Default: 600

How long temporarily cache holds modified items.
Only for cache_mode = temp.
Values from 5 to 1800.
The interval is used both to detect outdated values and to determine the frequency of verification.

The more values in the cache, the more resource-intensive the operation of clearing obsolete entries.
The purge period (aka cache storage time) should be chosen based on the expected load profile of the class so as to avoid cache overflow on the one hand (more costly periodic purging operation)

Field: opts.cache_limit
Mode: in
Type: int
Default: 1000

How many modified items are holded in temporarily cache.
Only fo cache_mode = temp.
Values from 100 to 100000.
If at the moment of periodic check there are more actual entries in the cache than the set value, those of them, which have been accessed most recently, are forcibly deleted from the cache.

The more values in the cache, the more resource-intensive the operation of clearing obsolete entries.
The replace, update, read_entity operations predominantly take a value from the cache.
The cache size limit should be determined based on the expected load profile per class.

Field: opts.security_filter_read
Mode: in
Type: array
Default: []

Filter for custom queries to select data from a collection, read a specific collection item, and search by key fields.

The format of the value corresponds to the format of the 'filter' parameter when selecting from a collection (read more).
Additionally, extended functions can be used in expressions.

Does not apply to requests from users who have the role 'admin'.
Does not apply to requests from scripts.
Does not apply to requests from Token-API if the integration channel is not assigned a user (opts.userid) on whose behalf requests and subscriptions are made, or if a role is assigned to that user or to the channel itself 'admin'.

Applies to HTTP REST API, Websocket REST API, change subscriptions and notifications.

Field: opts.security_filter_write
Mode: in
Type: array
Default: []

Filter for custom requests to create, replace, modify, delete a collection item.
Multiplied by the filter to read from the field 'opts.security_filter_read'.

The format of the value corresponds to the format of the 'filter' parameter when selecting from a collection (read more).
Additionally, the '$USER' function can be used to provide access to the data of the authorized user on whose behalf the read request is made.
See examples in 'opts.security_filter_read'.

Does not apply to read requests.
Does not apply to requests from users who have the role 'admin'.
Does not apply to requests from scripts.
Does not apply to requests from Token-API if the integration channel is not assigned a user (opts.userid) on whose behalf requests and subscriptions are made, or if a role is assigned to that user or to the channel itself 'admin'.

Applied in HTTP REST API, Websocket REST API.

Field: ext
Mode: inout
Type: object
Compound field

Allows you to extend the compound with arbitrary keys and values

Field: ext.ct
Mode: out
Type: date
Default: generated

Object creation time

Field: ext.lwt
Mode: out
Type: date
Default: generated

Time of last modification of the object

Field: ext.dirty_write_enabled
Mode: in
Type: bool
Default: false

Debug mode switch to speed up the class with the 'runtime' type without using transactions on write operations.

Special filter functions

Table 2. Special filter functions
Function Description and Examples

$USER

Provides access to the data of the authorized user on whose behalf the read request is made.

Examples:

  • ["$USER"] - user ID.

  • ["$USER", "id"] - user ID.

  • ["$USER", "login"] - user login.

  • ["$USER", "ext", "a", "b"] - value of the 'ext.a.b' field of the user account.

  • ["$USER", "security", "mandate"] - value of the 'security.mandate' field of the user account.

Allows you to access deep related user settings - a list of all user roles, including those built on the basis of joining groups, groups via groups, etc.

  • ["$USER", "ROLES"] - list of user roles (elements - rows - role names). Replaced by ["list","role1","role2",…​].

  • ["$USER", "ROLES::array"] - list of user roles (elements - strings - role names). Replaced by ["array","role1","role2",…​]. Used for application in functions 'hasAny', 'hasAll'.

Allows you to refer to the full-depth group list of the user:

  • ["$USER", "GROUPS"] - list of user groups (elements - rows - group identifiers). Replaced by ["list","uuid1","uuid2",…​].

  • ["$USER", "GROUPS::array"] - list of user groups (elements - strings - group identifiers). Replaced by ["array","uuid1","uuid2",…​]. Used for application in functions 'hasAny', 'hasAll'.

Allows you to access the list of subordinates. In the simplest case of full mutual access, all are subordinate to all. This is reflected by a single element "all" instead of several identifiers in the list.

  • ["$USER", "SUBORDINATES"] - list of subordinate user identifiers (elements - strings - user identifiers). Replaced by ["list","uuid1","uuid2",…​] or ["list","all"].

  • ["$USER", "SUBORDINATES::array"] - list of subordinate user identifiers (elements - strings - user identifiers). Replaced by ["array","uuid1","uuid2",…​] or ["array","all"]. Used for application in functions 'hasAny', 'hasAll'.

Allows you to select an aggregate value from the set of all entities: the user, all his roles and groups. The second-order function is used "DEEP".
Third order functions MAX, MIN, ANY are supported. By default ANY - the value is taken from the user’s account, if not there then from one of his roles, if not there then from one of his groups. If there is nowhere, then undefined.
The DEEP function makes sense for accessing the ext and security fields, which exist in all three classes and are filled in arbitrarily (ext is available for modification by the user via API, while security is not).
For example, to find the most affordable mandate possible.

Examples:

  • ["$USER", "DEEP", "MAX", "security", "mandate"] - The maximum value of the 'security.mandate' field from the user account, all its groups and roles.

  • ["$USER", "DEEP", "MIN", "ext", "priority"] - The minimum value of the 'ext.priority' field from the user account, all its groups and roles.

  • ["$USER", "DEEP", "ANY", "ext", "priority"] - value of the 'ext.a.b' field from the user account, if absent, from any of the roles, if absent, from any of the groups.

  • ["$USER", "DEEP", "ext", "a", "b"] - equivalent to the previous one - the value of the 'ext.a.b' field from the user account, if not present then from any of the roles, if not present then from any of the groups.

The field path can be of arbitrary depth - a comma-delimited list of nested keys.
When the next key is a string, the current value is treated as an object; when the key is a natural number, the current value is treated as an array.

If the specified field or path is not found, the undefined value is substituted, which can be processed by the 'isnull', 'isnotnull' functions and correctly converted to the 'NULL' value when making queries to specific DBMS types.

Depending on the type of the detected value, the transformation to the final result is performed: numbers remain numbers, strings remain strings, Boolean values remain Boolean values, arrays are replaced by ["list","elt1","elt2",…​], objects are replaced by undefined.

A complex expression example:

["or",
  ["in",
     "master",
     ["$USER","ROLES"]
  ],
  ["and",
    ["in",
      ["property","status"],
      ["list","in_work","in_review"]
    ],
    ["==",
      ["property","responsibleUserId"],
      ["$USER", "id"]
    ]
  ]
]

See also

API

  • Personalization
  • Increase font size
  • Decrease font size