Storage connection (storage)

Description

Defines access parameters for a specific storage used by the platform.
Defines the type (type). Data can be located in PostgreSQL, Kafka, or ClickHouse clusters. Files in s3, fs, nfs, fsync.
Sets the code (instance) and allows referencing it when configuring data model classes.
An element with instance = auto is created automatically.

Fields

Entity structure
{
  "id": uuid,
  "name": str,
  "instance": str,
  "params": object,
  "opts": {
    "title": str,
    "comment": str
  },
  "ext": {
    "ct": date,
    "lwt": date
  }
}
Table 1. Fields
Specification Description

Field: id
Mode: inout
Type: uuid
Default: generated

Identifier. Can be specified at creation, otherwise generated by the system.

Field: type
Mode: in
Type: str
Default: required

Code. Used to associate with other entities.

  • postgresql. A PostgreSQL relational DB instance. Several instances with the same instance value are considered replicas of each other.

  • kafka. A Kafka cluster. One entity defines several brokers of the cluster at once.

  • clickhouse. A server of the ClickHouse analytical columnar DB cluster. Several instances with the same instance value are considered servers of one cluster.

  • s3. S3 file storage. Can be used by the data model server (dms) and for storing call recordings.

  • nfs. File storage on a network disk accessible from any node as a folder on the current server.

  • fs. File storage organized on several servers where file server role instances are located (or middleware in its absence).

  • fsync. File storage in a synchronized folder. Synchronization to all servers of the site and other sites serving the domain occurs within 10 seconds.

Field: instance
Mode: in
Type: str
Default: required

Storage name. Used when binding to classes.

Storages with the value auto are generated and applied automatically if they are not explicitly present in the collection.
Data is placed in storages of type postgresql (for tabular data) and fs (for attachment files).

Field: params
Mode: in
Type: object
Default: required

Parameters of the specific storage.

Example (postgresql)
{
  "host": "192.168.1.221",
  "port": 5432,
  "login": ".....",
  "pwd": ".....",
  "database": "my_database",
  "max_connections": 50
}
Example (kafka)
{
  "endpoints": [
     { "host": "192.168.1.201", "port": 9092 },
     { "host": "192.168.1.202", "port": 9092 },
     { "host": "192.168.1.203", "port": 9092 },
     { "host": "192.168.1.204", "port": 9092 }
  ],
  "send_mode": "async",
  "partition_count": 2,
  "replication_factor": 2
}
Example (clickhouse)
{
  "endpoints": [
     { "host": "ch1.zone1,my_domain" },
     { "host": "ch2.zone1,my_domain" },
     { "host": "ch1.zone2,my_domain", "port": 8125 },
     { "host": "ch2.zone2,my_domain", "port": 8125 }
  ],
  "scheme": "https",
  "port": 8123,
  "login": ".....",
  "password": ".....",
  "cluster": "my_cluster"
}
Example (s3)
{
  "bucket": "my_bucket",
  "endpoint": "https://my_cloud.net",
  "prefix": ".....",
  "keyId": "my_key",
  "secretKey": "....."
}

Field: opts
Mode: in
Type: object
Compound field

Field: opts.title
Mode: in
Type: str
Default: empty

Arbitrary title

Field: opts.comment
Mode: in
Type: empty
Default: "

Arbitrary comment

Field: ext
Mode: inout
Type: object
Compound field

Allows extending the compound with arbitrary keys and values

Field: ext.ct
Mode: out
Type: date
Default: generated

Object creation time

Field: ext.lwt
Mode: out
Type: date
Default: generated

Time of last modification of the object

Connection parameters for storage types

Table 2. Connection parameters for storage types
Storage type Parameters

postgresql

* host — host;
* port — port;
* login — login;
* pwd — password;
* database — database name;
* ssl (optional) — secure connection flag: false | true. If ssl is not used, omit the parameter.
* auth_method (optional) — authorization mode when using ssl: cleartext, md5, scram-sha-256.
* max_connections (optional) — number of simultaneous connections, 10 by default; applied separately for user calls and for calls from the system and from tokens.
* endpoints (optional) — list of several connection points. The 'host' parameter and optionally 'port', 'login', 'pwd' can be specified in the list under the 'endpoints' key when there are several connection points. By default there is one connection with parameter values from the root section.

Secret (pwd, endpoints.*.pwd): ciphertext when read; the plaintext value is not returned via API and is decrypted by the service when used (Encryption and keys).

kafka

* endpoints — list of connection points. Each endpoint object contains:
. - host — host.
. - port — port. The standard KAFKA port is 9092.
* endpoints_for_clickhouse (optional) — list of connection points for clickhouse tables. Unlike the communication platform, clickhouse can connect in an alternative way (for example, without authorization over the local interface). This list is passed to it when creating tables with ENGINE=KAFKA. Each endpoint object contains: host, port.
* send_mode (optional) — message sending mode to the KAFKA broker. Value variants: 'sync' or 'async'. Default 'sync'.
* partition_count (optional) — number of topic partitions. It is useful to set a value matching the connected cluster. By default it is taken from the class, but its default value reflects the minimal single-server KAFKA deployment.
* replication_factor (optional) — number of topic replicas. It is useful to set a value matching the connected cluster. By default it is taken from the class, but its default value reflects the minimal single-server KAFKA deployment.
* security (optional) — secure connection settings. Object: {protocol, sasl_mechanism, login, password}. Variants of the 'protocol' field: plaintext, sasl_plaintext, ssl, sasl_ssl. For sasl protocol variants, an optional 'sasl_mechanism' field can be added with value variants: plain (default), 'scram_sha_256', 'scram_sha_512'. Also 'login' and 'password' (empty strings by default).
* domain (optional) — for cross-domain and cross-system processes, the ability to connect to send information to the database topics of another domain. For example, this can be useful for using a single archive data storage not divided by domains, or when organizing an ETL process to transfer data from PostgreSQL archive collections to clickhouse. The parameter is used only by the dynamic class service and is not used by the arbitrary event sending script component.
* topic (optional) — for cross-system processes when using the event generator from scripts, the ability to connect to send information to a specific externally created topic. For example, this can be useful for generating arbitrary events into a corporate KAFKA broker system from project scripts. The parameter is used only by the arbitrary event sending script component and is not used by the dynamic class service. In the latter case, the class.opts.custom_topic field is intended.
* ensure_topic_mode (optional) — topic check mode when connecting before the first send. 0 - do not check, do not create; 1 - check existence; 2 - create if absent. By default, the "Event generator" component and event sending use mode 1, and the DMS dynamic data model management service uses mode 2. Mode 0 can be useful for accounts with limited rights.

Secret (security.password): ciphertext when read; the plaintext value is not returned via API and is decrypted by the service when used (Encryption and keys).

clickhouse

* cluster — name of the clickhouse cluster from its configuration.
* host — connection host.
* login — login.
* password — password.
* port (optional) — connection port. Default 8123.
* scheme (optional) — connection scheme. Value variants: 'http' or 'https'. Default 'http'.
* endpoints (optional) — list of several connection points. The 'host', 'port', 'login', 'password', 'scheme' parameters can be specified in the list under the 'endpoints' key when there are several connection points; parameters not set are taken from the common section. By default there is one connection with parameter values from the root section.
* domain (optional) — for cross-domain and cross-system processes, the ability to connect to send information to the database topics of another domain. For example, this can be useful for using a single archive data storage not divided by domains, or when organizing an ETL process to transfer data from PostgreSQL archive collections to clickhouse.

Secret (password, endpoints.*.password): ciphertext when read; the plaintext value is not returned via API and is decrypted by the service when used (Encryption and keys).

s3

* bucket — bucket name.
* keyId — key ID.
* secretKey — secret string.
* endpoint (optional) — storage connection point (URL).
* region (optional) — region name for connecting to Amazon S3. The connection point is determined automatically.
* prefix (optional) — path prefix in the storage structure. Default empty string.
* timeout (optional) — HTTP request timeout in milliseconds. Default 10000 ms.

Secret (secretKey): ciphertext when read; the plaintext value is not returned via API and is decrypted by the service when used (Encryption and keys).

nfs

* configkey — configuration key defining the disk directory associated with the storage (inside the container). Value variants: 'globalshare', 'siteshare', 'local'.
* prefix (optional) — path prefix in the disk directory. Default empty string.

fs

* prefix (optional) — path prefix in the disk directory. Default empty string.

fsync

* prefix (optional) — path prefix in the disk directory. Default empty string.