Disk load

SSD vs HDD

The platform has 6-7 kinds of ROM (file system) usage. Each of them determines a particular read intensity, write intensity, the ratio of these indicators, the storage duration of written data, the frequency of access to it, the criticality of read access speed, and the criticality of write access speed.

These usage kinds are bound to specific (different) paths in the file system. More precisely, they are bound to aliases, and aliases in the configuration are bound to file system paths inside the container. When installing the system on a server, a docker volume is created for each of these partitions and attached to the container. Thus each of them can be mounted on different partitions.

If all disks are SSD with a high TBW value — that is good. It eliminates delay problems due to an "overflowing access queue" at the root. If there is a choice, then SSD.

Large-capacity server SSDs are quite expensive (of course, this is not about 1 TB but about a larger size and higher TBW), so HDD for some cases is a compromise.

  • On SSD — all the working infrastructure where there is a lot of reading, a lot of writing, but small volumes or a rewrite cycle — logs, conversation recording, mixing, object DB, script files, autoprovision, temporary web server files, and a hundred others,

  • On HDD — everything that accumulates. Conversation recording archive (if they are not sent to external S3 or NFS storage), log journal archive, possibly an archival DB.

The SSD partition is for fast access and intensive rewriting (despite the fact that SSD lifetime directly depends on rewrite intensity). Statistics of SSD usage by the platform show that writes are an order of magnitude more intensive than reads. This must be considered when choosing disks. Each intensively loaded system server can write around 100 GB of log journals per day, and as much again in the infrastructure. With conversation recordings it fits within 1 TB (rough estimate). Thus the lower segment of server SSDs with TBW ~ 7 PB — lasts at least 20 years.

The DB gradually rewrites its volume. The DB can be placed on HDD, but with intensive DB load this very factor will slow further system performance progress. If you are ready to spread different domains, and even collections within a domain, across different DB servers, then HDD may be an acceptable option. Also, speed can be increased by combining several HDDs into a long RAID-5 or its derivatives.

Examples

If postgres DB is used intensively and the HDD is loaded at 100%, the next read request may be delayed. If the system’s DB work intensity is like this, the options are: split the DB into parts and move them to different servers for different domains/collections, organize RAID-5, or move the DB to SSD.

At dialog creation with recording, a file is opened for writing synchronously. If the disk is loaded and pauses for more than 3 seconds, the call will not happen. Recording is done by servers with MG instances, and disks are loaded proportionally to the placement of instances across servers. If recording is done to an SSD partition, the problem is practically impossible with any number of calls — the computational limits of one server are reached earlier. With HDD — the opposite.

The system working directory is used to host the mnesia object DB. Therefore the working directory — on SSD. The ability to write quickly without a queue is almost critical for data integrity. If mnesia shares a disk with other processes, it may interfere with them, although it will not hinder the system itself — writes are always deferred, and reads are performed entirely into RAM at system start.

When a conversation has been mixed, it is moved to connected S3 or NFS storage or distributed across its own servers if storages are not connected. Here the queue interferes with no one, and access to listen to a conversation or view a log journal in the archive is not time-critical. HDD handles it calmly.

Conversation recording storage location

For HDD a size of 1 TB is a conditional minimum so that lack of free space does not bother for some time, and with statistics for several months one can orient: what real volume is needed for the prospect of storing log journals for N weeks and conversation recordings for M years. Obviously less is possible, but one must orient to the planned load.

Connecting external storage for archival conversations is more efficient and better managed in large systems. When using external storages, new storages can be created regularly, and current files will be moved there. If previous storages are still available, previous files will be read from them.

If they are left by default on servers with the platform, the storage volume is effectively single (since it is limited by docker volumes; this can also be developed, but it is not a priority). Data older than N years can be deleted. Or, if the server is physical, combine several disks into one volume — this must be done before installation or when reinstalling the system on a specific server.

File system partitions and their minimum sizes

The layout is the same on all servers, but in reality the load depends on the configuration.

The current installer provides the following root paths for partitions:

  • /usr/lib/era — component and application files. Including update archives. ~20 GB (SSD or HDD) OK.

  • /var/lib/era — working directory: object DB, mixing. ~80 GB OK.

  • /var/log/era — real-time log journals. ~100 GB.

  • /var/lib/era_files/rectemp — real-time recordings, stored for 6 hours. ~20 GB SSD.

  • /var/lib/era_files/recpath — mixing results, local per-domain storage (if external is not configured) — NFS storage is assumed. But in one-two server systems NFS is not used, and this is its own disk. It is for long-term recording storage in this case. ~20 GB if moved to external storages. Or ~1 TB if archives are kept here without NFS.

  • /var/lib/era_files/local — local directory for storages. Temporary files — extensively used by the web server. By default it hosts the attachment directory of all product data model collections (in particular, email messages). HDD is sufficient as long as it handles the load. If the system is extensively loaded in all directions, temporary files should be on SSD (because intensity is high), and collection attachments moved to other disks (because volume is large).

  • /var/lib/era_files/syncroot — 20 GB SSD. In reality 1-2 GB. This is data of the automatic synchronizer between servers. It holds script media files, public web pages, certificates, patches, autoprovision templates, static media files (for conferences, hold, parking, voicemail, numeral announcements), attachment files for domain entities (product layer custom applications, product layer microservices), and various project files — if storages were specially created here or placement here is done in scripts.

  • /var/lib/era_files/logstore — * HDD — archival log journal storage. Its size is configured in the configuration. 500 GB conditionally for 1 week at average intensity.

  • /var/lib/era_files/siteshare — NFS. Used in project settings as shared storage between servers of one site. For example in scripts — placed in one script, used in another.

  • /var/lib/era_files/globalshare — NFS. The same, but in multi-site systems between all sites.

  • /var/lib/era_files/a — additionally mounted partitions on different media so that load can be moved from one disk to another by changing the configuration on the fly. That is, they have no data of their own but can take over instead of the above.

  • /var/lib/era_files/b — same

  • /var/lib/era_files/c — same

Division into such small partitions carries potential difficulties, excluding the possibility of redistributing space. For example, somewhere there are no object DB roles at all, but a lot of log journals are written (for example SIP signaling traffic).

These paths are inside the container. On the host they all map to volumes during installation; by default host paths are inside /opt. The /usr/lib/era directory is not moved to a volume and remains inside the container, which is somewhere in the host system folders (/var/lib most likely)

By the way, real-time logs can be written to HDD, but ONLY if the HDD is dedicated to this task. If it is combined with the DB or with recording, harmful cross-dependencies on the access queue immediately appear.

Asynchronous writes for real-time logs, recording archive, log journal archive, object DB, mixer. Synchronous writes — for postgresql DB, dynamic object model attachments. Synchronous reads — for media files, dynamic model objects.

File system and inode

With standard layout, for example in ext4, the inode count limit must be remembered. If many small files are placed on a disk, inodes may run out before free space. Each directory and each file occupies one unit. For example, if the disk stores email message history — regardless of each message’s volume, more than 100 KB will actually be consumed per message due to several directories and the raw message data file.

The platform monitors inodes on disks where its partitions are located. At monitored thresholds the administrator receives a message within system_state.