v/1.12.0: detailed description

Upgrade to 1.12 - only from version 1.11.9 and later: first update the system to 1.11.9.

Version 1.12 changes the types of date-time fields in ClickHouse tables, masks psk in local.cfg.json and encrypts some of the settings secrets.

Versions before 1.11.9 do not read this data, 1.11.9 reads it - the rollback to it is saved. Rollback to version 1.11

Brief description — v/1.12.0.

Requires attention when updating (BREAKING CHANGE)

  • The administrator bot replaces the Telegram monitoring bot:

    • microservices im (on the central site) and dms are needed to work;

    • the bot and connected administrators of the previous Telegram bot are transferred automatically, including from versions up to 1.10; system_telegram_bot and telegram_bot_token parameters have been replaced with admin_imi_options.

  • If payloads_audio_offer is not set (for any role, including esg and sg), the G729 is no longer offered: add G729/8000 to the parameter if needed. The order of codecs in SDP may change: it is now determined by the order in the configuration, rather than the previous constant platform order (for example, [PCMA, PCMU, G722, CN] gives PCMA first instead of PCMU).

  • Codecs in the SDP of initiated calls (the payloads_audio_offer parameter of the roles b2b, conf, esg, ivr, prompt, sg, as well as payloads_video_offer) are offered in list order; basic PCMU, PCMA and telephone-event not listed are added after those listed. The - prefix excludes the codec from being added (- - all unspecified basic ones), -- - also from the SDP of an incoming call transferred to the other leg (with -- - in list order). CN and telephone-event are placed at the end. Read more in Codecs and Media Negotiation.

  • The provider (opts.payloads_audio) telephone-event can be excluded: -telephone-event/8000 or -*.

  • The list of provider codecs (opts.payloads_audio) is used only on the provider’s shoulder: internal shoulders are not limited to it (transcoding if necessary), the offer inside the platform contains the provider’s codecs from the list in the order of the provider. Without a list, with reinvite = 0, the provider’s offer is transmitted internally with its codecs, without the addition of payloads_audio_offer esg. If the provider’s SDP (including re-INVITE and INVITE with Replacements) does not have a codec from the list or there is not a single codec known to the platform, esg responds with 488 Not Acceptable Here instead of 415.

  • Codec names in parameters and in the provider are specified insensitive to case.

  • Provider: G723/8000 in opts.payloads_audio is not supported - when adding an error, the previously saved value is discarded.

  • Recording servers: when audio_formats is specified, exactly the specified codecs are offered in list order.

  • Secrets in domain settings are encrypted when saved and are not returned in clear form: keys and secrets of yandex_cloud, sber_salute, azure_cloud (including in accounts) and record_asr_options servers, password in project_postgresql_connstr. Previously saved values ​​are encrypted the next time the setting is changed. Scripts that read these values ​​through the domain.settings system variables receive them in encrypted form: the script sets the connection parameters independently.

  • Storages of type s3 (/rest/v1/domain/storages): the secret key secretKey is encrypted when saved and is not returned in clear form; previously stored keys are encrypted the next time the storage is changed.

  • Date functions in scripts (year, month, day, hour, minute, second, millisecond, dayofyear, dayofweek, weekofyear, tick, datetimeutc, datetimelocal, localtoutc, utctolocal) with an invalid date end with the error “Invalid datetime” instead of substitution 2000-01-01; valid_date for an invalid date returns false. An explicitly written date 1970-01-01 with zero time is valid.

  • Eval nesting in scripts is limited to 16 levels (error “eval: nesting is too deep”).

  • Copying a script (Save As and ext.copied_from): Only the ID of an existing script of the same type is accepted.

  • The join aggregation function (aggr parameter) now takes a delimiter as the first argument, like the general purpose join function and as stated in the help: ["join",", ",["property","name"]]. Queries with the same argument order need to be corrected. In PostgreSQL collections, join also joins non-string values.

  • The name of the dynamic data model class when created and renamed must contain a section (section/name); names without a section are reserved for the static model. Existing classes are not affected.

  • REST domain (/rest/v1/domain/domains): the solution field is no longer returned or accepted; deleted /rest/v1/domain/solutions.

  • IAM rules with authenticated mode again require authentication: a REST request without a session to such routes receives 401.

  • The web designer application event via websocket (send_app_event) checks user roles in the same way as HTTP access to the application; without a suitable role - error “Invalid app name. "Role mismatch."

  • SIP user without registration (reg=false): the number of devices when created and modified cannot exceed the lic.devices value set for it (the check did not work before). Saved objects are loaded as before.

  • An implicit call from conference to conference is blocked (the check did not work before).

  • Receiving mail via IMAP: the ssl parameter is taken into account (true by default; port 993 - always TLS); Previously, the connection was always established via TLS.

New

  • Administrators bot:

    • works in Telegram, MAX and VK through messenger accounts of the master domain; Only instant messengers with authentication of incoming messages are used;

    • the administrator connects the chat with a one-time code, confirming it in the administration application; the password is not sent to the messenger;

    • each administrator can exclude system status sections from notifications and select the message language (/lang command);

    • The status file is sent when problems appear and change; if problems remain unchanged, a reminder is sent; daily report time is configurable;

    • commands with parameters can be selected from the messenger menu;

  • The script components “Speech synthesis (Yandex Cloud SpeechKit)” and “Speech recognition (Yandex Cloud SpeechKit)” have been transferred to API v3 (gRPC). Recognition is performed streaming as the recording progresses, synthesis is played back without waiting for completion. Added new voices and synthesis roles, volume, voice pitch, caching; for recognition - normalization and literary design of text, single phrase mode, settings for determining the end of a phrase, text without normalization into a separate variable. The settings of existing scripts are applied without changes: Alyss’s voice is replaced by Dasha, emotions not supported by voice are not transmitted, the themes maps, dates, names, numbers are replaced by general. In setting up the yandex_cloud domain, service addresses are specified by the grpc_tts, grpc_stt fields; the uri_tts and uri_asr_short fields are no longer used.

  • SIP over WebSocket (ws/wss): CRLF-keepalive supported; Previously the connection was closed.

  • SIP over WebSocket (ws/wss) and mobile devices:

    • closing the previous connection of a device that reconnected with the same +sip.instance and reg-id no longer unregisters it;

    • SG checks WebSocket ping connections (role parameter sg ws_ping_interval, default 25 seconds): a device that goes missing without breaking the connection is removed in 50-75 seconds instead of three minutes;

    • a call to a lost connection immediately goes to another device connection, and without it the caller receives 480 instead of 503 (for redirection - condition dnd);

    • backup call to a device with multiple connections starts after 2 seconds and with the same Contact; before — 408 after 30 seconds;

    • a device that reconnects during an incoming call receives a call cancellation to the new connection, and UPDATE without SDP responds with 481 if the call is already canceled;

    • mobile application with push notifications whose connection is alive but not responding: after 2 seconds the device is woken up by a push notification and receives a call through the new connection; before — 408 after 30 seconds;

    • user failure (486, 6xx) on one device connection immediately ends the call on that device; The user’s other devices continue to ring.

  • Collection of trn logs and call sequence diagram (/api/monitor/v1/trace):

    • available even after clearing the connection cache (about 3 hours): call connections are taken from the callid logs of microservices (depth - parameter days), logs already deleted from the nodes - from the logstore if resources are free (parameter logstore);

    • the order of the arrows corresponds to the order of events: the response is not shown before the request, the discrepancy between microservice clocks is taken into account;

    • the assembly loads microservices less; parameter time (call time) reduces it further.

  • Master domain subscriptions to the event class callevents or confevents (or a separate event of the class) with objects any receive events from all platform domains: a listener for all telephony events via the websocket Token-API. The master domain’s subscription to these object classes is rejected.

  • Aggregating queries to collections (aggr parameter): added the count_distinct function - the number of different values in a group, for example, unique entities by day.

  • Static files of the web server (help, admin panel, role-playing application resources) support HTTP Range: audio and video rewinding works.

  • SG, ESG, redirect and WS begin to accept requests for addresses from the sip_ifaces, http_ifaces, https_ifaces settings that appeared on the server after the role was launched, within 30 seconds; Previously, a node restart was required.

  • External Web Server Hosts (ws/ExternalRoutedHosts): The archive_referers record or route mode saves client referrals (IP, URL, utm tags, Referer, User-Agent) to a new collection ws/ArchiveReferers. Off by default.

  • Messenger channels (uc/imchannels) are also available in the master domain.

  • Messenger messages (im/Messages): sendOptions (disable_notification, parse_mode) are taken into account in Telegram.

  • Added info and message operations to /rest/v1/uc/sipusers/<id>/device; the help describes resync, event, headers, content.

  • Service tasks (service/svctasks): the dtstart and dtstop fields are optional - an unspecified value does not limit the period of activity. A regular task without them is executed according to a schedule, a one-time task is executed immediately after switching on. Previously, the current time was substituted, and a task created without these fields would not be launched. The dates of existing tasks do not change.

  • System state: comparison of platform files on servers (version_files_diffs, version_all_files_diffs, levels 1–3) is performed in fractions of a second instead of tens of seconds — file size and modification time are compared, backup and build directories are not taken into account; On a single-server system the comparison is not performed.

Fixes and improvements

  • BYE the caller before answering (after 180/183) cancels the call in the same way as CANCEL; before - 481, and the call continued.

  • ESG: requests within a conversation before the ACK are queued, as in B2B, and processed sequentially when all preparatory states are completed.

  • B2B: INFO without DTMF is sent to the other side as SIP INFO (was 408).

  • ESG: fixed transmission of X-Era-Extaccount with urlencode for values, allowing to avoid special characters in the providers' code.

  • Subscription (BLF) to external numbers from the settings blf_external_numbers and chief_mobile of the manager’s kit works; previously rejected.

  • SIP: fixed presence notifications (PIDF), including for BLF Cisco phones (cpim-pidf+xml); REGISTER without Contact via SG receives 200; SUBSCRIBE with Expires 1..29 — 423 Interval Too Brief; to the outer direction - route error instead of 500.

  • B2B: fixed reinvite and dialog termination when media migration fails; if the domain center did not respond to the request for forwarding rules, the call is processed without forwarding (previously - a call processing error); a transferred call (REFER) when the remote B2BUA is not available is executed without connection with the original dialogue (previously - an error); INVITE+Replaces to an external direction returns 403 when there is a route error.

  • ESG: normalization script without variables or interrupted does not delay the call by 10 s; fixed RADIUS billing for a server without a port.

  • IVR: fixed Replacements in the ringing state and before the response (the replaced INVITE responds with 487); a media mode mismatch on REFER is returned to the script as operation_failed.

  • Music on hold: if music_on_hold is missing or incorrect, the default melody plays.

  • Fixed: esg with reinvite = 0 responded to re-INVITE and INVITE with Replaces codecs that were not in the proposal. Now the response contains only offer codecs, in the absence of general ones - 488 Not Acceptable Here (re-INVITE is rejected without disconnecting the call).

  • Fixed: when sending re-INVITE via esg without transcoding, codecs removed by the new proposal remained in the offer and response.

  • The SDP response contains dynamic formats (eg telephone-event) under proposal numbers (RFC 3264). If the calling parties advertise different numbers for the same codec, the media gateway matches them by codec name.

  • Media Gateway: improved mechanism for switching sending to another socket when the incoming stream changes.

  • Fixed: substituting {A} in the SIP user display name modifier (displayname of an address book contact by number) always gave an empty string.

  • SG, mutual supervision of instances (reregister mode): when an SG instance crashes, a command for unscheduled re-registration is again sent to the devices registered through it; Previously, after a service process crash, the mode could stop working until the node was restarted.

  • ACME for automatic certificate generation: failed preparation is repeated after 10 s; Fixed restoring certificates from backup.

  • Fixed the selection of the leading site in an intersite quorum when dividing a network.

  • DMS microservice:

    • less processor load during idle time: schedulers do not keep cores in active waiting;

    • subscriptions to model events (modelevents) with the filter “property equals value” (including inside “and”) are selected by index, filters are collected once when subscriptions change - with hundreds of subscriptions per class, preparing notifications is tens of times faster;

    • class processes are created during the first operation, added when there is a long or long wait for operations in the queue (short bursts of fast operations are not taken into account) and are eliminated after a lull without stopping processing (the order of operations on one entity is preserved): idle classes do not hold processes (role parameter class_workers_lazy and parameters class_workers_grow_*, class_workers_shrink_*);

    • reads of class collections are transferred to the least loaded read process (previously - random);

    • in case of short-term unavailability of a class (pause during overload, cleaning, loading), requests wait up to 5 seconds in the order they were received, and do not receive an error; repeated requests to the same entity are performed in the order of the original request;

    • synchronous requests to a class that came during its loading or pause during overload are executed after it in the order of arrival (previously they were lost without a response).

  • The domain license (dql.domains) when reduced or exceeded no longer unloads all child domains (previously it was enough for the reserved licowned value to exceed the new license):

domains created earlier work within the limits of the license; the rest, as well as domains whose allocated licenses do not fit into the available ones, do not start and are displayed blocked (opts.isblocked, ext.licblocked), the blocking is not saved; the administrator can block a running domain and unblock a domain blocked by license; licenses of blocked domains are not consumed;

  • when increasing the license, domains are launched automatically; licowned exceeding the license is reduced to it.

    • Fixed HEAD requests to REST: public resources responded with 401, /public/version - 405.

    • Fixed API responses that gave 500 or a break: list of sessions when the domain did not respond; query DC collection by field for empty collection; stopping the script when the script node is unavailable; rest on websocket with classpath not collections; to an unknown binding command and switching the conference in an invalid state - an immediate error, not a timeout.

    • Fixed delivery of some events via websocket subscriptions and subscription to model events (modelevents) with a short class path.

    • Websocket proxying takes into account the port from the destination address host:port.

    • The CCS connection_failure event contains sipcode and reason.

    • The statesevents.user_state_changed event is delivered to subscriptions (webhooks, websocket, Kafka) in the correct JSON with the presence field; Previously it came as invalid_data.

    • Webhooks: a subscription with objects of the form {"type": "sipuser", "sipuser": "sip1"} again filters events (since 1.11.0 no events were received for such subscriptions), types other, conf, provider, user are also supported; subscription to modelevents via webhooks is rejected - websocket is used for them; in the GET response, the expires field is the remaining subscription time in seconds.

    • Websocket: responses with an error containing details (for example, incorrect password when logging in) again come with a code and error_details (internal error 1501 was returned since 1.11.0).

    • Websocket: login using an unfound or expired session (sessionid, rsessionid) without a login and password returns the error “Session not found or expired.” (msg_code 10025) instead of the domain requirement.

    • Webhooks subscriptions (/rest/v1/service/webhooks): expires greater than 86400 is reduced to 86400; the applied value is returned in the response in the expires field and determines the extension period.

    • Subscriptions to model events: a string mask (exargs.mask) together with the class security policy does not lead to a subscription error; The ext.http_notify_format web hook format is also used for /api/token/v1/<token>/subscribe, the unknown format value is the JSON body.

    • Fixed: a request to a web server with a body of unknown length (Transfer-Encoding: chunked, HTTP/2 without Content-Length) when limiting the request rate from the address was enabled, ended with error 500.

    • Websocket API (rest): hc and setup operations for classes with access restrictions (x_security) for a user without the admin role — hc returns a class change mark, setup is disabled (previously the request ended with an internal error).

    • REST API of the dynamic data model: when restarting the DMS microservice, including while classes are running, requests receive error 503 (service unavailable) instead of 500.

    • Fixed: The system/audit/RestrictedLogons, system/licensing/Indicators and telephony/ArchiveDialogs service collection classes were recreated when changing domain classes and when starting servers.

    • Collections with transactionlog storage (Kafka + ClickHouse):

  • datetime is stored in UTC with milliseconds, as in PostgreSQL; reading and filters by interval do not depend on the time zone of the ClickHouse server. Tables created earlier are transformed when the class is loaded: the partition field remains accurate to the second;

  • unspecified fields of types uuid and enum are returned as missing, rather than as a null identifier and an empty string;

  • ClickHouse login and password are sent in the request headers: the password can contain any characters.

    • Fixed downloading a file from properties of the filelink type in multi=true in collections of the dynamic data model.

    • Requests to a collection of a dynamic data model that came during the loading of its class are executed after loading; Previously, with several such requests, the part ended with a timeout.

    • Dynamic model collections with ets storage: new classes are launched again (from 1.10.5 the class was not loaded, requests received “DataModel server is not loaded”); page-by-page snapshot reading returns pages in sorted order and does not mix selections with different filters.

    • Login through a service authentication script (by password or token) works with any form of the script result; Previously, in some cases the login ended with an internal error.

    • Login using a password through a service authentication script: a login accepted by the script, but not in the domain, is rejected as an incorrect password; Previously, a request to /api ended with error 500, REST ended with error 500 “User is gone”.

    • /api requests without a session accept the login domain in the domain parameter (together with login, pwd); td remains a synonym and, when both are passed, defines the entry domain.

    • Calling the server side of the shelf application (service application script) via HTTP /api/roleapp/v1/<appkey> works; Previously, the request always failed.

    • The dc_read and dc_find script functions use the id field to find objects of those classes where the record key is not id (users, groups, roles, etc.); Previously, an empty result was returned.

    • Scripts: script components have been corrected in terms of tolerance to incorrect values, which lead to interruption of the component and termination of the script.

    • Scripting: The LDAP component now supports multi-line filters and not(…​), the mail receiving component decodes =?charset?…​?= headers.

    • Receiving mail via IMAP: closing the connection by the mail server does not crash the script component, which was previously capable of crashing the script.

    • Proxying external web server hosts (ws/ExternalRoutedHosts): fixed cookie transmission; cookies and platform session token are not transferred to the destination server. If the destination server is the platform itself (for example, logging in through an external OAuth provider on the route host), the session is sent with the route parameter pass_session.

    • Proxying external web server hosts: the cert route parameter (verification of the destination server certificate: verify_none, PEM or its path) is used in proxy_http and proxy_websocket; was previously ignored.

    • Several vulnerabilities were discovered and corrected when working with files through the API of authorized users and integration channels: temporary websocket session files (file and subdirectory names are limited to the session folder); file APIs (directory traversal via .. and symbolic links); system API of role applications (bypassing url_mask with double-encoded segments).

    • Fixed deleting a folder in file storage (REST fs): without a specified root, empty directories above the root could be deleted.

    • Fixed a vulnerability in the script editor: operations with attached script files only accept the file name, without the path.

    • The web phone API vulnerability (/api/webphone/v1) has been fixed: a request with a login like sip:…​ was executed without authorization and made it possible to check the passwords of SIP accounts of any domain; command clicktocall removed.

    • Uploading a file via a link (websocket token-API, file_upload_static): the destination path and file name cannot contain the . and .. segments - previously the file could be written outside the specified section.

    • Anti-brute force protection: an unknown token (Authorization: Bearer) on REST API routes with a session is counted as a failed attempt, as an incorrect password (previously it was not counted if the session cookie was not passed).

    • Fixed: downloading a folder as an archive (DOWNLOADZIP) occasionally returned an empty response; simultaneous downloads on the same server could receive someone else’s archive.

    • Errors in creating directories, copying and deleting files (NFS storage, mixer, temporary files) are no longer reported as success: for example, if copying a conversation recording fails, downloading it returns an error rather than an empty file.

    • Fixed: data synchronization between systems (ETL, survival mode) did not transfer address book contacts.

    • Data synchronization between systems (ETL, survival mode): objects and attachments not recorded in the storage are taken into account - the class is included in the list of unsuccessful ones in the standby mode state, a warning is written to the log with the first error; The number of repetitions of an unsuccessful class is limited (previously, a class with a permanent error was repeated indefinitely).

    • Fixed platform installation with multiple database connection strings (installation failed).

    • Fixed update preparation if the update file name contains special characters.

    • Product installation and update are not interrupted by an error if any domain object is updated with warnings.

    • Diagram of a call based on logs of versions before 1.9.7: a message without an address does not make the diagram erroneous.

    • Dialog log (SORM3/ArchiveDialogs): events are given a real date instead of 1970-01-01.

    • Kafka for generic events: random and random_by_key partitioning modes work.

    • The service for distributing telephony provider accounts among ESG instances now takes into account the excludedomains ESG configuration option.

    • Survival mode: fixed token verification for multiple addresses.

    • Voicemail: Fixed message being marked as listened via REST.

    • Voicemail: nightly cleaning no longer crashes on an empty directory, previously it was written to the log.