Rollback to version 1.11
Overview
Versions 1.12 and newer change data on first start that versions 1.11 to 1.11.9 do not read:
-
class tables with ClickHouse storage (transactionlog): date-time fields get the type
DateTime64(3, 'UTC'), the partition field —DateTime('UTC'); -
psk in
local.cfg.jsonis masked; -
secrets of the
yandex_cloud,sber_salute,azure_cloud,record_asr_optionssettings, the password inproject_postgresql_connstr, and the s3 storagesecretKeyare encrypted when saved.
Version 1.11.9 reads this data. Rollback from 1.12 and newer is possible only to 1.11.9 and newer.
Update procedure
-
Update the system to 1.11.9 with a warm update (in the same container).
-
Update to 1.12 with a warm update. From versions below 1.11.9, updating to 1.12 and newer is rejected.
-
Update to 1.13 — according to its release notes (backups before updating are mandatory).
After warm updates the directories of both versions remain in /usr/lib/era_platform, and rollback is performed by switching the symlink.
Rollback in the same container
On each cluster server:
docker exec -it <ContainerName> ls -l /usr/lib/era_platform/
docker exec -it <ContainerName> ln -sfn /usr/lib/era_platform/<Directory1.11.9> /usr/lib/era_platform/era_current
docker restart <ContainerName>
The 1.12 and newer startup script does not start nodes if the version in /usr/lib/era is below 1.11.9.
The container log then contains the line ERA server is not started: platform in /usr/lib/era is older than 1.11.9.
Rollback on another container
After a cold upgrade to 1.12 and newer, rollback starts a container from the 1.11.9 and newer image rather than the previous container of an earlier version: there is no protection against starting it.
When rolling back from 1.13, the DB and /var/lib/era backups made before the upgrade are restored first.
Rollback to a version below 1.11.9
Performed before starting the previous version.
psk (node cookie). The node cookie is set by psk from /var/lib/era/_workdir/erainstaller/local.cfg.json.
Version 1.12 stores psk masked (ha_v=4;…), while versions below 1.11.9 take the value from the file as is: the central node receives an invalid cookie and an incomplete startup command.
Set the original psk value (specified during installation) manually in the file on each server.
Settings and s3 storage secrets. After rollback, values saved in 1.12 are visible as ha_v=4;…: they must be entered again.
ClickHouse. Affects only systems with classes stored in ClickHouse. Without conversion, such classes do not start in versions below 1.11.9 (the partition field change is rejected); data is not lost. The conversion returns field types to the 1.11 form; milliseconds are discarded. A repeated update to 1.12 will convert the types again.
-
Back up the tables (for example,
ALTER TABLE <db>.<table> FREEZE). -
Generate conversion commands with the
gen.sqlquery:SELECT concat('ALTER TABLE `', c.database, '`.`', c.table, '` ', arrayStringConcat(groupArray(concat('MODIFY COLUMN `', c.name, '` DateTime')), ', '), ' SETTINGS mutations_sync = 2;') FROM system.columns AS c INNER JOIN system.tables AS t ON t.database = c.database AND t.name = c.table WHERE c.type IN ('DateTime64(3, \'UTC\')', 'DateTime(\'UTC\')') AND t.engine LIKE '%MergeTree' GROUP BY c.database, c.table FORMAT TSVRawFor a ClickHouse cluster,
ON CLUSTER <Cluster>is added to each command after the table name. -
Execute the commands and verify that re-running
gen.sqlreturns an empty result:CH="clickhouse-client --host <Host> --user <User> --password <Password>" $CH --multiquery < gen.sql > alter.sql $CH --multiquery < alter.sql $CH --multiquery < gen.sql
Distributed tables, Kafka queues, and materialized views are recreated by the previous version itself.
Post-rollback verification
-
The system state has normalized (/system/state).
-
Classes with ClickHouse storage are accessible: reading the collection via the REST API returns data.