Rollback after an irreversible upgrade
Overview
Usually rollback is stopping the new container and starting the previous one (step 5): database changes between versions are backward compatible. A database backup before an upgrade is always recommended.
Some transitions change data irreversibly — they are marked in the version release notes.
Such a transition is from a version below 1.13 to 1.13 and above: on first start the new version changes the forms of storing secrets in the database and files of the working directory /var/lib/era (mic configuration, psk in local.cfg.json), and the previous version cannot work with them.
For such transitions a database backup is mandatory, and rollback is its restoration and starting the previous container; data written after the upgrade is lost.
The procedure for such transitions is below.
The examples below are for the era_11 instance (directory /opt/era_11), the new container era_12, and a server with address 192.168.10.127.
If there are several servers, container and working directory actions are performed on each server, and database actions once.
Working directory options
The new container uses data volumes (syncroot, recpath, a, b, c, local, logstore, etc.) at the previous paths.
The options differ only in the working directory volume lib (/var/lib/era in the container).
| A. Shared working directory | B. Working directory copy | |
|---|---|---|
New container |
The same |
A copy of |
Before upgrade |
Copy the mic directory and |
Copy the entire |
On rollback |
Restore the mic directory and |
Nothing: the previous container works with its own untouched directory |
Disk space |
Megabytes |
The size of |
Option B is more reliable: on rollback the previous container gets the working directory (configuration, local.cfg.json, mnesia tables) in the state at the time of the database backup.
Option A is suitable if there is not enough disk space.
A. Shared working directory
Before starting the new container (the previous one is stopped), copy the mic directory — on servers with the mic role (the directory name is mic<number>@<address>, see ls /opt/era_11/lib/_workdir/), and local.cfg.json — on each server:
cp -a /opt/era_11/lib/_workdir/mic1@192.168.10.127 /opt/era_11/mic_before_upgrade
cp -a /opt/era_11/lib/_workdir/erainstaller/local.cfg.json /opt/era_11/local.cfg.json_before_upgrade
On rollback, after stopping the new containers and restoring the database, restore them:
rm -rf /opt/era_11/lib/_workdir/mic1@192.168.10.127
cp -a /opt/era_11/mic_before_upgrade /opt/era_11/lib/_workdir/mic1@192.168.10.127
cp -a /opt/era_11/local.cfg.json_before_upgrade /opt/era_11/lib/_workdir/erainstaller/local.cfg.json
Without this the mic of the previous version will take the configuration changed by the new version: it is newer than the one restored from the database.
If the copy was not taken, the mic files can be taken from the backup that the new version mic makes itself on first start, before changing files: /opt/era_11/syncroot/secure/cfg/mic1@192.168.10.127/<previous version>_<UTC date and time>/ (the last 5 copies are kept; for versions below 1.13 — the directory before_<date and time>).
Take the earliest directory after the last start of the previous version.
The files are copied into the mic directory over the existing ones, and the platform_version marker file in the mic directory is deleted — otherwise, on a repeated upgrade, the new version will not make a copy again.
The local.cfg.json file is not included in this copy: it contains the node cookie (psk), which has no place in the shared synchronization directory.
The new version masks psk in it, and the previous version uses the mask as the node cookie: the cluster works if the new version started on all servers (the mask is the same).
But a new server installed with a psk known to the administrator will not connect to such a cluster, and if the new version did not start on all servers, the server cookies differ.
Therefore local.cfg.json is copied before the upgrade and restored on rollback on each server (the commands above).
In option B this is not required: the previous working directory with the open psk is not changed.
B. Working directory copy
Before starting the new container (the previous one is stopped), copy the directory; the /opt/era_12/lib directory must not exist before copying:
mkdir -p /opt/era_12
cp -a /opt/era_11/lib /opt/era_12/lib
In the new container start command (step 5) replace only the working directory volume:
--volume=/opt/era_12/lib:/var/lib/era:rslave
Do not move or rename the volume directories: when the previous container starts, an empty directory will be created in place of the missing one, and the installer will perform primary initialization (firstinit) on the existing database.
After a successful upgrade the working directory becomes /opt/era_12/lib: installer commands and the next upgrade use it (varlibvoldir parameter, container start command), and the previous directory is deleted together with the previous container.
Database backup
The copy is taken after stopping the previous platform container (step 4) and before starting the new one: thus it contains no records made after stopping.
Database in a container
Stop the PostgreSQL container and copy the cluster directory:
docker stop postgres_11
cp -a /opt/era_11/postgresql /opt/era_11/postgresql_before_upgrade
docker start postgres_11
Restoration (the platform container is stopped):
docker stop postgres_11
mv /opt/era_11/postgresql /opt/era_11/postgresql_failed_upgrade
cp -a /opt/era_11/postgresql_before_upgrade /opt/era_11/postgresql
docker start postgres_11
Database on the host
Stop the cluster with the installer unit and copy the cluster directory (path per the OS matrix):
systemctl stop postgresql_<version>_era_11.service
cp -a <cluster directory> <cluster directory>_before_upgrade
systemctl start postgresql_<version>_era_11.service
Restoration is the same as for a container: stop, replace the cluster directory with the copy, start.
If the cluster cannot be stopped, the copy is taken from a running cluster via pg_basebackup (a replication user is required) or pg_dumpall; the dump is restored into an empty cluster.
Streaming replication
The copy is taken from the master, as for a database on the host.
Restoration:
-
Stop the platform containers on all servers.
-
Restore the master cluster directory from the copy and start the master.
-
Re-fill each replica from the master with the installer operation switch_to_slave: it moves the current replica directory to backup and copies the database from the master.
-
Start the previous platform containers.
Replicas must not be left as is: after the upgrade they contain new version data.
Rollback procedure
-
Stop the new containers on all servers:
docker stop era_12. -
Restore the database from the copy (see above).
-
Option A: restore the mic directory (on servers with the mic role) and
local.cfg.json(on each server). Option B: nothing. -
Start the previous containers on all servers:
docker start era_11. -
Verify startup as in step 6.
After verification the new container can be deleted.
Consequences of rollback
-
Data written after the upgrade is lost.
-
Administrator connections to the notification system made or migrated by the new version must be made again.
-
Files added by the new version to the shared synchronization directory (for example, the installation key in
:SYNC/secure/keys) do not interfere with the previous version. -
The installation key is created in a separate step after the upgrade (more); create it when rollback is no longer planned.