High Availability

Overview

Storware Backup & Recovery can be deployed in an active/passive cluster using Pacemaker and Corosync.

This chapter describes configurations with two or three cluster hosts. Each host runs a local MariaDB instance and has the Storware Backup & Recovery Server and Node components installed.

Pacemaker manages the floating IP address and the Storware services. Only one host runs the Storware Server and Node services at a time. MariaDB remains running on all hosts to maintain database replication.

Users and Storware Nodes connect to the Server through a shared hostname that resolves to the floating IP address.

Component	Two-host configuration	Three-host configuration
Storware Server and Node services	Active on one host	Active on one host
MariaDB	Running on both hosts	Running on all three hosts
Database replication	Bidirectional replication	Circular replication
Client access	Shared hostname and floating IP	Shared hostname and floating IP
Backup storage	Accessible from both hosts	Accessible from all three hosts

Important: Database replication in this configuration is asynchronous. Failover does not guarantee that the destination database contains every transaction committed on the previous active host. Pacemaker service monitoring alone does not verify replication consistency or select the most up-to-date database.

Example Environment

The examples use the following addresses:

Host or resource	Hostname	IP address
First cluster host	node1.example.com	10.1.1.2
Second cluster host	node2.example.com	10.1.1.3
Third cluster host — three-host configuration only	node3.example.com	10.1.1.4
Cluster access address	storware.example.com	10.1.1.5/24
NFS server used in this example	—	10.1.1.1

Replace these values with the addresses and hostnames used in your environment.

Complete Common Preparation, follow the section for your selected cluster size, and then continue with Configuring Cluster Resources.

Two and Three node cluster

Prerequisites

Before starting, prepare:

• Two or three hosts running a supported operating system. The commands in this chapter target Red Hat Enterprise Linux 9 on x86_64.

• The same Storware Backup & Recovery version and compatible MariaDB versions on all hosts.

• Access to the required operating system, High Availability, Storware, and MariaDB repositories.

• Unique hostnames and working name resolution between all hosts.

• An unused floating IP address that can be assigned to any cluster host.

• A shared hostname resolving to the floating IP address.

• A server certificate valid for the shared hostname.

• Backup destinations accessible from every host that can run the Storware Node service.

• Network connectivity for cluster communication, database replication, HTTPS access, and backup storage.

• A fencing mechanism appropriate for the physical or virtual infrastructure.

• Administrative access to all hosts.

All shell commands in this chapter are executed as root.

This procedure assumes a new cluster. When using an existing Storware deployment as the source, first back up its database and configuration and stop all backup, restore, and administrative activity.

Common Preparation

1. Install Storware Server and Node

Run on every cluster host.

Follow the Server and Node installation instructions for your operating system, including repository configuration and operating system preparation.

During installation:

• Do not start vprotect-server or vprotect-node.

• Do not register the Node yet.

• Complete any required operating system reboot before configuring the cluster.

Ensure that both Storware services are stopped and disabled:

systemctl disable --now vprotect-server vprotect-node

Pacemaker will manage these services after the cluster resources have been created.

MariaDB must remain available independently on every host:

systemctl enable --now mariadb

Do not disable MariaDB as part of disabling the Storware services. Both database replication configurations require MariaDB to run on all participating hosts.

2. Configure Name Resolution

Run on every cluster host.

Configure DNS or add the following entries to /etc/hosts:

10.1.1.2 node1.example.com node1
10.1.1.3 node2.example.com node2
10.1.1.5 storware.example.com storware

For a three-host cluster, also add:

10.1.1.4 node3.example.com node3

Ensure that each cluster hostname resolves to its actual network address. Remove any conflicting mapping of a cluster hostname to a loopback address, while preserving the standard localhost entries.

3. Install Cluster Packages

Run on every cluster host.

Enable the RHEL 9 High Availability repository:

subscription-manager repos \
  --enable=rhel-9-for-x86_64-highavailability-rpms

Install the cluster packages:

dnf install -y pacemaker pcs resource-agents

Install the fence agent required by your infrastructure.

Enable the PCS management service:

systemctl enable --now pcsd

Set a password for the hacluster account:

passwd hacluster

Use the same password on all cluster hosts.

PCS authentication is configured later with pcs host auth. Passwordless SSH between every pair of hosts is not required for PCS cluster authentication.

4. Configure Network Access

Run on every cluster host.

Allow cluster communication:

firewall-cmd --permanent --add-service=high-availability

Allow MariaDB replication from the participating cluster hosts. For example, add a rule for each peer, replacing <PEER_IP>:

firewall-cmd --permanent \
  --add-rich-rule='rule family="ipv4" source address="<PEER_IP>/32" port port="3306" protocol="tcp" accept'

Apply the changes:

firewall-cmd --reload

Apply the rules to the firewall zone associated with the relevant interface.

Also allow the HTTPS and storage connections required by the deployment.

If HTTPS access uses the installation’s port redirection from TCP 443 to TCP 8181, configure local access to the floating IP:

firewall-cmd --permanent --direct --add-rule ipv4 nat OUTPUT 0 \
  -p tcp -o lo -d 10.1.1.5/32 --dport 443 \
  -j REDIRECT --to-ports 8181

firewall-cmd --reload

This rule applies to deployments using that port-redirection model. Adapt it if HTTPS is provided by a reverse proxy or another configuration.

Keep the firewall and SELinux configured according to the supported installation requirements. Disabling them is not part of this procedure.

5. Configure Shared Backup Storage

Run on every cluster host.

Configure access to the selected backup destination. Each host must have the required connectivity, credentials, and permissions.

For filesystem destinations, use the same path on every host.

The following example mounts an NFS export at /vprotect_data.

Install the NFS client and create the mount point:

dnf install -y nfs-utils
mkdir -p /vprotect_data

Add the following entry to /etc/fstab:

10.1.1.1:/vprotect /vprotect_data nfs defaults,_netdev 0 0

Mount and verify the filesystem:

mount /vprotect_data
findmnt /vprotect_data

Run once, from one cluster host:

mkdir -p /vprotect_data/backup/filesystem
mkdir -p /vprotect_data/backup/dbbackup

Configure ownership and permissions so that the vprotect account can access these directories from every host. Ensure consistent UID/GID mapping and appropriate NFS export permissions.

The shared storage must remain available when the active cluster host changes. Its availability is a separate dependency from the Storware cluster.

Creating a directory on NFS does not make it a synthetic backup destination. Configure synthetic destinations according to their storage backend requirements.

6. Configure HTTPS and Synchronize Server Configuration

Prepare on node1.example.com, then apply to the other hosts.

Configure HTTPS for storware.example.com using the procedure described in Enabling HTTPS Connectivity for Nodes.

Synchronize the Server configuration and certificate material required by your Storware release. Depending on the release, these files can include:

/opt/vprotect/server/quarkus.properties
/opt/vprotect/server/vprotect.env
/opt/vprotect/server/log4j2-server.xml
/opt/vprotect/server/keystore.jks
/opt/vprotect/server/vprotect-keystore.jks

For releases using a local license file, also synchronize:

/opt/vprotect/server/license.key

Preserve file ownership and restrictive permissions when transferring configuration files and private keys.

Review host-specific settings after copying. In this architecture, each Storware Server must connect to its local MariaDB instance.

Configure the same application database password on every host, matching the password in quarkus.properties.

For an existing vprotect database account using password authentication, execute locally on each MariaDB instance:

ALTER USER 'vprotect'@'localhost'
  IDENTIFIED BY '<VPROTECT_DB_PASSWORD>';

Use the account host value created by the installation if it differs from localhost.

7. Prepare MariaDB for Replication

Run on every cluster host.

Add or update the following settings in the [mysqld] section of /etc/my.cnf.d/server.cnf:

[mysqld]
log-bin=mysql-bin
relay-log=relay-bin
log-slave-updates=ON
binlog_format=ROW
replicate-do-db=vprotect

Assign a unique server_id:

Host	Setting
node1.example.com	server_id=10
node2.example.com	server_id=20
node3.example.com	server_id=30

Preserve other settings required by the Storware installation.

Ensure that MariaDB accepts replication connections on the host’s network address. A configuration restricted to 127.0.0.1 will prevent remote replication.

Restart MariaDB:

systemctl restart mariadb

Do not change lower_case_table_names on an already initialized database as part of this procedure. Its value must be consistent across hosts and established during database initialization. MariaDB Documentation

Create a replication account locally on each database:

CREATE USER 'replicator'@'10.1.1.%'
  IDENTIFIED BY '<REPLICATION_PASSWORD>';

GRANT REPLICATION SLAVE ON *.*
  TO 'replicator'@'10.1.1.%';

Use a strong password and restrict the permitted source addresses to the replication network or individual peers.

Because replication is restricted to the vprotect database, database accounts and grants must be maintained locally on every host.

8. Initialize the Databases from a Common Source

Use node1.example.com as the authoritative source.

Keep Storware services stopped on all hosts throughout database initialization and replication setup. Prevent other processes, scheduled database events, and administrators from modifying the application database during this period.

Create a database dump:

umask 077

mariadb-dump -u root -p \
  --single-transaction \
  --routines \
  --events \
  --triggers \
  --databases vprotect \
  > /root/vprotect-ha.sql

Copy the dump to the second host:

scp /root/vprotect-ha.sql node2.example.com:/root/

For a three-host cluster, also copy it to the third host:

scp /root/vprotect-ha.sql node3.example.com:/root/

On each destination host, import the dump through a local MariaDB session:

mariadb -u root -p

SET SESSION sql_log_bin = 0;
SOURCE /root/vprotect-ha.sql;
SET SESSION sql_log_bin = 1;

Disabling binary logging in this import session prevents the initialization data from later circulating through the replication topology.

The destination databases must contain the same application data and schema as the source before replication begins. This step is intended for newly prepared destination databases, not for merging independent deployments.

After completing all imports and local account configuration, run the following on each host:

SHOW MASTER STATUS;

Record the File and Position values separately for each host.

Use those values in the appropriate configuration below. Do not copy example positions from another environment, and do not allow application writes between this step and completion of replication setup.

Two-Node Cluster Configuration

1. Configure Bidirectional Database Replication

The databases replicate changes in both directions:

Replica	Replication source
node1.example.com	node2.example.com
node2.example.com	node1.example.com

On node1.example.com:

CHANGE MASTER TO
  MASTER_HOST='10.1.1.3',
  MASTER_PORT=3306,
  MASTER_USER='replicator',
  MASTER_PASSWORD='<REPLICATION_PASSWORD>',
  MASTER_LOG_FILE='<NODE2_BINLOG_FILE>',
  MASTER_LOG_POS=<NODE2_BINLOG_POSITION>;

On node2.example.com:

CHANGE MASTER TO
  MASTER_HOST='10.1.1.2',
  MASTER_PORT=3306,
  MASTER_USER='replicator',
  MASTER_PASSWORD='<REPLICATION_PASSWORD>',
  MASTER_LOG_FILE='<NODE1_BINLOG_FILE>',
  MASTER_LOG_POS=<NODE1_BINLOG_POSITION>;

These commands assume that no replication channel has previously been configured.

Start replication on both hosts:

START SLAVE;
SHOW SLAVE STATUS\G

Verify that both replication threads are running:

Slave_IO_Running: Yes
Slave_SQL_Running: Yes

Check for replication errors and confirm that both databases have reached the recorded source positions before continuing.

2. Create the Pacemaker Cluster

Run on node1.example.com.

Authenticate both hosts:

pcs host auth node1.example.com node2.example.com -u hacluster

Create the cluster:

pcs cluster setup sbrcluster \
  node1.example.com \
  node2.example.com

Start the cluster and enable startup at boot:

pcs cluster start --all
pcs cluster enable --all

Check its status:

pcs status
pcs quorum config
pcs quorum status

3. Configure Fencing and Verify Quorum

Configure and test fencing for both hosts using the fence agent appropriate for your infrastructure.

Retain the quorum behavior appropriate for the two-host configuration generated by PCS. Do not set no-quorum-policy=ignore merely because the cluster contains two hosts.

Two-host quorum behavior differs from ordinary majority voting. Initial startup can also require both hosts to have been visible before the cluster becomes quorate. Verify startup, isolation, and recovery behavior for your deployment. Red Hat Documentation

Continue with Configuring Cluster Resources.

Three-Node Cluster Configuration

1. Configure Circular Database Replication

The databases form a replication ring:

Replica	Replication source
node1.example.com	node3.example.com
node2.example.com	node1.example.com
node3.example.com	node2.example.com

The log-slave-updates setting allows replicated changes to be forwarded to the next database in the ring.

On node1.example.com:

CHANGE MASTER TO
  MASTER_HOST='10.1.1.4',
  MASTER_PORT=3306,
  MASTER_USER='replicator',
  MASTER_PASSWORD='<REPLICATION_PASSWORD>',
  MASTER_LOG_FILE='<NODE3_BINLOG_FILE>',
  MASTER_LOG_POS=<NODE3_BINLOG_POSITION>;

On node2.example.com:

CHANGE MASTER TO
  MASTER_HOST='10.1.1.2',
  MASTER_PORT=3306,
  MASTER_USER='replicator',
  MASTER_PASSWORD='<REPLICATION_PASSWORD>',
  MASTER_LOG_FILE='<NODE1_BINLOG_FILE>',
  MASTER_LOG_POS=<NODE1_BINLOG_POSITION>;

On node3.example.com:

CHANGE MASTER TO
  MASTER_HOST='10.1.1.3',
  MASTER_PORT=3306,
  MASTER_USER='replicator',
  MASTER_PASSWORD='<REPLICATION_PASSWORD>',
  MASTER_LOG_FILE='<NODE2_BINLOG_FILE>',
  MASTER_LOG_POS=<NODE2_BINLOG_POSITION>;

Start replication on all three hosts:

START SLAVE;
SHOW SLAVE STATUS\G

Verify on every host:

Slave_IO_Running: Yes
Slave_SQL_Running: Yes

Check replication errors and confirm that every database has reached its source’s recorded position.

Important: A failed database host breaks the replication ring. Pacemaker does not automatically repair replication channels or reconnect the remaining databases into a new topology. Restore or reconfigure replication before relying on further failovers. mariadb.com

2. Create the Pacemaker Cluster

Run on node1.example.com.

Authenticate all three hosts:

pcs host auth \
  node1.example.com \
  node2.example.com \
  node3.example.com \
  -u hacluster

Create the cluster:

pcs cluster setup sbrcluster \
  node1.example.com \
  node2.example.com \
  node3.example.com

Start the cluster and enable startup at boot:

pcs cluster start --all
pcs cluster enable --all

Check its status:

pcs status
pcs quorum config
pcs quorum status

3. Configure Fencing and Verify Quorum

Configure and test fencing for all three hosts.

Retain the normal majority-quorum behavior. With one vote per host, two communicating hosts constitute a majority; one isolated host does not.

Do not disable fencing simply because the cluster contains three hosts. Quorum and fencing address different parts of failure handling.

Continue with the common resource configuration below.

Configuring Cluster Resources

Complete this section for either cluster size.

Before proceeding, verify that:

• Database initialization and replication have completed.

• All cluster hosts are online.

• Fencing is configured and tested.

• The cluster has quorum.

• Shared storage is accessible.

• Server configuration and certificates are synchronized.

• The local database credentials work on every host.

RHEL HA deployments require a fencing device for each cluster node. Use the platform-specific configuration procedure rather than disabling STONITH. docs.redhat.com

1. Configure Resource Defaults

Run once, from node1.example.com.

pcs resource defaults update \
  migration-threshold=3 \
  failure-timeout=30s

These defaults control how recorded resource failures affect placement and when they expire. Adjust them to the operational requirements of your environment.

2. Create the Floating IP and Server Resources

Temporarily place the cluster in maintenance mode while defining the initial resources:

pcs property set maintenance-mode=true

Create the floating IP resource:

pcs resource create sbr-vip ocf:heartbeat:IPaddr2 \
  ip=10.1.1.5 \
  cidr_netmask=24 \
  op monitor interval=30s \
  --group sbr-group

Create the Server resource:

pcs resource create sbr-server systemd:vprotect-server \
  op start timeout=300s \
  op stop timeout=300s \
  op monitor interval=30s timeout=300s \
  --group sbr-group

Prefer the first host for the initial startup:

pcs constraint location sbr-group \
  prefers node1.example.com=100

Return the cluster to managed operation:

pcs property set maintenance-mode=false

Check the resources:

pcs status

Once the Server is running, verify access through:

https://storware.example.com

The resources belong to the same group, so the floating IP and Server are colocated. Group members start in the order in which they were added and stop in reverse order.

MariaDB is not part of this active/passive group. It runs independently on every host.

3. Configure Storware Node Registration

On every host, import the Server certificate using the documented certificate-trust procedure:

/opt/vprotect/node/scripts/node_add_ssl_cert.sh \
  storware.example.com 443

Verify the certificate before trusting it.

Two-Node Configuration: Shared Node Identity

The original two-host configuration uses a single Storware Node identity shared between both hosts.

Register the Node once on node1.example.com:

vc node inst register \
  --name=sbr-ha-node \
  --login='<ADMIN_LOGIN>' \
  --password='<ADMIN_PASSWORD>' \
  --apiurl=https://storware.example.com:443/api

With the Node service stopped on both hosts, copy its session file:

scp /opt/vprotect/node/.session.properties \
  node2.example.com:/opt/vprotect/node/.session.properties

Preserve the original file permissions and ensure that the file is owned by vprotect on the destination.

Only the cluster-managed active host may run this shared Node identity.

Three-Node Configuration: Separate Node Identities

The original three-host configuration registers a separate Storware Node on each host.

Run on every host, specifying its unique Node name:

vc node inst register \
  --name='<UNIQUE_NODE_NAME>' \
  --login='<ADMIN_LOGIN>' \
  --password='<ADMIN_PASSWORD>' \
  --apiurl=https://storware.example.com:443/api

Use names such as node1, node2, and node3.

Do not copy .session.properties between separately registered Nodes.

Configure Node configurations and backup destination availability so that the Node on each possible active host can execute the required tasks.

4. Add the Node Service to the Cluster

Run once, from a cluster host.

After registration is complete, add the Node resource:

pcs resource create sbr-node systemd:vprotect-node \
  op start timeout=300s \
  op stop timeout=300s \
  op monitor interval=30s timeout=300s \
  --group sbr-group

The resource group now starts in this order:

1. Floating IP address.

2. Storware Server.

3. Storware Node.

Verify the configuration:

pcs status
crm_verify -L

Only one host should run the complete resource group.

The systemd resource agents monitor service state. They do not verify that a database replica is current, that the application API is ready, or that backup storage remains accessible. Production failover procedures must account for these dependencies.

Verification and Failover Testing

1. Verify Normal Operation

Confirm that:

• All expected cluster hosts are online.

• The cluster has quorum and fencing is operational.

• The floating IP, Server, and Node resources run on the same host.

• MariaDB runs on every host.

• Database replication has no errors.

• The web interface is accessible through the shared hostname.

• The active Storware Node can access its backup destinations.

• A backup and a test restore complete successfully.

2. Test a Planned Switchover

First, allow active backup and restore tasks to finish.

For a controlled database handover, stop the Storware resource group:

pcs resource disable sbr-group --wait=300

Verify that the database on the intended destination has applied all changes from the previous active host. Check replication positions and errors; running replication threads alone do not prove that the replica is current.

Place the previous active host in standby:

pcs node standby node1.example.com

Re-enable the resource group:

pcs resource enable sbr-group --wait=300

Verify that it starts on another eligible host:

pcs status

Check web access, Node availability, backup destination access, and database replication again.

When the original host is ready to rejoin resource scheduling:

pcs node unstandby node1.example.com

Returning a host from standby makes it eligible to host resources. Whether resources move back depends on the configured placement preferences and stickiness.

For a three-host configuration, repeat the test to verify operation on each host.

3. Test Failure Handling Separately

A planned standby test does not validate fencing or unexpected host failure.

During a controlled maintenance window, test the agreed failure scenarios, including host loss and cluster-network interruption. Verify that the previous active host is isolated before another host runs the Storware services.

After an unexpected failure, assess database consistency and replication topology before further failover or failback.

Do not assume that interrupted backup or restore tasks resume automatically.

Maintenance and Useful Commands

Run cluster-wide administration commands from one cluster host.

Operation	Command
Display cluster status	pcs status
Display quorum status	pcs quorum status
Put a host in standby	pcs node standby node1.example.com
Remove a host from standby	pcs node unstandby node1.example.com
Stop Storware resources	pcs resource disable sbr-group --wait=300
Start Storware resources	pcs resource enable sbr-group --wait=300
Temporarily stop managing the resource group	pcs resource unmanage sbr-group
Resume management of the resource group	pcs resource manage sbr-group
Stop cluster services on one host	pcs cluster stop node1.example.com
Stop cluster services on all hosts	pcs cluster stop --all
Start cluster services on all hosts	pcs cluster start --all
Clear failure history for a corrected resource	pcs resource cleanup <RESOURCE_ID>

Unmanaging a resource does not stop it. Use the appropriate maintenance procedure before manually operating services or installing updates.

Before restoring a host to service after maintenance, verify its configuration, storage access, database consistency, and replication state.

For planned operating system shutdown, stop or relocate the resources first and confirm the resulting cluster state.