Linux integration

1.5.5 High Avaiability Failover

The high avaiability failover design strictly depends on your IT structure, so it is advisable to coordinate with your system administrator for a proper Failover design.

However on this chapter we'll propose a minimal three node cluster example that hosts two windows virtual machines:


3-Node HA Cluster Setup - Quick Reference Guide


Architecture Overview


  • node1 (172.31.99.145): Quorum-only node
  • node2 (172.31.99.144): Active node with DRBD
  • node3 (172.31.99.146): Active node with DRBD
  • VM Storage: 60GB DRBD replicated between node2/node3
  • OS: Ubuntu 24.04 with XFCE4 desktop


Setup Steps


1. Network Configuration (All Nodes)

bash
# Configure /etc/hosts on all 3 nodes
sudo nano /etc/hosts
```
Add:
```
172.31.99.145  node1 vm-quorum
172.31.99.144  node2 sp
172.31.99.146  node3 dc1

2. Package Installation


On all 3 nodes:

bash
sudo apt update
sudo apt install -y pacemaker corosync pcs

On node2 and node3 only:

bash
sudo apt install -y drbd-utils qemu-kvm libvirt-daemon-system \
    libvirt-clients bridge-utils resource-agents-base \
    resource-agents-common resource-agents-extra kpartx

3. DRBD Setup (node2 and node3 only)


Create backing storage:


bash
sudo mkdir -p /var/lib/drbd-backing
sudo truncate -s 60G /var/lib/drbd-backing/vm-disk.img
sudo losetup -f /var/lib/drbd-backing/vm-disk.img
sudo losetup -a | grep vm-disk  # Should show /dev/loop15. if different, note the number which must then be inserted with the same name in the configuration file



Create DRBD configuration:

bash
sudo nano /etc/drbd.d/vm-disk.res
```
```
resource vm-disk {
  protocol C;
  
  disk {
    on-io-error detach;
  }
  
  on node2 {
    device    /dev/drbd0;
    disk      /dev/loop15;
    address   172.31.99.144:7788;
    meta-disk internal;
  }
  
  on node3 {
    device    /dev/drbd0;
    disk      /dev/loop15;
    address   172.31.99.146:7788;
    meta-disk internal;
  }
}

Initialize DRBD:

bash
sudo drbdadm create-md vm-disk
sudo drbdadm up vm-disk

# On node2 only - make Primary for initial sync
sudo drbdadm primary --force vm-disk

Make loop devices persistent:

bash
sudo nano /etc/systemd/system/drbd-loop.service
ini
[Unit]
Description=Setup DRBD Loop Device
Before=drbd.service
After=local-fs.target

[Service]
Type=oneshot
ExecStart=-/sbin/losetup -f /var/lib/drbd-backing/vm-disk.img
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
bash
sudo systemctl daemon-reload
sudo systemctl enable drbd-loop

4. Cluster Setup


On all 3 nodes:

bash
echo "hacluster:YourPassword" | sudo chpasswd
sudo systemctl enable pcsd
sudo systemctl start pcsd

On node1:

bash
sudo pcs host auth node1 node2 node3 -u hacluster -p YourPassword
sudo pcs cluster setup ha-vm-cluster node1 node2 node3
sudo pcs cluster start --all
sudo pcs cluster enable --all
sudo pcs property set stonith-enabled=false
sudo pcs property set no-quorum-policy=stop

5. VM Setup (node2 and node3)


Download Ubuntu ISO:

bash
cd /var/lib/libvirt/images
sudo wget https://releases.ubuntu.com/24.04/ubuntu-24.04.4-live-server-amd64.iso

Create VM XML:

bash
sudo nano /etc/libvirt/qemu/ubuntu-vm.xml
xml
<domain type='qemu'>
  <name>ubuntu-vm</name>
  <memory unit='GiB'>2</memory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset='utc'/>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/drbd0'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/ubuntu-24.04.4-live-server-amd64.iso'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
    </disk>
    <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
    <graphics type='vnc' port='-1' listen='0.0.0.0'/>
    <video>
      <model type='vga'/>
    </video>
    <console type='pty'>
      <target type='serial'/>
    </console>
  </devices>
</domain>

Install Ubuntu:

bash
# On node3
sudo virsh define /etc/libvirt/qemu/ubuntu-vm.xml
sudo virsh start ubuntu-vm
# Use virt-manager to complete installation

After installation, remove CD-ROM from XML and redefine on both nodes


6. Pacemaker Resources (node1)

bash
# Create DRBD resource
sudo pcs resource create drbd-vm-disk ocf:linbit:drbd \
    drbd_resource=vm-disk op monitor interval=60s

sudo pcs resource promotable drbd-vm-disk \
    meta promoted-max=1 promoted-node-max=1 clone-max=2 clone-node-max=1 notify=true

sudo pcs constraint location drbd-vm-disk-clone avoids node1

# Create VM resource
sudo pcs resource create ubuntu-vm ocf:heartbeat:VirtualDomain \
    config=/etc/libvirt/qemu/ubuntu-vm.xml \
    hypervisor="qemu:///system" \
    op start timeout=120s op stop timeout=120s op monitor interval=30s

# Add constraints
sudo pcs constraint colocation add ubuntu-vm with drbd-vm-disk-clone \
    INFINITY with-rsc-role=Master

sudo pcs constraint order promote drbd-vm-disk-clone then start ubuntu-vm

sudo pcs constraint location ubuntu-vm avoids node1

7. Verification

bash
# Check cluster status
sudo pcs status

# Test failover
sudo pcs node standby node3
sudo pcs node unstandby node3

# Check DRBD
sudo drbdadm status vm-disk


Key Commands Reference


Cluster Management:

bash
sudo pcs status                    # Check cluster status
sudo pcs resource cleanup          # Clean failed actions
sudo pcs node standby node3        # Put node in standby
sudo pcs node unstandby node3      # Bring node back

DRBD:

bash
sudo drbdadm status                # Check DRBD status
sudo drbdadm up vm-disk            # Start DRBD
sudo drbdadm down vm-disk          # Stop DRBD

VM:

bash
sudo virsh list --all              # List VMs
sudo virsh start ubuntu-vm         # Start VM
sudo virsh shutdown ubuntu-vm      # Shutdown VM