KoaVM
Guide pages

Guide · For administrators

High availability

If a node suddenly goes down (hardware fault, frozen kernel, power), its VPSs stay down until the node comes back. With high availability protected VPSs restart on another node, the reserve node, from the latest copy of their disks, with the same IP. At most the data written after the latest copy is lost.

Which VPSs are protected

  • From the plan: in Clients and plans, the plan's High availability option. It applies to all the plan's VPSs, including existing ones.
  • On the VPS: on the VPS page, High availability box › Settings: as the plan, yes or no. The choice on the VPS wins over the plan.

Only VPSs on KoaVM nodes with ZFS, on the node's default pool, can be protected. Provider VPSs (Hetzner Cloud, Contabo…) can't: the provider manages them.

The VPS list shows a shield next to protected VPSs: green when the copy is ready, orange while it is being made or when it is late. The customer sees on their VPS whether it is protected and how many minutes of data they can lose at most, nothing else.

The reserve node

The panel chooses it, in this order:

  1. a node where the VPS's IP can follow it: the same IP block linked to both nodes (for example a vRack, in Network and IPs), or a provider IP the panel moves through the API (for example OVHcloud failover IPs). IPv6 and additional IPs count too;
  2. a reachable node, not in maintenance, with ZFS;
  3. with room: RAM, vCPUs and disk with the node's overbooking, also counting the other protected VPSs of the same node that would restart there together, and the space of all the copies the node holds;
  4. in the same city, if there is one; then the same block before a moved IP (which adds 1–3 minutes), then the freest.

If no node is suitable the VPS is not protected and the box says why (usually: the IP block is not linked to any other node). A VPS never restarts with a different IP: it would not be reachable until the customer changed their DNS, so it would not really be protected.

You can change the reserve node from the box's Settings (the list says why the other nodes are not suitable). Changing node starts the copy from scratch.

Continuous copy

Every N minutes (10 by default, in High availability › Settings) the panel takes a ZFS snapshot of all the VPS's disks at the same instant and sends to the reserve node only the blocks changed since the previous copy, going through the panel like backup replication. The first copy is a full one.

  • On the reserve node the copy lives in <pool>/koavm-ha/vps-<id>-g<N>, outside Incus: there is no stopped instance that someone could start by mistake. The last 2 copies are kept.
  • On the VPS's node only the snapshot of the latest copy stays (@koavm-ha-…): it takes the space of the blocks changed since then.
  • A copy is like a sudden power-off: the data is what was on disk at that instant (modern databases recover like after a power cut).
  • At most 2 copies at a time in the whole panel. While a job that changes the disks runs on the VPS (reinstall, restore, disks, move) the copy waits; with other jobs (start, network…) it goes on. Restoring a snapshot or a backup starts the copy from scratch; after adding or removing a disk the copy runs again straight away.
  • If the latest copy is older than 3 intervals you get an alert (VPS group), with the space taken by snapshots on the VPS's node. If copies have been failing for 24 hours, or that space is more than a quarter of the VPS's disks (at least 5 GB), the panel removes the latest copy's snapshot from the VPS's node: the copy on the reserve node stays valid and the next copy is a full one.
  • If the VPS can no longer be protected (for example it moves to a non-default pool), if the reserve node is removed or no longer suitable, or if it hasn't answered for more than 6 hours, the copy on the reserve node is no longer valid: the panel removes it and picks another one.

The VPS box shows the reserve node, the latest copy, how much can be lost and the copy's size; Copy now runs it straight away.

When a node is really down

The panel checks the nodes every 15 seconds. It decides a node is down only if all of these are true:

  1. the panel has not reached the node's Incus API for at least N minutes (3 by default), counted since the panel started;
  2. the panel can't reach the node's SSH port either: if it answers, the server is on (Incus may have stopped with the VMs still running) and nothing happens;
  3. the panel reaches the other nodes: if it sees none of them, or more than half of the nodes don't answer, the problem is the panel or its network and nothing happens;
  4. at least one other node, asked over SSH, confirms it can no longer reach it: ports 8443 and SSH, ping and, with private networks, the same checks through the WireGuard tunnel and its latest handshake. If even one node still sees it, nothing happens.

A node in maintenance never restarts anything by itself.

The cross-check

Another node that "doesn't see it" counts as a confirmation only if it could see it before. Every 20 minutes, while the nodes answer, the panel runs a baseline check: each node tries to reach the nodes with protected VPSs and the panel records which methods work. When a node stops answering, another node's answer counts only if it saw the node in the baseline check and now tries all those methods again without success. This way a firewall between the nodes, a closed port or a command missing on the node (timeout, ping, wg) doesn't look like a node that is down. Without a valid baseline check (less than 24 hours old) the node stays unconfirmed and nothing restarts by itself.

On the High availability page every node with protected VPSs shows who sees it and how. If nobody does, open port 8443 (TCP) and ping between the nodes, or link the nodes with private networks (the tunnel gets through firewalls). Nodes prepared by the panel already open 8443 to the panel's other nodes that existed at that moment (with ufw or firewalld); for nodes added later, open the port by hand on the older nodes.

Automatic or after your confirmation

In High availability › Settings:

  • Alert and wait for confirmation (default): you get the "Node down" alert and the VPSs restart when you press Restart them on the High availability page. It is the default because the panel cannot power a server off by itself (no IPMI): if the node were only cut off from the network and came back, two copies of the same VPS could run for a few seconds. Check in the provider panel that the server is off (or turn it off), then confirm.
  • Automatic: VPSs restart by themselves as soon as the node is found to be down. Switch to it after testing HA with Switch to the reserve node and checking that the IPs follow the VPSs. A VPS whose latest copy is older than 3 intervals before the fault doesn't restart by itself: it waits for your confirmation (more data than expected would be lost).

If the panel isn't sure (node unconfirmed, another node still sees it, panel isolated) you can still restart the VPSs by ticking I checked: the node is down. Once the restarts are over, Restart them can be repeated for the VPSs still on the node that is down (for example when room has been freed or after an error).

The restart

VPSs restart in priority order (high, normal, low: set in the VPS settings) while there is room on their reserve node (RAM with overbooking and RAM actually in use; in an emergency vCPUs don't block). The ones that don't fit stay down and you get an alert.

For each VPS the panel: stops the copy in progress, creates the instance on the reserve node with the configuration saved at the latest copy (same cloud-init instance-id, so no first boot again; older VPSs that don't have one use the name, which stays the same), puts the copy's disks in place of its own (instant), only then moves the provider IP if needed, updates database, firewall and private networks, starts it if it was running and, if the gateway or MAC changed, fixes the network inside the VPS with the agent (like a move between nodes). Extra disks are the ones of the latest copy: a disk added later stays in the old copy (the job says so). If something goes wrong before the switch, the disks go back into the copy, the new instance is deleted and the IP goes back where it was (if the provider doesn't answer, the panel puts it back when the node returns). If you choose Email the client, the customer gets an email saying how many minutes of data may be missing.

Then the VPS protects itself again on another node; if there is no other one, when the node that went down comes back.

The fence: VPSs don't start by themselves

Protected VPSs have boot.autostart=false on the node: a node that boots again does not start them by itself (Incus normally starts the VMs that were running). The panel starts them, if they were running when the node went down and haven't restarted elsewhere. After a normal node reboot, protected VPSs therefore start a few tens of seconds later, and only if the panel is running.

When a node that was down comes back, the first thing the panel does, as soon as the node answers, is handle the old copies of the VPSs restarted elsewhere: it stops them all at once, then removes their network cards (and masks the one from the Incus profile), renames them (vps-12-old-…, extra disks too) and keeps them for 7 days, then deletes them. They hold the data written after the latest copy: to recover it, attach the disk to a service VPS on the node. You find them in High availability › Old copies (with Delete now). If the VPS protects itself towards that node again, the old copy is used as a base: it is copied locally on the node and only the differences go over the network.

Switch to the reserve node

From the VPS page, Switch to the reserve node moves the VPS to its reserve node without losing data: copy while running, shutdown, last copy (seconds), start on the reserve node. Then protection is reversed: the previous node becomes the reserve, starting from the disks it already has. Use it for node maintenance and to test high availability before switching to automatic mode. A VPS with snapshots can't be switched (snapshots are not in the copy): use a move between nodes, which takes them along, or delete them first.

Rescue, files from backups and grown disks

  • Rescue mode: copies go on while the VPS is in the rescue system (the VPS's disk is copied, never the rescue disk). If the node goes down, the VPS restarts on the reserve node normally, with its own system (running if the rescue system or the VPS before rescue was running): rescue mode is closed and the customer can open it again. The rescue disk stays on the old node and the panel deletes it when the node comes back.
  • Files from backups: the backup copy attached to the VPS is not part of the high availability copy. On a restart the files session is closed (it reopens in one click); the panel deletes the backup copy on the old node when the node comes back. No automatic exit and no alert from rescue mode or files from backups after the restart.
  • Switch to the reserve node doesn't start while the VPS is in rescue mode or has a files session open: the rescue disk and the backup copy are on the current node. Leave rescue mode or close the session, then try again.
  • Main disk grown while the VPS was running (plan change): the VPS restarts with the size set in the panel; if the latest copy was smaller the disk grows on the reserve node (the system grows the partition at boot, or the customer does) and the job says so. CPU and disk limits apply on the new node a few seconds after the restart.

Before rebooting a node

For planned maintenance put the node in maintenance (no automatic restarts) or switch its VPSs to their reserve nodes first. In automatic mode, a reboot that lasts longer than the waiting minutes would restart the VPSs elsewhere (losing the last few minutes).

What it does not protect

  • data written after the latest copy (at most the copy interval plus the copy time);
  • mistakes inside the VPS (deleted files, broken system, malware): use snapshots and backups;
  • two faults at once: the node and its reserve node, or more than half of the nodes;
  • provider VPSs, VPSs on a pool other than the default one, nodes without ZFS;
  • the VPS's snapshots: after a restart they stay only on the old copy;
  • network cards added by hand outside the panel and attached ISOs (they are not on the reserve node);
  • VPSs with a manually configured network (or Windows) if the gateway changes on the reserve node: they can't be protected.

API

PUT /api/v1/vps/{id}/ha with protect ("" as the plan, "on", "off"), reserve_id (0 = chosen by the panel) and priority (0 low, 1 normal, 2 high): to sell high availability as an option from billing. GET /api/v1/vps/{id} returns the status in ha.