Clusters and shared storage

If you have shared storage (Ceph, an iSCSI or Fibre Channel SAN, a TrueNAS), KoaVM uses it through an Incus cluster: several servers that see the same disks. A VPS on shared storage:

• moves between the cluster servers without copying the disk, even live, in a few seconds;
• if its server fails, restarts on another server with the same disk: nothing is lost.

In Incus shared storage works only inside a cluster: a Ceph pool is not shared between independent Incus servers.

Preparing the cluster

You create the cluster with the Incus tools; KoaVM connects to it.

1. On each server install Incus as for a regular node.
2. Create the cluster on the first server (incus admin init, answer yes to clustering) and join the others with a token (incus cluster add <name> on the first, then incus admin init on the others).
3. Add the shared pool: - Ceph: an existing Ceph cluster, or MicroCeph on the same servers (snap install microceph, microceph cluster bootstrap, microceph disk add). Then incus storage create remote ceph. With MicroCeph, each server needs ceph.conf and the client.admin key in /etc/ceph. - iSCSI, Fibre Channel or NVMe-oF SAN: the lvmcluster driver (with lvmlockd and sanlock). - TrueNAS: the truenas driver.
4. For the VPS network create an Incus bridge with the same name on all members (for example br-vps), or use IPs that follow the VPS (the same block linked to all members).

Connecting the cluster to KoaVM

Add each cluster server as a node (Nodes › Add node), with its address. The panel works out on its own that it is a cluster member: the Nodes page shows “member … · cluster …” and the shared pool is marked shared.

• Turn on the shared pool for VPSs (Nodes › Storage pools › For VPSs) on each member.
• In plans pick the preferred storage pool (e.g. remote): the plan's VPSs go on the shared pool.
• Link the VPS IP block to all members: that way the IP follows the VPS when it moves.

For shared storage the panel counts the space of the whole cluster, not of one node.

Moving a VPS within the cluster

With Move to another member of the same cluster, a VPS on the shared pool copies nothing: Incus only changes the server. If the VPS is running and live-movable it stays on; otherwise it stops, moves to the other server and starts again in under a minute.

Within the cluster only VPSs with the main disk on the shared pool, no extra disks and no GPU, keeping the same addresses, move without copying.

High availability without data loss

A protected VPS (see High availability) on the cluster's Ceph pool needs no copies: the reserve node is another member and protection is active right away. The VPS box says “at most you lose: nothing”.

When a member suddenly goes down:

1. the panel checks Incus and SSH as always, and also asks the other cluster members how they see it: if they mark it OFFLINE the failure is confirmed; if they still see it, no restart;
2. before restarting the VPSs, it fences the failed member off the storage (Ceph blocklist): a server that is only cut off from the network, but still running, can no longer write to the same disks. If fencing fails, the VPSs do not restart;
3. it moves the VPSs to the reserve member and starts them, with the same disk;
4. when the member comes back, the panel stops the leftover processes of the VPSs restarted elsewhere and removes the blocklist.

With “Switch to the reserve node” the VPS moves between members like with Move, live when possible.

For now high availability without copies works with Ceph. With lvmcluster and TrueNAS VPSs move within the cluster, but automatic protection will come once their fencing is tested.

Backups

VPSs on the shared pool back up with the Incus export (chunked archive on S3, FTP or another node), like VPSs on the other non-default pools.