Nodes

A node is a server with Incus. VPSs are created on the node with the most assignable RAM left (with its overbooking, see below), unless you choose one.

Connect a node

Add node offers three ways: A server you already have and Order a server set the node up on their own (see New nodes and automatic purchasing); Incus already set up (token) connects a server where Incus is already installed:

1. On the server, if the Incus API is not listening on the network yet, enable it:

incus config set core.https_address :8443

2. Create a token for the panel and copy it whole:

incus config trust add koavm

3. In Nodes › Add node paste the token, name the node (e.g. gra-01) and pick its city on the map.

The token contains the server certificate's fingerprint: the connection is encrypted and verified. The advanced options let you change the address, the VPS bridge (default br-vps) and the storage pool (default default).

The panel's SSH key

For some tasks the panel logs into the node over SSH: migrations, backups stored on a node, IPv6 setup, real RAM, VPS disk and CPU and disk and CPU limits (since 1.14). Add the panel's key (shown on the Migration page) to the node's /root/.ssh/authorized_keys, or to a user with passwordless sudo, and set user and port on the same page. The Test button checks that it works.

Status and resources

At the top right you choose the view: Cards (one card per node) or List (one row per node, like the Finder: click a column to sort, and the disclosure triangle opens the full card under the row). The panel remembers the choice.

For each node you see CPU, assigned RAM (against the assignable RAM with overbooking) and RAM really in use, swap (and how much is in use), used and assigned disk, vCPUs sold against those for sale, CPU really in use and running VPSs. The panel reads the nodes every 30 seconds; Refresh reads them right away. A node that does not answer is shown in red with the last error.

Overbooking

The panel counts the RAM, disk and vCPUs assigned to VPSs, not what they really use. Without overbooking it assigns up to 90% of the server RAM, up to the whole pool disk and one vCPU per thread.

With Nodes › Overbooking (or when you add the node) you choose how much extra to sell:

• Disk from 1× to 3×. ZFS disks only take what the VPSs actually write, so 1.5–2× is common.
• RAM from 1× to 2×. VPSs give the node back the RAM they free (see below), but a VPS that uses all its RAM keeps it. Above 1.2× you need swap.
• CPU from 1× to 16× (vCPUs sold for each server thread; 4× by default, also on nodes added before 1.5). VPSs take turns on the CPU and rarely all at once: 4× is the most common ratio. A VPS still cannot have more vCPUs than the server has threads.

There is also a real-usage brake (Settings › Nodes and overbooking, 85% by default). When the RAM, disk or CPU actually in use on a node reaches the threshold, the node gets no more VPSs, moves, migrations or bigger plans, even if overbooking would allow it. The CPU in use is the sum of what the running VPSs really use (sampled every 30 seconds). The node RAM and disk alerts stay as before.

RAM really in use

Incus counts VM memory as "cache" (it is shared memory), so on its own it reports far less than the truth. Since 1.5 the panel reads /proc/meminfo every minute from the nodes it can log into over SSH: in use = total RAM minus available RAM, plus the swap in use. Without SSH access the panel estimates (what Incus reports plus the RAM used inside the running VPSs) and says so next to the percentage: (estimate). The brake, the RAM alerts and the live map use this number.

RAM freed by VPSs goes back to the node

Without help a VM keeps forever the RAM it touched once, even after freeing it. Since 1.5 VPSs have free page reporting on the balloon device: the system inside the VPS tells the node which pages it freed and the node takes them back within seconds (test: a VPS using 3 GB and then freeing it went from 3.9 GB to 0.7 GB on the node). This is what makes RAM overbooking safe.

It applies to new VPSs and to existing ones from their next start or restart from the panel (a panel restart stops, changes and starts again; a reboot from inside the VPS is not enough). It is kept on moves, live ones included, and on restores. A hand-written raw.qemu.conf is left alone.

The same rules apply to automatic node purchasing: a node over the brake counts as full.

VPSs on a single chip

Some servers have several chips inside the same processor, each with its own threads and RAM (NUMA nodes: for example first-generation AMD EPYC, 4 chips of 8 threads and 32 GB). A free VPS can have its vCPUs on one chip and its RAM on another, and every memory access takes the long way. Since 1.5.1, on nodes with several chips the panel keeps each VPS on a single chip: in our tests RAM with several threads is 50% faster and without dips (single-thread speed does not change).

The chip is chosen by the panel, not by Incus: the one with the least assigned RAM (with the node's overbooking) and free vCPUs. Only VPSs that fit go there: a VPS with more vCPUs than a chip's threads, or more RAM than a chip has, stays free, and when the chips are full new VPSs stay free too. This matters because a VPS on a chip can only use that chip's RAM: an overfull chip would swap even with the others free.

It applies to new VPSs and to existing ones from their next start or restart from the panel. You turn it off from Overbooking › CPU chips (VPSs become free again from their next panel restart). A VPS on a chip can be live-moved only to a node with the same chips; a hand-written limits.cpu.nodes is left alone. The node card shows how many chips it has and how many VPSs are on a chip.

Swap: RAM can use the disk when needed

You choose it when the panel sets up the node, and you can change it later from Overbooking › Apply swap if the panel can log into the node over SSH:

• Yes, automatic: half of the server RAM, 2 to 32 GB;
• Yes, custom size: the GB you want (up to 256);
• No: VPS memory never goes to disk. If the server runs out of RAM, the system kills a VPS.

The panel swap is the file /var/lib/koavm-swap (ext4 or xfs needed). If the server already has swap, for example the provider's partition, the file only adds what is missing. vm.swappiness is 10: the server uses swap only when RAM runs short. With No the panel also turns off the existing swaps and comments them out in /etc/fstab.

Since 1.5, with swap on the panel also turns on compressed RAM (zswap, zstd compression): pages going to swap first stay compressed in RAM, 2–3 times smaller and up to 20% of the RAM, and reach the disk only when that part is full. It is much faster than the disk and wears the NVMe drives less. It comes back on at every boot (koavm-zswap service). On nodes added before 1.5 just press Apply swap once.

Swap is a safety net, not extra RAM: even on NVMe the disk is about 100 times slower than RAM. It is fine for RAM the VPSs took but do not use; if the VPSs really use it all at once they all slow down. That is what the real-usage brake is for.

Disks and ZFS cache

On ZFS nodes the panel keeps a cap on the cache (ARC): 10% of RAM, between 2 and 16 GB. Without a cap ZFS would use almost all free RAM and take it away from VPSs under load. The cap is applied by itself to every node (also those already connected) and survives reboots (/etc/modprobe.d/zfs-koavm.conf).

New Linux VPSs created from the catalog get a virtio-blk disk (vda in the system), faster than virtio-scsi; the others stay as they are. Plans with fast writes set sync=disabled on their VPSs' disks (see Plans and systems).

Since 1.5.2 fast writes can also be turned on for a whole node (Nodes › Fast writes): they apply to all its VPSs, including those on plans without them, within seconds and without restarts; a VPS moved to another node follows that node's rules. They help on nodes with desktop-class drives without power-loss protection, where every confirmed write waits for the disk (about 500 per second on gra-02 instead of 5,000): with fast writes they reach about 4,000, but if the node loses power suddenly every VPS loses the last few seconds of data. The node card shows it as "Fast writes".

What matters most for database writes is the disks: datacenter NVMe drives with power-loss protection confirm a write in 20–50 µs, desktop ones in about 1 ms.

More storage pools

Since 1.12 a node can have more than one storage pool, for example mirrored disks for normal VPSs and a separate disk for one service. The pool chosen when you connect the node (default unless you change it) stays the default: VPSs, extra disks, volumes, ISOs and templates go there.

You create the other pools on the node with Incus, for example:

zpool create cdn /dev/sdb1
incus storage create cdn zfs source=cdn

The panel finds them by itself within a minute and shows them on the node card, under Storage pools, with space used and disk assigned to VPSs. A new pool takes no VPSs until you tick its For VPSs box. A VPS goes on another pool:

• if the admin picks it when creating the VPS (the Storage pool field shows up when you pick a node with more than one pool turned on), or with "pool": "cdn" in POST /api/vps;
• if its plan has a preferred storage pool (Clients & plans › Plans, see Plans and systems): on nodes that have it turned on the plan's VPSs go there, on the others on the default pool;
• with Change pool on the VPS page (Storage pool row). Incus copies the disk with its snapshots; the VPS stays off while it copies and starts again by itself if it was on.

Each pool is counted on its own: assigned disk, the disk overbooking cap and the real-use brake apply per pool. If you move a disk by hand with Incus (incus move --storage, incus storage volume move), the panel notices at the next check, within 30 seconds, and updates the VPS pool.

Limits:

• extra disks and volumes stay on the default pool;
• a VPS outside the default pool does not move to other nodes (rebalancing included): bring it back to the default pool with Change pool first;
• backups of VPSs outside the default pool use the chunked archive, not ZFS replication;
• the node's default pool can only change when the node has no VPSs or disks.

Top users

When a node slows down, Top users in the node card shows the VPSs that used the most CPU and disk in the last 10 minutes: the quickest way to find the "noisy neighbour". For each VPS (with a link to its page):

• CPU: how many whole CPUs it used on average and which share of its vCPUs, with its limit;
• Disk: reads and writes in MB/s, operations per second (IOPS) and its per-disk limits, with the share of the limit used by the busiest disk; at the limit when on average it is at 90% or more.

The numbers come from the node, not from the agent inside the VPS (which the customer can stop or fake): every 30 seconds the panel reads over SSH, with a connection that stays open, the counters of the process of every running VPS. Without SSH access to the node the CPU comes from the agent (noted next to it) and the disk is missing. The Disk in the last 24 hours chart on each VPS page uses the same numbers. To brake the ones that overdo it there are the plans' disk and CPU limits; to keep an eye on them there is the Disk at its limit alert (see Alerts).

Disk and CPU limits on the node

Incus puts the processes of all VMs in the same group, the one of its service: there a single VPS cannot be limited. For the limits the panel puts every VPS of the node in its own cgroup v2 group (/sys/fs/cgroup/koavm/<vps>) and writes there:

• the maximum CPU use (cpu.max) and the weight (cpu.weight, 40 per vCPU: on a busy node each VPS counts for the vCPUs it has, as before; low priority counts a quarter, high double). The koavm group gets the sum of the VPS weights as its weight (at most 10,000), so all the VPSs together count as they did inside the Incus service;
• the disk limits (io.max), one line for every disk of the VPS: the main disk and the extra disks or volumes, that is the ZFS zvols and LVM logical volumes QEMU keeps open. Bandwidth (rbps, wbps) and operations per second (riops, wiops) apply together.

Every VPS of the node goes in, not only the limited ones, because otherwise the limited VPSs would count together as a single group. The panel recognises the process of each VPS from the Incus pid file (/run/incus/<vps>/qemu.pid) or from the name on the QEMU command line, and touches no other process.

The limits are applied in the same round that reads disk and CPU, every 30 seconds, and right after creation, start, a plan change, custom resources and disks added or removed; when you save a plan, on the nodes of its VPSs. The round only writes what changed. A VPS that starts has a new process: it runs without limits for one round at most (30 seconds), then goes back into its group. With the groups the disk chart also shows operations per second.

What is not limited:

• disks on dir pools are files, not block devices: the disk limit does not apply to them (the CPU limit does);
• with a kernel older than 6.4 the network vhost threads stay in the Incus group: the CPU they use for the network is not part of the VPS limit (Debian 13 has 6.12, Debian 12 has 6.1).

In Settings › Nodes and overbooking › VPS groups on the nodes you choose where to use them: only on the nodes with at least one VPS that has disk or CPU limits (default), on all of them, or never. When a node no longer uses them (or you choose "never") the panel puts the processes back into the Incus group, removes the limits and deletes the groups: the plans' limits do not apply. A node with cgroup v2 is needed (Debian 12 and 13 have it).

At the bottom of Top users you see whether the groups are active on the node. If the node does not accept a limit (for example a cgroup controller that cannot be enabled or a disk that refuses io.max) the panel writes it to its log and shows it there and on the Top users button of the node card.

Drive and RAM failures

Since 1.7 the panel checks every node every 15 minutes over SSH, without changing anything: the SMART data of each drive (health, wear, spare blocks, temperature, reallocated or unreadable sectors, media errors, last self-test), the ZFS pools and the system RAID (failed or missing drives, read, write and checksum errors, rebuilds), the RAM errors the kernel counts with ECC (corrected and uncorrected, per module) and the kernel log (disk I/O errors, hardware errors of the processor or RAM). If smartctl is missing it installs it (the smartmontools package, without recommended packages); nodes set up by the panel already have it.

The node card shows Drives and RAM as "fine", "warning" or "failing"; the button opens the details with the drives (model, serial number, wear, temperature, power-on hours, errors), the pools, the RAM and the latest error lines of the log, and Check now runs the check right away. A drive that disappears from the server stays in the list as "not answering" for 7 days: once the provider has replaced it, use Remove drives that no longer answer.

When something fails you get an alert (see Alerts): move the VPSs to another node with Drain node and ask the provider to check the server. Without ECC RAM, RAM errors can't be seen: the details say so.

Maintenance and retiring

• Enter maintenance: no new VPSs are created on the node; existing ones keep running. With high availability a node in maintenance never restarts its VPSs elsewhere by itself: put it in maintenance before rebooting it.
• Retire: only possible with no VPSs on the node. The panel also removes its own certificate from the server and, if the node is linked to a provider, can cancel the server.

After a node reboot Incus starts the VPSs that were running. VPSs protected by high availability don't start by themselves (boot.autostart=false): the panel starts them as soon as the node answers, if they haven't restarted elsewhere.

Provider nodes

If the server belongs to OVHcloud, Hetzner, Contabo or Scaleway (Dedibox) and you connected the provider account, the node is recognised by its main IP and its additional IPs become available for VPSs. See Network and IPs.