Guide · For administrators
Alerts
The panel checks every minute and alerts you when:
| Group | Alert |
|---|---|
| Nodes | node unreachable for a few minutes (and when it's back), node disk or RAM above the threshold, private network tunnels not ready; since 1.7 drive and RAM failures (see below) |
| Backup | backup failed |
| VPS | VPS in error after a creation, reinstall or deletion (and when it's fine again), move between nodes failed; since 1.11 disk almost full inside the VPS (see below); since 1.14 disk at its limit (off by default, see Thresholds); since 1.15 failed exit from rescue mode and VPS restarted with a backup copy (files from backups) |
| Updates | a new KoaVM version is out |
An open alert isn't repeated until the problem goes away; then the alert closes and, for nodes and VPSs, a "fine again" message arrives too. The Alerts page shows the open ones (you can close them by hand) and the history of the last 90 days.
Drive and RAM failures
From the node check every 15 minutes (see Nodes):
| Alert | When |
|---|---|
| Drive failing | SMART reports the drive is failing, spare blocks used up, drive switched to read-only, unreadable or unrecoverable sectors, reallocated sectors or media errors increasing, attribute below the manufacturer's threshold, failed self-test, wear above 100%, drive no longer answering |
| Drive to keep an eye on | wear above 80%, temperature above the limit, old reallocated sectors or media errors (not increasing) |
| Pool or RAID with problems | ZFS pool not ONLINE, read, write or checksum errors, data errors, RAID degraded or not active |
| Disk I/O errors | new lines in the kernel log (closes after 24 hours without new errors) |
| RAM errors | uncorrected errors since boot or processor hardware errors (MCE) |
| Corrected RAM errors | ECC corrected errors in the last 24 hours: VPSs are not affected, but if they keep coming the module needs replacing |
There is one alert per drive, pool or RAID; when the problem goes away (or the drive is replaced and removed from the list) the alert closes.
Disk almost full inside VPSs
Since 1.11 the panel measures the space inside running VPSs every 10 minutes, with the Incus agent: df on the real file systems of the OS (ext4, XFS, Btrfs, ZFS; no tmpfs, overlay, read-only or network disks). The space of the volume on the node is not enough: with ZFS and LVM thin the blocks of deleted files stay allocated until the OS trims them, so the disk would look fuller and fuller.
- Alert on the channels: when the space or the inodes of a disk go over the threshold (90% by default) you get "Disk almost full", with the disk, the free space and, if the disk is growing, when it fills up at that rate. It closes 5 points below the threshold.
- VPS threshold: on the VPS page, Disk space card, the administrator picks the general threshold for that VPS, its own threshold, or never.
- Email to the client: on the same card the client (or the administrator for them) asks for an email when a disk goes over the threshold they choose. It is sent once and armed again when the disk drops 5 points below the threshold.
The card shows each disk with its usage, free space, growth in the last 24 hours, the forecast and a chart of the last 7 days; Measure now measures right away. In the VPS list the Disk column shows the fullest disk of each VPS and the Disk full filter shows those over 90%. VPSs without the agent, with Windows or from providers are not measured.
Channels
In Alerts › Add channel:
- Email: one or more addresses, comma separated. They are sent with the panel's email settings.
- Webhook: the panel sends a JSON POST with
event,level,title,text,resolved,argsandurl. - Slack and Discord: the channel's webhook URL (Slack: Apps › Incoming Webhooks; Discord: Channel settings › Integrations › Webhooks).
- Telegram: the bot token (from @BotFather) and the chat (number or @channelname). Add the bot to the chat before testing.
For each channel choose the language (Italian or English) and the groups to send (none = all). Test sends a message right away; if a delivery fails, the error stays on the channel.
On the first start of 0.8, if the support email is set, the panel creates an email channel to that address on its own.
Thresholds
Node disk and RAM (real usage on the server, 85% and 90% by default) and after how many minutes a node is "unreachable" (3 by default). A disk or RAM alert closes when the value drops 5 points below the threshold.
Disk inside VPSs: measuring with the agent (it can be turned off: no measurements, no alerts) and the general alert threshold (90% by default; unticked means no alert, except for VPSs with a threshold of their own).
Disk at its limit (since 1.14, off by default): alerts when a VPS with disk limits keeps a disk at the maximum (at least 90% of the limit in every sample) for longer than the minutes you choose, from 5 to 240. The limit applies to each disk, so the busiest disk counts, not the sum of the disks. The alert says where the busiest disk is, how much the VPS reads and writes in total and the limits; it closes when it drops below 80%. It helps to see who needs a bigger plan.