Standby & failover
Keep a second server with a constant copy of a primary, see how far behind it is, and let it take over when the primary fails — by hand or automatically.
A standby is a second server that keeps a copy of a primary all the time. Websites' files, mailboxes, MariaDB databases and (when the primary is this panel's own server) the panel's own data are copied to it as they change. The screen shows how far behind each kind of data is, so you always know how fresh the copy is. When the primary fails, the standby takes over: you promote it with one button, or it does so by itself in automatic mode.
Where: Server Admin › Accounts › Standby & failover
Note: this menu entry appears only when at least one other server is joined under Servers. It is part of the High availability licence feature, which needs the Servers feature too.
Before you start
- The standby is a server joined under Servers that hosts no accounts of its own. While it is a standby, no new accounts are placed on it.
- Both servers need their public IP set on the Servers screen: each one lets the other in by it.
- The primary's SSH must listen on port 22 and allow root with keys (
PermitRootLoginnotno;forced-commands-onlyis enough). - The primary's mail certificate must be valid for its mail host name, since the standby checks it.
- MariaDB 10.11 or newer on both servers. Copying the databases turns on MariaDB's binary log on the primary, which restarts MariaDB once.
- For automatic failover, a third server joined under Servers: the witness (see below). A DNS cluster secondary is the best choice.
- Promoting needs two-step sign-in on your login.
Add a standby
- Click Create on the Add a standby card.
- Servers: choose the Primary (this panel's server or another joined one) and the Standby.
- Space: click Check space. The copy needs about everything the primary uses, plus a tenth; the standby must have that much free. Click Next once it says Enough space.
- First copy: click Start the first copy.
The first copy runs in the background and can take hours on a big server. Meanwhile the pair shows First copy running. When the files and the mail have been copied once, it becomes In step.
What is copied, and how
| Data | How | Usually behind by |
|---|---|---|
| Files | Each hosting account's home folder, as soon as something in it changes, plus a full pass every hour. The standby fetches it with rsync over SSH with a key of its own; on the primary that key can only read. | seconds |
| Each mailbox that changes, with Dovecot's own copy tool over an encrypted connection, protected by a secret that only this pair uses. | seconds | |
| MariaDB | Replication of every change, over an encrypted connection, as a database user only the standby may use. It starts from a consistent copy of all databases with their users. | seconds |
| Panel data | A copy of the panel's database every 5 minutes, sent over the signed link the servers joined with. Only when the primary is this panel's own server. | up to 5 minutes |
Homes of accounts deleted on the primary are removed from the standby by the next full pass. Mail waiting in the primary's queue is not copied; the senders try again.
The list
| Column | What it shows |
|---|---|
| Primary / Standby | The two servers, and when the pair was added. |
| Status | First copy running, First copy failed, In step, Falling behind (one kind is further behind than it should be), Not copying (one kind has an error or no recent copy), Promoting or Standby took over. Below it: Manual, or Automatic, after … s. |
| Files, Email, MariaDB, Panel data | How long the oldest change not copied yet has waited. Not copying shows the reason when you point at it; Not yet means no copy has finished; a dash means the pair copies no such data. |
| Last check | When the pair was last checked. |
Every pair is checked every 5 minutes; Check now checks them all straight away. A pair counts as falling behind when files or mail are more than 2 minutes behind, MariaDB more than a minute, or the panel data more than 15 minutes, and as not copying when a kind has not been copied for two hours.
Details shows what the last look at the primary saw (it answers, or since when it has not, and what the witness saw), the witness's name and the pair's history: when it was added, when the first copy finished or failed, each change of state and each promotion, with the reason. Copy again starts the first copy once more after it failed or after copying stopped.
Failover settings
Click Failover on a pair's row:
| Setting | What it does |
|---|---|
| Mode | Manual (the default): when the primary stops answering you get an alert and promote by hand. Automatic: the standby is promoted by itself when the primary and the witness agree it is gone. |
| Fail after (seconds) | How long the primary may not answer before the alert or the automatic promotion. Default 180 (3 minutes); 60 to 3600. |
| Witness | A third server that must not reach the primary either. Automatic picks a DNS cluster secondary first, else this panel's server when it is not in the pair, else another joined server. |
How the primary is watched. Every 20 seconds the standby tries the primary: HTTPS on the link between the servers, the web ports 80 and 443, mail on 25 and 993, and MariaDB. When none of them answers, the witness tries too.
The witness rule. An automatic promotion happens only when both the standby and the witness cannot reach the primary for the whole "fail after" time. If the witness can still reach it, the standby itself is probably cut off, and nothing is promoted — you get an alert instead. This stops a network cut from leaving two primaries that each take changes.
Alerts. The administrators get a notification and an email when a standby falls behind or stops copying, when
the primary stops answering, and when a standby is promoted (or a promotion fails). Choose where they go under
your notification settings. The event ha.failover also goes to webhooks and
script hooks.
Promote now
Click Promote now on the pair's row. The dialog says what happens and warns you when the witness can still reach the primary or could not be asked. Type the standby's name and click Promote now. Two-step sign-in is needed and the promotion is written to the audit log.
What the standby does:
- Stops copying and ends MariaDB replication; its databases become writable.
- Rebuilds every account from the panel data and the account settings it copied (websites, PHP, certificates, DKIM keys, mailboxes, FTP logins, cron jobs, apps, limits), and starts the web server, PHP, mail and MariaDB.
- Becomes the DNS master: in every zone, every record that named the primary's address — A and AAAA records,
ip4:/ip6:in SPF records and any other record holding the address — now names the standby's. The zones are published and the DNS cluster secondaries copy them at once. - The accounts now belong to the standby, which takes new accounts; the old primary does not.
When the primary is this panel's own server, the standby takes over the panel too: it starts again on the latest copy of the panel data (at most 5 minutes old) and becomes the main panel. Sign in there with the same logins. If this server is still up, its websites, mail and MariaDB stop at once.
When this panel's server is gone, open Standby & failover on the standby itself: it shows This server is the standby of … with its own Promote now.
The 5-minute TTL
While a pair is active every zone is published so that no record is kept longer than 5 minutes (TTL 300) by other DNS servers, so visitors follow a promotion within minutes. Records keep the TTL you gave them; it comes back when the last pair is removed.
Fencing: when the old primary comes back
A server that was primary asks its standby who is primary when it starts and every 20 seconds after. If the standby has taken over, the old primary stops its web server, PHP, mail and MariaDB and keeps them stopped, even after a restart, so the two servers never both take changes. Its Server Admin shows This server was replaced by ….
Fail back
To make the original server the primary again:
- On the old primary's banner click Make it the new standby, or on the main panel click Fail back on the pair's row (when the old primary was this panel's own server, join it under Servers first and choose it).
- The old primary is copied to again from the new one, as a first copy: the pair is reversed.
- When the pair is In step, choose a maintenance window and click Promote now, then Planned switch-over: both servers in step first. The current primary stops its websites and incoming mail, MariaDB stops taking changes, the standby copies the last of them, then it takes over. Websites and mail pause for a few minutes; no change is lost. If the standby can't catch up within 5 minutes, nothing is promoted and the primary takes changes again.
What customers notice
- Websites: unreachable from the moment the primary fails until the promotion plus the DNS change — about the "fail after" time plus a few minutes in automatic mode. Files changed in the last seconds before the failure may be missing.
- Email: senders keep mail in their queues and deliver it once DNS points at the standby. Mail that was only in the primary's own queue is delivered when the old primary comes back and is failed back. Mail apps keep working with the same server name.
- Passwords, mailboxes, databases and settings stay the same. Panel changes made in the last 5 minutes before the main panel failed may be missing.
- Let's Encrypt certificates are copied, so HTTPS keeps working.
Remove a pair
Click Remove on its row and confirm. Both servers stop copying and forget the pair's key, secrets and database user. The standby keeps the data it already has. If a server does not answer, the pair is still removed here and the message names the server, so you can remove it there again later.
A server that belongs to a pair can't be removed under Servers; remove its pair first. On the Servers screen a standby carries a Standby of … label, and Fleet shows each primary's Standby lag.