Quorum Is a Voting Problem
The first cluster most people build has two nodes, because two is what most people own. It works, right up until one of them reboots for updates, at which point the survivor refuses to start a single virtual machine and the whole thing looks broken.
It is not broken. It is doing exactly what it was told.
The arithmetic#
Corosync gives each node one vote and requires a strict majority to operate.
The rule is floor(votes / 2) + 1.
| Nodes | Votes | Majority needed | Failures survived |
|---|---|---|---|
| 2 | 2 | 2 | 0 |
| 3 | 3 | 2 | 1 |
| 4 | 4 | 3 | 1 |
| 5 | 5 | 3 | 2 |
Two nodes need both votes, so losing either one loses quorum. Note the fourth row: adding a fourth node to a three-node cluster buys nothing at all. It raises the majority from two to three while raising the available votes by one, and you still survive exactly one failure. Even-numbered clusters are the awkward case generally, not just at two.
Why not just let the survivor carry on#
Because it cannot tell the difference between "the other node is off" and "the other node is fine and I am the one who lost the network."
If both nodes assumed they were the survivor, both would start the same virtual machines against the same shared storage. That is split-brain, and it does not degrade gracefully – it corrupts. Refusing to operate without a majority is the cluster choosing downtime over data loss, which is the correct trade and the reason the behaviour is not configurable away.
The fix is a third voter, not a third server#
A QDevice is a quorum witness: a daemon running somewhere outside the cluster that holds a vote and nothing else. Two nodes plus a QDevice is three votes, so either node can fail and the survivor plus the witness still holds a majority.
The witness needs almost nothing. The
Proxmox documentation asks only
for network reachability and the corosync-qnetd package; a small VM or
container with 512 MB of RAM is sufficient, reachable on port 5403 from both
nodes.
The one rule that matters: it has to fail independently. A witness running as a virtual machine on one of the two nodes is not a third voter, it is a complicated way of giving one node two votes. Put it on a Raspberry Pi, a NAS, or anything else that does not share a power supply with the cluster.
Where this leaves a four-node cluster#
Four nodes survive one failure, same as three. The reason to run four is capacity, not availability, and if availability is the goal the honest options are five nodes or four plus a witness.
There is a useful operational distinction here. Planned maintenance is still a failure as far as quorum is concerned: before taking a node down, verify that the remaining voters can form a majority. That one check turns the behaviour from a surprise in the middle of patching into a constraint you can schedule around.
Sources#
- Proxmox VE, Cluster Manager
- Proxmox VE, pvecm documentation