High Availability

High availability is redundancy plus automated failover plus honest accounting of what “available” meant. The math is unforgiving: availability multiplies across components, so HA is a chain-of-weakest-links exercise — and for stateful services (databases), the hard part is never adding replicas, it’s election, split-brain prevention, and failover semantics.

The vocabulary that separates the designs:

• Replication: async (cheap, can lose acked writes on failover) vs sync/quorum (writes wait for consensus, no data loss, latency cost). For databases this is the trade-off decision.
• Election / quorum: who becomes primary when the primary dies — Patroni-style external consensus (etcd/DCS), Galera’s synchronous multi-master quorum, or K8s-native (the operator + API server as the DCS).
• Failover vs switchover: unplanned (race against data loss) vs planned maintenance (ordered, verified).
• Fencing: guaranteeing the old primary is really dead before promoting a new one — the split-brain killer. Often overlooked, always catastrophic when missed.
• Client rerouting: HA is end-to-end — DNS/proxy/VIP/service abstraction must follow the new primary (HAProxy and Reverse proxies are this layer; MaxScale for MariaDB, kube-proxy/services or a proxy for K8s).

Database High Availability

Databases make every HA problem harder: replicated state means failover decisions are data-loss decisions. The current best practice on Kubernetes:

• cloudnative-pg (OSS) (9.4k★, Go, Apache-2.0) — “the most popular Kubernetes Operator for PostgreSQL”: primary/standby streaming replication managed natively via the Kubernetes API (no Patroni/repmgr/Stolon sidecar — the operator is the HA manager: “if the primary instance fails, the operator elects a new primary, updates the cluster status, and orchestrates the transition”). Immutable application containers, declarative Cluster resource, rolling updates, volume snapshots.
• mariadb-operator (OSS) (1.0k★, Go, MIT) — the MariaDB counterpart with a topology menu: async replication, synchronous multi-master via Galera, multi-cluster replication for multi-region/DR/blue-green, MaxScale proxy integration for load balancing and failover/switchover; physical backups (mariadb-backup + VolumeSnapshots) and logical (mariadb-dump) to S3/Azure/PVC.

The pattern transfer: the operator pattern turns DBA HA procedures into a reconciled declarative resource (see Kubernetes Operators) — which is exactly why these repos double as reference material for the non-K8s world: the failover state machines, health-check criteria, and rolling-upgrade orderings are documented in code.

Non-K8s anchors: Patroni (etcd/Consul/ZooKeeper consensus for PG), Galera (quorum replication for MySQL/MariaDB), and the poor-man’s ladder — streaming replica + manual promotion — which is where most homelabs (and quite a few prod setups, embarrassingly) actually live.

Homelab read: single-node doombox means HA for stateful services is mostly backup discipline + reprovisioning speed (Homelabs); true DB HA needs ≥3 failure domains (quorum) — the Pi fleet could run a Galera/MariaDB trio experiment but the ops cost likely exceeds the availability gained. Mail servers and Reverse proxies layers face the same calculus.

Related: Kubernetes Operators · Reverse proxies (the rerouting layer) · State machines (failover is a statechart whether you admit it or not) · Uncloud/pi-cluster (OSS) (where the failure domains would come from)