When you first deploy PostgreSQL onKubernetes, nhiều người thường tập trung vào StatefulSet, PVC hay backup strategy, nhưng chỉ đến lúc nghĩ về sự cố thực tế mới thấy một câu hỏi khó hơn nhiều: nếu primary node bị crash thì điều gì sẽ xảy ra tiếp theo? Ai sẽ quyết định node nào được lên làm primary mới, và làm sao để quá trình đó diễn ra an toàn mà không gây split-brain? Đó là lúc Patroni trở thành một thành phần rất đáng để hiểu kỹ, không chỉ ở mức “biết dùng”, mà còn ở mức “biết vì sao nó hoạt động được”.

Trong bài viết này, mình sẽ cùng bạn đi qua cách Patroni triển khai high availability cho PostgreSQL trong Kubernetes, theo hướng dễ hình dung và bám sát cơ chế thực tế bên trong. Mục tiêu không phải chỉ để cấu hình Patroni chạy được, mà là để bạn hiểu rõ các khái niệm cốt lõi như leader lock, health check loop, leader race và failsafe mode.

Issue

Picture this: at 2 a.m. the primary node of a PostgreSQL cluster crashes. Applications start reporting connection errors and alerts pile up. The question is: who — or what — elects a new primary node before users notice anything is wrong?

PostgreSQL không có cơ chế HA tích hợp sẵn. Nếu primary node crash, toàn bộ cluster trở thành read-only hoặc ngừng hoạt động hoàn toàn cho đến khi DBA can thiệp thủ công. Trên Kubernetes, vấn đề này càng phức tạp hơn: pod có thể bị reschedule bất kỳ lúc nào, IP thay đổi, và không phải lúc nào cũng có DBA phản ứng kịp trong vài giây đầu tiên.

Đây chính là bài toán mà Patroni được tạo ra để giải quyết. Patroni là một high-availability (HA) solution cho PostgreSQL trên nhiều môi trường runtime khác nhau, sử dụng cơ chế failover tự động để đảm bảo cluster PostgreSQL hoạt động kể cả khi primary node gặp sự cố. Bài viết này giải thích cơ chế hoạt động bên trong của Patroni trong Kubernetes: từ cách health check định kỳ, cơ chế leader lock, đến cách phòng tránh race condition.

Summary

After reading this article, you will understand:

How Patroni structures its health check loop to keep the cluster operating normally.
The leader race: what qualifies a replica for promotion, and the order of election.
How Patroni prevents split-brain using the Kubernetes resourceVersion.
How failsafe mode works, and its limitations when the DCS fails.
Strengths, limitations, and when to choose Patroni (or not).

Glossary of terms

PostgreSQL cluster:A group of nodes deployed as 1 primary and N replicas.

A PostgreSQL cluster architecture with 1 primary and N replicas

Race condition:A condition that occurs when several processes try to change the same shared resource at once, producing unintended results.

resourceVersion:Kubernetes uses resourceVersion to implement optimistic concurrency. When a resource is updated, Kubernetes checks whether the resourceVersion in the request matches the resourceVersion currently stored.

If they match, the update is accepted and resourceVersion is set to the new value.
If they do not match, Kubernetes rejects the update and returns HTTP 409.

For example: suppose a Service resource currently has resourceVersion: 308. We update that Service but send a resourceVersion other than 308 (perhaps because someone else updated it first). The result: the request returns HTTP 409 and the update fails.

Patroni architecture and how it works

1. Overview of the model

Patroni uses a DCS (Distributed Configuration Store) as the source of truth for cluster metadata. Patroni supports several types of DCS: etcd, Consul, ZooKeeper, and Kubernetes ConfigMaps or Endpoints.

Patroni is deployed on every PostgreSQL node, so each node runs its own Patroni instance. The Patroni cluster mirrors the topology of the PostgreSQL cluster: one Primary and N Replicas. The Patroni Primary holds a leader lock stored in the DCS, and the Patroni Replicas watch that lock.

The role of each Patroni instance is determined via the DCS, which directly decides the role of its corresponding PostgreSQL instance: whichever Patroni instance holds the leader lock has its PostgreSQL instance promoted to Primary; the remaining Patroni instances run in the Replica role, and their corresponding PostgreSQL instances are Replicas.

Patroni deployment model

2. Health check loop

Patroni supports two mechanisms for storing the leader lock on Kubernetes: Endpoints and ConfigMaps. This article focuses on how a Kubernetes Endpoint acts as the DCS. When a Kubernetes Endpoint is used as the DCS, the leader lock is stored as follows:

$ kubectl get ep postgres-cluster -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    leader: postgres-cluster-0
    renewTime: "2026-05-10T09:03:36.094563+00:00"
    ttl: "30"
  name: postgres-cluster
  resourceVersion: "49818310"
...

The leader lock is in fact an Endpoints resource on Kubernetes, in which the renewTime annotation records the moment the Patroni Primary last updated the lock. The leader lock carries a TTL (time to live).

Patroni checks cluster state on a cycle (10 seconds by default, set by the loop_wait parameter). In each cycle, Patroni performs the following in order:

Retrieves information from DCS.
Check leader lock status:
1. If the lock is still valid: the Primary refreshes the lock and the Replicas take no action on it.
2. If the lock expires (or does not exist), the replicas all compete for a new lock — this is the leader race.

Example:The lock is renewed at 11:30:00 with a 30-second TTL. If it has still not been renewed by 11:30:30, the replicas begin a leader race.

3. Automatic failover

When the leader lock expires:

Each Patroni replica checks whether it is eligible to become the new leader, based on three conditions:
1. Lag:Replication lag does not exceed maximum_lag_on_failover.
2. Timeline:The node is on a WAL timeline compatible with the cluster.
3. Quorum (if synchronous mode is enabled): the node must be in the synchronous standby list, meaning it had received and written all WAL from the primary before the primary crashed.
The first Patroni replica to acquire the lock is promoted to Patroni primary. At that point Patroni calls pg_promote() to promote the corresponding PostgreSQL instance. The remaining nodes recognize the new leader and switch to watching that node's lock.

In parallel, if the former Patroni primary recovers from the failure and detects that a new leader already exists, Patroni automatically performs a demote: it stops PostgreSQL, runs pg_rewind to align the WAL timeline with the new primary (if enabled), and then restarts the node in standby mode.

How does Patroni prevent a leader race between Patroni replicas?

Patroni uses the Kubernetes resourceVersion mechanism to ensure that no race condition occurs during leader election. Each node sends a lock update request carrying the Endpoint's current resourceVersion:

If the resourceVersion matches, the request succeeds and the node becomes the new leader.
If the resourceVersion does not match, Kubernetes returns HTTP 409 (StatusConflict) and the node cannot become leader.

Case 1 – multiple nodes contending for the lock:The current resourceVersion is 100. Two replicas simultaneously send a request to update the endpoint with resourceVersion = 100. The request that arrives first succeeds and that node is promoted to primary. The remaining requests receive HTTP 409 because resourceVersion has changed.

Sequence diagram for case 1

Case 2 – Slow node:resourceVersion starts at 100. Node A updates successfully and becomes the leader, and resourceVersion increases to 101. Node B is lagging; when it performs its operation it reads the latest state with resourceVersion = 101, recognizes that a valid leader already exists, and continues to run as a Replica.

Sequence diagram for case 2

Case 3 – renewal request is delayed:If the Primary's renewal request is delayed long enough for the lock to expire, a leader race is triggered and a new node is elected, changing the resourceVersion. When the Primary's old request finally arrives, Kubernetes rejects it with HTTP 409 because the resourceVersion no longer matches. The old Primary realizes it has lost the lock and automatically demotes itself to Replica.

Sequence diagram for case 3

4. Failsafe mode

Failsafe mode is the second layer of protection when the DCS is unavailable. When it is enabled, the primary can retain its role provided it can still reach every other node in the cluster through Patroni's REST API.

Sequence diagram showing how failsafe mode works in Patroni

In this state, however, failover is no longer available. If the Primary fails while Failsafe is active, no Replica is promoted to take its place. The entire cluster becomes read-only.

What to consider when deploying PostgreSQL high availability with Patroni on Kubernetes

Strengths:

Integrates natively with Kubernetes, using Endpoints as the DCS without a separate etcd.
Fully automatic leader election.
Supports a range of HA options: failsafe mode, synchronous replication and custom bootstrap hooks.
Large community, extensive documentation.
Supports multiple DCS backends: etcd, Consul, ZooKeeper and Kubernetes Endpoints.

Limitations:

Dependence on the DCS: if the DCS fails, the cluster can fall into a read-only state — unless failsafe mode is enabled.
Operationally more complex than managed services (RDS, CloudSQL, AlloyDB).
You need to understand the ttl, loop_wait and maximum_lag_on_failover parameters to configure them correctly per workload.

Conclusion

If Patroni was previously just a name that kept appearing in PostgreSQL HA diagrams, we hope this article has made how it works considerably clearer. Rather than seeing Patroni only as an “automatic failover” tool, you should now have a clearer picture of how it monitors cluster state, elects a new leader when something fails, and works with Kubernetes to avoid dangerous situations such as split-brain. With that internal behavior understood, you can approach a Patroni deployment with more confidence — and judge more accurately whether it is the right fit for your Kubernetes environment.

Fundamentally, Patroni solves high availability for PostgreSQL by continuously maintaining a leader lock in the DCS and automatically triggering an election when that lock expires. The core mechanisms work together across three layers:

Health check loopso that cluster state is checked continuously. The Primary renews the lock, while the Replicas watch the lock to detect a failure early.
Leader raceoccurs when the lock expires. The Replicas then compete to become the new Primary based on conditions such as replication lag, WAL timeline and quorum. At the same time, Patroni uses the Kubernetes resourceVersion to ensure that only one node can acquire the lock, which prevents split-brain.
Failsafe Modeacts as the fallback layer when the DCS fails. In this state the primary can keep operating as long as it can still reach the entire cluster, but in exchange the system loses the ability to fail over automatically.

More broadly, Patroni suits systems that need a high degree of control and want to run PostgreSQL on Kubernetes themselves. For workloads that prioritize operational simplicity, managed services remain the option more worth considering.