Early Deadlock Detection with pg_stat_kcache and eBPF Uprobes

The dreaded deadlock. For anyone managing a high-traffic PostgreSQL database, it’s a familiar nemesis. The database logs a “deadlock detected” message, a transaction is unceremoniously aborted, and your application has to handle the fallout. This reactive cycle is frustrating; by the time you’re alerted, the damage is already done. Traditional postgres_metrics and monitoring tools are excellent at telling you that a deadlock occurred, but they fall short of explaining the subtle conditions that led to it.

What if you could move from post-mortem analysis to pre-mortem prediction? What if you could identify the “perfect storm” of conditions that foster deadlocks and intervene before they happen? This is the promise of a more advanced pg_observability strategy. By combining the deep resource insights of the pg_stat_kcache extension with the powerful real-time tracing capabilities of ebpf_postgres, you can build a system for proactive_monitoring that detects the precursors to deadlocks, offering a chance to fix the root cause rather than just cleaning up the aftermath.

The Anatomy of a Deadlock: Time is the Secret Ingredient

We often think of a deadlock as a simple, logical problem: Transaction A waits for B, and B waits for A. While this circular dependency is the definition, it overlooks the most critical ingredient: time. For a deadlock to occur, transactions must hold their locks for a long enough duration for a conflicting, circular wait to form. A transaction that acquires a lock and commits in a few milliseconds is highly unlikely to cause a deadlock. A transaction that holds a lock for hundreds of milliseconds or even seconds is a ticking time bomb.

So, what causes a transaction to hold locks for so long? It’s often not application-level pg_sleep() calls. The real culprits are hidden within the database’s execution plan:

1. Expensive I/O Operations: A query that needs to read many data blocks from disk will naturally take longer. The block_read_time is a major contributor to transaction duration.
2. High CPU Usage: Complex calculations, sorts, or aggregations can cause a high cpu_usage_query, extending the time a transaction remains active.

Herein lies the challenge. Standard PostgreSQL monitoring tools like pg_stat_activity might show you a lock_wait event, but they won’t tell you why the blocking transaction is taking so long. Is it performing a heavy I/O operation? Is it consuming excessive CPU? Without this context, you’re flying blind.

Uncovering Hidden Query Costs with pg_stat_kcache

This is where pg_stat_kcache becomes an indispensable tool. It’s a PostgreSQL extension that provides what standard tools lack: detailed, per-query resource usage statistics gathered directly from the Linux kernel. It hooks into the kernel’s process accounting to track, with remarkable accuracy, the I/O and CPU time consumed by every query.

When you enable postgres_kcache, you gain access to crucial metrics for each normalized query, including:

• reads and writes: The number of blocks read from and written to disk.
• read_time and write_time: The cumulative time spent waiting for those I/O operations to complete.
• user_cpu_time and system_cpu_time: The amount of CPU time the query consumed.

The power of pg_stat_kcache is fully realized when you join its view with pg_stat_statements. This allows you to create a latency_heatmap of your workload, pinpointing exactly which queries are the “heavy hitters” in terms of I/O and CPU. A query might have a low average execution time but a very high block_read_time, indicating that its performance is highly dependent on the cache hit rate. These are precisely the queries that, under certain conditions, can become slow, hold locks for too long, and catalyze a deadlock.

Seeing Locks in Real-Time with eBPF Uprobes

While pg_stat_kcache tells you the “who” (which queries are expensive), ebpf_postgres tells you the “when” and “how long” of lock contention. eBPF (extended Berkeley Packet Filter) allows you to safely and dynamically trace events in both the kernel and user-space. Using a postgres_uprobe (user-space probe), we can attach tracing logic to any function inside the running PostgreSQL binary itself—without restarts, recompilation, or significant performance overhead.

For lock_wait_prediction, we are interested in the functions responsible for acquiring locks, such as LockAcquire. We can use tools from the bcc_tools_postgres suite or simple bpftrace one-liners to hook into this function.

Imagine an eBPF script that does the following every time LockAcquire is called:

1. Records the timestamp at the start of the call.
2. Captures context: the Process ID (PID), the type of lock being requested, and the relation (table) it’s for.
3. Records the timestamp at the function’s exit.
4. Calculates the duration and emits an event if it exceeds a certain threshold (e.g., 10ms).

This eBPF_tracing_sql approach gives us a live stream of lock contention events. We can aggregate this data to build a lock_wait_histogram, which is far more powerful than a simple counter. It doesn’t just tell us that lock waits are happening; it tells us their distribution. Are most waits under 1ms, or do we have a long tail of waits extending into hundreds of milliseconds? This distribution is a powerful leading indicator of system-wide contention.

A Proactive Deadlock Detection Strategy

Now, we combine these two powerful tools into a unified strategy for deadlock_prediction.

Step 1: The Baseline (The “Who”) Continuously use pg_stat_kcache to maintain a ranked list of your most resource-intensive queries. Pay special attention to queries with high variance in their block_read_time or cpu_usage_query. These are your primary suspects—the queries most likely to cause trouble under load.

Step 2: Live Tracing (The “When”) Deploy eBPF uprobes to monitor the lock acquisition functions. You can configure them to only trace activity related to the “suspect” queries identified in Step 1, minimizing the overhead of data collection. The goal is to capture any instances where these specific queries hold locks for an unusually long time.

Step 3: Correlation and Prediction (The “Alert”) This is the heart of the proactive_monitoring system. The deadlock_alerts are no longer based on the deadlocks counter in pg_stat_database. Instead, you create a more intelligent alert based on leading indicators:

• Rule: “Trigger an alert if a transaction associated with a historically I/O-heavy query (from pg_stat_kcache) holds an EXCLUSIVE lock for more than 500ms (from the eBPF trace).”

This alert fires before a deadlock has a chance to form. It tells you that a potentially dangerous condition is unfolding right now. You can then immediately investigate the specific PID from the eBPF trace, look up its query in pg_stat_activity, and decide on a course of action—whether it’s optimizing the query, killing the specific session, or gathering more data—before it brings other parts of the application to a halt.

Moving Beyond Reactivity

The traditional approach to database monitoring often feels like driving by looking only in the rearview mirror. You see the problems after you’ve already passed them. By integrating kernel-level statistics from pg_stat_kcache with the live, function-level tracing of eBPF, you can finally start looking ahead. This advanced form of pg_observability allows you to understand the intricate dance between query resource consumption and lock contention, giving you the insight needed to build more resilient and performant database systems.

This powerful combination of tools and techniques provides the ground truth about your database’s performance. Netdata embraces this philosophy by correlating metrics from every layer of your stack—from the kernel to the application—onto a single screen. It gives you the high-granularity, real-time visibility needed to implement a proactive strategy, turning your monitoring platform into a predictive tool.

Start predicting problems instead of reacting to them. Try Netdata for free today.