High Availability on Netdata

Netdata Parents For Intelligent Observability

Thu, 18 Dec 2025 00:00:00 +0000

Streaming aggregators that centralize data from thousands of nodes while keeping intelligence distributed at the edge. Handle million metrics per second with minimal resources, active-active clustering for high availability, and ML-powered insights without centralized bottlenecks.

Zero-Downtime Monitoring For Real-Time Deployments

Thu, 18 Dec 2025 00:00:00 +0000

Netdata’s distributed architecture eliminates single points of failure in monitoring, delivering sub-2-second visibility with automatic failover, zero data loss, and production-safe resource overhead—ensuring complete observability during the moments that matter most.

Elasticsearch Yellow Cluster: Unassigned Shards Fix

Sun, 07 Sep 2025 00:00:00 +0000

You run a health check on your production cluster and the result comes back: status: yellow. It’s not the dreaded red status, so your application is likely still serving requests, but this is a critical warning sign. An Elasticsearch yellow cluster status is a direct indication that your data’s high availability is compromised. While all your primary shards are active, one or more replica shards have failed to be assigned to a node. Ignoring this warning can lead to data loss if another node fails, or worse, it could be a symptom of a network partition risking an Elasticsearch split-brain.

Consul Service Discovery Failures: Causes & Fixes

Wed, 03 Sep 2025 00:00:00 +0000

It’s a scenario that keeps DevOps and SRE teams up at night: your application logs fill with connection errors, services start failing, and you realize a critical component can’t find the database it depends on. The culprit? A breakdown in your service discovery mechanism. For many, that mechanism is HashiCorp Consul, the backbone of modern microservice architectures. When Consul falters, your entire ecosystem can become unstable.

Understanding how to diagnose these failures is crucial. The problem often lies deep within the operational layers—agent communication issues, misconfigured health checks, or disruptions in the gossip protocol that maintains cluster state. In this guide, we’ll dissect the most common causes of Consul service discovery failures, providing you with the tools to troubleshoot and resolve them. More importantly, we’ll show you how to shift from a reactive, fire-fighting mode to a proactive one, using comprehensive monitoring to build a truly resilient Consul deployment.

Redis Sentinel Failover & Split-Brain Recovery Guide

Sun, 24 Aug 2025 00:00:00 +0000

You’re on call. An alert fires—your Redis master node is unreachable. Your heart rate quickens. You’ve set up Redis Sentinel for high availability, but is it working? Did the failover succeed? Or worse, are you now in a Redis cluster split-brain situation where two nodes think they’re the master, leading to data inconsistency and eventual loss? In these critical moments, blindly trusting the automation isn’t enough; you need to verify what’s happening.

Ecommerce Infrastructure: Components & Benefits

Tue, 29 Apr 2025 00:00:00 +0000

The world of online shopping has exploded, offering convenience and accessibility like never before. Businesses can reach global audiences, and consumers can browse and buy from anywhere. But behind every successful online store, from small boutiques to giants like Amazon, lies a complex system working tirelessly: the ecommerce infrastructure. Without this foundation, websites crash during peak traffic, customer data gets compromised, and orders get lost – leading to frustrated customers and lost revenue.

What Is Database Clustering? Types & Benefits

Thu, 06 Mar 2025 00:00:00 +0000

Database clustering is a robust strategy employed to enhance the performance, scalability, and availability of databases by orchestrating the distribution of data across multiple servers. This setup doesn’t just streamline handling big data; it also keeps things running smoothly, even if some nodes go down. That way, your service stays up and available no matter what happens.

By diving into the various types of database clustering architectures, in this article we explain how each setup addresses specific needs and challenges within IT environments. We’ll also examine the tangible benefits that database clustering brings to businesses, from improved data redundancy to enhanced query response times.

6 + 1 Effective Strategies to Reduce Unplanned Downtime

Wed, 30 Oct 2024 00:00:00 +0000

For DevOps and SRE teams, unplanned downtime may be a nightmare since it can ruin everything, from customer satisfaction to corporate operations. With more and more systems relying on constant availability, any unplanned downtime must be eliminated to ensure reliability of service. In the article below, we will discuss how to utilize infrastructure monitoring, monitoring tools, development and operations best practices in order to minimize unplanned downtime.

What is Unplanned Downtime

Unplanned downtime occurs when an application or system or infrastructure component fails without notice, causing an interruption. These disruptions could be due to a network issue, or a problem in the software or hardware or even human error. The business expenses are often substantial in terms of revenue loss and negative publicity. So, putting a plan in place to reduce downtime is critical.

How To Achieve High Availability In CI/CD With Observability

Sun, 09 Jun 2024 00:00:00 +0000

Your CI/CD pipeline is the backbone of your software delivery process. When it works, code flows smoothly from commit to production. But what happens when it breaks? A failed pipeline means stalled feature releases, delayed bug fixes, and frustrated developers unable to ship their work. To prevent this, you need to treat your CI/CD infrastructure with the same rigor as your production applications, and that starts with making it highly available.

Netdata Parents (Streaming and Replication)

Fri, 30 Jun 2023 00:00:00 +0000

What are they and why do we need them?

A “Parent” is a Netdata Agent, like the ones we install on all our systems, but is configured as a central node that receives, stores and processes metrics data from other Netdata “Child” nodes in our infrastructure.

Netdata Parents are flexible. You can have one big active-active cluster of Netdata Parents, or you can spread a lot of independent Parents across the infrastructure.

Infinite Scalability: Monitoring Without Limits

Thu, 04 May 2023 00:00:00 +0000

Scalability is crucial for monitoring systems as it ensures that they can accommodate growth, maintain performance, provide flexibility, optimize costs, enhance fault tolerance, and support informed decision-making, all of which are critical for effective infrastructure management.

Most monitoring solutions struggle with scalability, mainly because of:

High data volume and velocity: Monitoring systems generate vast amounts of data and as the infrastructure grows, so does the volume and velocity of these data.
Resource constraints: Scalability requires efficient resource utilization, leading to bottlenecks and performance issues as the monitored environment grows.
Architectural limitations: Monitoring systems are usually designed with certain architectural constraints that limit their scalability. Most open source solutions rely on monolithic or centralized architectures that can become overwhelmed at scale.

For open source solutions scalability has always been a challenge, increasing their complexity significantly (check for example the scalability issues of Prometheus), while for commercial solutions it usually results in increased data collection to visualization latency and cost.

Server Uptime Monitoring: Core Benefits For High Performance

Tue, 02 May 2023 00:00:00 +0000

Server uptime monitoring tracks the availability and reliability of servers within your infrastructure.

What Is Server Uptime Monitoring?

Server uptime monitoring is the process of continuously tracking the operational status of your servers to ensure optimal performance and availability for users.

With Netdata, you gain access to real-time, high-resolution monitoring that goes beyond basic checks, providing a detailed overview of your entire infrastructure.

Why is data replication important?

Wed, 12 Oct 2022 00:00:00 +0000

High availability. This is what every monitoring tool needs to ensure that you never compromise on IT infrastructure visibility. On top of high availability, do you really want to enable all available features on your production system? It is important for the monitoring tool to have a low footprint on your CPU consumption and memory usage. Let’s dive deeper into the recommended way of configuring Netdata to ensure high availability and a low resource footprint through data replication.

Ceph FS_DEGRADED: standby MDS failed to take over a rank

Mon, 01 Jan 0001 00:00:00 +0000

FS_DEGRADED fires when at least one CephFS rank is failed or damaged and a standby did not promote. Clients can usually still reach the filesystem through surviving ranks, but you are running without the failover reserve the MDS cluster was sized to provide.

FS_DEGRADED is the precursor to MDS_ALL_DOWN. If the last active rank fails before you restore a healthy standby, CephFS becomes fully unavailable. The playbook classifies FS_DEGRADED active for more than 120 seconds as a TICKET. If CephFS is a primary storage interface, treat 120 seconds as the upper bound on response time, not a soft target.

CockroachDB clock skew cascade: how shared NTP drift causes quorum loss

Mon, 01 Jan 0001 00:00:00 +0000

Multiple CockroachDB nodes crashed overnight. The logs show “clock synchronization error: this node is more than 500ms away from at least half of the known nodes.” You restart them, and they crash again. Some ranges are now unavailable. The cluster is losing quorum.

A shared NTP failure caused multiple nodes to drift past CockroachDB’s self-termination threshold in quick succession. Single-node clock skew is bad but recoverable. Multi-node skew from a shared NTP source can take down quorum faster than the cluster can heal.

CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload

Mon, 01 Jan 0001 00:00:00 +0000

When a CockroachDB node dies, every client connected to it reconnects at the same time. Without jittered backoff in client connection pools, hundreds or thousands of new connections land on the surviving nodes within seconds. The survivors are not overwhelmed by additional query load. They are overwhelmed by connection overhead: per-connection goroutines, session memory allocations, TLS handshakes, and SQL planner initialization.

The signature is a 2-3x spike in sql_conns on surviving nodes, with a simultaneous jump in sys_goroutines and sql_mem_root_current. Latency rises across all nodes, not just those receiving the flood. The storm usually self-resolves as pools stabilize. But if memory pressure reaches OOM, one node failure can cascade into a multi-node outage.

CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping

Mon, 01 Jan 0001 00:00:00 +0000

Lease transfer spikes, briefly unavailable ranges, and client errors such as ambiguous results or connection resets indicate node liveness failure. In the logs, nodes transition to not-live and back within seconds. When the cluster decides a node cannot renew its liveness heartbeat, it redistributes leases. If the node recovers fast enough to renew but not fast enough to stay healthy, it flaps: an oscillating state more destructive than a clean outage because it repeatedly interrupts in-flight work and prevents stable failover.

CockroachDB replica unavailable: lost quorum and stuck Raft groups

Mon, 01 Jan 0001 00:00:00 +0000

ranges_unavailable has gone nonzero. Clients are seeing “replica unavailable” errors. Some portion of your keyspace cannot be read or written. This is an active availability incident.

In CockroachDB, every range (a ~512 MiB slice of the keyspace) is replicated across multiple nodes. A write requires quorum acknowledgment from a majority of replicas before it commits. When too few replicas are reachable, the range loses quorum and cannot serve reads or writes. The ranges_unavailable metric tracks exactly this condition: ranges with no leaseholder or with lost Raft quorum.

CockroachDB under-replicated ranges: ranges_underreplicated and the healing margin

Mon, 01 Jan 0001 00:00:00 +0000

When ranges_underreplicated rises above zero, CockroachDB has ranges with fewer live replicas than the configured replication factor. The replicate queue is already working to heal them by transferring Raft snapshots to new target nodes, but until healing completes, those ranges have reduced or zero fault tolerance margin.

The key operational question is not whether under-replicated ranges exist (they will, transiently, after almost any node event), but whether the cluster is healing fast enough to stay ahead of the next failure. This is the healing margin: the buffer between the current replication state and the point where a range loses quorum and becomes unavailable.

Elasticsearch master instability: frequent elections and metadata overload

Mon, 01 Jan 0001 00:00:00 +0000

Index creation requests time out. _cluster/health hangs or returns timeouts. The node listed by _cat/master changes every few minutes outside planned maintenance. Shard allocation stalls, and new indices stay red or unassigned even though all data nodes are reachable. These symptoms indicate a master node that cannot keep up with cluster state updates, triggering repeated elections and leaving the cluster without stable coordination.

This is metadata overload. The elected master maintains the cluster state: a heap-resident data structure describing every index, shard, mapping, alias, pipeline, and node. On every change, the master serializes and publishes the state to all nodes. Updates are processed serially, so any delay in serialization, heap allocation, or node acknowledgment blocks subsequent metadata operations. When metadata churn is high or the state is oversized, the master falls behind, pending tasks accumulate, and if the master misses enough heartbeat checks, remaining master-eligible nodes trigger a new election. Until a stable master converges, writes, allocations, and administrative operations stall.

Elasticsearch master_not_discovered_exception: no elected master and stalled writes

Mon, 01 Jan 0001 00:00:00 +0000

HTTP 503 and master_not_discovered_exception mean bulk indexing, index creation, mapping updates, and shard allocation checks are being rejected. The cluster cannot process writes or administrative work. The data nodes may be healthy, but without an elected master, the cluster cannot update shard routing, publish state changes, or acknowledge document writes. The root cause is usually one of four problems inside the master-eligible cohort.

Elasticsearch 7.0 and later use a consensus protocol called Zen2 for master election. Only master-eligible nodes vote. The elected master maintains the cluster state, which describes every index, shard, mapping, alias, and node. That state is serialized and published to all nodes on every change. Without a master, state updates halt and the default cluster.no_master_block rejects all operations until election completes.

Elasticsearch node left the cluster: fault detection, reallocation, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Your cluster health turned yellow and number_of_nodes dropped by one. The master logs a NODE_LEFT event, shards are unassigned, and the remaining nodes absorb extra load. In the next minute, the allocator decides whether to move data. Misread the cause and a transient restart becomes an expensive reallocation storm, or a genuine hardware failure goes unaddressed while replicas rebalance.

This guide covers how Elasticsearch decides a node is gone, what happens to its shards, and how to recover without deepening the incident.

HAProxy backend losing servers: active server count and cascade risk

Mon, 01 Jan 0001 00:00:00 +0000

A backend can lose half its servers and HAProxy will still report the BACKEND aggregate row as UP. As long as one server is available, the rollup status says traffic is being served, and status-based dashboards stay green. Meanwhile the surviving servers are absorbing redistributed load, their session counts are climbing, and you are one health check failure away from a full 503 outage for that backend.

This is the leading edge of the backend collapse cascade: initial server failure, traffic redistribution, survivor overload, survivor health check failures, more redistribution, total collapse. The operators who catch it early are watching the active server count and the load on survivors, not the aggregate status.

Kafka ISR shrinking: IsrShrinksPerSec, flapping, and the cascade to offline

Mon, 01 Jan 0001 00:00:00 +0000

IsrShrinksPerSec is climbing on your leaders and UnderReplicatedPartitions is no longer zero. If it is flapping – shrinks followed by expands every few minutes – the path ends with OfflinePartitionsCount rising and acks=all producers throwing NotEnoughReplicasException. This guide covers that path: how a lagging follower becomes a cluster-wide problem, how to separate flapping from one-way degradation, and how to stop the cascade before partitions go offline.

What this means

Kafka leaders maintain an In-Sync Replica set (ISR): followers that have fetched within replica.lag.time.max.ms (default 30 seconds since Kafka 2.5.0; 10 seconds before 2.5.0). When a follower stops fetching or falls behind beyond that window, the leader removes it. IsrShrinksPerSec measures the velocity of these removals. IsrExpandsPerSec measures replicas catching up and rejoining.

Kafka min.insync.replicas and acks: configuring durability you actually have

Mon, 01 Jan 0001 00:00:00 +0000

Most operators set producers to acks=all and assume the cluster acks only when every replica has the message. It does not. With acks=all, the broker waits only for the current in-sync replica set (ISR). Because the ISR shrinks dynamically when followers lag, a partition with replication factor three can have an ISR of one – the leader itself. Without raising min.insync.replicas from its default, the leader acks with zero followers caught up. Your durability guarantee collapses to leader-only persistence, and you only find out when the leader dies and data is missing.

Kafka NotEnoughReplicasException: acks=all writes rejected below min.insync.replicas

Mon, 01 Jan 0001 00:00:00 +0000

Producers are throwing org.apache.kafka.common.errors.NotEnoughReplicasException or NotEnoughReplicasAfterAppendException, and acks=all writes are failing while acks=1 or acks=0 writes may still succeed. The affected partitions no longer have enough in-sync replicas to satisfy min.insync.replicas. The immediate operational question is whether the ISR shrink is a transient recovery blip or a sustained degradation that will block writes until you fix the follower.

What this means

The leader tracks followers caught up within replica.lag.time.max.ms (default 30s, changed from 10s in Kafka 2.5.0) in the In-Sync Replica set (ISR). For acks=all, the leader waits for all current ISR members before acknowledging the producer. If the ISR size drops below min.insync.replicas, the leader rejects the produce request. The broker-level default for min.insync.replicas is 1, so a lone leader can acknowledge alone. In practice, with replication.factor=3 and acks=all, set min.insync.replicas=2 so a single follower loss blocks writes instead of silently weakening durability.

Keepalived Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Keepalived Monitoring

What Is Keepalived?

Keepalived is a robust software used primarily for high-availability and load-balancing on Linux systems. It leverages VRRP (Virtual Router Redundancy Protocol) to increase network uptime by offering failover protocols. By monitoring network interfaces, Keepalived helps in ensuring that servers are capable of handling traffic efficiently.

Monitoring Keepalived With Netdata

Monitoring Keepalived with Netdata is a seamless process that ensures you have real-time insights into the performance and availability of your network infrastructure. Netdata uses an openmetrics (Prometheus) exporter to retrieve Keepalived metrics. This means that, to monitor Keepalived, Netdata can accept data from any Prometheus exporter, allowing you to benefit from automated dashboards and alerts without needing a dedicated Prometheus server or Grafana. This integration allows for comprehensive monitoring of Keepalived metrics, ensuring your high-availability setup runs smoothly.

Kubernetes Controller-Manager Leader Election Failures

Mon, 01 Jan 0001 00:00:00 +0000

Your Deployment has stopped scaling. Nodes cordoned hours ago are still draining. Garbage collection is paused, and orphaned volumes are not being cleaned up. The kube-controller-manager runs these reconciliation loops, and in an HA cluster only the leader performs work. When leader election fails, the controller-manager exits, and the control plane stops acting on desired state. Existing workloads keep running, but nothing new is managed.

The kube-controller-manager coordinates through a Lease object in the coordination.k8s.io API group. The leader must renew the lease before --leader-elect-renew-deadline (default 10 seconds) elapses. The lease itself expires after --leader-elect-lease-duration (default 15 seconds). Renewal is attempted every --leader-elect-retry-period (default 2 seconds). If a write to etcd is too slow, if the API server is saturated, if RBAC is stripped, or if the election timing is misconfigured, the leader loses the lock, logs leaderelection lost, and exits. During the gap, no instance holds a valid lease, so controllers stop reconciling.

MongoDB no primary / election storm: repeated elections and write outages

Mon, 01 Jan 0001 00:00:00 +0000

Applications log “not primary” errors. rs.status() shows a different PRIMARY than thirty seconds ago. MongoDB logs repeat "Starting an election" and "Stepping down". Each election costs 2-12 seconds of write unavailability. More than two in ten minutes is an election storm.

This pattern is more dangerous than a single failover because it creates rolling write outages that do not self-stabilize. Drivers reconnect, retry buffers fill, and application latency degrades even when a primary exists. Root causes usually fall into three categories: the primary is too slow to answer heartbeats, the network is dropping or delaying packets between members, or a misconfigured priority is forcing a healthy primary to step down.

MongoDB not master error: writes hitting a non-primary node after failover

Mon, 01 Jan 0001 00:00:00 +0000

A node restart, network partition, or planned stepdown triggers a MongoDB election. Seconds later, application logs show NotWritablePrimary (code 10107) or the legacy string not master and slaveOk=false. Writes fail against a node that used to be PRIMARY, even though the cluster has elected a new one.

This guide covers how to find the root cause and stop it from recurring.

What this means

MongoDB replica sets elect exactly one PRIMARY at a time. When a failover occurs, the old primary steps down and a secondary is promoted. Application drivers discover the new topology through the replica set seed list and refresh their connection pools automatically. Between stepdown and election completion, there is a brief window with no writable primary. After the new primary is elected, drivers should route writes there.

MongoDB not primary and secondaryOk=false: reading from a secondary and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

Your application logs show NotPrimaryNoSecondaryOk (code 13435) with the message "not master and slaveOk=false". Metrics show read failures against a specific host. The mongod process is running, replica set heartbeats are clean, and replication lag looks normal. The cluster is not down. The error is a routing decision: a client sent a read to a replica set member that is not the primary, without declaring that reading from a non-primary is acceptable.

MongoDB rollback after failover: silent data loss and the rollback directory

Mon, 01 Jan 0001 00:00:00 +0000

A replica set member in ROLLBACK state, or an application reporting vanished documents after failover, means a former primary held writes that never reached a majority. When that node rejoins, MongoDB erases the divergent history and writes the removed data to files under <dbPath>/rollback/. The application may have received acknowledgment for those writes. With w:1, acknowledgment meant only that the primary applied the write. It did not guarantee replication to a majority or survival through failover. That is silent data loss.

MySQL GTID errant transactions: detecting replication divergence

Mon, 01 Jan 0001 00:00:00 +0000

Before a planned failover, or after an orchestrator aborts with a GTID consistency error, replication can appear healthy: SHOW REPLICA STATUS reports both threads running, Seconds_Behind_Source is near zero, and the error log is quiet. Yet comparing GTID sets between source and replica reveals mismatching numbers. Errant transactions are GTIDs present in a replica’s gtid_executed set that the source never generated.

Ordinary lag resolves as the replica catches up. Errant transactions represent true divergence. Promoting a replica with extra GTIDs propagates those transactions into the new source and downstream replicas, creating split-brain that is expensive to reverse. This guide covers distinguishing benign lag from dangerous divergence, pinpointing offending GTIDs, and deciding between empty-transaction injection and a full rebuild.

Patroni Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Patroni Monitoring

What Is Patroni?

Patroni is an open-source high-availability solution for PostgreSQL databases, providing automatic failover and reliable replication solutions. It is designed to be easy to configure and highly reliable, making it a popular choice for database administrators who require robust failover management.

Monitoring Patroni With Netdata

Monitoring Patroni with Netdata gives users real-time insights into their Patroni clusters. By utilizing an openmetrics (Prometheus) exporter like Patroni Exporter, Netdata can seamlessly ingest metrics from any Prometheus exporter, allowing for automated dashboards, alerts, and more—all without the need for a Prometheus server or Grafana setup. This approach ensures that you have everything you need to monitor Patroni efficiently with minimal setup.

PostgreSQL replication lag: detection, diagnosis, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Replication lag is the distance between the last WAL record generated on the primary and the last record applied on a replica. In asynchronous streaming replication, a few seconds of lag is normal. When lag grows without bound, your recovery point objective becomes fiction.

Lag often grows silently. Replication processes stay connected, WAL streams flow, and uptime checks stay green while the byte gap creeps from megabytes to gigabytes. Promoting a replica that is hours behind destroys the consistency your application assumes.

RabbitMQ network partition detected: split-brain in cluster_status

Mon, 01 Jan 0001 00:00:00 +0000

You ran rabbitmqctl cluster_status (or your monitoring scraped /api/nodes) and saw the “Network Partitions” section listing node names, or a non-empty partitions array on one or more nodes. The cluster has split: some nodes can no longer see each other over the Erlang distribution link, and each side now has its own view of the world.

This is a paging condition. What happens next depends on cluster_partition_handling: the two sides may be accepting writes independently and diverging (the ignore default), the minority side may have frozen and stopped serving clients (pause_minority), or nodes may be about to restart themselves (autoheal). Each outcome has a different blast radius, and the wrong response makes it worse.

RabbitMQ node down: telling a dead broker apart from a partitioned one

Mon, 01 Jan 0001 00:00:00 +0000

A node has disappeared from your cluster view. The first question is not “how do I bring it back” but “what actually happened to it.” A dead broker (VM crashed, process killed, host gone) and a partitioned broker (Erlang node still running but cut off from its peers) look almost identical from one vantage point and completely different from another. The recovery steps are opposite: you restart a dead node, but restarting a partitioned node mid-partition can make split-brain worse.

RabbitMQ node is quorum critical: checking before a rolling restart

Mon, 01 Jan 0001 00:00:00 +0000

You are about to restart a RabbitMQ node: a rolling upgrade, an OS patch, an instance resize. Before you stop it, one question decides whether this is routine maintenance or a queue outage: if this node goes down right now, does any quorum queue or stream lose its online majority?

RabbitMQ ships a purpose-built check for exactly this. rabbitmq-diagnostics check_if_node_is_quorum_critical returns unhealthy when stopping the target node would drop a quorum queue below the number of online members it needs to accept writes. The same logic is exposed over HTTP at GET /api/health/checks/node-is-quorum-critical, which makes it usable from automation, load balancer health gates, and CI-driven upgrade pipelines.

Redis cluster_slots_pfail > 0: impending node failure in a cluster

Mon, 01 Jan 0001 00:00:00 +0000

cluster_slots_pfail > 0 means at least one hash slot is mapped to a node that a peer suspects is down. In Redis Cluster, PFAIL is unilateral: any node raises it when another stops answering gossip PINGs for longer than cluster-node-timeout. Slots continue to serve traffic; the cluster has not yet agreed the node is dead.

Brief spikes are expected during background saves, AOF rewrites, or any main-thread freeze. Sustained non-zero values indicate a real problem: network partition, node crash, or overload. If the majority of masters confirm the suspicion within twice cluster-node-timeout, PFAIL escalates to FAIL. The affected slots become unavailable until a replica wins election. In a three-master cluster, losing two primaries leaves the survivor without quorum. The cluster enters a zombie state where no failover can proceed. Investigate PFAIL while you still have quorum and before automatic escalation.

Redis CLUSTERDOWN / cluster_state:fail: slot coverage and recovery

Mon, 01 Jan 0001 00:00:00 +0000

CLUSTERDOWN The cluster is down means at least one of the 16384 hash slots lacks a healthy master. With cluster-require-full-coverage yes (the default), a single missing slot blocks all writes. This guide covers diagnosing the root cause, recovering safely, and preventing recurrence.

What this means

Redis Cluster shards the keyspace across 16384 hash slots. Each slot must be assigned to a master node that is reachable and healthy to count toward cluster_slots_ok. When cluster_slots_assigned drops below 16384, or cluster_slots_fail becomes non-zero because a node has been marked FAIL by quorum, the cluster transitions to cluster_state:fail. Clients receive CLUSTERDOWN for operations hashing to affected slots.

Redis READONLY You can't write against a read only replica - causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Your application hits (error) READONLY You can't write against a read only replica. Writes fail; reads work. The connected Redis instance thinks it is a replica, so it rejects mutating commands. The replica is behaving correctly. The problem is a write-capable client routed to a node that is not the current primary. This typically happens in three situations: a routing bug that sends writes to a replica endpoint, stale client topology after a failover or upgrade, or an instance that was accidentally demoted at runtime.

SQL Server AG send and redo queues growing: replication lag and failover RTO

Mon, 01 Jan 0001 00:00:00 +0000

Two queues decide whether your Always On Availability Group can actually fail over: the send queue (log generated on the primary but not yet shipped to the secondary) and the redo queue (log received by the secondary but not yet replayed). When either grows without bound, replication lag is the visible symptom, but the hidden cost is failover RTO. On forced or automatic failover, the new primary must drain the entire redo queue before it accepts writes, so a queue that looks tolerable during steady state can turn a 30-second failover into a 30-minute one.

SQL Server Availability Group not synchronizing: NOT_HEALTHY replicas and failover risk

Mon, 01 Jan 0001 00:00:00 +0000

The symptom arrives as an alert or a dashboard color change: a synchronous-commit secondary replica is reporting synchronization_health_desc = NOT_HEALTHY or connected_state_desc = DISCONNECTED in sys.dm_hadr_availability_replica_states. The primary is still accepting writes, but the protection you assumed is degraded or gone.

In synchronous-commit mode, the primary waits for the secondary to harden log records before acknowledging commits. When the secondary drops or stops keeping up, the primary either continues unprotected or stops accepting writes entirely, depending on required_synchronized_secondaries_to_commit. Either way, your recovery point objective and your recovery time objective are both at risk.

SQL Server database in SUSPECT or RECOVERY_PENDING: an offline database and how to recover it

Mon, 01 Jan 0001 00:00:00 +0000

A production database is showing state_desc = SUSPECT or RECOVERY_PENDING in sys.databases. Applications cannot open connections to that database. Users are seeing login failures, query timeouts, or generic “database cannot be opened” errors. The SQL Server instance itself is up, and every other database on it may be fine.

RECOVERY_PENDING rarely means corruption. It usually means SQL Server could not get the resources it needed during recovery: a missing file, a full log volume, a permissions change, or a transient I/O failure at startup. SUSPECT is more serious because recovery actually ran and failed, but it still does not automatically mean data loss. The wrong move is to jump straight to DBCC CHECKDB with REPAIR_ALLOW_DATA_LOSS. The right move is to fix the underlying resource, re-run recovery, and only fall back to repair or restore when that fails.

SQL Server instance down: no response on port 1433 and where to look first

Mon, 01 Jan 0001 00:00:00 +0000

Your availability probe just fired: TCP connect plus SELECT 1 against port 1433 has failed three or more times over at least 60 seconds. Before you restart anything, separate the two failure modes that get lumped together as “SQL Server is down”. They have different causes, different fixes, and different blast radii.

Mode one: no TCP connect at all. The listener is not accepting connections on 1433 (or the named instance’s dynamic port). The service is stopped, the host is down, the network path is broken, or the listener is misconfigured, commonly after an AlwaysOn failover. The engine is not there to talk to.

vSphere HA 'Insufficient resources to satisfy configured failover level': admission control

Mon, 01 Jan 0001 00:00:00 +0000

The error “Insufficient resources to satisfy configured failover level for vSphere HA” is admission control refusing a VM power-on, vMotion, or reservation change because granting it would leave the cluster without enough spare capacity to honor the configured HA failover policy. Admission control is doing its job: protecting the restart guarantee after a host failure.

The cluster may physically hold more capacity than admission control lets you commit. A cluster with 500 GHz of CPU and 2 TB of RAM may only let you deploy against roughly 70% of that, with the rest held in reserve so HA can restart protected VMs after a host failure. Operators who bought hardware expecting to use all of it hit this wall during provisioning and reach for the disable switch.

ZooKeeper quorum loss: no leader elected and every write is failing

Mon, 01 Jan 0001 00:00:00 +0000

Every write to your ZooKeeper ensemble is timing out. Clients report ConnectionLoss and SessionExpired. Downstream systems that depend on ZK for coordination, such as Kafka controller elections or HBase region assignment, are cascading into failure. On the surviving ZK nodes, ruok still returns imok. The process is alive; the ensemble is not.

Quorum loss is ZooKeeper’s worst-case availability scenario. When fewer than floor(N/2)+1 voting members can communicate, no leader can be elected and every write fails. Surviving nodes sit in LOOKING state, unable to make progress through ZAB.