Chrony on Netdata

Chrony Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Chrony Monitoring

What Is Chrony?

Chrony is a versatile software suite for maintaining the system clock accuracy on your Linux distributions. Acting as an implementation of the Network Time Protocol (NTP), Chrony ensures your systems are synchronized to accurate time sources, which is crucial for various server operations. It is highly adaptive, providing swift synchronization capabilities, making it particularly beneficial for systems with sporadic network connectivity or those that endure extreme delays.

CockroachDB clock skew cascade: how shared NTP drift causes quorum loss

Mon, 01 Jan 0001 00:00:00 +0000

Multiple CockroachDB nodes crashed overnight. The logs show “clock synchronization error: this node is more than 500ms away from at least half of the known nodes.” You restart them, and they crash again. Some ranges are now unavailable. The cluster is losing quorum.

A shared NTP failure caused multiple nodes to drift past CockroachDB’s self-termination threshold in quick succession. Single-node clock skew is bad but recoverable. Multi-node skew from a shared NTP source can take down quorum faster than the cluster can heal.

CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes

Mon, 01 Jan 0001 00:00:00 +0000

You see this fatal log line on a CockroachDB node:

clock synchronization error: this node is more than 500ms away from at least half of the known nodes

The process exits immediately. If the node is managed by systemd, Kubernetes, or a process supervisor, it restarts and crashes again. The crash-loop continues until the clock problem is fixed.

CockroachDB uses Hybrid Logical Clocks (HLC) to enforce serializable consistency across distributed transactions. HLC combines physical wall-clock time with a logical counter. Every node must agree on time within a bounded window, controlled by --max-offset (default 500ms). When a node detects its clock has drifted beyond 80% of --max-offset relative to a majority of peers, it calls log.Fatal and exits rather than risk serving stale reads or assigning timestamps that conflict with committed transactions.

CockroachDB clock_offset_meannanos high: catching clock drift before self-termination

Mon, 01 Jan 0001 00:00:00 +0000

clock_offset_meannanos measures the mean clock offset between a CockroachDB node and its peers. When it climbs, you are on a path that ends in silent performance degradation from widened read uncertainty windows, or a node self-terminating to preserve data consistency.

The thresholds are unforgiving. CockroachDB uses a default --max-offset of 500ms. A node self-terminates when its mean offset exceeds 80% of that value (400ms) relative to at least half of its peers. But a constant 200ms offset, stable and below any threshold, still doubles the uncertainty interval for every read. Transactions silently restart more often, P99 read latency creeps up, and nobody suspects the clock.

CockroachDB ReadWithinUncertaintyInterval restarts: the near-diagnostic signal of clock skew

Mon, 01 Jan 0001 00:00:00 +0000

When CockroachDB reports readwithinuncertainty as a transaction restart cause, you have a clock synchronization problem. This restart cause does not appear in meaningful quantities for any other reason. Any sustained nonzero rate, even well below the self-termination threshold, indicates that NTP or the underlying clock source is not keeping node clocks aligned.

The error stems from CockroachDB’s Hybrid Logical Clock (HLC) design. Each transaction receives a timestamp from its gateway node’s HLC, which combines physical wall-clock time with a logical counter. When a read on one node encounters a write from a transaction that started on a different node, and the two nodes’ clocks are not aligned, the database cannot determine which transaction began first. CockroachDB conservatively restarts the reading transaction at a higher timestamp. Under SERIALIZABLE isolation, this restart requires client-side retry logic and adds directly to tail latency.

NTP drift on network devices: the silent killer of event correlation

Mon, 01 Jan 0001 00:00:00 +0000

Clock drift on network devices produces no visible symptom. The device stays up, interfaces carry traffic, BGP sessions remain Established, SNMP keeps responding. The damage surfaces hours or days later, in a postmortem where two devices’ timestamps disagree by hundreds of milliseconds and the analyst cannot reconstruct the event sequence. Every cross-device correlation in the monitoring stack depends on accurate, monotonic time across every collector and every polled device.

The telemetry itself looks fine. Syslog messages arrive with timestamps. Flow records carry timestamps. BGP NOTIFICATION traps are time-stamped. The problem is that those timestamps are wrong relative to each other, and nothing flags the discrepancy. A device 200 milliseconds off its peers produces records that technically arrive but correlate poorly with records from correctly synchronized devices.