Netdata: Monitoring and troubleshooting transformed on Netdata

Native macOS Monitoring: Logs, Sensors, GPU & Hardware Health

Wed, 15 Jul 2026 00:00:00 +0000

We’ve overhauled macOS monitoring in the latest Netdata release. Netdata already collects system metrics on Macs at per-second resolution; this release completes the picture with logs and hardware telemetry, areas that previously required users to run CLI tools like log show and powermetrics. The new collectors read this data through Apple’s own frameworks, allowing users to trace application and OS errors and catch hardware issues early.

9 Best Container Monitoring Tools & How To Choose

Thu, 09 Jul 2026 00:00:00 +0000

SolarWinds NPM Alternative: With Per-Second Metrics

Mon, 29 Jun 2026 00:00:00 +0000

Replace 5-minute SNMP polling with per-second network monitoring, ML anomaly detection, and zero-template device discovery.

SolarWinds NTA Alternative: NetFlow Traffic Analyzer

Mon, 29 Jun 2026 00:00:00 +0000

Replace SolarWinds NTA with Netdata for modern, per-second NetFlow analysis, interactive Sankey diagrams, street-level maps, and zero-config deployment.

SolarWinds Observability Alternative: Real-Time & AI

Mon, 29 Jun 2026 00:00:00 +0000

Netdata is the modern alternative to SolarWinds Hybrid Cloud Observability: per-second resolution, edge-native ML on every metric, zero-config deployment, and flat per-node pricing with full data sovereignty.

SolarWinds Orion Alternative: One Agent, Full Stack

Mon, 29 Jun 2026 00:00:00 +0000

Replace SolarWinds Orion Platform with Netdata: one agent, per-second visibility, ML anomaly detection, AI troubleshooting, and network flow analysis without stacked module licenses.

SolarWinds SAM Alternative: Server & App Monitoring

Mon, 29 Jun 2026 00:00:00 +0000

Fleet Observability: Linux Edge Device Monitoring

Sun, 28 Jun 2026 00:00:00 +0000

It feels less like managing devices and more like remote babysitting. You check the dashboard, everything is green, and then a customer in the field tells you a device has been down for two days. At a handful of servers, the rare failure is an event. Across thousands of distributed Linux endpoints — robots in warehouses, EV chargers across a city, kiosks in retail, IoT gateways in the field — the rare failure becomes a daily occurrence, and the tools built for a datacenter quietly stop telling you the truth.

Distributed & Edge Fleet Monitoring

Thu, 25 Jun 2026 00:00:00 +0000

Netdata brings real-time, AI-powered observability to distributed fleets of Linux endpoints — robots, kiosks, POS terminals, IoT gateways, and remote servers — keeping data and intelligence at the edge where it belongs.

EV Charging Network Monitoring

Thu, 25 Jun 2026 00:00:00 +0000

Netdata gives charge point operators per-second visibility into every charger controller, with edge-resident storage and streaming that survives flaky site connectivity.

Retail POS and Kiosk Fleet Monitoring

Thu, 25 Jun 2026 00:00:00 +0000

Netdata gives per-second visibility into every POS terminal and self-service kiosk across thousands of stores, with edge-resident monitoring that survives flaky in-store networks.

Robot Fleet Monitoring with Per-Second Edge Intelligence

Thu, 25 Jun 2026 00:00:00 +0000

Netdata runs on each robot, collecting per-second metrics for compute, memory, storage, thermals, and network — keeping thousands of devices observable even on intermittent site networks, without centralized SaaS.

School Safety Device Fleet Monitoring

Thu, 25 Jun 2026 00:00:00 +0000

Netdata runs on every campus device, giving IT teams per-second visibility into compute, connectivity, and OS health across thousands of school safety devices and endpoints.

What Is Edge Monitoring? Edge Observability Explained

Thu, 25 Jun 2026 00:00:00 +0000

Edge monitoring means running the full observability pipeline - collection, storage, anomaly detection, alerting, and dashboards - on or near the monitored device itself, instead of shipping all raw telemetry to a central system first. Each node in the fleet collects and processes its own metrics locally, and only forwards what is needed upstream. This model is essential for distributed fleets of thousands of devices where centralizing all telemetry is economically and technically impractical.

NetFlow vs sFlow vs IPFIX: Differences & When To Use

Wed, 24 Jun 2026 00:00:00 +0000

NetFlow, sFlow, and IPFIX are three network traffic export protocols that give visibility into traffic flows without requiring full packet capture. NetFlow and IPFIX aggregate packets into flow records on the device before exporting them, providing exact (if unsampled) accounting. sFlow takes a fundamentally different approach: it exports every Nth packet header plus interface counters, making it stateless and scalable but statistical rather than exact. IPFIX is the IETF-standardized evolution of NetFlow v9, adding vendor-neutral extensibility through template-based information elements.

Network Monitoring Dashboard

Wed, 24 Jun 2026 00:00:00 +0000

Netdata’s Network Monitoring Dashboard unifies SNMP device metrics, NetFlow/sFlow traffic analysis, SNMP traps, and live topology into a single real-time, interactive view.

Network Topology Mapping Explained

Wed, 24 Jun 2026 00:00:00 +0000

Network topology mapping is the process of discovering and visualizing how devices, endpoints, and services connect across a network. It captures both physical (Layer 2) relationships - which switch port links to which device - and logical (Layer 3) relationships - how subnets, routes, and autonomous systems reach each other. A topology map can range from a hand-drawn diagram to a live, continuously updated graph built from protocol data such as LLDP, CDP, ARP, FDB, OSPF, and BGP.

Real Time Network Monitoring: Topology, NetFlow, SNMP

Wed, 24 Jun 2026 00:00:00 +0000

Interface counters tell you a port is busy. Bytes in, bytes out, errors, drops. That’s enough to know a link is saturated, but not enough to know which conversations are saturating it, which devices are involved, or how a problem propagates across your network. For that you’ve traditionally needed dedicated network performance monitoring tools, usually expensive, usually a separate console from the rest of your monitoring.

Today we’re closing that gap. Netdata has had solid network interface monitoring for a long time through its native collectors and SNMP support. We’ve now built out the rest of the picture, and it adds up to NPM-class network monitoring: live network topology, NetFlow and sFlow traffic analysis, SNMP device monitoring across 200+ vendor profiles, SNMP trap handling, and a dedicated network monitoring dashboard. All of it runs alongside the infrastructure, application, and container metrics Netdata already collects, on the same timeline, in the same platform.

SNMP Monitoring Guide: How To Monitor Network Devices

Wed, 24 Jun 2026 00:00:00 +0000

To monitor SNMP devices you enable an SNMP agent on the device, open UDP/161 from the monitoring host, point your monitoring tool at the device with the correct credentials, and poll interface and health counters on a fixed interval. SNMP works by having a central manager query an agent that exposes numeric objects (OIDs) defined in MIBs, with optional push notifications (traps) for event-driven signals. The bulk of day-to-day network monitoring (interface utilization, errors, port state, CPU, memory) comes from periodic polling of those OIDs.

SNMP Traps vs Polling: What Is The Difference?

Wed, 24 Jun 2026 00:00:00 +0000

SNMP polling is a pull model where a monitoring server periodically queries a device over UDP port 161 to read metrics and state. SNMP traps are a push model where the device itself sends an unsolicited notification over UDP port 162 the instant an event occurs. The two are not competing choices: mature network monitoring uses polling for continuous metrics and traps for immediate event alerts.

What is SNMP polling?

Polling is the classic request-response model at the heart of SNMP-based monitoring. A central manager (your monitoring system) sends a GET, GETNEXT, or GETBULK request to an SNMP agent running on a router, switch, UPS, or other networked device. The agent responds with the requested values.

What Is NetFlow? Network Flow Monitoring Explained

Wed, 24 Jun 2026 00:00:00 +0000

NetFlow is a network protocol, originally developed by Cisco, for collecting metadata about IP traffic flows as they pass through a router, switch, or firewall. A flow is a unidirectional sequence of packets that share the same key fields - classically the 5-tuple of source IP, destination IP, source port, destination port, and protocol. NetFlow does not capture packet payloads; it exports compact metadata records that a collector stores and analyzes, giving you visibility into who is talking to whom, how much bandwidth they are using, and when.

5 Best SolarWinds Alternatives for 2026

Tue, 23 Jun 2026 00:00:00 +0000

As organizations modernize their infrastructure and embrace cloud-native architectures, traditional monitoring solutions are showing their age. SolarWinds, while a long-established player in IT management, was designed for an era of static, on-premise infrastructure. In 2026, teams are seeking alternatives that can keep pace with dynamic, distributed systems—and they’re finding better options.

What Is SolarWinds? Is It Still The Right Choice?

SolarWinds Platform (formerly Orion) has been a cornerstone of IT monitoring for decades, offering comprehensive coverage of network devices, servers, applications, and databases. It’s particularly strong in traditional enterprise environments with extensive hardware monitoring needs.

NetFlow Traffic Analyzer

Tue, 23 Jun 2026 00:00:00 +0000

Real-time NetFlow, IPFIX, and sFlow analysis with top talkers, Sankey diagrams, geographic traffic maps, and automatic GeoIP/ASN/NetBox enrichment — built into the Netdata Agent.

Network Topology Viewer

Tue, 23 Jun 2026 00:00:00 +0000

Netdata Network Topology Viewer maps live TCP/UDP connections between processes, containers, and endpoints, plus SNMP device fabric via LLDP, CDP, BGP, and OSPF.

SNMP & Network Device Monitoring

Tue, 23 Jun 2026 00:00:00 +0000

Real-time SNMP device monitoring with auto-matched vendor profiles, ML anomaly detection, and zero-config discovery.

SNMP Trap Monitoring

Tue, 23 Jun 2026 00:00:00 +0000

Native SNMP trap receiver with MIB-resolved decoding, storm controls, health alerts, and SIEM forwarding — built into Netdata.

SolarWinds Price Increases 2026: What Customers Need to Know

Tue, 23 Jun 2026 00:00:00 +0000

If you’re a SolarWinds customer facing renewal, you’ve likely noticed significant changes to pricing and licensing terms in 2024-2025. You’re not alone. At Netdata, we’ve been speaking with dozens of SolarWinds customers who are reassessing their monitoring strategies in light of these changes. This post provides factual information about what’s changed, the real impact on organizations, and a practical framework for evaluating your path forward.

What’s Happening with SolarWinds Pricing?

In February 2025, SolarWinds was acquired by private equity firm Turn/River Capital in a $4.4 billion transaction. As is common with PE-backed acquisitions, this has led to significant changes in pricing and business terms. Based on customer reports and public information, renewal prices have increased by 100-300% for many customers. One customer on the SolarWinds community forum reported their renewal more than doubled, a 225% increase from the previous year.

Grafana vs Datadog: The honest comparison in 2026

Sat, 30 May 2026 00:00:00 +0000

High Cardinality Metrics At Scale: A Better Playbook

Sat, 30 May 2026 00:00:00 +0000

The “high cardinality is expensive” sentence has become observability’s version of “in this economy”: said so often that nobody questions whether it’s true. Every vendor pricing page invokes it. Every glossary article repeats it. Every architecture diagram shows aggregation buffers placed before the storage layer.

“High cardinality is expensive” is not a fact about the universe; it’s a fact about one architectural choice: centralizing time-series storage and querying it through an index that scales with unique series. Once you accept that choice, everything follows. You pay per metric, you drop labels you wish you’d kept, you pre-aggregate before storage, and you discover that the bug you were debugging only existed at the full resolution you already threw away.

How To Reduce Alert Fatigue With Anomaly Detection

Sat, 30 May 2026 00:00:00 +0000

Most alert-fatigue tools manage noise after the fact. Netdata prevents it by running 18 ML models per metric with consensus voting — anomalies only fire when multiple models agree, suppressing the false positives that drive on-call burnout.

The best AI-powered observability platforms in 2026

Sat, 30 May 2026 00:00:00 +0000

The best DevOps monitoring tools in 2026

Sat, 30 May 2026 00:00:00 +0000

The Best Infrastructure Monitoring Tools In 2026

Sat, 30 May 2026 00:00:00 +0000

The best open-source observability tools in 2026

Sat, 30 May 2026 00:00:00 +0000

Netdata Skills: Teach Your AI Coding Agent To Monitor

Wed, 20 May 2026 00:00:00 +0000

There’s a growing ecosystem of AI coding agents: Claude Code, Cursor, Copilot, Codex, Gemini CLI, Windsurf, and others. They’re good at writing code, but they don’t inherently know how to instrument that code for observability, configure monitoring infrastructure, or troubleshoot production systems using real telemetry data. That knowledge lives in documentation, runbooks, and the heads of your senior SREs.

We’ve open-sourced a repository that encodes this knowledge into a format AI agents can use directly. netdata/skills is a collection of agent skills, published in the open agentskills.io format, that teach AI coding agents how to set up Netdata, instrument applications with OpenTelemetry, build collector pipelines, troubleshoot 49 specific technologies, and verify everything against live data via MCP.

OpenTelemetry Backend

Wed, 20 May 2026 00:00:00 +0000

An open OTEL backend with native OTLP ingestion, per-second granularity, ML anomaly detection, and infrastructure correlation. Traces coming soon. No per-metric pricing. No vendor lock-in.

OpenTelemetry and Netdata, Today

Fri, 15 May 2026 00:00:00 +0000

OpenTelemetry has become the default way to instrument applications and ship telemetry. The hard part has never been the data model. It’s been picking a backend that handles OTLP without quietly turning into a per-metric bill or a black box that swallows your data.

Netdata is a native OTLP backend. Stand up an OpenTelemetry Collector with any of its hundreds of receivers, point its OTLP exporter at Netdata, and you get per-second charts, ML anomaly detection on every signal, AI-assisted troubleshooting, and infrastructure correlation, with no per-metric, per-series, or per-host charges. Metrics and logs work today. Trace support is coming soon.

Dashboard Playlists: Cycle Through Dashboards in TV Mode

Tue, 12 May 2026 00:00:00 +0000

When we shipped TV mode, we heard almost immediately: “Great, but I have five dashboards and one screen.” A single dashboard on a wall display covers one view of your infrastructure. If you want to rotate between your network overview, database health, application metrics, and infrastructure summary, someone has to walk over and click, or you’re buying more screens.

Dashboard playlists solve this. You can now select a sequence of dashboards to cycle through in TV mode, with a configurable rotation interval. Set it up once, open the TV mode URL on your display, and the screen rotates through your chosen dashboards on its own.

Azure Local Migration: Monitor Both Sides In One View

Mon, 11 May 2026 00:00:00 +0000

More organizations are moving workloads from Azure public cloud to Azure Local (formerly Azure Stack HCI) than most people realize. The reasons vary: data sovereignty requirements, latency-sensitive workloads that need to be closer to the edge, cost optimization for predictable workloads where reserved cloud capacity doesn’t make financial sense, or regulatory constraints that require data to stay on-premises.

Geo Maps: See Where Your Infrastructure Lives

Sat, 09 May 2026 00:00:00 +0000

When your infrastructure is spread across regions, data centers, branch offices, or edge locations, knowing where a node is physically located matters more than people usually admit. During an incident, “the node in the Singapore POP” communicates faster than a hostname. When you’re planning capacity, seeing geographic clustering tells you something that a flat list of nodes doesn’t. When a subset of your fleet starts misbehaving, the first question is often “is this regional?”

NVIDIA DCGM Collector: Deep GPU Monitoring For AI

Mon, 04 May 2026 00:00:00 +0000

GPU infrastructure is expensive and increasingly central to production workloads. Whether you’re running ML training jobs, inference serving, video transcoding, or HPC workloads, understanding what your GPUs are actually doing, and what’s going wrong when performance degrades, is not optional. The problem is that NVIDIA’s Data Center GPU Manager (DCGM) exposes an enormous amount of telemetry, but getting that data into a monitoring system in a useful, organized way has traditionally required significant setup and custom dashboarding work.

Misconfigured Alert Detection: Tuning Made Easy

Tue, 28 Apr 2026 00:00:00 +0000

Netdata ships with hundreds of stock alerts. They cover a wide range of infrastructure conditions and they’re designed with sensible defaults. But “sensible defaults” and “correct for your environment” are not the same thing. A CPU threshold that’s perfectly reasonable for a build server might generate constant noise on a machine running batch jobs. An alert that’s critical for production might be irrelevant in staging, where it fires daily and everyone ignores it.

Azure Monitor Collector: Monitor Azure Infrastructure

Mon, 27 Apr 2026 00:00:00 +0000

If you’re running infrastructure on Azure, you’ve probably dealt with the split between your Azure-native monitoring and the rest of your stack. Your VMs, databases, and Kubernetes clusters generate platform metrics through Azure Monitor, but those metrics live in a separate world from the OS-level, application, and on-prem metrics you’re already watching in Netdata. You end up checking two (or more) places during incidents, building mental bridges between dashboards that don’t talk to each other.

Database Performance Monitoring: 14+ DBs Supported

Fri, 24 Apr 2026 00:00:00 +0000

Netdata has always collected database metrics: connections, throughput, replication lag, buffer cache hit ratios, and so on. These tell you that something is wrong, but they don’t tell you why. When your PostgreSQL response time spikes, the metric alone doesn’t tell you which query is responsible. For that, you’ve traditionally needed to SSH into the box, connect to the database, and run diagnostic queries manually. Or set up a separate database monitoring tool entirely.

Nagios Plugins: Run Existing Checks & Custom Scripts

Wed, 22 Apr 2026 00:00:00 +0000

A lot of teams have a collection of Nagios plugins and custom monitoring scripts that have been running reliably for years. Some are standard community plugins for checking disk health or SSL certificate expiry. Others are homegrown Bash or Python scripts that check something very specific to the business: whether an API endpoint returns the right payload, whether a batch job completed on time, whether a queue depth is within bounds. These scripts work, they’re battle-tested, and nobody wants to rewrite them.

Secrets Management: Remove Credentials From Configs

Mon, 20 Apr 2026 00:00:00 +0000

If you’re running Netdata collectors that connect to databases, APIs, or other authenticated services, there’s a good chance you have passwords sitting in plain-text configuration files right now. It works, but it’s the kind of thing that makes security teams nervous and makes credential rotation painful. Every password change means editing config files and restarting collectors.

Netdata now supports secrets management natively. Instead of putting credentials directly in your collector configurations, you reference them using a resolver syntax, and Netdata resolves the actual values at runtime from whatever source you choose: environment variables, files on disk, the output of a command, or a centralized secret store like HashiCorp Vault or AWS Secrets Manager.

Netdata Complete Tutorial by LearnLinux.tv

Thu, 16 Apr 2026 00:00:00 +0000

LearnLinux.tv’s Jay LaCroix built a complete video course on Netdata. Eight episodes taking you from first install to production best practices.

Smarter Alerts: Test, Review & Preview Schedules

Thu, 16 Apr 2026 00:00:00 +0000

Alert fatigue usually isn’t caused by one thing. It’s the accumulation of thresholds that are slightly too sensitive, alerts that fire during known maintenance windows, and historical patterns that nobody has the tools to review easily. Fixing it requires better visibility into how alerts actually behave over time, and a way to test changes before they hit production.

We’ve shipped three improvements to alerting in Netdata that address different parts of this problem: the ability to evaluate alert definitions against historical data before deploying them, a timeline view of alert transitions, and a schedule preview for recurring silencing rules.

TV Mode: Put Your Dashboards on the Big Screen

Tue, 14 Apr 2026 00:00:00 +0000

One of the most common requests we’ve gotten since launching custom dashboards is deceptively simple: “How do I put this on a TV?” Teams want their dashboards on wall-mounted screens in NOCs, war rooms, and open office spaces. The dashboard is already built. The data is already there. They just need a way to display it on a screen that nobody is logged into, without exposing the full Netdata Cloud interface.

New Custom Dashboards: Metrics, Logs & Live Commands

Sun, 12 Apr 2026 00:00:00 +0000

Custom dashboards in Netdata have always let you pull charts together on-the-fly into a single view. That’s useful, but it’s also limited. In practice, when you’re running an incident or reviewing a service, you don’t just want charts. You want to see the output of top alongside your CPU metrics. You want slow query logs next to your database latency charts. You want an infrastructure summary card that tells you how many nodes in a room are healthy without having to click through to find out.

Alert Acknowledgement: Mark It as Seen, Keep Working

Fri, 10 Apr 2026 00:00:00 +0000

If you’ve ever opened the alerts tab during a busy period, you know the problem. There are alerts you’ve already looked at, alerts someone on your team is handling, and alerts that fired on a known issue that’s being worked on. They all sit together in the same list alongside the new ones you haven’t seen yet. There’s no way to say “I’ve seen this, move on” without silencing or disabling the alert entirely, which is a much heavier action than the situation calls for.

Expanded Chart View: Investigate Without Leaving the Chart

Wed, 08 Apr 2026 00:00:00 +0000

Charts in Netdata have always been interactive. You can zoom, pan, select time ranges, and see per-second granularity across thousands of metrics. But when you spotted something interesting, the next steps usually meant leaving the chart: opening another tab to check a related metric, navigating to the correlation tool, or pulling up a different time range for comparison. The investigation workflow lived outside the chart, even though the chart was where the investigation started.

Monitor Your Azure to Azure Local Migration

Wed, 08 Apr 2026 00:00:00 +0000

Netdata gives migration teams a single pane of glass across Azure cloud resources and Azure Local cluster nodes. Per-second metrics, edge-based ML, and 800+ collectors mean you can baseline source workloads, validate destination hardware, run parallel cutovers, and operate the new environment without changing tools.

Conversations: Ask Netdata About Anything You're Looking At

Thu, 02 Apr 2026 00:00:00 +0000

Netdata AI can already troubleshoot your alerts and generate Insights reports. What it couldn’t do, until now, was have a back-and-forth conversation. You could get a one-shot analysis, but you couldn’t ask follow-up questions, pull in additional context, or go from a quick question to a full investigation without starting over.

We’ve added a conversational layer to Netdata AI. You’ll notice a new blue chat icon throughout Netdata Cloud, on charts, in the alerts table, on Insights reports, and in the reports list. Click it, and you’re in a conversation where the thing you clicked on is already the context. No copy-pasting metric names, no explaining what you’re looking at.

Node Groups: Organize Infrastructure Into Views

Wed, 01 Apr 2026 00:00:00 +0000

When you’re managing a handful of nodes, the flat list in the nodes tab works fine. When you’re managing hundreds or thousands, it becomes a wall of hostnames. You end up applying the same filters repeatedly: all the production database servers, all the nodes in eu-west, all the Kubernetes workers in the staging cluster. The filters work, but they don’t persist, and there’s no way to share them with the rest of your team.

Netdata Product Roadmap

Fri, 13 Mar 2026 00:00:00 +0000

Netdata’s product roadmap and strategic investment areas. From AI-native observability to full-stack signal coverage, see where Netdata is headed.

Introducing the Netdata Cloud MCP Server

Fri, 27 Feb 2026 00:00:00 +0000

The Netdata Cloud MCP Server is now available — giving AI agents and assistants direct access to your Netdata through a single endpoint at app.netdata.cloud/api/v1/mcp.

AI Is Changing How We Monitor Infrastructure

If you’re an engineer in 2026, chances are AI is already part of your daily workflow, whether that’s a general-purpose assistant like ChatGPT, Claude, or Gemini that you bounce questions off, or a coding agent like Claude Code, Codex, Cursor, or Windsurf that writes and debugs code alongside you. These tools are incredibly powerful, but until now, they’ve been blind to what’s actually happening on your infrastructure.

Netdata for Homelabs

Wed, 18 Feb 2026 00:00:00 +0000

Netdata’s Homelab plan gives you enterprise-grade monitoring at a price that respects your hobby. Per-second metrics, ML anomaly detection, unlimited custom dashboards, and zero configuration — for $90/year.

Howard Conference & Expo 2026: Smarter Observability

Tue, 03 Feb 2026 00:00:00 +0000

The Netdata team will be at the Howard Conference and Expo “Game On” event, February 24-26, 2026 at the Grand Hotel Marriott Resort in Fairhope, Alabama. We’re looking forward to meeting IT leaders and practitioners to talk about real-time observability—what it actually looks like in practice, and where traditional monitoring falls short.

What We’ll Be Showing

Stop by our booth to see Netdata in action and chat with our team about what you’re dealing with in your own environment.

India DevOps Show 2026: Modern Observability Recap

Tue, 03 Feb 2026 00:00:00 +0000

DevOps has fundamentally transformed how organizations build and deliver software. But as deployment velocity increases and infrastructure becomes more dynamic, the gap between shipping code and truly understanding system behavior continues to widen. Teams need observability that keeps pace with their pipelines, not tools that slow them down or break the budget.

Netdata is proud to participate as a Silver Partner at the 10th Edition India DevOps Show 2026, taking place on February 13, 2026 at Aloft ORR Hotel, Bengaluru. We’re excited to engage with India’s vibrant DevOps community and share our vision for efficient, intelligent observability.

Tech Show London 2026: Cloud & AI Observability Recap

Tue, 03 Feb 2026 00:00:00 +0000

The intersection of cloud and AI is creating unprecedented infrastructure complexity. As organizations race to deploy AI workloads alongside traditional cloud services, the demand for intelligent, high-fidelity observability has never been greater. Understanding what’s happening across your entire stack, in real time, is no longer a luxury, it’s a necessity.

That’s why the Netdata team is excited to be part of Tech Show London 2025, taking place March 4-5 at ExCeL London. We’ll be in the Cloud & AI Infrastructure zone, ready to show you how modern observability should work.

Database Monitoring Software Without Query Languages

Tue, 27 Jan 2026 00:00:00 +0000

Netdata gives DBAs complete database visibility with per-second query performance, replication lag tracking, lock analysis, and connection pool monitoring across MySQL, PostgreSQL, SQL Server, Oracle, and MongoDB - all from one dashboard with zero configuration.

Netdata vs ScienceLogic | Monitoring Tools Comparison

Wed, 14 Jan 2026 00:00:00 +0000

Comprehensive comparison of Netdata and ScienceLogic Skylar One platforms, highlighting real-time per-second visibility, transparent pricing, and zero-configuration deployment advantages over enterprise-focused 5-minute polling.

Netdata vs SolarWinds | Monitoring Tools Comparison

Tue, 30 Dec 2025 00:00:00 +0000

Introducing Real-Time Conversations with Netdata AI

Tue, 23 Dec 2025 00:00:00 +0000

Over the past few months, we’ve seen incredible adoption of our AI Investigations and Insights reports. Teams are using them to automate the deep, thoughtful analysis required for complex post-mortems, capacity planning, and performance optimization. These comprehensive reports are fantastic when you need a well-researched, shareable document.

But what about the moments during an investigation? What about the rapid-fire “what if” questions and the quick exploration of hypotheses that happen in the heat of the moment? For that, you need speed and interactivity. You need a partner you can have a real-time dialogue with.

Update Your Netdata Agent

Fri, 19 Dec 2025 00:00:00 +0000

About Us

Thu, 18 Dec 2025 00:00:00 +0000

Meet the team behind Netdata - engineers who transformed observability by distributing intelligence to the edge, making infrastructure transparent for teams worldwide.

Access Control: Secure Observability Without Risk

Thu, 18 Dec 2025 00:00:00 +0000

Secure your observability infrastructure with Netdata’s edge-native access control: granular RBAC, automated SSO provisioning, true multi-tenancy through Spaces and Rooms, comprehensive audit logging, and SOC 2 Type 2 certification - all while keeping your data on your infrastructure.

AI Co-Engineer For Instant Root Cause Insights

Thu, 18 Dec 2025 00:00:00 +0000

Netdata’s AI Co-Engineer combines edge-native machine learning with flexible AI integration, providing instant expert-level insights while keeping your data sovereign and secure.

AI Reporting Software For Observability Data

Thu, 18 Dec 2025 00:00:00 +0000

Generate professional infrastructure reports in minutes, not hours. Netdata’s AI reporting delivers automated insights across metrics, logs, and alerts - from capacity planning to root cause analysis - with zero configuration and complete data sovereignty.

AIOps Platform: Edge-Native ML, Zero Configuration

Thu, 18 Dec 2025 00:00:00 +0000

Enterprise AIOps intelligence without enterprise complexity. Edge-native ML, automated insights, and transparent pricing deliver operational excellence from day one.

Algorithmic Dashboards For Every Metric

Thu, 18 Dec 2025 00:00:00 +0000

Netdata’s algorithmic dashboards transform observability from a months-long configuration project into instant, comprehensive visibility. Every metric visualized automatically, every relationship revealed through intelligent point-and-click analysis, every engineer productive from day one.

Anomaly Advisor: Root Cause In Seconds, Not Hours

Thu, 18 Dec 2025 00:00:00 +0000

Netdata Anomaly Advisor uses edge-native machine learning with 18-model consensus to eliminate 99% of false positives while surfacing root causes in the top 30-50 metrics from thousands collected. Get sub-2-second correlation analysis at any scale without configuration, training delays, or specialist expertise.

Anomaly Detection Software: 99% Fewer False Positives

Thu, 18 Dec 2025 00:00:00 +0000

Production-ready ML anomaly detection from installation. 18 consensus models per metric achieve 99% false positive reduction in anomaly detection while catching issues competitors miss. Zero configuration. Zero false promises.

Anomaly Detection With 99% Fewer False Positives

Thu, 18 Dec 2025 00:00:00 +0000

Netdata trains 18 independent ML models per metric at the edge, achieving 99% false positive reduction in anomaly detection through unanimous consensus - all included at no additional cost.

Application Performance Monitoring Software

Thu, 18 Dec 2025 00:00:00 +0000

Netdata revolutionizes application performance monitoring with distributed edge intelligence, zero-configuration deployment, and AI-powered troubleshooting that delivers enterprise-grade observability at a fraction of traditional APM costs.

AWS Monitoring Software With 1 Second Visibility

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers true real-time AWS monitoring with 1-second granularity, ML-powered anomaly detection, and predictable per-node pricing - eliminating CloudWatch’s complexity and cost unpredictability.

Azure Monitoring Tool With Real-Time Visibility

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers true real-time Azure monitoring with per-second granularity, 90% lower costs, and zero learning curve. Monitor Azure VMs, containers, databases, and applications with automated dashboards, ML anomaly detection, and AI-powered root cause analysis.

Blast Radius Detection For Faster Incident Response

Thu, 18 Dec 2025 00:00:00 +0000

Netdata reveals blast radius dynamically through real-time anomaly correlation and ML-powered pattern recognition, showing the complete story from first failure to full impact in seconds.

Built-In MCP For AI Assistants To Query Metrics & Logs

Thu, 18 Dec 2025 00:00:00 +0000

Transform infrastructure troubleshooting with AI assistants that understand your systems. Netdata’s built-in MCP server provides real-time metrics, ML anomaly detection, logs, and live system state - all accessible through natural language queries.

Cloud Monitoring Software With Real-Time Metrics & AI

Thu, 18 Dec 2025 00:00:00 +0000

Transform cloud operations with Netdata’s distributed observability platform. Get per-second insights, AI-powered troubleshooting, and predictable costs - all while keeping your data sovereign.

Cloud On-Premises Solution With Complete Observability

Thu, 18 Dec 2025 00:00:00 +0000

Self-hosted Netdata Cloud for air-gapped environments and regulated industries. Complete control plane within your datacenter, zero external dependencies, full feature parity with SaaS. SOC 2 Type 2 certified observability that meets DORA, NIS2, HIPAA, and FedRAMP requirements by design.

Container Monitoring Software With Zero Configuration

Thu, 18 Dec 2025 00:00:00 +0000

Transform container monitoring with Netdata’s edge-native platform. Get per-second metrics, automated dashboards, ML anomaly detection, and AI troubleshooting - all without complex setup or unpredictable costs.

Custom Dashboards With Real-Time Precision

Thu, 18 Dec 2025 00:00:00 +0000

Netdata’s algorithmic dashboards eliminate the false choice between pre-built rigidity and custom complexity. Each chart is a complete analytical tool equivalent to 25+ Grafana charts, providing 360° views with simple point-and-click. No query languages. No dashboard building. No maintenance burden.

Data Center Monitoring Software With Real-Time Metrics

Thu, 18 Dec 2025 00:00:00 +0000

Transform data center operations with Netdata’s real-time monitoring platform. Per-second metrics, automated ML anomaly detection, and AI-powered troubleshooting deliver 80% faster MTTR at 90% lower cost than traditional solutions.

Data Sovereignty With Metadata-Only Cloud

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers complete observability while keeping every metric and log exactly where you need it: on your infrastructure, in your jurisdiction, under your control. Meet GDPR, NIS2, DORA, and HIPAA requirements through architecture, not configuration.

Database Monitoring Software With Real-Time Visibility

Thu, 18 Dec 2025 00:00:00 +0000

Real-time database monitoring with AI-powered troubleshooting, zero configuration, and predictable costs. Monitor 15+ database platforms with per-second granularity and ML-based anomaly detection.

DevOps Monitoring Software With ML Anomaly Detection

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers complete observability for DevOps teams with per-second metrics, zero-configuration deployment, and AI-powered root cause analysis. Monitor everything from bare metal to Kubernetes with a single platform that replaces 7 tools while reducing costs by 90%.

Distributed Observability For Scale, Speed & Control

Thu, 18 Dec 2025 00:00:00 +0000

Collect unlimited metrics at per-second granularity with zero configuration. Netdata’s edge-native architecture eliminates cardinality cost traps while delivering ML-based anomaly detection on every metric.

Distributed Observability Without Centralized Pipelines

Thu, 18 Dec 2025 00:00:00 +0000

Netdata’s distributed architecture processes observability data at the edge, eliminating centralized bottlenecks while delivering per-second insights, automatic ML anomaly detection, and 90% cost reduction compared to traditional pipeline solutions.

eBPF Network Monitoring With Real-Time Insights

Thu, 18 Dec 2025 00:00:00 +0000

Get kernel-level network insights - bandwidth, connections, retransmissions - embedded in your infrastructure observability platform. Zero eBPF expertise required. Zero configuration. Zero surprise bills.

Edge Observability Without Cloud Dependency

Thu, 18 Dec 2025 00:00:00 +0000

Monitor thousands of edge nodes with per-second precision, <5% CPU overhead, and 90% cost reduction. Netdata’s edge-native architecture delivers complete visibility with autonomous operation, local ML, and zero configuration.

Eliminate Log Pipelines: Native Log Query At Scale

Thu, 18 Dec 2025 00:00:00 +0000

Netdata’s Zero Pipeline Logs architecture eliminates traditional log aggregation pipelines by leveraging native system formats. Query logs directly where they live - no shipping, parsing, or indexing required - achieving 90% cost reduction and sub-2-second latency at any scale.

Enterprise Monitoring At Scale Without Bottlenecks

Thu, 18 Dec 2025 00:00:00 +0000

Experience observability that grows naturally with your infrastructure - no rewrites, no bottlenecks, no compromise. Monitor everything, everywhere, at per-second resolution with predictable costs that scale with nodes, not data volume.

Financial Transaction Monitoring Software

Thu, 18 Dec 2025 00:00:00 +0000

Transform financial services operations with per-second monitoring, AI-powered troubleshooting, and zero-pipeline logs. Built for compliance, optimized for performance, designed for lean teams.

Gaming Infrastructure Monitoring & Observability

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers true real-time monitoring for gaming infrastructure with per-second metrics, edge-based ML anomaly detection, and AI-powered troubleshooting - enabling lean teams to maintain 99.9% uptime while reducing monitoring costs by 90%.

GCP Monitoring Tool With Per-Second Visibility

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers true real-time GCP monitoring with per-second granularity, zero-configuration deployment, and predictable per-node pricing - solving the cost unpredictability and complexity that plague organizations.

Hetzner Monitoring With Real-Time Observability

Thu, 18 Dec 2025 00:00:00 +0000

Complete monitoring solution for Hetzner infrastructure with 1-second granularity, automated hardware health tracking, and AI-powered troubleshooting. Eliminates native monitoring gaps while reducing costs by 90%.

High Cardinality Protection For Unlimited Metrics

Thu, 18 Dec 2025 00:00:00 +0000

Automated multi-layer protection handles extreme cardinality at the edge, enabling unlimited observability without manual tuning or cost explosions.

HPC Monitoring Software With Per-Second Metrics

Thu, 18 Dec 2025 00:00:00 +0000

Transform HPC operations with distributed edge-native monitoring that delivers sub-2-second insights, 90% cost reduction, and linear scalability to 100,000+ nodes - without the complexity.

Hybrid Cloud Monitoring Solution At 90% Lower Cost

Thu, 18 Dec 2025 00:00:00 +0000

Transform hybrid cloud monitoring with Netdata’s distributed edge-native architecture. Get per-second visibility, ML-powered insights, and 90% cost savings without pipelines, sampling, or vendor lock-in.

Infrastructure Health Monitoring Software

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers HIPAA-compliant, real-time observability for healthcare with 90% cost savings, per-second monitoring, and zero-configuration deployment. Monitor EHRs, medical devices, and critical infrastructure with edge-based ML and complete data sovereignty.

Infrastructure Monitoring Software With AI & ML

Thu, 18 Dec 2025 00:00:00 +0000

Transform infrastructure monitoring with Netdata’s edge-native platform. Get per-second visibility, ML anomaly detection on every metric, and AI-powered troubleshooting - all while keeping your data sovereign and reducing costs by 90%.

Infrastructure Observability Without Instrumentation

Thu, 18 Dec 2025 00:00:00 +0000

Complete infrastructure visibility without touching code. Netdata monitors processes, containers, networks, and logs automatically through kernel-level instrumentation and intelligent auto-discovery.

Instant Observability UI For Every Engineer

Thu, 18 Dec 2025 00:00:00 +0000

Algorithmic dashboards, ML-powered insights, and point-and-click analysis that turns 100,000+ metrics into actionable intelligence in seconds.

IoT Monitoring Solution With Full Data Sovereignty

Thu, 18 Dec 2025 00:00:00 +0000

Real-time IoT monitoring with per-device metrics, ML-powered anomaly detection, and transparent pricing. Monitor thousands of sensors, gateways, and edge devices with <2s latency.

Join Us

Thu, 18 Dec 2025 00:00:00 +0000

Build infrastructure observability at massive scale with Netdata. Remote-first culture, open source core, 76K+ GitHub stars. Own the full stack, ship code used by millions, work with world-class engineers.

Kubernetes Monitoring Tool Without Centralized Data

Thu, 18 Dec 2025 00:00:00 +0000

Transform Kubernetes observability with Netdata’s edge-native architecture. Get per-second visibility, ML anomaly detection on every metric, and AI-powered root cause analysis - all at 90% lower cost than traditional solutions.

Linux Monitoring Software With Per-Second Metrics

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers complete Linux monitoring with per-second granularity, edge-based ML anomaly detection, and zero-pipeline logs - all with zero configuration and predictable per-node pricing.

Live Tab - Real-Time Netdata Functions

Thu, 18 Dec 2025 00:00:00 +0000

The Live tab provides real-time access to Netdata Functions - specialized routines from collectors that deliver live information from your monitored nodes, including database monitoring, network topology maps, process explorers, and diagnostics.

LLM Monitoring Platform With Per-Second Visibility

Thu, 18 Dec 2025 00:00:00 +0000

Real-time infrastructure monitoring for LLM deployments. Track GPU utilization, container resources, database performance, and system metrics with per-second granularity and ML-powered anomaly detection.

Log Management Software With 90% Cost Reduction

Thu, 18 Dec 2025 00:00:00 +0000

Eliminate expensive log pipelines and centralized clusters. Netdata queries systemd-journal and Windows Event Logs directly at the edge, delivering 90% cost savings, sub-second queries, and complete data sovereignty - all while maintaining superior analysis accuracy than traditional platforms.

Manufacturing Monitoring Software Without Downtime

Thu, 18 Dec 2025 00:00:00 +0000

Transform manufacturing operations with distributed observability that processes data at the edge. Monitor production metrics, predict equipment failures, and troubleshoot issues 80% faster - all while keeping costs predictable and data sovereign.

Microservices Monitoring Tool Without Sampling

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers comprehensive microservices observability with per-second metrics, ML-based anomaly detection, and AI-powered troubleshooting - all without the complexity, cost overruns, or tool sprawl that plague traditional solutions.

Monitoring Agents For Accurate Production Insights

Thu, 18 Dec 2025 00:00:00 +0000

Discover how Netdata Agents revolutionize infrastructure monitoring with edge-native intelligence, per-second granularity, and zero-configuration deployment. Complete observability in 60 seconds.

MSP Monitoring Software With Multi-Tenant Isolation

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers the most cost-effective, scalable observability platform for MSPs - solving tool sprawl, alert fatigue, and unpredictable costs with true real-time monitoring, physical multi-tenancy, and edge-based ML.

Multi-Cloud Monitoring Tools With Predictable Pricing

Thu, 18 Dec 2025 00:00:00 +0000

Break free from fragmented multi-cloud monitoring. Netdata delivers unified per-second visibility across AWS, Azure, GCP, and on-premises with edge-native ML, zero configuration, and transparent per-node pricing - eliminating tool sprawl and surprise bills.

Multi-Tenant Observability Access Without Data Egress

Thu, 18 Dec 2025 00:00:00 +0000

Edge-intelligent alerting that delivers component-level precision through 400+ pre-configured templates, zero-configuration deployment, and sub-2-second evaluation latency - all while maintaining visibility during network partitions.

Native iOS & Android Apps With AI Troubleshooting

Thu, 18 Dec 2025 00:00:00 +0000

Transform on-call monitoring with Netdata’s mobile apps for iOS and Android. Get push notifications for infrastructure alerts, AI-powered root cause analysis, and real-time dashboard access - all at a fixed, predictable price that includes mobile access for your entire team.

Netdata Parents For Intelligent Observability

Thu, 18 Dec 2025 00:00:00 +0000

Streaming aggregators that centralize data from thousands of nodes while keeping intelligence distributed at the edge. Handle million metrics per second with minimal resources, active-active clustering for high availability, and ML-powered insights without centralized bottlenecks.

Netdata vs Amazon CloudWatch | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata provides 90% cost reduction vs CloudWatch with superior real-time monitoring (1-second vs 1-60 minutes), comprehensive ML on all metrics, and zero-configuration simplicity. Deploy in 60 seconds and eliminate CloudWatch’s unpredictable costs while gaining better visibility.

Netdata vs Azure Monitor | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs BetterStack | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata and BetterStack serve different observability needs. Netdata excels at deep infrastructure monitoring with per-second metrics and ML intelligence, while BetterStack focuses on incident coordination and uptime monitoring. Learn which solution fits your technical requirements.

Netdata vs CardinalHQ | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata’s edge-native monitoring platform versus CardinalHQ’s AI-powered observability optimization layer. Learn which solution fits your infrastructure needs.

Netdata vs Catchpoint | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers real-time infrastructure visibility with per-second metrics, ML-based anomaly detection, and AI-powered troubleshooting - the essential internal monitoring layer that external testing platforms cannot provide.

Netdata vs Checkmk | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Checkmk comparison: Real-time monitoring (1-second vs 60-second), zero configuration (vs 700+ rule sets), AI-powered troubleshooting, 90% cost reduction, and operational simplicity. See why cloud-native teams choose Netdata for modern infrastructure monitoring.

Netdata vs Coralogix | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata and Coralogix serve complementary observability needs. While Coralogix excels at centralized log analytics and SIEM, Netdata provides unmatched real-time infrastructure monitoring with per-second granularity, zero-configuration deployment, and transparent per-node pricing. Learn how Netdata addresses common Coralogix pain points including steep learning curves, slow production rollouts, unpredictable costs, and UI performance issues.

Netdata vs Coroot | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Coroot comparison: production-ready observability platform vs emerging Kubernetes tool. Learn why Netdata provides superior maturity, universal coverage, and operational simplicity for real-world infrastructure.

Netdata vs Cribl | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata provides real-time infrastructure monitoring with ML-based anomaly detection and built-in dashboards. Cribl optimizes log pipelines and routes data for cost efficiency. Learn how these complementary solutions work together for complete observability.

Netdata vs Dash0 | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Dash0: Real-time edge intelligence with predictable pricing versus OpenTelemetry-native centralized monitoring. Compare features, costs, and capabilities to choose the right observability platform.

Netdata vs Datadog | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Dynatrace | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata provides superior infrastructure monitoring at a fraction of Dynatrace’s cost, with true per-second granularity and complete data sovereignty. Learn when to use each platform and how a hybrid approach delivers complete observability at significantly lower total cost.

Netdata vs EdgeDelta | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Complete edge-native observability vs telemetry pipelines. See how Netdata provides real-time monitoring, ML-powered insights, and automated dashboards without the complexity of data routing infrastructure.

Netdata vs ELK Stack | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

A comprehensive comparison between Netdata and the ELK Stack for infrastructure monitoring and observability. Learn which solution fits your real-time monitoring, log management, and troubleshooting needs.

Netdata vs Fluentd | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata provides complete real-time observability with metrics, logs, ML anomaly detection, and AI troubleshooting in a single platform. Fluentd excels at log collection and routing. Learn how they complement each other and when Netdata replaces entire monitoring stacks.

Netdata vs Grafana | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata and Grafana monitoring platforms, highlighting real-time performance, cost efficiency, and operational simplicity.

Netdata vs Graylog | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata and Graylog serve different but complementary purposes in modern observability stacks. Netdata provides real-time infrastructure metrics monitoring with ML-based anomaly detection, while Graylog offers centralized log management for security analysis and forensic investigations. Together, they deliver complete observability.

Netdata vs Grepr | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Guance | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs HetrixTools | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

When HetrixTools alerts you’re down, Netdata shows you why. Compare external uptime monitoring with internal infrastructure observability powered by ML and AI.

Netdata vs Honeycomb | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Compare Netdata and Honeycomb for infrastructure monitoring. Learn how Netdata provides comprehensive system visibility, predictable per node pricing, and zero-configuration deployment—solving the gaps in Honeycomb’s application-centric observability.

Netdata vs IBM Instana | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Icinga | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata and Icinga monitoring platforms, highlighting real-time performance analysis, automated deployment, ML-based anomaly detection, and modern observability capabilities.

Netdata vs Lansweeper | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Last9 | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs LibreNMS | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Compare Netdata and LibreNMS monitoring platforms. Learn how Netdata delivers per-second real-time monitoring, AI-powered anomaly detection, and native Windows support while LibreNMS excels at SNMP network device monitoring.

Netdata vs LogicMonitor | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata and LogicMonitor monitoring platforms, highlighting real-time performance, cost efficiency, and deployment simplicity advantages.

Netdata vs Logz.io | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata provides real-time, edge-native observability with predictable per-node pricing and zero-configuration deployment. Unlike Logz.io’s centralized log aggregation model with volume-based billing, Netdata distributes intelligence to your systems - delivering 90% cost reduction, 80% faster MTTR, and complete data sovereignty.

Netdata vs ManageEngine | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs ManageEngine: Real-time observability comparison for cloud-native infrastructure. See how per-second monitoring, distributed architecture, and zero-config deployment transform operations.

Netdata vs Mezmo | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata and Mezmo observability platforms, highlighting real-time monitoring capabilities, cost efficiency, and deployment simplicity.

Netdata vs Monit | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Compare Netdata’s real-time observability platform with Monit’s lightweight process supervision. Learn how Netdata provides per-second monitoring, ML anomaly detection, AI troubleshooting, and unlimited scalability versus Monit’s basic service checks and 1,000-host ceiling.

Netdata vs N-able: Real-Time Monitoring Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata provides per-second infrastructure monitoring with ML-based anomaly detection and AI troubleshooting - capabilities N-able’s 5-10 minute intervals and basic dashboards can’t match. See how Netdata solves the monitoring gaps N-able customers experience daily.

Netdata vs Nagios | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Netmon: Real-Time Observability Compared

Thu, 18 Dec 2025 00:00:00 +0000

Netdata transforms infrastructure monitoring with real-time per-second data collection, ML-based anomaly detection, and AI-powered troubleshooting - replacing traditional appliance-based solutions with modern cloud-native observability.

Netdata vs New Relic | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Observium | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Observium comparison: Real-time full-stack observability with ML and AI versus SNMP-based network device monitoring. See how Netdata’s distributed architecture, per-second granularity, and automated intelligence deliver comprehensive infrastructure visibility that network-only monitoring cannot provide.

Netdata vs Prometheus: No PromQL Complexity

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Prometheus comparison: Real-time monitoring with zero configuration, built-in ML, and 90% lower costs. See how Netdata solves Prometheus pain points while maintaining enterprise-grade capabilities.

Netdata vs PRTG | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Rakuten SixthSense | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata provides per-second real-time infrastructure monitoring with AI-powered troubleshooting for 90% less than traditional APM tools. Compare features, pricing, and capabilities to see why infrastructure teams choose Netdata over Rakuten SixthSense.

Netdata vs Sensu | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Sensu: Real-time monitoring that just works vs a platform in decline. See why thousands of teams are switching to Netdata for instant deployment, beautiful dashboards, and 90% cost savings.

Netdata vs Sentry | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata and Sentry serve different but complementary purposes in modern observability. Netdata provides real-time infrastructure monitoring with ML-based anomaly detection, while Sentry excels at application error tracking. Learn how using both tools together delivers complete visibility at lower total cost.

Netdata vs SigLens | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata and SigLens observability platforms. Learn how these complementary tools work together to provide complete infrastructure monitoring and log analysis at a fraction of traditional enterprise costs.

Netdata vs SigNoz | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata and SigNoz observability platforms, highlighting operational simplicity, real-time performance, and cost efficiency for infrastructure monitoring teams.

Netdata vs Splunk | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs StackState | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Real-time monitoring in 60 seconds vs months of topology setup. Netdata provides per-second observability, ML anomaly detection, and AI troubleshooting at 90% lower cost - perfect for lean teams who need comprehensive monitoring without enterprise complexity.

Netdata vs Sumo Logic | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata and Sumo Logic: real-time monitoring, pricing models, deployment complexity, and use cases. Learn which platform fits your DevOps and observability needs.

Netdata vs Uptime Kuma | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs UptimeRobot | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata and UptimeRobot serve different purposes: UptimeRobot monitors external availability while Netdata provides deep infrastructure diagnostics. Learn when to use each tool and how Netdata’s component-level alerts and ML-based anomaly detection provide deeper visibility.

Netdata vs VictoriaMetrics | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata vs VictoriaMetrics covering real-time monitoring, ML anomaly detection, visualization, scalability, and total cost of ownership.

Netdata vs Virtana: Real-Time Operations Monitoring

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers per-second operational monitoring for engineering teams, while Virtana provides strategic cost optimization for IT executives. Learn how these complementary platforms serve different needs within modern organizations.

Netdata vs WhatsUp Gold | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs WhatsUp Gold: Real-time observability with per-second granularity, zero configuration, and built-in ML/AI vs traditional Windows-centric network monitoring. See how Netdata delivers 90% cost reduction and 80% faster MTTR for modern infrastructure.

Netdata vs Zabbix | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Netdata vs Zenoss | Monitoring Tools Comparison

Thu, 18 Dec 2025 00:00:00 +0000

Comprehensive comparison of Netdata and Zenoss monitoring platforms, highlighting Netdata’s advantages in real-time performance, ease of use, resource efficiency, and transparent pricing.

Network Monitoring Software For Education

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers comprehensive observability for educational institutions with zero-configuration deployment, complete data sovereignty, and 90% cost savings. Monitor everything from learning management systems to research clusters with per-second precision.

Observability Cost Control With Per-Node Pricing

Thu, 18 Dec 2025 00:00:00 +0000

Transform observability from unpredictable cost center to fixed operational expense. Netdata’s distributed architecture and transparent pricing deliver 90% cost savings while providing unlimited metrics, logs, and per-second visibility.

Open Source

Thu, 18 Dec 2025 00:00:00 +0000

The world’s most popular open source monitoring platform. Deploy complete infrastructure observability in 60 seconds with zero configuration. Trusted by millions of engineers worldwide.

Operations Center Monitoring & 24/7 NOC Teams

Thu, 18 Dec 2025 00:00:00 +0000

Real-time observability platform designed for 24/7 operations centers. Per-second monitoring, zero-configuration deployment, and AI-powered troubleshooting reduce MTTR by 80%.

Our Values and Promises

Thu, 18 Dec 2025 00:00:00 +0000

Discover the 12 principles that guide every decision at Netdata. From radical transparency to edge-native intelligence, learn how we invest engineering effort to eliminate complexity and align our success with your operational needs.

Per-Second Observability Without Cardinality Limits

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers true real-time monitoring at planetary scale through distributed edge intelligence - maintaining per-second granularity and sub-2-second latency whether you’re monitoring 10 nodes or 100,000, with 90% cost reduction and 80% MTTR improvement.

Point-and-Click Analysis Without Query Languages

Thu, 18 Dec 2025 00:00:00 +0000

Experience genuine queryless observability with Netdata’s algorithmic dashboards and NIDL framework. Each chart delivers the equivalent of 25+ Grafana visualizations through simple point-and-click navigation - making expert-level analysis accessible to every engineer from day one.

Proxmox Monitoring With Real-Time Observability

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers enterprise-grade Proxmox monitoring with per-second granularity, automated dashboards, and ML anomaly detection - all without the complexity or cost of traditional solutions.

Query Systemd-Journal Logs Without Pipelines

Thu, 18 Dec 2025 00:00:00 +0000

Netdata revolutionizes systemd-journal log management by querying logs directly where they live, eliminating costly pipelines and delivering faster queries with zero configuration. Get complete visibility into Linux system logs with AI-powered troubleshooting, unified metrics correlation, and 90% cost reduction compared to traditional log management platforms.

Real-Time GPU Monitoring For AI Infrastructure

Thu, 18 Dec 2025 00:00:00 +0000

Monitor AI training clusters, inference APIs, and ML workloads with Netdata’s edge-native platform. Per-second visibility, ML-powered insights, predictable pricing.

Real-Time Infrastructure Monitoring For Freelancers

Thu, 18 Dec 2025 00:00:00 +0000

Real-time infrastructure monitoring built for technical freelancers managing multiple client environments. Zero configuration, AI-powered insights, and predictable costs let you focus on delivering value, not managing monitoring tools.

Real-Time Infrastructure Monitoring For Tech Industry

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers per-second observability for technology infrastructure with ML-based anomaly detection, zero-configuration deployment, and predictable per-node pricing - solving the cost, complexity, and visibility challenges facing modern tech teams.

Real-Time Infrastructure Observability For CISOs

Thu, 18 Dec 2025 00:00:00 +0000

Transform infrastructure security monitoring with real-time visibility, edge-based ML anomaly detection, and zero-configuration deployment. Netdata empowers CISOs to detect threats faster, reduce operational burden, and maintain data sovereignty - all at 90% lower cost than traditional SIEMs.

Real-Time Infrastructure Troubleshooting With AI

Thu, 18 Dec 2025 00:00:00 +0000

Transform troubleshooting from hours to minutes with Netdata’s real-time observability platform. Get per-second metrics, automatic anomaly detection, and AI-guided investigations—all with zero configuration and 90% cost savings.

Real-Time Monitoring & ML Detection For Sysadmins

Thu, 18 Dec 2025 00:00:00 +0000

Purpose-built observability for sysadmins: per-second metrics, automatic dashboards, ML-powered anomaly detection, and predictable pricing. Monitor everything from bare metal to Kubernetes without learning query languages or building dashboards.

Real-Time Observability For 24/7 Operations

Thu, 18 Dec 2025 00:00:00 +0000

Real-time observability built for continuous operations. Per-second metrics, ML anomaly detection, and AI root cause analysis keep your infrastructure running around the clock.

Real-Time Observability For Developers Who Ship Fast

Thu, 18 Dec 2025 00:00:00 +0000

Stop context switching between tools. Netdata provides developers with real-time infrastructure visibility, AI-assisted debugging, and zero-configuration monitoring—all in one unified platform that integrates directly into your IDE and workflow.

Real-Time Observability For Government & Public Sector

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers enterprise-grade observability for government agencies with per-second monitoring, configurable multi-year log retention, edge-based ML anomaly detection, and 90% cost savings vs. legacy monitoring - all while keeping data sovereign on your infrastructure.

Real-Time Observability For Platform Engineers

Thu, 18 Dec 2025 00:00:00 +0000

Empower platform engineering teams with Netdata’s distributed observability platform. Get per-second visibility, automated dashboards, ML anomaly detection, and predictable costs - all without query languages or complex pipelines.

Real-Time Observability For SRE Teams

Thu, 18 Dec 2025 00:00:00 +0000

Transform SRE operations with Netdata’s edge-native observability platform. Get per-second visibility, ML anomaly detection on every metric, and AI-powered troubleshooting at 90% lower cost than traditional solutions.

Real-Time Troubleshooting With Sub-2-Second Latency

Thu, 18 Dec 2025 00:00:00 +0000

Interactive debugging with per-second precision and full historical context. Netdata transforms troubleshooting through edge-native ML, automated correlation, and AI-powered analysis.

Red Hat OpenShift Monitoring Tool At Scale

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers real-time, AI-powered observability for Red Hat OpenShift with 90% cost savings, per-second metrics, ML anomaly detection on every metric, and zero-configuration deployment. Eliminate Prometheus memory exhaustion, Loki query timeouts, and RHACM complexity.

Retail Monitoring Software Without PromQL

Thu, 18 Dec 2025 00:00:00 +0000

Transform retail operations with Netdata’s edge-native observability platform. Monitor POS systems, payment processing, store networks, and inventory infrastructure in real-time with per-second granularity, ML-powered anomaly detection, and predictable per-node pricing.

Root Cause Analysis (RCA) With 80% MTTR Reduction

Thu, 18 Dec 2025 00:00:00 +0000

Netdata’s edge-native ML detects anomalies as they happen, automated correlation surfaces root causes in the top 30-50 results, and AI explains incidents in plain English - achieving 80% MTTR reduction at 90% lower cost.

Service Mesh Observability Without Overhead

Thu, 18 Dec 2025 00:00:00 +0000

Netdata revolutionizes service mesh observability with true real-time monitoring, automatic anomaly detection, and transparent pricing - solving the cost explosion, performance overhead, and complexity that plague traditional solutions.

Synthetic Monitoring Solution For APIs & Services

Thu, 18 Dec 2025 00:00:00 +0000

Real-time synthetic monitoring for APIs, certificates, DNS, and network health - unified with infrastructure observability for faster troubleshooting and predictable costs.

Team Collaboration With Real-Time Observability

Thu, 18 Dec 2025 00:00:00 +0000

Unite teams with real-time visibility, intelligent collaboration features, and enterprise-grade access control - all while maintaining data sovereignty and predictable costs.

Telecom Network Monitoring Software At Scale

Thu, 18 Dec 2025 00:00:00 +0000

Transform telecom operations with distributed edge monitoring that delivers complete infrastructure visibility, ML-powered anomaly detection, and zero-configuration deployment in minutes.

Text-To-Alert: Create Alerts From Natural Language

Thu, 18 Dec 2025 00:00:00 +0000

Netdata has an incredibly powerful alerting engine. But this can sometimes be a double-edged sword: the flexibility to build incredibly specific, intelligent alerts is immense, but mastering its syntax can feel like learning a new language. We’ve heard this from so many of you. You tell us that configuring alerts is often the steepest part of the learning curve, a task that falls to the one “Netdata expert” on the team who has spent the time digging through the documentation.

Tiered Retention With 0.6 Bytes/Sample Storage

Thu, 18 Dec 2025 00:00:00 +0000

Netdata redefines tiered retention through edge-native architecture that updates all storage tiers simultaneously during collection, eliminating batch jobs and maintenance overhead while providing 15× better efficiency than traditional systems.

Unified Observability Platform With Predictable Costs

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers unified observability through edge-native architecture - metrics, logs, and AI-powered insights at per-second granularity without the complexity, cost explosion, or tool sprawl of traditional platforms.

VMware Monitoring Software Without Centralized Data

Thu, 18 Dec 2025 00:00:00 +0000

Netdata delivers per-second VMware monitoring with zero configuration, built-in ML anomaly detection, and 90% cost savings. Monitor vSphere, ESXi hosts, VMs, and hybrid clouds with true real-time visibility.

Web Server Monitoring Software For Complete Visibility

Thu, 18 Dec 2025 00:00:00 +0000

Transform web server monitoring with Netdata’s edge-native platform. Get per-second metrics, automated root cause analysis, and complete infrastructure visibility without complex setup or unpredictable costs.

Windows Event Log Monitoring Software

Thu, 18 Dec 2025 00:00:00 +0000

Netdata revolutionizes Windows Event Log monitoring by combining infrastructure metrics with native log access, ML anomaly detection, and AI troubleshooting - delivering comprehensive observability at a fraction of traditional SIEM costs.

Windows Server Monitoring Software At Scale

Thu, 18 Dec 2025 00:00:00 +0000

Transform Windows Server monitoring with Netdata’s edge-native platform. Get per-second visibility, automatic discovery, and AI-powered troubleshooting at a fraction of traditional costs.

Zero Configuration Observability In 60 Seconds

Thu, 18 Dec 2025 00:00:00 +0000

True zero-configuration observability: from installation to troubleshooting production issues in 60 seconds, without specialized skills or months of integration work.

Zero Data Storage In SaaS With Unified Visibility

Thu, 18 Dec 2025 00:00:00 +0000

Access your entire infrastructure from anywhere while all observability data stays on-premises. Unlimited horizontal scalability with predictable per-node pricing. Role-based access control, dynamic organization, infrastructure-level dashboards, centralized alerts, and managed AI troubleshooting.

Zero-Downtime Monitoring For Real-Time Deployments

Thu, 18 Dec 2025 00:00:00 +0000

Netdata’s distributed architecture eliminates single points of failure in monitoring, delivering sub-2-second visibility with automatic failover, zero data loss, and production-safe resource overhead—ensuring complete observability during the moments that matter most.

Dashboard Gaps During Outages: Research Report

Mon, 01 Dec 2025 00:00:00 +0000

Research Notice: This document was compiled through online research conducted on December 1, 2025. It serves as reference material for our blog post: Monitor Everything is an Anti-Pattern!. Sources are cited inline and summarized at the end.

Outage Incidents With Monitoring Gaps, Missing Dashboards & Quantified Impact

Executive Summary

This report documents multiple real-world outage incidents where monitoring systems failed to collect critical data, engineers created dashboards ad-hoc during crises, and teams implemented systematic improvements post-incident. The research reveals consistent patterns across major technology companies including Cloudflare, Datadog, GitLab, AWS, Azure, PagerDuty, and Google SRE, with quantified financial impacts reaching $2 million per hour for high-impact outages.

Monitor Everything is an Anti-Pattern!

Mon, 01 Dec 2025 00:00:00 +0000

Bullshit and nonsense.

But let’s take it from the beginning.

The industry’s story goes something like this:

“Monitor everything is universally recognized as an anti-pattern.”
“You’ll drown in metrics, burn out your engineers, and blow your budget.”
“Just focus on 3–10 signals — the Four Golden Signals, RED, USE — and ignore everything else.”
“Trust us, you don’t want that much telemetry.”

(true, read the whole story here)

Then, in the same breath:

Why 'Monitor Everything' Is An Anti-Pattern: Research

Mon, 01 Dec 2025 00:00:00 +0000

Research Notice: This document was compiled through online research conducted on December 1, 2025. It serves as reference material for our blog post: Monitor Everything is an Anti-Pattern!. Sources are cited inline and summarized at the end.

TL;DR

“Monitor everything” is universally recognized as an anti-pattern by SRE experts, observability leaders, and major tech companies. The core reasons are:

Metric Fatigue: Teams become overwhelmed by excessive data, unable to identify critical signals
Alert Fatigue: 63% of organizations face 1,000+ daily alerts with 72-99% false positives, costing $300,000+/hour in missed incidents
Lack of Actionability: 97% of alerts are non-actionable noise rather than signals requiring response
High Costs: Organizations spend 20-40% of cloud budgets on observability (vs. optimal 10-15%), with cardinality explosions creating exponential cost increases
Employee Burnout: Costs $4,000-$21,000 per employee annually, totaling $5M+ for 1,000-person companies
System Complexity: Monitoring systems themselves become fragile, requiring constant maintenance
Monitoring Tools Creating Problems: Monitoring agents can cause the latency outliers they’re meant to detect

Expert Consensus: Focus on 3-10 key metrics (Google’s Four Golden Signals, RED Method, USE Method) that indicate symptoms rather than attempting comprehensive monitoring of all possible metrics.

Gartner IOCS 2025: Tackling Observability Overspend

Fri, 07 Nov 2025 00:00:00 +0000

The observability market is facing a paradox. As organizations spend more than ever on monitoring tools, their infrastructure complexity continues to grow, and incident resolution times often remain stubbornly high. Teams are drowning in data, struggling with tool sprawl, and facing unpredictable, budget-breaking bills.

This challenge, how to gain better visibility without spiraling costs, is one of the most critical conversations for IT leaders today.

That’s why the Netdata team is heading to Las Vegas for the Gartner IT Infrastructure, Operations & Cloud Strategies (IOCS) Conference from December 9-11, 2025. We’ll be there to discuss this challenge head-on and share our vision for a more efficient and intelligent future for observability.

ServiceNow Integration: Streamline Incident Response

Fri, 07 Nov 2025 00:00:00 +0000

When a critical alert fires at 2 AM, the last thing your on-call engineer should be doing is manual administrative work. Yet, for many teams, that’s exactly what happens. You see the alert in your monitoring tool, then you have to switch contexts, open a new browser tab, log into your ITSM platform, and manually create an incident—all while your systems are failing.

This “swivel-chairing” between tools is slow, error-prone, and a significant drag on your Mean Time to Resolution (MTTR).

SOC 2 Type 2: Validated Security Controls Over Time

Thu, 16 Oct 2025 00:00:00 +0000

We’re excited to share that Netdata has successfully achieved SOC 2 Type 2 attestation.

Following a five-month audit conducted by Sensiba LLP, we can now confirm that our security controls work consistently in practice. The audit covered the period from April 1 to August 31, 2025, and tested whether our controls operated effectively throughout that entire timeframe.

Back in April, we announced our SOC 2 Type 1 attestation, which validated that our security controls were properly designed at a specific point in time. We also mentioned we were entering the monitoring period for Type 2. Today we can share the results.

Automate Infrastructure Analysis With AI Reports

Tue, 23 Sep 2025 00:00:00 +0000

The least exciting part of an operations or SRE role is often the manual, repetitive task of generating reports. It’s the Monday morning scramble to summarize weekly infrastructure health for the team, or the end-of-quarter push to build a capacity planning document. This is boilerplate work that pulls you away from critical engineering tasks.

We believe that if a process is repeatable, it should be automated.

That’s why we’re introducing Scheduled AI Investigations and Insights. This new capability builds directly on our existing AI tools, allowing you to set your most important analyses on a recurring schedule. It’s like setting up a cron job for your infrastructure reporting, letting your Co-SRE do the heavy lifting for you.

Elasticsearch Yellow Cluster: Unassigned Shards Fix

Sun, 07 Sep 2025 00:00:00 +0000

You run a health check on your production cluster and the result comes back: status: yellow. It’s not the dreaded red status, so your application is likely still serving requests, but this is a critical warning sign. An Elasticsearch yellow cluster status is a direct indication that your data’s high availability is compromised. While all your primary shards are active, one or more replica shards have failed to be assigned to a node. Ignoring this warning can lead to data loss if another node fails, or worse, it could be a symptom of a network partition risking an Elasticsearch split-brain.

Docker Layer Caching in CI Pipelines Cut Build Times by 70 %

Sat, 06 Sep 2025 00:00:00 +0000

You push a one-line code change, and your CI/CD pipeline kicks off. You grab a coffee, come back, and it’s still running, stuck on the docker build step. Slow container builds are a silent productivity killer, delaying feedback, slowing down deployments, and frustrating developers. In an ephemeral CI environment where every job starts with a clean slate, Docker often has to rebuild your entire application image from scratch, every single time.

Linux Cgroups V2 Memory Throttling & OOM Fix

Fri, 05 Sep 2025 00:00:00 +0000

Your critical service is lagging. Users are complaining about timeouts. You check your orchestration platform and see the dreaded OOMKilled status on a container. You dive into the node’s logs (dmesg) and confirm it: the kernel’s Out-of-Memory (OOM) killer has claimed another victim. The immediate fix is easy—restart the container, maybe give it more memory—but the real question remains unanswered: why did it happen? Was it a sudden memory leak, a traffic spike, or something more subtle?

Designing Error Budget Policies For SLOs At Scale

Thu, 04 Sep 2025 00:00:00 +0000

In every engineering organization, there’s a constant, fundamental tension: the push to ship new features versus the need to maintain a stable, reliable service. Move too fast, and you risk outages that erode user trust. Move too slowly, and you risk being outpaced by the competition. For years, this balancing act was managed by intuition, late-night heroics, and tense priority meetings. Site Reliability Engineering (SRE) offers a better way: the error budget.

Consul Service Discovery Failures: Causes & Fixes

Wed, 03 Sep 2025 00:00:00 +0000

It’s a scenario that keeps DevOps and SRE teams up at night: your application logs fill with connection errors, services start failing, and you realize a critical component can’t find the database it depends on. The culprit? A breakdown in your service discovery mechanism. For many, that mechanism is HashiCorp Consul, the backbone of modern microservice architectures. When Consul falters, your entire ecosystem can become unstable.

Understanding how to diagnose these failures is crucial. The problem often lies deep within the operational layers—agent communication issues, misconfigured health checks, or disruptions in the gossip protocol that maintains cluster state. In this guide, we’ll dissect the most common causes of Consul service discovery failures, providing you with the tools to troubleshoot and resolve them. More importantly, we’ll show you how to shift from a reactive, fire-fighting mode to a proactive one, using comprehensive monitoring to build a truly resilient Consul deployment.

AI Troubleshooting GA With On-Demand Credits

Tue, 02 Sep 2025 00:00:00 +0000

Since launching our AI investigations and insights in a research preview, one thing has become clear: automated root cause analysis delivers a significant return on investment. Teams have confirmed that instant insights don’t just save a few minutes; they fundamentally shorten incident response cycles, free up valuable engineering hours, and reduce the business impact of downtime.

The preview successfully demonstrated this value, with 10 free AI sessions per month allowing teams to integrate AI into their workflows. Now, based on the success and maturity of the capabilities, we are proud to announce that Netdata’s AI investigations and insights are graduating from research preview to General Availability.

Fix NGINX 502/504 Double-Proxy Errors With CloudFront

Tue, 02 Sep 2025 00:00:00 +0000

You’ve set up a modern, resilient architecture: NGINX as your robust origin server and Cloudflare as your global CDN and security layer. Then, it happens. A visitor reports seeing a dreaded white screen with “502 Bad Gateway” or “504 Gateway Timeout.” The immediate question is, who’s to blame? Is Cloudflare having an edge_error, or is your NGINX origin server failing? This confusion is a common pitfall in a double_reverse_proxy setup where requests pass through multiple layers.

PostgreSQL 17 Cardinality Estimation & Tuning Tips

Mon, 01 Sep 2025 00:00:00 +0000

You’ve seen it before: a query that runs in milliseconds on your staging server takes minutes to execute in production. Or a seemingly simple JOIN causes the query_planner to choose a disastrously slow Nested Loop over a much faster Hash Join. In almost every case, the root cause of these performance mysteries is not a bug in PostgreSQL, but a flaw in its understanding of your data. This is the challenge of cardinality_estimation. The planner is only as smart as the statistics it’s given, and when its row_estimation is wrong, the consequences are severe.

Using FOR UPDATE SKIP LOCKED For Queue Workflows

Fri, 29 Aug 2025 00:00:00 +0000

One of the most common and powerful patterns in modern application development is the job queue. Whether you’re sending emails, processing images, or running complex calculations, offloading tasks to background workers is essential for building responsive and scalable systems. Many developers reach for dedicated queueing software like RabbitMQ or Redis, but for many use cases, your primary PostgreSQL database already has all the tools you need to build a robust, transactional, and incredibly performant job_queue_postgres.

SSL/TLS Handshake Failures Causing NGINX 502 Errors

Wed, 27 Aug 2025 00:00:00 +0000

You’ve set up NGINX as a reverse proxy, pointing it to a secure upstream service. You refresh your browser, and instead of your application, you’re greeted by a stark “502 Bad Gateway” page. You check your upstream service; it’s running perfectly. Puzzled, you turn to the NGINX error logs and find a cryptic message that seems to complicate things even further: SSL_do_handshake() failed.

This scenario is frustratingly common. A 502 error suggests a problem with the upstream server, but in many cases, NGINX itself is failing to establish a secure connection with that server. The true culprit is an SSL/TLS handshake failure, often masked by this generic HTTP error code. Understanding the root cause is key to a quick resolution, preventing prolonged downtime and tedious debugging sessions.

How Autovacuum Causes PostgreSQL Deadlocks

Tue, 26 Aug 2025 00:00:00 +0000

You’ve meticulously optimized your application queries. Your transaction logic is sound. Yet, under heavy load, your system seizes up, logging the dreaded “deadlock detected” error. You dig into the logs, expecting to find two application transactions locked in a deadly embrace, but instead, you find a surprising culprit: one of the participants is the PostgreSQL autovacuum process. How can a routine maintenance task, designed to keep the database healthy, be the cause of a production-stopping postgres_deadlock?

Real-World PostgreSQL Deadlock Examples & Fixes

Mon, 25 Aug 2025 00:00:00 +0000

It’s 3 AM. The pager screams. Your application is throwing a cascade of errors, and users are reporting that the system is completely frozen. You dive into the logs and see the same ominous message repeating over and over: ERROR: deadlock detected. A PostgreSQL deadlock is one of the most abrupt and disruptive failures a database can experience. It’s not a performance degradation; it’s a hard stop where two or more transactions are locked in a fatal embrace, each waiting for a resource the other holds.

Redis Sentinel Failover & Split-Brain Recovery Guide

Sun, 24 Aug 2025 00:00:00 +0000

You’re on call. An alert fires—your Redis master node is unreachable. Your heart rate quickens. You’ve set up Redis Sentinel for high availability, but is it working? Did the failover succeed? Or worse, are you now in a Redis cluster split-brain situation where two nodes think they’re the master, leading to data inconsistency and eventual loss? In these critical moments, blindly trusting the automation isn’t enough; you need to verify what’s happening.

Fix NGINX 503 Errors From Misconfigured limit_req

Sat, 23 Aug 2025 00:00:00 +0000

You’ve done the responsible thing. To protect your application from abusive bots and prevent any single user from overwhelming your services, you’ve implemented rate limiting in NGINX. You add the limit_req_zone and limit_req directives, push the configuration, and watch. But instead of seeing a drop in malicious traffic, your monitoring dashboards light up with a sea of red. A massive 503 Service Unavailable spike appears, and legitimate users are complaining they can’t access your site. Your shield has become a weapon turned against yourself.

Prometheus Alertmanager: Noise Reduction Rules

Fri, 22 Aug 2025 00:00:00 +0000

It’s 3 AM, and a torrent of PagerDuty notifications floods your phone. A single network partition has triggered a cascade, making every service instance that can’t reach the database fire its own individual alert. You’re drowning in hundreds of notifications, all pointing to the same root cause. This is alert fatigue, a critical problem for any SRE or on-call engineer. When you’re constantly bombarded with low-signal noise, you risk becoming desensitized, potentially overlooking the one critical alert that signals a major incident.

Tune NGINX Timeouts: Eliminate Upstream Errors

Thu, 21 Aug 2025 00:00:00 +0000

The 504 Gateway Timeout error is a familiar and frustrating sight for anyone managing web applications. It signifies a breakdown in communication, but not between the user and your server. Instead, it means NGINX, acting as your trusty reverse proxy, gave up waiting for a response from a backend, or “upstream,” service. This could be your Node.js application, a PHP-FPM process, or a Python microservice.

While your first instinct might be to blame the application, the root cause is often a simple configuration mismatch. NGINX has built-in timers to protect itself from unresponsive backends. When a backend process takes too long—longer than NGINX is configured to wait—NGINX proactively closes the connection and serves the dreaded 504 error. Understanding and tuning directives like proxy_read_timeout and fastcgi_read_timeout is the key to resolving these issues and building a more resilient infrastructure.

Linux Load Average Spikes: IO Wait & Bottlenecks

Tue, 19 Aug 2025 00:00:00 +0000

An alert fires: “High load average on production server.” Your heart rate quickens. You SSH into the machine and run a command like top, only to be confused. The CPU usage is hovering at 10%, but the load average is sky-high. What’s going on? If the CPU isn’t busy, what is the system “loaded” with? This common scenario highlights one of the most misunderstood metrics in Linux performance troubleshooting: the system load average.

Kubernetes CrashLoopBackOff: Causes & Quick Fixes

Mon, 18 Aug 2025 00:00:00 +0000

You’ve deployed your application to Kubernetes, but a quick check on your pods reveals the dreaded CrashLoopBackOff status. Your pod is stuck in a restart loop, and your service is down. This isn’t an error itself, but a status indicating that Kubernetes is trying to start a container, it crashes, and Kubernetes waits an exponentially increasing amount of time before trying again. This back-off mechanism prevents a faulty pod from overwhelming the cluster with constant restart attempts.

Jenkins Declarative Pipeline: Parallelism Fixes

Sun, 17 Aug 2025 00:00:00 +0000

You’ve embraced the power of Jenkins Declarative Pipelines, and now you’re looking to take the next step in CI/CD optimization: running stages in parallel. The promise is alluring—slashing your build times by running tests, linting, and security scans simultaneously. You eagerly refactor your Jenkinsfile, commit the change, and watch your pipeline trigger. But instead of blazing-fast execution, your build gets stuck, silently hanging with no obvious error.

This frustrating scenario is often caused by two insidious problems: executor starvation and node label mismatch. While the parallel step in Jenkins is incredibly powerful, using it naively can lead to deadlocked pipelines and misconfigured jobs that fail in subtle ways. Understanding how Jenkins allocates resources is the key to unlocking true parallelism without the headaches. This guide will walk you through these common pitfalls and provide the best practices to avoid them.

Buffering keepalive and Hidden 500-Series Errors in NGINX

Sat, 16 Aug 2025 00:00:00 +0000

Your NGINX instance is humming along, handling traffic flawlessly. Then, a sudden traffic spike or a deployment of a new feature that handles larger files occurs, and your monitoring dashboards light up with 500-series errors. You check the NGINX error logs, but find nothing conclusive—just generic 502 Bad Gateway or 504 Gateway Timeout messages that don’t point to a root cause. These are the hidden, frustrating errors that often stem not from your application code, but from poorly tuned NGINX buffering and keepalive settings.

Fix Helm Chart Rollback & Pending-Upgrade Failures

Fri, 15 Aug 2025 00:00:00 +0000

It’s a scenario that every Kubernetes operator dreads. A production deployment has gone wrong, you confidently initiate a Helm rollback, and then… nothing. The command hangs, and a quick check reveals the dreaded pending-rollback status. Your application is now in a broken state, and you’re blocked from deploying any new fixes. This is more than a minor inconvenience; it’s a critical failure that can leave your services unstable and your deployment pipeline paralyzed.

Fix NGINX 502 Bad Gateway In Kubernetes Ingress

Wed, 13 Aug 2025 00:00:00 +0000

You’ve deployed your application to Kubernetes, and everything seems fine until a user reports the dreaded 502 Bad Gateway or 504 Gateway Timeout. In a traditional VM setup, your first step would be to SSH into the server and check the NGINX configuration and logs. But in the ephemeral, abstracted world of Kubernetes, the problem is rarely that simple. A gateway error from the NGINX Ingress Controller is often a symptom of a deeper issue within the cluster’s networking or health management systems.

Blue-Green And Canary Deployments With NGINX

Tue, 12 Aug 2025 00:00:00 +0000

The moment of truth arrives. You’ve tested the new version of your application, the container image is pushed, and the deployment pipeline is ready. You click “deploy,” and a wave of anxiety hits. Will this be a smooth, zero-downtime rollout, or will your dashboards soon light up with 502 Bad Gateway and 504 Gateway Timeout errors? For many teams using advanced deployment strategies like Blue-Green or Canary, this fear is all too real.

Docker Compose Networking: Service Discovery & Ports

Sun, 10 Aug 2025 00:00:00 +0000

You’ve meticulously crafted your docker-compose.yml file, your services build correctly, and the startup process finishes without a single error. Yet, when your application tries to connect to its database, you get a “Connection refused” or “Host not found” error. This is a classic and frustrating scenario in Docker Compose networking. Your containers are running, but they’re isolated in their own digital worlds, unable to communicate.

These issues almost always boil down to two core mysteries: service discovery failed events, where containers can’t find each other, and docker compose port conflict problems, where port mappings are misunderstood or misconfigured. To build reliable containerized applications, you must look beyond the docker-compose.yml file and understand the virtual network that Docker creates. This guide will demystify that network, providing you with the tools and techniques to troubleshoot and resolve these common connectivity challenges.

Metric Cardinality In Observability: Strategies

Sun, 10 Aug 2025 00:00:00 +0000

It’s a story familiar to any SRE or DevOps engineer. You add a seemingly innocuous label to a key metric—user_id, request_id, container_id—to gain deeper insight. Suddenly, your monitoring bill skyrockets, your Prometheus_TSDB instance starts gasping for memory, and dashboards slow to a crawl. You have just triggered a label_explosion, the single biggest challenge in modern metrics-based observability: metrics_cardinality.

High cardinality isn’t an edge case; it’s the new normal in a world of microservices, containers, and complex user interactions. As the number of unique time series grows into the millions or even billions, it places immense pressure on monitoring systems, impacting storage_costs, query performance, and the fundamental ability to scale.

Early Deadlock Detection With pg_stat_kcache & eBPF

Sat, 09 Aug 2025 00:00:00 +0000

The dreaded deadlock. For anyone managing a high-traffic PostgreSQL database, it’s a familiar nemesis. The database logs a “deadlock detected” message, a transaction is unceremoniously aborted, and your application has to handle the fallout. This reactive cycle is frustrating; by the time you’re alerted, the damage is already done. Traditional postgres_metrics and monitoring tools are excellent at telling you that a deadlock occurred, but they fall short of explaining the subtle conditions that led to it.

Save Hours on Troubleshooting with Automated Investigations

Mon, 04 Aug 2025 00:00:00 +0000

How many times has your team stared at a dashboard, pointed to a spike, and asked a question that charts alone can’t answer? “What was the real impact of that deployment?” “Why are our Kubernetes pods in the us-east-1 cluster suddenly crashing?” “Are we wasting money on overprovisioned servers?”

Answering these questions is the real work of operations and SRE. It often kicks off a time-consuming scramble, sending engineers down rabbit holes for hours, days, or even weeks. You dig through logs, correlate metrics across services, and piece together clues from Slack conversations and Jira tickets.

Netdata Now Troubleshoots Your Alerts for You

Sun, 03 Aug 2025 00:00:00 +0000

The 2 AM pager alert. For anyone in Ops, SRE, or IT administration, those words trigger a familiar sense of dread. An alert has fired. Is it a real fire, or another false alarm waking you from a dead sleep? The pressure is on. Every minute of downtime costs money and reputation, but troubleshooting a complex system when you’re sleep-deprived is a Herculean task.

This cycle is a massive drain on engineering resources. The daily grind of sifting through alerts, trying to distinguish signal from noise, and manually correlating metrics to find a root cause consumes countless hours. This constant firefighting leads to alert fatigue, where even critical notifications start to get ignored. The core questions are always the same: Is this a real problem? What is the potential impact? Why did this trigger? What do I do next? Answering them is a slow, manual, and often stressful process.

GitLab Runner Executor Failures: Docker & Kubernetes

Thu, 24 Jul 2025 00:00:00 +0000

You’ve been there. You push your code, a new CI/CD pipeline kicks off in GitLab, and you wait for that satisfying green checkmark. Instead, you get a dreaded red ‘X’. The pipeline failed. Digging into the job logs, you find a cryptic message: “ERROR: Job failed (system failure): prepare environment: exit code 1”. The culprit is often the GitLab Runner executor—the very engine responsible for running your jobs—failing in its environment.

Apache Kafka Consumer Lag: Troubleshooting & Fixes

Wed, 23 Jul 2025 00:00:00 +0000

You’ve built a powerful, real-time data pipeline with Apache Kafka, but suddenly, things grind to a halt. The dreaded consumer lag is exploding, alerts are firing, and your downstream applications are starved for data. This scenario is all too common for teams running Kafka at scale. Often, the culprit is a subtle interplay between consumer group rebalancing, partition assignment, and suboptimal consumer configurations. Understanding these mechanics is not just about fixing a problem; it’s about building resilient, high-throughput streaming systems from the ground up.

Apache mod_rewrite Debugging & Log Analysis

Wed, 23 Jul 2025 00:00:00 +0000

You’ve crafted what seems like a perfect rewrite rule, uploaded it to your .htaccess file, and refreshed your browser, only to be met with a “Too many redirects” error. You’re stuck in an Apache redirect loop, a frustrating and all-too-common problem when working with Apache mod_rewrite. This powerful module can manipulate URLs in almost any way imaginable, but its complexity means that a small mistake in a regular expression or a missing condition can lead to unexpected behavior, broken links, or infinite loops.

AWS ELB 5xx Surge Analysis: Health & Draining

Wed, 23 Jul 2025 00:00:00 +0000

The alert notification chimes, and your dashboard lights up with an ELB 5xx error rate spike. Your application, running behind an AWS Application Load Balancer (ALB), is suddenly returning 502, 503, or 504 errors. This is a high-stakes scenario where every second of downtime impacts users. The load balancer is often just the messenger; the real culprit lies somewhere in your backend infrastructure. Is a target group unhealthy? Are your connections timing out? Is a misconfigured deployment process wreaking havoc?

25 NGINX Directives to Audit Before Opening a Support Ticket

Wed, 09 Jul 2025 00:00:00 +0000

You’ve built a robust application, but it’s sluggish or throwing intermittent errors. Your first instinct might be to blame the code, but often the culprit lies in the intricate web of your NGINX configuration. Before you spend hours debugging your application or drafting a support ticket, a thorough NGINX config check can save you time and reveal simple yet impactful misconfigurations.

An NGINX configuration audit isn’t just for troubleshooting; it’s a proactive step toward better performance, tighter security, and greater stability. Many common issues related to performance bottlenecks, security vulnerabilities, or proxy errors stem from overlooked or poorly optimized directives. This checklist covers 25 essential directives you should review. Working through this list can help you pinpoint problems, implement best practices, and gain a deeper understanding of how NGINX operates.

Short vs Long Transactions In PostgreSQL Benchmarks

Wed, 09 Jul 2025 00:00:00 +0000

Every developer working with a database faces a fundamental design choice: how to structure their transactions. Should a complex business operation, involving multiple database writes, be wrapped in a single, all-or-nothing transaction? Or is it better to break it down into a series of smaller, faster commits? On the surface, the long_running_transaction seems safer, guaranteeing perfect atomicity. But as traffic grows, a mysterious performance degradation can set in. Latency spikes, throughput drops, and users start seeing errors. The culprit is often not the queries themselves, but the invisible contention they create.

Netdata vs Auvik: Real-Time Auvik Alternative

Tue, 24 Jun 2025 00:00:00 +0000

Netdata vs Auvik comparison: real-time full-stack observability with ML vs network-centric MSP SaaS with NCM.

Netdata vs Cacti: Real-Time Cacti Alternative

Tue, 24 Jun 2025 00:00:00 +0000

Netdata vs Cacti comparison — real-time full-stack network observability vs legacy SNMP graphing.

Netdata vs ManageEngine OpManager: Which Is Better?

Tue, 24 Jun 2025 00:00:00 +0000

Netdata vs ManageEngine OpManager — real-time, unified, ML-powered observability vs traditional modular NMS.

Netdata vs NeDi: Network Discovery Tool Comparison

Tue, 24 Jun 2025 00:00:00 +0000

Compare Netdata and NeDi for network monitoring, topology, flow analysis, and full-stack observability.

Network Device Auto-Discovery

Tue, 24 Jun 2025 00:00:00 +0000

Automatically discover, classify, and monitor every SNMP-reachable network device with zero manual MIB mapping.

Netdata Implements MCP Protocol

Wed, 18 Jun 2025 00:00:00 +0000

Update (February 2026): Netdata now also provides MCP via Netdata Cloud at app.netdata.cloud/api/v1/mcp for infrastructure-wide access (Business/Homelab plan). See MCP documentation for details.

We are excited to announce that Netdata has officially implemented the Model Context Protocol (MCP), joining the forefront of AI-powered infrastructure monitoring.

By enabling direct connections between AI assistants and your data sources and tools, the Model Context Protocol is a new open standard that builds a crucial link between artificial intelligence and practical systems. Instead of the AI providing generic suggestions that don’t understand your environment, MCP enables it to communicate and then interact with your infrastructure data in real time. Being among the first monitoring platforms to adopt this groundbreaking protocol, Netdata is at the forefront of intelligent observability. With the help of this integration, traditional monitoring becomes a dialogue that your entire technical team is able to participate in. We encourage you to dive deeper into the technical foundations of MCP, explore Anthropic’s comprehensive explanation, and discover how this protocol is reshaping the future of AI-data interaction.

How Can Generative AI (Gen AI) Be Used In Cybersecurity

Tue, 10 Jun 2025 00:00:00 +0000

Generative AI has exploded into the mainstream, changing how we create content, write code, and interact with technology. But beyond its creative applications, what is generative AI doing in the high-stakes world of cybersecurity? The answer is complex. GenAI is rapidly becoming one of the most powerful tools for security professionals, but it’s also a formidable weapon in the hands of their adversaries.

For security teams grappling with an ever-expanding attack surface and increasingly sophisticated threats, GenAI offers a way to automate, predict, and respond faster than ever before. However, understanding how generative AI can be used in cybersecurity requires looking at both sides of the coin—its potential to fortify our defenses and its capacity to create entirely new kinds of attacks.

How To Find And Fix Memory Leaks in C or C++

Tue, 10 Jun 2025 00:00:00 +0000

Your application feels sluggish. It runs perfectly after a restart, but over hours or days, it slows to a crawl before eventually crashing. If you’re working with C or C++, this behavior is a classic symptom of a memory leak—a silent bug that can drain system resources and destabilize your services. Because these languages put memory management directly in your hands, understanding how to find and fix memory leaks is a critical skill for building robust, long-running applications.

How To Fix Packet Loss - A Step-by-Step Guide To Reduce It

Tue, 10 Jun 2025 00:00:00 +0000

You have a fast internet connection, but your online game is lagging, your video call is choppy, or your application feels unresponsive. You run a speed test, and everything looks fine—low ping, high bandwidth. So, what’s the problem? The likely culprit is packet loss, a frustrating and often misunderstood network issue that can occur even on the best internet connections.

Unlike high latency (ping), which is a measure of delay, packet loss meaning refers to data that never arrives at its destination at all. These lost pieces of data, or “packets,” must be re-sent, creating hitches, stutters, and lag. This guide will walk you through what causes this issue and provide a clear, step-by-step process for how to fix packet loss.

How To Secure Sensitive Data In Cloud Environments

Tue, 10 Jun 2025 00:00:00 +0000

Migrating to the cloud offers unparalleled benefits in scalability, flexibility, and collaboration. However, this shift also introduces new complexities and risks, especially when it comes to protecting your most valuable asset—your data. Securing sensitive data in the cloud is not a one-time task; it’s a continuous process that requires a deep understanding of your cloud environment, robust security practices, and a proactive mindset.

Many organizations mistakenly assume their cloud service provider (CSP) handles all aspects of security. In reality, security is a shared responsibility. While the CSP is responsible for the security of the cloud (i.e., the physical data centers and underlying infrastructure), you, the customer, are responsible for security in the cloud. This includes securing your data, applications, and user access. Let’s explore the essential strategies for how to secure sensitive data in cloud environments.

Kubernetes Security Posture Management (KSPM) Explained

Tue, 10 Jun 2025 00:00:00 +0000

Kubernetes has become the de facto standard for container orchestration, offering incredible power and scalability. But this complexity introduces a massive attack surface. A single misconfiguration in a YAML file, an overly permissive role, or a vulnerable container image can create a critical security gap. Traditional security tools, designed for static infrastructure, struggle to keep up with the dynamic, ephemeral nature of Kubernetes environments.

This is where Kubernetes Security Posture Management (KSPM) comes in. It’s not just another security tool; it’s a specialized practice designed to continuously monitor, assess, and enforce security policies across your entire Kubernetes ecosystem. For any team running mission-critical workloads on Kubernetes, understanding and implementing KSPM is no longer a luxury—it’s a necessity.

OpenSearch vs Elasticsearch Which One Is Better In 2025?

Tue, 10 Jun 2025 00:00:00 +0000

If you’re building a system for log analytics, application search, or security monitoring, you’ve almost certainly faced a critical decision: OpenSearch vs Elasticsearch. For years, Elasticsearch was the undisputed king, but since 2021, its open-source fork, OpenSearch, has emerged as a powerful and popular alternative. What began as a dispute over licensing has evolved into two distinct projects with different philosophies, features, and performance characteristics.

Choosing between them is no longer a simple matter. It’s a strategic decision that impacts everything from your budget and licensing compliance to your system’s performance and future scalability. As we move through 2025, the paths of these two search engines have diverged enough that making an informed choice is more important than ever. This guide breaks down the key differences to help you decide which platform is right for your needs.

What Is A Flaky Test How To Detect Fix & Avoid Them

Tue, 10 Jun 2025 00:00:00 +0000

You push a new feature, all local tests pass, and you open a pull request. The continuous integration (CI) pipeline kicks off, but a few minutes later, you see a dreaded red ‘X’. A test failed. You scrutinize your code, find nothing wrong, and re-run the job. This time, it passes with a green checkmark. If this scenario feels familiar, you’ve encountered a flaky test.

A flaky test is a test that exhibits non-deterministic behavior—it can both pass and fail across multiple runs without any changes to the code or its environment. While it might seem like a minor annoyance, test flakiness is a significant problem that can erode your team’s confidence in your test suite, slow down development velocity, and ultimately allow real bugs to slip into production. Understanding what causes these inconsistencies is the first step toward building a more reliable and trustworthy testing process.

What Is A Memory Leak In Java How To Detect And Fix Them

Tue, 10 Jun 2025 00:00:00 +0000

Your Java application runs smoothly after a fresh deploy, but over hours or days, its performance steadily degrades. Response times creep up, garbage collection pauses become longer and more frequent, and then, the inevitable happens: the application crashes, logging a fatal OutOfMemoryError. This classic scenario is often the calling card of a subtle but dangerous problem—a memory leak.

Even though Java features automatic memory management via its garbage collector (GC), applications are not immune to leaks. A Java memory leak occurs when objects are no longer in use by the application, but the GC is unable to reclaim their memory because they are still being referenced. Over time, these orphaned objects accumulate, consuming the available heap space and leading to performance degradation and eventual failure. Understanding how to detect and fix these leaks is a critical skill for any Java developer.

What Is Distributed Tracing How It Works And Use Cases

Tue, 10 Jun 2025 00:00:00 +0000

Your team has deployed a new feature, but users are reporting that a specific action in your application is painfully slow. You look at the metrics for your frontend service, and they seem fine. You check the logs for the user authentication service, and there are no errors. The database CPU usage is normal. So where is the bottleneck? In a modern distributed architecture built with microservices, a single user request can trigger a complex chain reaction across dozens of independent services. Finding the root cause of a problem can feel like searching for a needle in a global haystack.

Agentless Network Monitoring A Complete Guide To Understand

Mon, 09 Jun 2025 00:00:00 +0000

As your infrastructure grows, so does the complexity of monitoring it. The traditional approach often involves deploying a software “agent” on every server, virtual machine, and container. While effective, this can quickly become a management nightmare. You have to install, configure, update, and secure hundreds or even thousands of agents, each consuming precious CPU and memory on the hosts you’re trying to monitor. What if there was a less intrusive way?

Garbage Collection In Java What It Is and How It Works

Mon, 09 Jun 2025 00:00:00 +0000

One of the most powerful features of the Java platform is its automatic memory management. Unlike languages like C or C++, where developers must manually allocate and deallocate memory, Java handles this process for you through a process called garbage collection (GC). This frees developers to focus on application logic rather than the complexities of memory management, which is a major reason for Java’s enduring popularity.

But what exactly is garbage collection in Java, and how does it work under the hood? While it’s an automatic process, a solid understanding of the Java garbage collector is crucial for writing high-performance, stable applications and for troubleshooting memory-related issues like the dreaded OutOfMemoryError.

How To View Docker Container Logs A Step-by-Step Guide

Mon, 09 Jun 2025 00:00:00 +0000

Your containerized application is misbehaving. The service is unresponsive, or worse, it’s crash-looping. As a developer or SRE, your first instinct is to ask, “What do the logs say?” For applications running in Docker, accessing and understanding container logs is the most fundamental troubleshooting skill you can possess. These logs are the raw, unfiltered story of what your application is doing, thinking, and feeling.

But docker logging is more than just a single action. It’s a comprehensive system with layers of functionality, from quickly tailing real-time output to implementing robust, production-grade log management strategies. This guide will walk you through everything you need to know to effectively view docker container logs, starting with the basics and progressing to the best practices that will keep your applications observable and your on-call nights quiet.

Logging As A Service (LaaS): Simplify Log Management

Mon, 09 Jun 2025 00:00:00 +0000

In the world of modern software development, logs are the lifeblood of observability. They are the detailed, chronological record of every event, error, and transaction that occurs within your applications and infrastructure. When something goes wrong, logs are the first place developers and SREs turn to for answers. But with the rise of microservices, containers, and distributed cloud architectures, the sheer volume and complexity of log data have exploded.

The old way of managing logs—SSHing into a server and using grep to search through a text file—simply doesn’t scale. This approach becomes an impossible, time-consuming, and error-prone task when you have hundreds of services spread across dozens of servers. This is the problem that Logging as a Service (LaaS) was created to solve.

What Is AIOps (Artificial Intelligence For IT Operations)

Mon, 09 Jun 2025 00:00:00 +0000

Your team is drowning in data. Your monitoring systems generate thousands of alerts every day, most of which are just noise. When a real issue strikes your complex, distributed application, engineers scramble, jumping between a dozen different dashboards to piece together clues. This frantic, reactive firefighting is the reality for many IT Operations, DevOps, and SRE teams. It’s stressful, inefficient, and unsustainable.

This is where AIOps (Artificial Intelligence for IT Operations) enters the picture. It’s not just another industry buzzword; it’s a fundamental shift in how we manage technology. AIOps is the practice of applying artificial intelligence—specifically machine learning and big data analytics—to automate and streamline every aspect of IT operations. By intelligently processing the massive volumes of data your systems generate, AIOps platforms help you move from a reactive state of constant crisis to a proactive, predictive model of management.

What Is Container Orchestration & Why Do We Need It

Mon, 09 Jun 2025 00:00:00 +0000

If you’re building modern applications, you’re likely using containers. They’re lightweight, portable, and provide a consistent environment for your code. Deploying a single container is simple. Managing a handful is manageable. But what happens when your application grows from a few containers to hundreds or even thousands, all working together as a complex system of microservices?

This is where manual management breaks down. You face a storm of questions: How do you deploy updates without downtime? How do you scale to handle a sudden traffic surge? What happens if a server fails in the middle of the night? Answering these questions manually is an operational nightmare. This is precisely the problem that container orchestration was created to solve.

What Is OpenTelemetry How It Works Benefits and Use Cases

Mon, 09 Jun 2025 00:00:00 +0000

If you manage modern applications, you’re dealing with complexity. Your system isn’t a single program running on one server; it’s a distributed web of microservices, serverless functions, and third-party APIs running across a hybrid cloud infrastructure. Understanding what’s happening inside this complex system is a monumental challenge. Each component and vendor has its own proprietary way of exporting performance data, leaving you to juggle dozens of agents and formats, leading to data silos and vendor lock-in.

What Is Remote Infrastructure Management (RIM)

Mon, 09 Jun 2025 00:00:00 +0000

The days of managing servers neatly racked in a single on-premise data center are over for most businesses. Today’s IT landscape is a sprawling mix of public cloud instances, private cloud infrastructure, edge devices, and SaaS applications. This distribution is powerful, but it creates a significant management challenge: How do you maintain control, ensure uptime, and secure an infrastructure that’s physically everywhere and nowhere at once?

This is the problem that Remote Infrastructure Management (RIM) solves. It’s a strategic approach that moves IT operations from a hands-on, on-site model to a centralized, remote-first paradigm. For DevOps and SRE teams, understanding RIM is no longer optional; it’s fundamental to building and maintaining resilient, scalable, and efficient systems in the modern era.

Whats The Difference Between PostgreSQL vs MySQL

Mon, 09 Jun 2025 00:00:00 +0000

When building an application, one of the most fundamental decisions you’ll make is choosing the right database. For many developers, this choice boils down to two of the most popular open-source relational databases in the world: PostgreSQL and MySQL. Both are powerful, reliable, and backed by strong communities, but they operate on different philosophies and excel in different areas.

Making the right choice between PostgreSQL vs MySQL isn’t about finding a single “best” database, but understanding which one aligns with your application’s specific needs. This guide breaks down the critical differences in their architecture, features, performance characteristics, and ideal use cases to help you make an informed decision for your next project.

Introducing Netdata Insights

Tue, 27 May 2025 00:00:00 +0000

We’ve been thinking a lot about synthesis lately.

Netdata already samples every metric every second at the edge. Engineers told us the remaining pain point was synthesis, the ability to pull hours or days or months of high‑resolution time‑series into a concise explanation they could hand to a teammate (or use themselves to debug faster).

You know the pattern. An incident happens, and suddenly you’re context-switching between dozens of dashboards, trying to reconstruct a timeline. Or you need to write a capacity planning report, and you’re copy-pasting screenshots into slides, manually correlating trends across different retention windows. The raw data is there, but the synthesis step (the part where you turn metrics into narrative) doesn’t scale.

What Is Canary Deployment? Benefits, Metrics & Setup

Tue, 27 May 2025 00:00:00 +0000

Releasing new software versions can be a nerve-wracking experience. Even with rigorous testing, the real world of production traffic often uncovers unforeseen issues. A problematic deployment can lead to downtime, frustrated users, and a frantic scramble to roll back. This is where a canary deployment strategy shines, offering a more cautious and controlled approach to rolling out updates.

Instead of a big-bang release, a canary release exposes the new version to a small subset of users first, allowing you to monitor its performance and gather feedback before a full-scale rollout. This technique significantly de-risks the deployment process, especially in complex environments like Kubernetes.

What Is Structured Logging And How To Utilize It Effectively

Tue, 27 May 2025 00:00:00 +0000

Dealing with gigabytes of plain text log files can feel like searching for a needle in a haystack, especially when critical systems are down. Unstructured logs make it incredibly difficult to quickly pinpoint issues, correlate events, or gain meaningful insights from the vast amount of data your applications generate. If you’ve ever struggled to filter logs by a specific customer ID or trace a transaction across multiple services, you understand the pain. This is precisely where structured logging steps in, transforming your log data from a chaotic stream of text into a valuable, queryable asset.

Nodejs Memory Leak How To Identify Debug And Avoid Them

Mon, 26 May 2025 00:00:00 +0000

In the fast-paced world of Node.js development, performance and reliability are non-negotiable. However, a silent saboteur often lurks in the shadows – the Node.js memory leak. These insidious issues can gradually degrade your application’s performance, leading to slowdowns, crashes, and frustrated users. Understanding how to effectively identify, debug, and prevent memory leaks is a critical skill for any developer, DevOps engineer, or SRE working with Node.js. This guide will walk you through the intricacies of Node.js memory management and equip you with the knowledge to tackle these challenging problems.

What Is A Bare Metal Server? Benefits & Use Cases

Mon, 26 May 2025 00:00:00 +0000

In a world increasingly dominated by virtualized cloud environments, the concept of a bare metal server might seem like a throwback. However, these physical powerhouses continue to play a critical role in modern IT infrastructure, offering distinct advantages for specific workloads. Understanding what is a bare metal server and why you might choose one over virtualized alternatives is key to designing efficient and high-performing systems.

What Is A Bare Metal Server? Definition & How It Works

At its core, a bare metal server is a physical computer server dedicated entirely to a single customer or tenant. Unlike virtual machines (VMs) that share the resources of a physical host through a hypervisor layer, a bare metal server gives the user direct, unmediated access to the underlying hardware – the “bare metal.” This means you have exclusive use of the server’s CPU, RAM, storage, and network interface cards.

Application Monitoring Best Practices In 2025

Sun, 25 May 2025 00:00:00 +0000

In today’s digitally driven world, application performance is not just a technical concern; it’s a critical business imperative. Users expect applications to be fast, reliable, and seamlessly available. Any deviation can lead to frustrated users, lost revenue, and damaged reputation. This makes application monitoring a cornerstone of successful software development and operations. It’s the continuous process of collecting, analyzing, and acting on data to ensure your applications meet performance, availability, and user experience expectations.

The Three Pillars Of Observability Logs Metrics And Traces

Sun, 25 May 2025 00:00:00 +0000

In the realm of modern, complex IT systems, especially those involving microservices, cloud-native architectures, and distributed environments, understanding what’s happening internally is paramount. This is where observability comes into play. Observability isn’t just about monitoring; it’s the ability to infer the internal state and health of a system by examining its outputs. To achieve this, we rely on what are commonly known as the three pillars of observability: logs, metrics, and traces.

What Is Event Correlation Benefits Use Cases And Techniques

Sun, 25 May 2025 00:00:00 +0000

In today’s complex and dynamic IT environments, organizations are inundated with a massive volume of events generated by countless sources – applications, servers, network devices, security systems, and more. This flood of data, while rich in potential insights, can easily become overwhelming. The core challenge lies in sifting through this “sea of data” to identify events that truly matter and understand their relationships. This is precisely where event correlation becomes indispensable. It’s the process of sensing and analyzing relationships between disparate events to uncover meaningful patterns, diagnose root causes, and enable proactive responses.

What Is Synthetic Transaction Monitoring

Sun, 25 May 2025 00:00:00 +0000

In today’s digital-first world, the performance and availability of web applications are paramount to business success. Users expect seamless, fast, and reliable interactions, and any hiccup can lead to frustration, lost revenue, and a tarnished brand reputation. This is where synthetic transaction monitoring (STM) emerges as a critical proactive strategy. Unlike monitoring that relies on actual user traffic, STM uses simulated user journeys to continuously test and validate application functionality, performance, and availability, even before real users encounter issues.

What Is Blue-Green Deployment? Benefits & Process

Sat, 24 May 2025 00:00:00 +0000

Software teams today face the challenge of releasing updates quickly without disrupting users. Traditional deployment methods often cause downtime and raise the risk of bugs reaching production. Blue green deployment offers a solution: by running two identical environments and switching traffic seamlessly, teams can deliver new versions with minimal risk and zero downtime. In this guide, we’ll explain what blue green deployment is, how it works, its benefits and challenges, and the best practices for implementing it.

LLM Observability and Monitoring A Comprehensive Guide

Wed, 21 May 2025 00:00:00 +0000

The rapid proliferation of Large Language Models (LLMs) like GPT and LLaMA is transforming how businesses operate, from enhancing office productivity with tools like Microsoft Copilot to innovative fraud management at Stripe. However, deploying and managing these powerful LLM applications in production environments presents unique hurdles. The sheer size of these models, their complex architectures, and often non-deterministic outputs make them challenging to manage. If you’re a developer, DevOps engineer, or SRE, you understand that ensuring consistent performance, security, and accuracy in LLM-driven systems requires a new level of insight. This is where LLM observability and LLM monitoring become indispensable.

Understanding Digital Experience Monitoring DEM

Sun, 18 May 2025 00:00:00 +0000

In today’s digital-first world, the way users interact with your online services defines their perception of your brand. A slow-loading webpage, a confusing mobile app, or an unavailable critical feature can quickly lead to frustration and churn. This is where Digital Experience Monitoring (DEM) becomes indispensable, offering a critical lens into how users truly perceive and interact with your digital platforms. Understanding and optimizing these interactions is no longer a luxury but a necessity for businesses aiming to thrive.

Understanding Error Budgets And Their Importance In SRE

Sun, 18 May 2025 00:00:00 +0000

In the quest for flawless digital experiences, the reality is that 100% uptime is an elusive, if not impossible, goal. Systems inevitably encounter issues, and services can experience disruptions. This is where the concept of an error budget becomes a cornerstone for modern Site Reliability Engineering (SRE) and DevOps practices. Understanding and effectively managing your error budget can mean the difference between fostering innovation and constantly firefighting.

So, what is an error budget? Simply put, it’s the quantifiable amount of unreliability or downtime that a service can tolerate over a specific period without breaching its Service Level Objectives (SLOs) or upsetting users. It’s the acknowledged margin for error, a critical component in balancing the drive for new features with the imperative of maintaining a stable and reliable service. For SRE teams, the sre error budget is not just a metric- it’s a vital tool for decision-making.

A Comprehensive Guide To Database Performance Optimization

Sat, 17 May 2025 00:00:00 +0000

Sluggish database performance can be a silent killer for applications, leading to frustrated users, missed opportunities, and a direct hit to your bottom line. In today’s data-intensive environments, ensuring your database operates at peak efficiency isn’t just a technical task- it’s a critical business imperative. If you’re grappling with slow queries, high resource consumption, or concerns about data scalability, this guide will walk you through essential database optimization strategies.

TL;DR Summary

Database performance issues usually come down to a few repeat offenders: slow or poorly designed queries, CPU pressure, disk I/O bottlenecks, weak indexing, and locking or concurrency contention.
To manage and improve performance, track core metrics like response time, throughput, scalability, and resource utilization (CPU, memory, disk I/O), then optimize iteratively rather than treating tuning as a one-time fix.
The biggest wins typically come from query tuning (EXPLAIN/EXPLAIN ANALYZE, rewriting queries, caching) and smart indexing (right index types, best-practice column selection, and regular maintenance).
For growth and resilience, use techniques like partitioning, sharding, connection pooling, read replicas, HA setups, and archiving via history tables, backed by continuous monitoring (for example, real-time per-second visibility and alerts) to catch issues before users feel them.

Understanding The Roots Of Database Performance Issues

Before diving into solutions, it’s crucial to understand what database performance truly means and what typically causes it to degrade. Performance refers to the speed and efficiency with which your database handles queries, transactions, and data retrieval. When performance suffers, it’s often due to one or more common culprits.

OpenShift vs Kubernetes What Are The Differences

Fri, 16 May 2025 00:00:00 +0000

Container orchestration is a cornerstone of modern cloud-native application development, and when it comes to managing containers at scale, two names often dominate the conversation- OpenShift vs Kubernetes. While both platforms are designed to automate the deployment, scaling, and management of containerized applications, they have distinct characteristics and cater to slightly different needs. Understanding the difference between OpenShift and Kubernetes is crucial for organizations making strategic decisions about their containerization strategy.

SOC 2 Type 1: Committed To Security & Trust

Fri, 16 May 2025 00:00:00 +0000

We are pleased to announce that Netdata has successfully achieved SOC 2 Type 1 attestation!

Following an independent examination performed by AssuranceLab CPAs LLC, the report confirms that—as of April 25, 2025—the design of Netdata’s controls meets the Security, Availability, and Confidentiality Trust Services Criteria defined by the AICPA.

At Netdata, the security and integrity of the monitoring data our users entrust to us are paramount. This significant milestone, validated through a rigorous, independent third-party audit conducted by AssuranceLab, formally attests to the robustness of our security controls and practices as designed and implemented at a specific point in time.

What Is Apache Kafka Used For Everything You Need To Know

Fri, 16 May 2025 00:00:00 +0000

In today’s data-driven world, applications generate vast streams of information that need to be processed and acted upon in real time. Managing these high-velocity, high-volume data flows presents a significant challenge for developers, DevOps engineers, and Site Reliability Engineers (SREs). This is precisely where Apache Kafka shines, offering a powerful platform for building robust, scalable, and real-time data pipelines. Understanding what Apache Kafka is used for and how its underlying Kafka technology works is crucial for anyone looking to harness the power of event streaming.

What Is Incident Management Benefits Process Best Practices

Wed, 07 May 2025 00:00:00 +0000

When your critical services face unexpected disruptions, the clock starts ticking. For developers, DevOps engineers, and Site Reliability Engineers (SREs), understanding what is incident management is paramount. A slow or disorganized response not only impacts users but can also strain resources and damage your organization’s reputation. Effectively managing these events is key to maintaining system stability and ensuring business continuity.

Incident management is the set of actions an organization takes to identify, analyze, correct, and prevent future occurrences of service disruptions or losses in operations. An “incident,” in ITIL terms, is any event that disrupts, or could disrupt, a service. This could range from a complete application outage to a web server running slowly, impacting productivity and posing a risk of total failure. The primary goal of IT incident management is to restore normal service operation as quickly as possible and minimize the adverse impact on business operations.

What Is Database Concurrency? Problems & Control Techniques

Sun, 04 May 2025 00:00:00 +0000

Imagine trying to book the very last seat on a popular flight online. At the exact same moment, another person clicks “confirm” for the same seat. How does the system ensure only one booking goes through and the database remains accurate? This scenario highlights the core challenge of database concurrency.

In today’s world, almost every application interacts with databases accessed by multiple users or processes simultaneously. Database concurrency is the ability of a Database Management System (DBMS) to handle these simultaneous operations efficiently while maintaining data integrity and consistency. For developers, DevOps engineers, and SREs, understanding concurrency is vital for building reliable and performant applications. Let’s explore what concurrency entails, the problems it can cause if unmanaged, and the techniques used to control it.

Normalized vs Denormalized - Choosing The Right Data Model

Sat, 03 May 2025 00:00:00 +0000

When designing databases, one of the fundamental decisions you’ll face is how to structure your data. Two primary approaches dominate this discussion: normalization and denormalization. Choosing between a normalized vs denormalized model significantly impacts data integrity, storage efficiency, and query performance. Understanding this trade-off is crucial for developers, database administrators, and SREs responsible for building and maintaining reliable, efficient systems.

Getting the data model right from the start can save considerable headaches down the line. Let’s explore what normalized data and denormalized data mean, their respective strengths and weaknesses, and how to decide which strategy best fits your specific needs.

Cloud Managed Services: Definition, Types & Benefits

Fri, 02 May 2025 00:00:00 +0000

Moving to the cloud offers incredible benefits like scalability and flexibility, but managing cloud infrastructure isn’t always simple. Configuring networks, ensuring security, optimizing costs, performing maintenance, and staying compliant can quickly become overwhelming, especially for growing teams or those new to the cloud landscape. This is where cloud managed services come into play.

Understanding managed cloud services is essential for developers, DevOps engineers, and SREs. It represents a strategic approach to handling cloud complexity, allowing technical teams to offload operational burdens and focus on innovation and core business objectives. Let’s explore what these services entail, how they work, and their potential advantages and disadvantages.

How To Check Your Firewall Logs On Windows

Thu, 01 May 2025 00:00:00 +0000

Every Windows system comes equipped with a built-in firewall, a critical component of its security posture. The Microsoft Defender Firewall (previously Windows Firewall) acts as a gatekeeper, controlling incoming and outgoing network traffic based on predefined rules. While it diligently protects your system, its default configuration doesn’t tell you much about the traffic it’s allowing or blocking.

This is where Windows Firewall logs come in. These logs record detailed information about the firewall’s activity, providing invaluable insights for troubleshooting network connectivity problems, identifying potential security threats, and ensuring compliance. For developers, DevOps engineers, and SREs, knowing how to check firewall logs is a fundamental skill for maintaining secure and reliable Windows environments, whether on workstations or Windows Server instances. This guide will walk you through enabling, locating, interpreting, and managing these essential logs.

Cloud Workload - Definition, Types & Challenges

Wed, 30 Apr 2025 00:00:00 +0000

Cloud computing has transformed how businesses operate, offering unprecedented scalability, flexibility, and access to powerful resources. As organizations increasingly migrate applications and services to the cloud, you’ll frequently encounter the term “cloud workload.” But what is a cloud workload exactly? Understanding this concept is crucial, especially for DevOps engineers, SREs, and developers tasked with deploying, managing, and optimizing applications in cloud environments.

A workload in cloud computing is essentially the specific amount of processing or computing task assigned to or running on cloud resources at any given time. It’s the fundamental unit of work – an application, a service, a set of processes – that consumes cloud resources like compute, storage, and networking. Misunderstanding or mismanaging these workloads can lead to performance issues, security vulnerabilities, and runaway costs. This guide dives into the cloud workload definition, explores the various types of workloads, and highlights the common challenges associated with their management.

Industrial Remote Monitoring: Process & Examples

Wed, 30 Apr 2025 00:00:00 +0000

Imagine needing to know the exact operating temperature of a critical pump in a remote processing plant, or wanting to predict if a crucial piece of machinery on your factory floor is about to fail – all without sending a technician on-site. This is the power of industrial remote monitoring. In today’s competitive landscape, industries from manufacturing to energy to logistics are under constant pressure to improve efficiency, reduce costs, and ensure safety. Remote machine monitoring provides the critical visibility and data needed to achieve these goals.

Ecommerce Infrastructure: Components & Benefits

Tue, 29 Apr 2025 00:00:00 +0000

The world of online shopping has exploded, offering convenience and accessibility like never before. Businesses can reach global audiences, and consumers can browse and buy from anywhere. But behind every successful online store, from small boutiques to giants like Amazon, lies a complex system working tirelessly: the ecommerce infrastructure. Without this foundation, websites crash during peak traffic, customer data gets compromised, and orders get lost – leading to frustrated customers and lost revenue.

Database Backup - Types, Process and Benefits

Mon, 28 Apr 2025 00:00:00 +0000

Data is often called the lifeblood of modern organizations. From customer details and financial records to application configurations and operational logs, databases store the critical information that powers business operations. But what happens if that data is lost due to hardware failure, accidental deletion, software corruption, or a cyberattack? Without a safety net, the consequences can be catastrophic, leading to costly downtime, reputational damage, and potentially irreparable business harm. This is where database backup becomes indispensable.

Server Security - What It Is & Why It Is So Important

Fri, 25 Apr 2025 00:00:00 +0000

Servers are the workhorses of the modern digital world. They store critical data, run essential applications, host websites, and power the services we rely on every day. But with great power comes great responsibility – specifically, the responsibility of server security. An unsecured server is like leaving your front door wide open, inviting potential threats that can lead to data breaches, service disruptions, and significant financial or reputational damage.

For developers, DevOps engineers, and Site Reliability Engineers (SREs), understanding and implementing robust server protection is not just an IT task; it’s fundamental to building reliable, trustworthy, and resilient systems. Whether you’re managing a single web server or a complex distributed infrastructure, knowing how to secure a server is paramount. This guide will walk you through what server security entails, why it’s critically important, and the essential server security best practices you need to implement.

What Is Web Server Capacity Planning & How Does It Work?

Thu, 24 Apr 2025 00:00:00 +0000

Imagine launching a major marketing campaign or experiencing peak holiday traffic, only to have your website slow to a crawl or crash entirely. Users encounter frustrating error messages, abandon their carts, and your business suffers. This scenario often happens when web servers are at capacity, unable to handle the incoming load. The solution? Proactive web server capacity planning.

For DevOps engineers, SREs, and system administrators, capacity planning isn’t just a “nice-to-have”; it’s a fundamental practice for ensuring the reliability, performance, and availability of web services. It’s about understanding your current server capacity, anticipating future needs, and ensuring you have the right resources in place before demand overwhelms your infrastructure. This guide explores what web server capacity planning involves, why it’s critical, and the process for implementing it effectively.

Container vs VM - Which Is Better Option For You

Tue, 22 Apr 2025 00:00:00 +0000

Deploying applications efficiently and reliably often requires isolating them from the underlying infrastructure and from each other. Virtualization makes this possible by creating virtual representations of computing resources. Two leading technologies dominate this space: containers and virtual machines (VMs). While both offer isolation and deployment benefits, they operate fundamentally differently. Choosing between a container vs VM approach is a critical decision that impacts performance, resource usage, security, and deployment speed.

Infrastructure Monitoring vs Application Monitoring

Tue, 22 Apr 2025 00:00:00 +0000

Navigating the divide between infrastructure monitoring vs application monitoring requires a strategic approach. Infrastructure monitoring assesses system health, such as network bandwidth adequacy. Is your network performing optimally? On the other hand, application monitoring focuses on software performance issues. What causes slow response times?

How can your organization merge these monitoring capabilities to proactively identify and address potential issues?

Let’s find out.

What Is Infrastructure Performance Monitoring (IPM)?

Infrastructure Performance Monitoring (IPM) is a critical facet of IT operations. It focuses on the continuous oversight of essential system components such as hardware, cloud networks, and overall system performance. IPM involves real-time data analysis to ensure system health and efficiency, with a focus on identifying and pinpointing issues that can affect system performance and stability.

What Is Real-Time Monitoring? 5 Benefits & How It Works

Mon, 21 Apr 2025 00:00:00 +0000

Real-time monitoring is an essential tool for any business aiming to safeguard its digital operations and boost network performance. If your goal is to maintain smooth operations and robust security, you shouldn’t overlook real-time data monitoring, as it is crucial for detecting and addressing issues instantly, keeping your systems efficient and protected.

In this article, we will explore how real-time monitoring works and its key benefits. Understanding these elements can significantly enhance your approach to network management and operational efficiency.

What Is Cloud Workload Protection

Sun, 20 Apr 2025 00:00:00 +0000

The shift to cloud computing offers incredible advantages in scalability, agility, and innovation. Businesses leverage cloud platforms like AWS, Azure, and GCP to build and deploy applications faster than ever before. However, this migration introduces new complexities, particularly around securing the actual applications and processes running in these dynamic environments – the cloud workloads.

Traditional security approaches focused on protecting the network perimeter are no longer sufficient. Cloud workload protection (CWP) has emerged as a critical security strategy designed specifically for the unique nature of cloud and hybrid environments. It focuses on securing the workload itself, regardless of where it runs. Understanding what CWP is, why it’s essential, and the role of a Cloud Workload Protection Platform (CWPP) is vital for anyone responsible for cloud workload security. This guide explores the core concepts, benefits, and challenges of CWP.

What Is Network Congestion & How To Fix It

Sun, 20 Apr 2025 00:00:00 +0000

Site congestion found. Let’s fix this before it gets worse! Network congestion can cripple your digital operations, slowing down processes and frustrating users. What causes congestion in your network? How does it affect your system, and what can you do to resolve – or prevent – these traffic jams? Keep your network up and running. Here is how!

What Is Network Congestion?

Network congestion is like a traffic jam on your data highway; it occurs when there’s more data trying to travel across a network than the available bandwidth can handle. This overload leads to annoying performance issues such as increased latency, jitter, packet loss, and reduced throughput.

What Are Windows Event Logs? The Ultimate Guide

Fri, 18 Apr 2025 00:00:00 +0000

What could the dream of every IT professional be? Systems running flawlessly, no question about that. In reality, though, crashes, errors, and performance issues are inevitable. When problems arise, Windows event logs provide you with a detailed record of what happened, helping you diagnose and resolve issues efficiently.

Windows powers about 71.9% of desktops worldwide, making it the most widely used operating system. Moreover, Windows servers generate a massive amount of event logs. Some can log up to 500 events per second. With this kind of data flow, managing and analyzing logs efficiently is crucial to keeping your systems running smoothly and securely.

All Types Of Databases: Advantages & Examples

Thu, 17 Apr 2025 00:00:00 +0000

This is a tech-savvy exploration of database types! From the structured precision of relational models to the dynamic versatility of NoSQL, we’re diving deep into each DB type, unpacking their perks, and dishing out some examples to help you pick the perfect fit for your digital toolbox.

What Is A Database?

A database is a structured collection of stored data, that allows easy access, management, and updating. They come in various types to suit different needs; from hierarchical databases that organize data in a tree-like format to relational databases that use tables and relationships among those data. Databases are essential to build any tech product from small mobile apps to large-scale enterprise systems. They help you handle, search, and efficiently utilize large data sets.

FreeBSD vs Linux - Which Is Better?

Tue, 15 Apr 2025 00:00:00 +0000

When choosing an operating system for servers, embedded systems, or even desktops, developers and system administrators often encounter two powerful, free, and open-source Unix-like options: FreeBSD and Linux. Both share a common heritage tracing back to the original UNIX, but they have evolved along different paths, resulting in distinct philosophies, architectures, and strengths. The FreeBSD vs Linux debate isn’t about one being definitively “better” but rather understanding which is better suited for your specific needs.

Real-Time Data Visualization: Examples & Use Cases

Tue, 15 Apr 2025 00:00:00 +0000

Data is increasingly generated at an unprecedented velocity. From application logs and system metrics to user interactions and IoT sensor readings, the sheer volume can be overwhelming. Simply collecting this data isn’t enough; the real value lies in understanding it as it happens. This is where real-time data visualization comes into play. It transforms relentless streams of raw data into clear, intuitive visuals, allowing you to grasp trends, spot anomalies, and make informed decisions instantly.

What Is Network Security Monitoring - A Comprehensive Guide

Mon, 14 Apr 2025 00:00:00 +0000

Imagine attackers silently infiltrating your network, hiding like the Greeks inside the Trojan Horse. They could remain undetected for weeks or even months, mapping your systems, stealing sensitive data, and preparing for a larger attack. By the time you realize they’re inside, significant damage might already be done. This scenario highlights a critical challenge in modern cybersecurity: perimeter defenses like firewalls are essential, but they aren’t foolproof. You need visibility inside your network to detect threats that slip through.

Deployment Automation: Tools, Benefits & Practices

Wed, 02 Apr 2025 00:00:00 +0000

Getting new features and bug fixes from a developer’s machine into the hands of users quickly and reliably is paramount when it comes to successful software development. Manual deployment processes, however, are often slow, error-prone, and stressful.

Manual tasks in application deployments frequently lead to configuration errors and make the software deployment process a time consuming process, especially for complex deployments. This is where deployment automation comes in – a crucial practice in modern DevOps and agile methodologies.

Monitoring Netdata Restarts: A Reliable Solution

Thu, 06 Mar 2025 00:00:00 +0000

For a tool like Netdata, monitoring crashes and abnormal events extends far beyond bug fixing—it’s essential for identifying edge cases, preventing regressions, and delivering the most dependable observability experience possible. With millions of daily downloads, each event provides a vital signal for maintaining the integrity of our systems.

The Challenge with Traditional Solutions

Over the years, we’ve evaluated many monitoring tools, each with significant limitations:

Tool	Strengths	Limitations
Sentry	• Comprehensive error tracking features • Detailed stack traces	• Per-event pricing model becomes prohibitive at scale • Forces sampling which reduces visibility into critical issues • Compromises complete error capture for cost control
GCP BigQuery & Similar	• Powerful query capabilities • Flexible data processing • High scalability potential	• Complex reporting setup and maintenance • Significant costs at high event volumes • Requires specialized technical expertise
Other Solutions	• Various specialized features • Some open-source flexibility	• Either too inflexible for custom requirements • Or prohibitively expensive at full-capture scale • Often require compromising between detail and cost

We consistently encountered these core challenges:

What Is Database Clustering? Types & Benefits

Thu, 06 Mar 2025 00:00:00 +0000

Database clustering is a robust strategy employed to enhance the performance, scalability, and availability of databases by orchestrating the distribution of data across multiple servers. This setup doesn’t just streamline handling big data; it also keeps things running smoothly, even if some nodes go down. That way, your service stays up and available no matter what happens.

By diving into the various types of database clustering architectures, in this article we explain how each setup addresses specific needs and challenges within IT environments. We’ll also examine the tangible benefits that database clustering brings to businesses, from improved data redundancy to enhanced query response times.

Docker Monitoring Tool With Unlimited Containers

Mon, 27 Jan 2025 00:00:00 +0000

Netdata brings the simplicity Docker promised to monitoring. One command install, instant visibility into every container’s CPU, memory, disk, and network. No agents per container, no complex pipelines, no surprise bills.

Infrastructure Monitoring For Network Engineers

Mon, 27 Jan 2025 00:00:00 +0000

Netdata gives network engineers the full picture: live topology maps, NetFlow/sFlow/IPFIX flow analysis, SNMP monitoring with 100+ device profiles, a native SNMP trap receiver, per-second interface metrics, and ML that learns your traffic patterns. Deploy in minutes and stop hearing about outages from users.

Netdata vs Prometheus: A 2025 Performance Analysis

Thu, 23 Jan 2025 00:00:00 +0000

When it comes to infrastructure monitoring, performance, scalability, and efficiency are critical considerations. In this blog post, we revisit two widely adopted open-source monitoring solutions: Netdata and Prometheus. Both tools have introduced notable improvements in their latest versions, emphasizing scalability and enhanced efficiency.

In our previous analysis, we explored key differences between these systems, focusing on resource consumption and data retention. This follow-up expands on that comparison by subjecting both tools to a significantly larger workload. With the number of monitored nodes increased to 1000, containers to 80k, and metrics ingestion reaching 4.6 million metrics per second, we examine how each system performs under these demanding conditions, focusing on CPU utilization, memory requirements, disk I/O, network usage, and data retention during data ingestion.

Long-Term Data Storage and Retention in Netdata

Tue, 21 Jan 2025 00:00:00 +0000

Netdata’s database engine (dbengine) provides a sophisticated multi-tiered storage system designed for efficient long-term data retention while maintaining high granularity. This article explores the technical details of how Netdata handles metric storage, the advantages of its distributed architecture, and how to configure it for your specific needs.

Database Engine Architecture

Netdata’s database engine provides a sophisticated, efficient solution for long-term metric storage through:

Intelligent multi-tiered storage architecture
Efficient compression and caching mechanisms
Flexible retention strategies
Distributed deployment options

This design enables organizations to maintain detailed historical data while optimizing storage use and maintaining query performance.

Network Monitoring Software With Real-Time Visibility

Wed, 15 Jan 2025 00:00:00 +0000

Real-time network monitoring with per-second visibility, ML-powered anomaly detection, and zero-configuration deployment.

Best Of Category Badges Earned In 2024: G2 & Capterra

Mon, 23 Dec 2024 00:00:00 +0000

Netdata Featured with Multiple “Best Of” Category Badges in 2024

As we are close to the end of this year, we are thrilled to announce that Netdata has been recognized with multiple “Best of” badges from Gartner Digital Markets brands: Capterra, Software Advice, and GetApp, leading software recommendation search engines.

This “Best of” badges program is an independent assessment that evaluates user reviews to help buyers identify the highest-rated software companies in specific categories that offer the most popular solutions.

Getting Started With Netdata: Real-Time Monitoring

Thu, 19 Dec 2024 00:00:00 +0000

Now you can start monitoring thousands of metrics in real-time, detecting anomalies throughout your infra, and troubleshooting issues even mid-crisis! Watch this 1-minute video for a quick intro! Get ready to be blown away!

To fully utilize Netdata for monitoring your infrastructure, do these:

① Deploy Netdata

Install Netdata to your systems.

Netdata runs on Linux, Windows, FreeBSD and MacOS and can be installed on bare-metal servers, cloud VMs, Kubernetes, even weak IoT devices. We have carefully optimized it to be the fastest and most advanced monitoring you will ever need, and at the same time be extremely friendly, polite and respectful to your production systems and applications.

Real-Time Windows Server Monitoring Best Practices

Tue, 17 Dec 2024 00:00:00 +0000

If you are looking to monitor your Windows Server Machines, this webinar will guide you through the latest strategies for real-time observability, system and infrastructure optimization. Featuring a hands-on live demo and insights from industry experts, this session will be full of actionable techniques to help you gain deeper visibility into your Windows infrastructure, troubleshoot faster, and improve performance with ease.

Customize Your Netdata Experience with Favorites

Mon, 09 Dec 2024 00:00:00 +0000

Make Netdata yours with Favorites

Monitoring is personal. Different systems, workloads, and use-cases mean different charts matter to different users. Netdata now makes it easier than ever to customize your experience by letting you favorite charts you care about most.

How it works

You’ll now see a heart icon next to any chart or section of charts. Click it, and it instantly gets added to your favorites. favorites appear right above your system overview metrics, so you see what matters most—first.

20 Best DevOps, SRE & Observability Conferences 2025

Wed, 04 Dec 2024 00:00:00 +0000

Every year, DevOps, SRE Sysadmin & IT leaders gather at conferences across the world to share not only knowledge but also the latest trends, tools, and strategies for achieving greater insight and control over complex systems. This guide brings you a comprehensive list of DevOps, SRE & observability events that can help you stay ahead in a rapidly evolving field. Whether you’re exploring the latest in tools, cloud observability, or AI-driven insights, this guide will lead you to the perfect event to expand your knowledge and network with industry leaders!

SolarWinds Migration Program | Netdata

Tue, 12 Nov 2024 00:00:00 +0000

KAUST Case Study: System Stability For Academia

Mon, 11 Nov 2024 00:00:00 +0000

Empowering System Stability in a Complex Research Environment

The Systems Research Group at King Abdullah University of Science and Technology (KAUST) manages a diverse and expansive computing environment essential for advancing scientific research. Given the critical nature of this environment, ensuring uptime and rapid troubleshooting is essential. However, without a robust monitoring solution, pinpointing performance issues or hardware failures was a slow, often reactive process that challenged the team’s ability to maintain a smooth operational flow.

Native Windows Agent: Real-Time Windows Monitoring

Fri, 08 Nov 2024 00:00:00 +0000

We are pleased to announce a significant advancement in system monitoring: the launch of Netdata’s first-ever Native Windows Agent. This release represents a major step forward in our mission to provide comprehensive and efficient monitoring solutions across all platforms. With the introduction of the native Windows agent, we are extending our robust monitoring capabilities to Windows environments, enabling seamless and unified monitoring across diverse infrastructures. This development is a direct response to the needs of our user community, who have expressed a strong demand for a powerful and intuitive Windows monitoring solution.

Accurate Process Monitoring with Netdata

Mon, 04 Nov 2024 00:00:00 +0000

Understand why tracking cumulative resource consumption is crucial for accurate process monitoring.

Accurate Process Monitoring with Netdata: Why Tracking Cumulative Resource Consumption Matters

Tracking the cumulative resource consumption of processes, including short-lived and exited children, is a rare feature in monitoring tools – and it’s one of the standout capabilities Netdata offers.

Most mainstream solutions, like Datadog’s process monitoring and Prometheus’s Node Exporter, focus on active processes and only collect metrics per PID. Even specialized process monitors (top, htop, etc.) face the same limitations. They capture snapshots of currently running processes and often rely on fixed sampling intervals, which are too slow to catch very short-lived tasks. This approach falls short when trying to capture the resource footprint of dynamic applications, shell scripts, and complex process hierarchies.

Linux Load Average Myths and Realities

Sun, 03 Nov 2024 00:00:00 +0000

When it comes to monitoring system performance on Linux, the load average is one of the most referenced metrics. Displayed prominently in tools like top, uptime, and htop, it’s often used as a quick gauge of system load and capacity. But how reliable is it? For complex, multi-threaded applications, load average can paint a misleading picture of actual system performance.

In this article, we’ll dive into the myths and realities of Linux load average, using insights from Netdata’s high-frequency, high-concurrency monitoring setup. Through this journey, we’ll uncover why load average spikes can occur even under steady workloads, and why a single metric is rarely enough to capture the true state of a system. Whether you’re a system administrator, developer, or performance enthusiast, this exploration of load average will help you interpret it more accurately and understand when it may—or may not—reflect reality.

Fix NGINX 500 Internal Server Error: Easy Steps

Thu, 31 Oct 2024 00:00:00 +0000

When managing a website or web application through NGINX, it’s quite likely you’ve stumbled upon the feared 500 Internal Server Error at least once. This error often causes a moment of panic as it typically indicates a problem lurking within the server’s operations. Regrettably, one finds that the message falls short in offering sufficient context to aid in uncovering the underlying reason.

No need to fret! This guide is here to take you step by step through the usual culprits behind a 500 error when using NGINX and, even better, how to solve it. If you’re facing a problem like not having the right permissions, a setup mistake, or something different, don’t worry—we have all the steps you need to fix it.

6 + 1 Effective Strategies to Reduce Unplanned Downtime

Wed, 30 Oct 2024 00:00:00 +0000

For DevOps and SRE teams, unplanned downtime may be a nightmare since it can ruin everything, from customer satisfaction to corporate operations. With more and more systems relying on constant availability, any unplanned downtime must be eliminated to ensure reliability of service. In the article below, we will discuss how to utilize infrastructure monitoring, monitoring tools, development and operations best practices in order to minimize unplanned downtime.

What is Unplanned Downtime

Unplanned downtime occurs when an application or system or infrastructure component fails without notice, causing an interruption. These disruptions could be due to a network issue, or a problem in the software or hardware or even human error. The business expenses are often substantial in terms of revenue loss and negative publicity. So, putting a plan in place to reduce downtime is critical.

12 Benefits You Get by Scaling with Netdata

Wed, 23 Oct 2024 00:00:00 +0000

80% of decision-makers globally acknowledge that digital infrastructure is essential for reaching business goals. However, IT infrastructure is becoming increasingly distributed and complex. Organizations are managing hundreds—even thousands—of nodes across cloud, on-premise, and edge environments. This predicament makes effective monitoring across all systems more essential than ever. In turn, this drives the demand for valuable real-time insights through scalable monitoring solutions like Netdata.

If you’re managing a large infrastructure but haven’t fully embraced Netdata, now is the time to reconsider. Let’s take a look at the benefits you’ll get, below.

5 Best Datadog Alternatives For Monitoring & Observability

Tue, 15 Oct 2024 00:00:00 +0000

As businesses rely more on infrastructure monitoring, observability tools have become essential for keeping systems secure, responsive, and efficient. In this evolving monitoring landscape, Datadog is known as a leading analytics and monitoring tool, capable of collecting essential performance metrics from servers, databases, applications, and other IT infrastructures. But at what cost?

What Is Datadog? Is It Your Only Option?

Datadog is a cloud-based monitoring platform that helps organizations track the performance of their infrastructure, applications, and logs. It provides visibility across systems to ensure efficient operations and identify issues quickly.

ilert Integration: Streamline Monitoring & Response

Mon, 14 Oct 2024 00:00:00 +0000

Netdata now integrates with ilert, a leading incident response platform. With this integration, the incident management features and alerting capabilities of ilert and the real-time systems monitoring provided by Netdata can be leveraged. By combining both systems, users can not only monitor their infrastructure with fine detail as never before, but also assure the responsiveness of critical alerts to the correct teams swiftly.

What is ilert?

ilert is an end-to-end platform for alerting, on-call management, and status pages, built for the new-age DevOps and SRE teams. It streamlines the entire incident management process by automating key aspects and providing powerful tools to improve efficiency, such as automated on-call duty, multi-level escalations, alert grouping, postmortem document creation, and much more. ilert integrates with monitoring systems, like Netdata, and sends alerts through multiple channels (SMS, phone, push, Slack, Microsoft Teams, etc.), ensuring that teams can promptly respond to issues before they impact service.

What Is Application Performance Monitoring (APM)?

Thu, 03 Oct 2024 00:00:00 +0000

Ensuring that applications are functioning as expected is essential in the software-driven world of today. One sub-standard performance and your app can fail to win people over, ultimately drive them away or be the reason for debilitating first impressions. This is where Application Performance Monitoring (APM) comes in and helps developers, DevOps teams, and especially an SRE (Site Reliability Engineer) to monitor their application running live on systems, allowing them to identify the issues faster before they impact users.

ChileAtiende Case Study: 25% Less AWS Downtime

Tue, 24 Sep 2024 00:00:00 +0000

Proactive Infrastructure Management for Citizen Services

ChileAtiende, a government organization dedicated to delivering essential digital services, faced a growing challenge: ensuring the stability and reliability of its complex infrastructure. With an ever-expanding digital presence, the team needed an efficient way to monitor a hybrid environment that included containers, databases, and web services like nginx and Apache.

The critical pain point for ChileAtiende was ensuring system reliability through real-time monitoring and quick response capabilities. They sought a solution that would allow them to anticipate potential problems and address them before they impacted the public.

TMB Case Study: Barcelona Transport Monitoring

Fri, 20 Sep 2024 00:00:00 +0000

Monitoring a Complex, Heterogeneous Infrastructure

Transports Metropolitans de Barcelona (TMB), the public transportation provider in Barcelona, manages a vast and varied infrastructure hosted on AWS and on bare metal. Monitoring this infrastructure presents significant challenges due to its heterogeneous nature—spanning Linux, Windows, Kubernetes, cloud, and on-premises environments.

One of the main obstacles for TMB was finding a single tool capable of handling the breadth of this environment effectively. As the infrastructure evolved, it became essential to have a holistic view of all systems to avoid operational silos and improve response times to issues.

What is Alert Fatigue and How to Prevent It

Fri, 20 Sep 2024 00:00:00 +0000

What is Alert Fatigue?

This is called alert fatigue where engineers especially DevOps SREs and on-call teams become numb to the many notifications from their monitoring tools. Instead of quickly responding to important issues, engineers may start ignoring or missing crucial alerts due to the overwhelming number of notifications they receive. That is no small issue because that causes slower incident response times, possible service outages, and less reliability.

Robust monitoring is a must-have in the high-speed world of cloud infrastructure and microservices. However, when every little spike sets off an alarm, it is hard to separate the wheat from the chaff, so to speak. So in the end, alerts lose their meaning and it becomes a society of ignoring alarms, even when they shouldn’t be ignored.

What Is Observability? Definition, Benefits & How It Works

Fri, 20 Sep 2024 00:00:00 +0000

What Is Observability?

Observability is the process of trying to figure out how a system works on the inside, by looking at what it does on the outside. It has even grown to be a central theme in today’s technology stacks where it is used to guarantee system availability and speed, particularly in intricate distributed architectures. But it is not just about the data, it is about the knowledge derived from the data that would help keep the system healthy, diagnose problems, and optimize performance.

What Is Uptime Monitoring? All SREs & DevOps Teams Must Know

Thu, 19 Sep 2024 00:00:00 +0000

Uptime Monitoring Explained

Keeping an eye on how well and how often servers, apps, services, and all the parts of your system are up and running is what uptime monitoring is all about. For people in Site Reliability Engineering (SRE) and DevOps teams, making sure everything works almost all the time is super important. Keeping your services up and running means users run into less trouble and enjoy a more seamless connection without outages. This cuts down on the chance of expensive interruptions in business.

Synthetic Checks: Definition, Examples & Benefits

Wed, 18 Sep 2024 00:00:00 +0000

What Are Synthetic Checks?

Synthetic checks are proactive tests that simulate user interactions or network requests to monitor the availability and performance of services. Rather than waiting for an issue to be reported by a user, synthetic checks continuously run automated checks to detect problems before they affect real users. This makes them an essential tool for DevOps and Site Reliability Engineers (SREs) who aim to maintain a reliable and responsive infrastructure.

Infrastructure Monitoring: Key Benefits, Types & Use Cases

Tue, 17 Sep 2024 00:00:00 +0000

What Is Infrastructure Monitoring?

Infrastructure monitoring is the practice of continuously tracking the performance, availability, and overall health of IT systems such as servers, databases, networks, cloud services and so many more. To put it simply, it’s about having clear visibility into how all the components of your applications and your entire infrastructure in general, are behaving. This is why we need and use the so-called monitoring / observability tools.

Important Changes to the Netdata Agent Dashboard

Mon, 12 Aug 2024 00:00:00 +0000

Important Notice: These changes ONLY impact users of the Netdata Agent Dashboard not connected to Netdata Cloud.

Dear Netdata Community,

We are writing to inform you of upcoming changes to the Netdata Agent Dashboard, which will take effect in the coming weeks. This change impacts users from the soon to be released Netdata v2.0 onwards (and also on the Netdata v1.47 nightly releases). Currently, the Open-Source Netdata Agents allow unauthorized and unlimited Agent dashboard access. From Netdata v2.0 onwards, all Netdata Dashboards (Agent and Cloud) will offer exactly the same functionality under the same policy. Netdata Agent Dashboard will use Netdata Cloud as an SSO provider, ensuring dashboard access is authenticated and validated by Netdata Cloud, users will have the option to proceed with an unauthorized local dashboard but this will no longer be the default.

Analyze & Reduce Disk I/O Bottlenecks: Top Techniques

Mon, 01 Jul 2024 00:00:00 +0000

Understanding Disk I/O Bottlenecks

Disk I/O (Input/Output) bottlenecks can severely impact the performance of your applications and services. These bottlenecks occur when the disk subsystem cannot keep up with the read/write requests from the CPU or memory, leading to slow response times and degraded performance. As a DevOps or SRE professional, understanding how to analyze and reduce these bottlenecks is crucial for maintaining optimal system health.

Identifying Disk I/O Bottlenecks

Before you can tackle disk I/O bottlenecks, you need to identify them accurately. Effective identification involves using a combination of monitoring tools and understanding key metrics.

Motohunt Case Study: Automotive Industry Monitoring

Thu, 27 Jun 2024 00:00:00 +0000

Navigating System Complexity with Netdata

Motohunt is a big player in the automotive technology sector, delivering innovative products to the moto lovers. The organization’s dedication to maintaining a healthy production environment and understanding data evolution across systems underscores its commitment to excellence. Netdata emerged as a key player in addressing Motohunt’s primary challenge of monitoring system health.

“Netdata’s out-of-the-box functionality and real-time analytics have been instrumental in maintaining our system’s health. Its value efficiency and comprehensive coverage ‘just works,’ providing us with immediate insights without a hefty price tag.”

Webinar: Maximize Uptime With Powerful Monitoring

Thu, 27 Jun 2024 00:00:00 +0000

Selecting the right observability solution is essential for maintaining the performance and reliability of your IT infrastructure. With countless options available, making an informed decision can be challenging. This webinar will provide you with a comprehensive guide to navigating this complex process.

We’ll explore the critical factors to consider, such as scalability, ease of use, integration capabilities, and cost-effectiveness. You’ll gain insights into best practices for evaluating and implementing monitoring tools that align with your business goals and technical requirements.

High CPU Usage Detected How To Fix CPU Overload

Mon, 10 Jun 2024 00:00:00 +0000

You get an alert: “High CPU usage detected on server-db-01.” Your application feels sluggish, users are reporting timeouts, and the system is becoming unresponsive. A maxed-out CPU is a clear sign of trouble, grinding your operations to a halt and putting your service reliability at risk. But what does high CPU usage actually mean, and more importantly, how do you fix it?

Sustained high CPU usage, often pegged at 100%, means your server’s processor is completely saturated. It’s trying to handle more tasks than it’s capable of, leading to performance degradation, increased latency, and potential crashes. For DevOps engineers, SREs, and developers, quickly diagnosing and resolving a CPU overload is a critical skill. This guide will walk you through identifying the culprits and restoring your system to optimal health.

How To Achieve High Availability In CI/CD With Observability

Sun, 09 Jun 2024 00:00:00 +0000

Your CI/CD pipeline is the backbone of your software delivery process. When it works, code flows smoothly from commit to production. But what happens when it breaks? A failed pipeline means stalled feature releases, delayed bug fixes, and frustrated developers unable to ship their work. To prevent this, you need to treat your CI/CD infrastructure with the same rigor as your production applications, and that starts with making it highly available.

Key Observability Metrics | Infrastructure & APM Monitoring

Thu, 06 Jun 2024 00:00:00 +0000

What Is Observability? The Fundamentals

Noone can argue that observability is crucial for maintaining the health and performance of applications and infrastructure. Observability refers to the ability to measure and understand the state of a system based on the outputs it produces. This is extremely important for identifying, diagnosing, and resolving issues effectively and efficiently.

Observability is essential for DevOps and SRE teams as it provides a comprehensive, overall view of the infrastructure’s health, enabling proactive maintenance and quicker incident response. It involves collecting and analyzing a variety of data types, including logs, metrics, and traces, to gain insights into system behavior and it can help discover possible anomalies throughout the whole infrastructure.

Introducing Netdata's Dynamic Configuration Manager

Wed, 05 Jun 2024 00:00:00 +0000

We are thrilled to unveil the latest addition to the Netdata platform: the Dynamic Configuration Manager. This powerful new feature revolutionizes how you manage your monitoring and alerting configurations, making it easier and more efficient than ever before.

Key Features of the Dynamic Configuration Manager

Create and Modify Alerts from Every Chart
- You can now create and modify alerts directly from any chart on your dashboard or from the dedicated Alerts tab. This streamlined process allows for quick adjustments and ensures your monitoring is always aligned with your current needs.
Configure Collectors from the Integrations Section on the Dashboard
- Currently available for go.d collectors, this feature lets you configure collectors straight from the Integrations section. This means you can quickly identify what Netdata can monitor and set up your configurations in one go, without having to dig through multiple settings pages.
Submit Configurations to Multiple Nodes with One Click
- Managing configurations across a large infrastructure can be time-consuming. With the Dynamic Configuration Manager, you can now submit configurations to multiple nodes simultaneously, saving you valuable time and effort.
Construct and Copy Configurations for IaC Solutions
- For those using Infrastructure as Code (IaC) solutions, this feature allows you to construct and copy configurations easily, integrating them into your IaC workflows. This ensures your configurations are consistent and reproducible across different environments.

How to Use the Dynamic Configuration Manager

To help you get started with the Dynamic Configuration Manager, we’ve put together a quick guide using the Netdata demo environment.

Effective Strategies for Managing PostgreSQL Deadlocks

Mon, 03 Jun 2024 00:00:00 +0000

Managing PostgreSQL deadlocks is critical for maintaining database performance and ensuring smooth application operations. Deadlocks occur when two or more transactions hold locks that the other transactions need, creating a cycle of dependencies with no resolution. This guide explores effective strategies for preventing and resolving deadlocks in PostgreSQL, helping DevOps and Site Reliability Engineers (SREs) ensure optimal database performance.

Understanding PostgreSQL Deadlocks

What Are Deadlocks?

A deadlock in PostgreSQL happens when two or more transactions block each other, each waiting for the other to release a lock. This mutual blocking results in a standstill where none of the transactions can proceed. PostgreSQL has mechanisms to detect and resolve deadlocks by aborting one of the transactions, but this can still lead to performance issues and data inconsistencies.

How to automate adding nodes to rooms in Netdata?

Tue, 28 May 2024 00:00:00 +0000

How we organize nodes (and the Netdata agents that are running on those nodes) across different rooms should reflect our architectural decision because the room is a logical container with its own user members and notification rules. So if we are monitoring large infrastructure we should be consistent with these rules and one way to achieve this is to choose automation. Netdata Cloud Terraform Provider lets you automate this by provisioning all the cloud resources and giving you the credentials to spin up the Netdata Agents. In this article, we will concentrate on how in practice we can organize and assign nodes across different rooms in two scenarios, in each of them I’m using non-production installation of the Netdata Agents:

How To Fix 504 Gateway Timeout Errors In NGINX

Mon, 27 May 2024 00:00:00 +0000

Encountering a ‘504 Gateway Timeout’ error in Nginx can be frustrating, especially when it disrupts the availability of your web applications. This error indicates that the server, while acting as a gateway or proxy, did not receive a timely response from the upstream server. This guide will help DevOps and Site Reliability Engineers (SRE) diagnose and fix these errors efficiently.

TL;DR

A 504 Gateway Timeout means Nginx, acting as a proxy, didn’t get a response from the upstream server in time, so the problem sits in the backend, the network, or your timeout settings rather than in Nginx itself.
Diagnose before you change anything: check the Nginx and upstream logs, measure backend response time with curl, and test connectivity with ping, traceroute, or mtr.
The fastest stopgap is raising Nginx’s timeout directives, but the durable fix is making the upstream faster through resource scaling, query optimization, and load balancing.
Adding Nginx caching reduces how often requests hit the upstream at all, which keeps a strained backend responsive and cuts down 504s.k.

Understanding The ‘504 Gateway Timeout’ Error

The ‘504 Gateway Timeout’ error typically occurs in Nginx when:

How to Troubleshoot Slow Queries in MongoDB

Mon, 27 May 2024 00:00:00 +0000

MongoDB is a popular NoSQL database known for its flexibility and scalability. However, as with any database system, you might encounter slow queries that can impact the performance of your application. In this guide, we’ll walk you through the steps to troubleshoot and optimize slow queries in MongoDB, ensuring your database runs efficiently.

Understanding the Basics

Before diving into troubleshooting, it’s important to understand some basic concepts in MongoDB:

Collections and Documents: MongoDB stores data in collections, which are analogous to tables in relational databases. Each collection contains documents, which are JSON-like data structures.
Indexes: Indexes improve query performance by allowing the database to quickly locate the data without scanning every document.
Query Plans: MongoDB evaluates different ways to execute a query and chooses the most efficient plan.

Identifying Slow Queries

Using the slowms Parameter MongoDB logs operations that take longer than a specified threshold. By default, this threshold (slowms) is set to 100 milliseconds. You can adjust this setting to catch slower operations more effectively:

DevOps Best Practices Playbook For Performance

Wed, 22 May 2024 00:00:00 +0000

In software development and IT operations, DevOps has become a central practice, facilitating collaboration, increasing efficiency, and ensuring high-quality software delivery. However, the range of DevOps operations can be intimidating for beginners. This playbook aims to simplify DevOps by outlining best practices for developers and teams, enabling more efficient workflows and operational excellence.

Devops Meaning: Bridging Software Development & IT Operations

DevOps is a cultural and technical shift that bridges the historically isolated worlds of software development and IT operations. The goal is to construct a fully automated pipeline for developing, testing, delivering, and maintaining software. Think of it as running a race where each runner hands off the baton seamlessly, boosting overall speed and efficiency.

How To Speed Up Windows 10 & 11: Performance Tips

Fri, 17 May 2024 00:00:00 +0000

Dealing with a slow computer is frustrating. Whether you’re compiling code, running virtual machines, analyzing data, or just trying to browse documentation, sluggish performance pc issues can significantly hinder your productivity and break your focus. Waiting for applications to load or the system to respond eats up valuable time and can turn simple tasks into chores. Fortunately, there are numerous ways to speed up computer performance on Windows.

This guide provides actionable tips to optimize pc responsiveness, ranging from simple maintenance routines and software adjustments to more impactful hardware upgrades. We’ll cover how to improve computer performance and get your Windows system running smoothly again.

What Is An HPC Cluster - Key Components & How It Works

Wed, 01 May 2024 00:00:00 +0000

Imagine needing to analyze petabytes of genomic data, simulate the airflow over a new aircraft wing, or predict complex financial market movements. A single computer, no matter how powerful, would struggle or take an impractically long time. This is where High-Performance Computing (HPC) clusters come in. They provide the immense computational power needed to tackle problems far beyond the reach of standard computing. If you’re stepping into roles involving large-scale data processing or complex simulations, understanding HPC clusters is essential.

Costa Tsaousis Interview: Homelab Show Insights

Fri, 26 Apr 2024 00:00:00 +0000

Costa Tsaousis, founder and chief visionary of Netdata, recently shared his expertise on real-time monitoring’s pivotal role in modern IT landscapes during an episode of ‘The Homelab Show’ on YouTube. If you missed the live stream, here’s an essential summary of the discussion’s key points.

The Necessity of Real-Time Monitoring

Nowadays, data flows incessantly and operational demands are continuous, the importance of real-time monitoring cannot be overstated.

Decentralized Monitoring Explained

Thu, 11 Apr 2024 00:00:00 +0000

Introduction to Distributed Observability

Users often find themselves puzzled by the concepts of decentralized or distributed monitoring. This confusion is likely due to many monitoring systems claiming distributed capabilities, making it challenging to discern how Netdata stands out.

To grasp the distinction, we must delve into the evolution of monitoring systems.

When the first monitoring systems were created, about 20-25 years ago, they were built as SNMP collectors. The monitoring application was installed on a server, configured to discover network devices via SNMP, pulling data once every minute, per device. Simultaneously, the monitoring system was exposing a daemon to collect SNMP traps (key events generated by the network devices, pushed to the monitoring system).

Netdata is the only real-time monitoring solution: Justified

Wed, 10 Apr 2024 00:00:00 +0000

In the digital era, where data flows like a ceaseless river, real-time monitoring stands as a pivotal technology, allowing organizations to not only keep pace but also to deeply understand the intricate dance of their operational ecosystems. This technology is not just about keeping tabs; it’s about gaining a profound, almost intuitive sense of the micro-worlds within which systems, containers, services, and applications pulse and thrive.

Real-time monitoring is the art and science of tracking system performance, activities, or transactions continuously and automatically, providing the ability to analyze and visualize data the moment it’s generated.

Understanding Monitoring Tools

Wed, 10 Apr 2024 00:00:00 +0000

If you care about operational excellence when it comes to your IT infrastructure, the role of monitoring systems is pivotal. As we navigate through the myriad of available monitoring tools, it becomes essential to understand the distinct architectures, styles, and focal points of various monitoring solutions, as well as the time-to-value they offer. This blog post aims to demystify the landscape of monitoring systems, providing a comprehensive overview that categorizes these tools into four primary architectural design principles.

SafetyDetectives: An Interview-With-Costa-Tsaousis

Mon, 01 Apr 2024 00:00:00 +0000

In a recent conversation with SafetyDetectives, Costa Tsaousis, CEO and founder of Netdata, shares insights into the inception and evolution of Netdata, a game-changing monitoring solution. With a background in fintech and a passion for real-time data processing, Tsaousis was driven to create Netdata in response to the significant gaps he identified in traditional monitoring tools. Emphasizing the importance of real-time data, comprehensive metrics collection, and the innovative use of machine learning, Tsaousis discusses how Netdata is setting new standards in the monitoring industry. His vision for Netdata not only challenges the status quo but also introduces a novel approach to cybersecurity, making it an essential tool for organizations worldwide.

Overcoming Monitoring Challenges in Shared VM Environments

Thu, 28 Mar 2024 00:00:00 +0000

Optimizing Resource Management in Complex VM Ecosystems

ID5 Web Solutions, a web solutions provider in Brazil, faced significant challenges in managing its shared VM environment. With a diverse clientele depending on their infrastructure, the need for precise and efficient resource usage identification per user was paramount. The company sought to not only allocate workloads more effectively but also to ensure fair pricing strategies based on actual consumption.

The complexity of monitoring several shared VMs and the necessity for detailed user-specific resource usage insights posed substantial hurdles. Additionally, ID5 Web Solutions aimed to incorporate a seamless alerting mechanism via Discord, enhancing their response capabilities to incidents and optimizing service availability.

Transforming Server Monitoring across Diverse Hosts

Thu, 28 Mar 2024 00:00:00 +0000

Mastering Monitoring Complexity

URIOS, operating in the financial services sector, faced a significant challenge in monitoring its diverse Linux server environments. Spread across various infrastructures and hosts, the complexity of managing these servers was growing. The organization required a solution that could centralize and unify its monitoring efforts, offering precise and numerous indicators to stay ahead of potential issues.

The multitude of infrastructures and hosting providers made it difficult for them to maintain a consistent monitoring approach. They soon realized that they needed a tool that could bring everything together in a cohesive manner.

TTIC Case Study: Efficient System Monitoring

Thu, 28 Mar 2024 00:00:00 +0000

Enhancing System Management with Scarce Time Resources

At the Toyota Technological Institute at Chicago, the challenge of system administration is uniquely compounded by the scarcity of time. As a one-man IT department, Adam Bohlander juggles various responsibilities, making efficient time management crucial. The institute’s evolving needs demand a monitoring solution that aligns with its mission of leading in computer science and information technology research and education.

“I find the install quick and easy via ansible (my own playbooks) and defaults to be quite sane. This allows me to use it without investing that much time.”

Manage Netdata Cloud with Terraform

Wed, 27 Mar 2024 00:00:00 +0000

We proudly announce the release of the Netdata Cloud Terraform Provider.

It’s a step forward to make our platform more automated and compliant with the modern Infrastructure as Code approach. Terraform is one of the leaders in the IaC tools with a rich ecosystem of providers and modules, now you can put a puzzle with Netdata Cloud to your stack.

The initial iteration of the Netdata Cloud Terraform Provider supports the following resources:

University of Calgary: Netdata in Academia

Tue, 26 Mar 2024 00:00:00 +0000

Empowering Academic Research with Real-Time Monitoring

At the Machine Learning Lab at the University of Calgary, managing a robust infrastructure of servers and workstations is critical for advancing their research in machine learning (ML). Assistant Professor Yani Ioannou oversees this infrastructure, ensuring that these essential resources remain operational and are used to their fullest potential. The lab’s challenge lies not just in the maintenance of these resources but in minimizing downtime to keep the research moving forward.

Enhancing E-Learning Through Proactive Monitoring

Wed, 20 Mar 2024 00:00:00 +0000

Tackling E-learning Monitoring Challenges

Ingenium Digital Learning, a major player in the e-learning sector, faces unique challenges in delivering high-quality educational services online. One primary concern they had was managing disk space usage and CPU load effectively, as users continuously upload data without a clear view of their capacity limits. Additionally, unexpected spikes in server load, often due to unannounced large-scale events, further complicate resource management.

“Clients do not warn us of big events, which impacts the load on the servers. Users can upload data to our servers but don’t have a view on their total capacity.”

Dynatrace vs Datadog vs Instana vs Grafana vs Netdata!

Sun, 17 Mar 2024 00:00:00 +0000

In this post, we delve into the comparative analysis of the commercial offerings of five leading monitoring solutions—Dynatrace, Datadog, Instana, Grafana, and Netdata. Our objective is to unravel the intrinsic value each of these services offers when applied to a real-world scenario. To accomplish this, we employed trial subscriptions of these services to monitor a set of Ubuntu servers and VMs, each hosting a pair of widely-used applications: NGINX and PostgreSQL, along with a couple of Docker and LXC containers. Additionally, we extended our monitoring to physical servers to evaluate the efficacy of these tools in capturing hardware and sensor data along with VMs monitored from the host.

Monitoring Industrial Equipment Manufacturing with Netdata

Thu, 14 Mar 2024 00:00:00 +0000

Empowering Precision and Efficiency

Autonoma Technologies GmbH, a leader in the industrial equipment manufacturing sector, has been at the forefront of integrating digital solutions to address the complexities of modern industrial challenges. Monitoring many different services without clear insights into critical metrics for each, Autonoma turned to Netdata for a solution. The need for deep knowledge across various services to identify and monitor critical metrics was a significant hurdle.

With a primary focus on monitoring vital systems like disk space, RabbitMQ, VerneMQ, NGINX, Redis, Fail2ban, and Kubernetes clusters, Autonoma required a robust and flexible tool to keep its production setup fast and reliable. Netdata became their solution of choice, offering out-of-the-box analysis for specific services that drastically cut down the time and effort needed for creating and adapting dashboards.

New Streamlined Plan Structure

Wed, 06 Mar 2024 00:00:00 +0000

UPDATE: Netdata is introducing a streamlined plan structure, sunsetting Early Bird plans on 13-03-2024.

As the landscape of real-time monitoring evolves, so does the diversity and complexity of use cases that our community brings to Netdata. Our mission has always been to democratize monitoring by making it accessible, powerful, and scalable for everyone. With the rapid growth of our user base and their expanding needs, it’s become clear that our plan structure must evolve to maintain this mission sustainably.

ACPro Case Study: Effortless Manufacturing Monitoring

Tue, 27 Feb 2024 00:00:00 +0000

Unifying Monitoring Across Diverse Environments

ACPro faced the challenge of overseeing a wide array of in-house applications scattered across numerous servers, all without a monitoring solution in place. This lack of visibility made it difficult to maintain the high uptime and availability standards that are critical in the manufacturing industry. The search for a monitoring tool that was both comprehensive and user-friendly led ACPro to Netdata, which stood out for its intuitive interface and extensive, yet easily digestible, metrics.

InRento Case Study: Fintech Cloud Monitoring Wins

Tue, 27 Feb 2024 00:00:00 +0000

Simplifying Complex Cloud Environments

The migration of Inrento’s infrastructure to the cloud introduced a new set of challenges, particularly in monitoring multiple servers with diverse roles. The transition from a single server environment to a distributed application on AWS required a solution that not only automated the monitoring process but also made it accessible and understandable to developers accustomed to a monolithic setup.

“Netdata enables automated integration of new servers into our architecture. As we experiment with various configurations, we can leverage automated configuration that is prepared beforehand.”

Reduced Costs & Kubernetes Efficiency

Tue, 27 Feb 2024 00:00:00 +0000

Simplifying Complexity with Netdata

At 7 Oaks Group, our journey with Netdata began out of curiosity and a desire to move away from the frustrations of complex monitoring setups. Our infrastructure, primarily composed of Kubernetes clusters and a selection of EC2/VM instances, required a monitoring solution that could provide instant, meaningful insights without the hassle of intricate configuration. The immediate value we discovered in Netdata was nothing short of revolutionary, offering us a level of insight into our systems that was both effortlessly attainable and profoundly impactful.

Easy Doc: Revolutionizing IT Operations with Netdata

Tue, 13 Feb 2024 00:00:00 +0000

Overcoming IT Operational Challenges

At Easy Doc Soluções Integradas, navigating the complexities of docker swarm cluster infrastructure presented significant challenges, particularly in monitoring the health and availability of containers and cluster nodes. The primary pain point was the difficulty in detecting and diagnosing the reasons behind API unavailability, especially since container restarts resulted in the loss of crucial logs. This obstacle hindered our ability to maintain high levels of service availability and performance, directly impacting our operational efficiency and customer satisfaction.

Streamlining IT Operations with Netdata

Tue, 13 Feb 2024 00:00:00 +0000

Addressing Monitoring Complexity

The main challenge for the organization was ensuring up-to-date, comprehensive monitoring across a diverse infrastructure, complicated by different operating systems, security requirements, and network segmentation. The varied nature of their environment made it difficult to maintain a unified view of system health and performance, often leading to delayed root cause analysis and inefficient troubleshooting processes.

Netdata’s ease of deployment and out-of-the-box functionality have been game-changers, offering immediate visibility into their systems’ health with minimal setup. Its comprehensive monitoring capabilities allowed them to quickly identify and address issues, reducing the time and effort previously required for these tasks.

Upcoming Homelab Plan

Wed, 07 Feb 2024 00:00:00 +0000

UPDATE: On the 2024-02-08 A new Netdata Cloud Homelab plan will become available. This is aimed for home users and students.

What you need to know?

New Plan alert: We’re introducing a dedicated plan on Netdata Cloud—the Homelab plan. It’s tailored to meet the needs of home users and students, offering unrestricted access to Netdata features without the limitations seen in the Community plan.
Exclusively for Personal Use: The Homelab plan is designed for personal, non-commercial use only. To qualify, users will need to self-certify as a home user or student during the sign-up process.

Codyas: Global Server Management with Netdata

Thu, 01 Feb 2024 00:00:00 +0000

Overcoming Server Management Hurdles

Codyas faces the intricate challenge of managing a vast array of servers, equipped with different technologies for various clients around the globe. The primary obstacle has been the allocation of limited time and personnel towards monitoring tasks, which detracted from the organization’s core focus on development and customer service.

“Netdata’s ease of installation and configuration has been a game-changer for us, allowing quick setup and efficient monitoring across our diverse server landscape,”

Enhancing Research Through Advanced Monitoring

Thu, 01 Feb 2024 00:00:00 +0000

Helping Scientists focus on the Science

At Manaaki Whenua – Landcare Research, managing multi-user Linux workstations for scientific computing presented a unique challenge. Tasked with overseeing system utilization without a dedicated role in the organization, the need for a solution that was intuitive and efficient became paramount. Netdata’s simplicity in setup and the ability to monitor systems remotely via the cloud addressed these challenges head-on, offering a seamless solution for workstation management.

From Bottlenecks to Breakthroughs

Thu, 01 Feb 2024 00:00:00 +0000

Simplifying IT Monitoring Challenges

At LANCOM Systems, the transition to a more integrated and efficient IT monitoring system was driven by a desire to streamline operations and overcome the challenges associated with complex system monitoring. The company faced significant hurdles in integrating services within Docker containers and adapting to the programming language Go, despite a strong preference for Python. This challenge was further compounded by the need for a comprehensive monitoring solution that could offer extensive out-of-the-box features without necessitating extensive development work.

Leica Biosystems: Advancing Cancer Diagnostics

Thu, 01 Feb 2024 00:00:00 +0000

Navigating Offline Monitoring Challenges

Leica Biosystems, a leader in pathology and cancer diagnostics, faced a unique challenge: monitoring and observability in strictly offline environments. The need for a dashboard providing system insights in an offline capacity was paramount, especially given the constraints of remote access impossibility due to the offline nature of the devices used.

“Monitoring complexity in offline environments was a significant hurdle. Netdata’s ability to export metrics allowed us to achieve the observability we needed without the need for online connectivity.”

Monitoring on Autopilot

Thu, 01 Feb 2024 00:00:00 +0000

Overcoming Scalability Issues

Simon Gajdosik, the driving force behind Webnestify, identified scalability and maintainability as primary challenges in their monitoring strategy. The traditional approach with Grafana and Prometheus, although effective, became untenable at scale. Netdata Cloud provided the much-needed solution, offering automated monitoring that effortlessly scales with the business, eliminating the complexity of manual configurations.

Enhancing Operations with Netdata

Webnestify’s adoption of Netdata Cloud provided significant operational enhancements:

Time Savings: The automated service discovery and configuration capabilities of Netdata have saved Webnestify at least 10 hours weekly, allowing the team to focus more on business growth rather than infrastructure maintenance.
Productivity Improvements: With the efficiency gains from Netdata, Webnestify has increased its client onboarding capacity, directly contributing to business expansion.
Issue Resolution: The advanced alerting system of Netdata enabled proactive anomaly detection, preventing potential downtimes for critical ecommerce platforms and thereby ensuring uninterrupted service.
Cost Reduction: Shifting from a dedicated Grafana server to Netdata resulted in a 40% reduction in annual monitoring costs, translating to significant financial savings.
Enhanced Security: The ability to whitelist parent node IP addresses and use private IPs for child nodes improved the overall security posture.
Improved Troubleshooting and Metrics Accuracy: The intuitive dashboards and accurate metrics from Netdata have simplified the process of identifying and resolving issues, making troubleshooting 50% easier.
Downtime Reduction and SLA Improvements: With Netdata, Webnestify has seen a 20% reduction in downtime and significantly improved its SLA compliance by resolving issues more efficiently.

“A standout moment for us was when Netdata’s anomaly detection alerted us to a potential issue, allowing us to avert a crisis on a server hosting over 100 high-traffic ecommerce sites. This instance underscored the indispensable role of Netdata in our operations,”

Optimizing The FinTech Industry

Thu, 01 Feb 2024 00:00:00 +0000

Optimizing The FinTech Industry: Yellow Brick Road’s Monitoring Journey with Netdata

Yellow Brick Road, a leader in the financial services industry, embarked on a mission to fine-tune its system and kernel parameters to minimize network latency crucial for quick trading decisions. The challenge was not just in identifying what to measure but also in how to effectively use the data obtained.

The Netdata Solution

Netdata Cloud came to the rescue, offering comprehensive baseline monitoring that was both easy to deploy and cost-efficient. The adoption of Netdata, provided the team at Yellow Brick Road with the necessary tools to monitor their Nomad, Consul, bare metal, and systemd applications efficiently.

Real-Time Insights for Reliable Web Hosting

Thu, 01 Feb 2024 00:00:00 +0000

Simplifying Monitoring for Enhanced Performance

Trillium was confronted with the challenge of finding an effective monitoring solution that could be easily integrated with their systems, despite the non-technical background of some team members. The company aimed to uphold its commitment to outstanding service uptime and performance—a critical aspect in the hosting industry. The search for a monitoring tool that could provide comprehensive insights into the health of their infrastructure, particularly RAM and CPU usage, and uptime tracking, led them to Netdata.

Streamlining VM Monitoring with Netdata

Thu, 01 Feb 2024 00:00:00 +0000

Simplifying Virtual Machine Monitoring

Comtegra, a leader in IT solutions for storage and virtualization, faced a significant challenge in managing a dynamic environment of virtual machines (VMs). The frequent deployment and deletion of VMs, often within a single day, created a complex scenario that was both time-consuming and difficult to monitor. This challenge was particularly pronounced for Mariusz Mieszkowski, a storage specialist at Comtegra, who noted the inefficiency of performance testing and overall infrastructure management under these conditions.

Transforming Container and Server Health Insights

Thu, 01 Feb 2024 00:00:00 +0000

Revolutionizing Infrastructure Monitoring

Nodecraft faced significant challenges in monitoring its ever-expanding fleet of dedicated servers and containers. The complexity of tracking system health and hardware performance became a bottleneck, impacting its ability to deliver the seamless gaming experiences Nodecraft is known for. With a small development team, the creation and maintenance of a custom solution were out of the question. Nodecraft’s journey towards finding a solution led it to Netdata, which has been transformative.

Transforming critical software development

Thu, 01 Feb 2024 00:00:00 +0000

Revolutionizing Vehicle Automation

ADASTEC, a frontrunner in automated driving software for commercial vehicles, utilizes Netdata to ensure the highest standards of reliability and efficiency in their operations. Specializing in SAE Level-4 automation, ADASTEC empowers OEMs to craft modern, automated, and connected vehicles. The challenge of distinguishing crucial alerts within their software stack posed a significant hurdle, as Tayfun Yurdaer, a DevOps specialist at ADASTEC, explains. The essential task was identifying alerts pivotal for the mission-critical applications to respond appropriately, amidst the complexity of their software environment.

Empowering Digital Transformation

Tue, 30 Jan 2024 00:00:00 +0000

Maximizing Resources, Elevating Services

The Falkland Islands Government faced critical challenges in gaining visibility into the resource consumption of its digital assets. This lack of clarity hindered the ability to efficiently manage alerts and optimize resource usage across digital platforms. The integration of better monitoring tools was imperative to overcoming these obstacles and ensuring the seamless provision of government services.

“Not having a clear picture on the resource consumption of our digital assets was a major challenge. With Netdata agent, I am now able to get a comprehensive set of data about each of my nodes, which has revolutionized how we manage resources and handle alerts.”

Revolutionizing Device Monitoring at SafeSize

Tue, 30 Jan 2024 00:00:00 +0000

Enhancing Efficiency & Transparency in Retail

SafeSize, a trailblazer in the retail technology sector, leverages innovative 3D foot scanning and shoe recommendation solutions to enhance the shopping experience in both physical and online stores. However, managing and monitoring devices installed in diverse customer environments posed significant challenges. These devices, crucial for delivering SafeSize’s cutting-edge service, operate in varied network environments with limited access, necessitating a monitoring solution that is both efficient and resource-light.

Revolutionizing Server Monitoring at Tilkal

Tue, 30 Jan 2024 00:00:00 +0000

Tackling Server Monitoring Complexity

At Tilkal SAS, the challenge of aggregating infrastructure information from numerous standalone Linux servers into a singular platform was significant. The complexity was further compounded by the need to incorporate custom metrics emitted by proprietary services, alongside standard system statistics. This requirement for a holistic view of both standard and custom data sources necessitated a versatile and powerful monitoring tool.

“The data we wanted to collect was not only standard monitoring stats but also custom stats emitted by our own services. Netdata’s ease of deployment and its cloud app’s capability to consolidate server stats in one place were exactly what we needed.”

Simplifying Complex Monitoring for Educational Institutions

Tue, 30 Jan 2024 00:00:00 +0000

Navigating a Complex Monitoring Terrain

The Fundación Universitaria Iberoamericana (FUNIBER), with its expansive network of institutions and students across Spain and beyond, faces the unique challenge of managing and configuring monitoring tools across a dynamically evolving IT infrastructure, compounded by the task of integrating data from disparate sources. The organization’s IT landscape is characterized by rapid changes, requiring flexible and adaptable monitoring solutions. Compatibility issues with legacy systems and proprietary data formats further complicate the seamless integration of monitoring data, presenting significant obstacles to maintaining a cohesive and efficient monitoring strategy. Jesús Ramos, a seasoned DevOps professional at FUNIBER, found himself at the crux of these challenges.

BPF vs eBPF: Key Differences Explained For DevOps & SREs

Mon, 29 Jan 2024 00:00:00 +0000

Introduction To BPF & eBPF

For DevOps and Site Reliability Engineers (SREs), understanding and applying the right tools for monitoring and troubleshooting is crucial. Among the various technologies available, BPF (Berkeley Packet Filter) and its extended version, eBPF (extended Berkeley Packet Filter), have emerged as powerful tools. But what exactly are BPF and eBPF, and how do they differ? This article will delve into the basics of both, highlight their differences, and explore their applications in modern DevOps and SRE practices, with a special focus on observability.

Enhancing Digital Media Excellence with Netdata

Mon, 29 Jan 2024 00:00:00 +0000

Navigating Network Complexity

SHED encountered a critical pain point in swiftly and accurately identifying bandwidth hogs within their network. This was a significant issue, as monitoring these elements was essential to pinpoint devices, applications, or users causing excessive bandwidth consumption. The lack of real-time visibility and comprehensive insights into data transfer rates and traffic patterns was a major hurdle in their ability to proactively manage and optimize network resources. This limitation directly impacted their network performance, leading to potential slowdowns or bottlenecks, affecting the efficiency of their operations. Moreover, this challenge posed risks in resource allocation and maintaining a smooth user experience. Additionally, this issue had implications for security, as unidentified bandwidth hogs could indicate security threats or unauthorized activities. Thus, SHED Montreal’s foremost challenge was the effective monitoring of file server and switch activities to swiftly identify and mitigate bandwidth hogs, crucial for ensuring optimal network performance, resource allocation, and maintaining a secure infrastructure.

SRE vs DevOps: Key Differences, Synergies & Roles Explained

Mon, 29 Jan 2024 00:00:00 +0000

If you’re new to IT operations and software development, you’ve likely encountered the terms SRE (Site Reliability Engineering) and DevOps. Both play crucial roles in modern tech organizations, but understanding the differences between SRE and DevOps can be confusing. This article aims to clarify these concepts, explain their roles, and show how they work together to enhance software delivery and system reliability.

A Brief History Of DevOps & SRE

To better understand the philosophies behind DevOps and Site Reliability Engineering (SRE), it helps to look at how each approach emerged.

What Is Cardinality In Databases: A Comprehensive Guide

Mon, 29 Jan 2024 00:00:00 +0000

An Introduction To Cardinality

Cardinality is a fundamental concept in databases that plays a crucial role in designing efficient databases and optimizing query performance. For those new to databases, understanding what is cardinality is essential for effective data management. This comprehensive guide will explain the meaning of cardinality, its types, and its impact on database performance, making it accessible for both beginners and intermediate users.

What Is Cardinality In Databases?

Cardinality in databases refers to the uniqueness of data values contained in a column. It essentially measures how many distinct values exist in a column compared to the total number of rows in a table.

What Is Cloud Management? Ensure Security & Optimize Costs

Mon, 29 Jan 2024 00:00:00 +0000

Cloud computing has become a cornerstone for businesses of all sizes. As more organizations migrate their workloads to the cloud, understanding cloud management and how to maximize efficiency is crucial. But what is cloud management exactly? This article will explore the concept, dive into the various aspects of cloud management, and provide practical tips on how to enhance efficiency, especially for beginners.

What Is Cloud Management?

Cloud management involves the control, orchestration, and administration of cloud computing resources and services. It encompasses a range of tasks, from deploying and monitoring applications to ensuring security and compliance. Effective cloud management is essential for optimizing performance, controlling costs, and maintaining security across cloud environments.

What Is Continuous Profiling & Why It Matters

Mon, 29 Jan 2024 00:00:00 +0000

Introduction to Profiling: A Key Tool for Developers

Profiling is like a health checkup for your software. Just as doctors use various tests to understand how your body is performing, developers use profiling to understand how their applications are running. This process helps to find out which parts of the application are working well and which parts need improvement.

What Is Profiling?

Profiling, in simple terms, is the process of measuring how your program uses resources like CPU time and memory. Think of it as a way to see inside your application and understand what’s going on.

IoT Monitoring Challenges: Key Issues & How To Overcome Them

Thu, 11 Jan 2024 00:00:00 +0000

With the increasing prevalence of IoT devices, which are being used in a wide range of applications, from smart homes and cities to industrial and agricultural systems, monitoring thei performance and health is extremely important. However, it’s essential to remember that monitoring IoT devices involves more than just tracking device-level data. In addition, monitoring data from the IoT platform or application layer is equally important.

We’ll explore some of these topics in more detail and explain how Netdata can play an essential role in the monitoring of such devices, including some hints on how it can be set up for maximum performance in such scenarios.

Introducing Netdata’s Alerts Configuration Manager

Thu, 21 Dec 2023 00:00:00 +0000

Netdata introduces its latest feature, the Alerts Configuration Manager, transforming the way users configure and manage alerts in their Netdata environment. This powerful tool integrates directly into the Netdata Dashboard, offering a streamlined and intuitive interface for both novice and experienced users.

What is the Alerts Configuration Manager?

The Alerts Configuration Manager is an innovative feature available to users with Business subscriptions. It allows for the creation and customization of alerts directly from the Netdata Dashboard, employing a user-friendly UI wizard. This tool simplifies alert configuration, making it accessible and straightforward, even for those who are not deeply technical.

Cost Transparency: The True Cost Of Monitoring

Fri, 01 Dec 2023 00:00:00 +0000

Businesses are increasingly reliant on monitoring tools to ensure the seamless performance and reliability of their systems. However, the true cost of implementing and maintaining these tools is often obscured by hidden expenses. Our previous blog delved into the concealed costs associated with various monitoring solutions, such as Prometheus & Grafana (Open Source Monitoring) and commercial platforms like Datadog, Dynatrace, and NewRelic. These costs can manifest in various forms - from complex setups and maintenance to additional charges for advanced features.

Understanding the Netdata Methodology

Sat, 04 Nov 2023 00:00:00 +0000

In the dynamic landscape of modern infrastructure and multi cloud environments, observing and understanding system performance requires a new breed of tools—ones that keep pace with the ’living’ nature of modern infrastructure. This is the inflection point at which Netdata steps in, and aims to bring a fresh perspective to monitoring.

What does Netdata do differently

There are key “cultural” faults that we believe hold the observability industry back. Let’s take a look through the prism of these faults and understand what Netdata is doing differently to address them.

Netdata Best Practices

Fri, 03 Nov 2023 00:00:00 +0000

Effective system monitoring is non-negotiable in today’s complex IT environments. Netdata offers real-time performance and health monitoring with precision and granularity. But the key to harnessing its full potential lies in the optimization of your setup. Let’s ensure you are not just collecting data, but doing it in the most optimal way while gaining actionable insights from it.

The starting point for optimization is a robust setup. Netdata is engineered for minimal footprint and can run on a wide range of hardware—from IoT devices to powerful servers. Time for a deep dive into each of these key areas and what the best practices you should follow, if you are serious about monitoring and optimizing your Netdata monitoring setup:

System Operators: The Role Of SysOps In IT Infrastructure

Fri, 03 Nov 2023 00:00:00 +0000

System operators know the drill: as the complexity of systems scales, so does the deluge of logs. Traditionally, taming this relentless tide demands a concoction of costly tools and laborious configurations—until now. The dynamic duo of systemd-journal and Netdata is revolutionizing log management, turning what was once a Herculean task into a streamlined, powerful, and surprisingly straightforward process.

Efficient Handling of Volume and Velocity

systemd-journal is built to manage the deluge of data that systems generate, without buckling under the speed and volume of incoming logs. It captures logs at the source, facilitating direct and immediate processing. By utilizing the journal’s native mechanism to send logs to a central server, system operators can bypass the complexities of traditional log centralization methods.

Upcoming Changes To Netdata Cloud Plans

Thu, 02 Nov 2023 00:00:00 +0000

UPDATE: On the 2023-11-08 Node and Dashboard limits will be applied on the Netdata Cloud Community plan, while all current features of the Community plan will remain the same.

What you need to know?

On Netdata Cloud Free Community plan, the number of active nodes that can be concurrently visualized on the Netdata dashboards, as well as the number of active custom dashboards for accounts created after 2023-11-07 will be subject to limits.

Netdata vs Prometheus

Sat, 28 Oct 2023 00:00:00 +0000

In an era dominated by data-driven decision making, monitoring tools play an indispensable role in ensuring that our systems run efficiently and without interruption. When considering tools like Netdata and Prometheus, performance isn’t just a number; it’s about empowering users with real-time insights and enabling them to act with agility.

There’s a genuine need in the community for tools that are not only comprehensive in their offerings but also swift and scalable. This desire stems from our evolving digital landscape, where the ability to swiftly detect, diagnose, and rectify anomalies has direct implications on user experiences and business outcomes. Especially as infrastructure grows in complexity and scale, there’s an increasing demand for monitoring tools to keep up and provide clear, timely insights.

Discover The New Netdata!

Fri, 27 Oct 2023 00:00:00 +0000

Missed the last Netdata updates? Here is what is new:

Explore your systemd-journal logs with Netdata

Netdata got a systemd-journal logs explorer to analyze your systemd-journal logs, directly on their sources. By just installing Netdata on any systemd based system, Netdata automatically finds all the journal sources and presents a powerful dashboard to explore, search, filter and analyze your logs. It works on both individual servers and journal centralization servers.

Improve Your Security With systemd-journal & Netdata

Tue, 24 Oct 2023 00:00:00 +0000

systemd journals play a crucial role in the Linux system ecosystem, and understanding the importance of the logs contained within is essential for both system administrators and developers.

For those unfamiliar, systemd is an init system employed by Linux distributions, initiates the user space and oversees all ensuing processes. One of its key components, systemd journal, assumes a central role in logging system activities and messages, delivering a host of benefits to both system administrators, developers and cyber security engineers. The systemd journal functions as a logging system that gathers, archives, and oversees log messages and event data originating from a diverse array of system components, encompassing the kernel, system services, applications, and user activities.

Monitoring vs Observability: Key Differences

Tue, 24 Oct 2023 00:00:00 +0000

As systems increasingly shift towards distributed architectures to deliver application services, the roles of monitoring and observability have never been more crucial. Monitoring delivers the situational awareness you need to detect issues, while observability goes a step further, offering the analytical depth to understand the root cause of those issues.

Understanding the nuanced differences between monitoring and observability is crucial for anyone responsible for system health and performance. In dissecting these methodologies, we’ll explore their unique strengths, dive into practical applications, and illuminate how to strategically employ each to enhance operational outcomes.

Exploring systemd journal logs with Netdata

Thu, 12 Oct 2023 00:00:00 +0000

Today, we released our systemd journal plugin for Netdata, allowing you to explore, view, search, filter and analyze systemd journal logs.

Like most things about Netdata, this is a zero-configuration plugin. You don’t have to do anything apart from installing Netdata on your systems.This is key design direction for Netdata, since we want Netdata to be able to help even if you install it mid-crisis, while you have an incident at hand.

systemd journal logs

Mon, 09 Oct 2023 00:00:00 +0000

“Why bother with it? I let it run in the background and focus on more important DevOps work.” — a random DevOps Engineer at Reddit r/devops

In an era where technology is evolving at breakneck speeds, it’s easy to overlook the tools that are right under our noses. One such underutilized powerhouse is the systemd journal. For many, it’s a mere tool to check the status of systemd service units or to tail the most recent events (journalctl -f). Others who do mainly container work, ignore even its existence.

Netdata Cloud On Prem

Tue, 26 Sep 2023 00:00:00 +0000

We at Netdata understand that infrastructure monitoring can be a complex maze—high costs, specialized skill sets, scalability, data silos, and more. That’s why we have always aimed to streamline and modernize this critical operation. Today, we’re thrilled to announce the launch of Netdata Cloud On Prem, a ground-breaking solution designed for robust on-prem infrastructure monitoring - it comes with all the Netdata Cloud features you love but fully on prem.

Netdata Cloud On-Prem

While Netdata Cloud never stores any of your metric data on the cloud and just streams it ephemerally while you view a chart, the demand for on premise infrastructure monitoring has never been more pressing. Many large enterprises, governmental organizations, research institutes and critical infrastructures require a level of data privacy, security, and customization that only an on-prem solution can offer.

Netdata QoS Classes monitoring

Tue, 26 Sep 2023 00:00:00 +0000

Netdata monitors tc QoS classes for all interfaces.

If you also use FireQOS it will collect interface and class names.

There is a shell helper for this (all parsing is done by the plugin in C code - this shell script is just a configuration for the command to run to get tc output).

The source of the tc plugin is here. It is somewhat complex, because a state machine was needed to keep track of all the tc classes, including the pseudo classes tc dynamically creates.

Netdata, Prometheus, Grafana Stack

Tue, 26 Sep 2023 00:00:00 +0000

In this blog, we will walk you through the basics of getting Netdata, Prometheus and Grafana all working together and monitoring your application servers. This article will be using docker on your local workstation. We will be working with docker in an ad-hoc way, launching containers that run /bin/bash and attaching a TTY to them. We use docker here in a purely academic fashion and do not condone running Netdata in a container. We pick this method so individuals without cloud accounts or access to VMs can try this out and for it’s speed of deployment.

Process Monitoring vs Console Tools: A Comparison

Tue, 26 Sep 2023 00:00:00 +0000

Netdata reads /proc/<pid>/stat for all processes, once per second and extracts utime and stime (user and system cpu utilization), much like all the console tools do.

But it also extracts cutime and cstime that account the user and system time of the exit children of each process. By keeping a map in memory of the whole process tree, it is capable of assigning the right time to every process, taking into account all its exited children.

Our first ML based anomaly alert

Wed, 13 Sep 2023 00:00:00 +0000

Over the last few years we have slowly and methodically been building out the ML based capabilities of the Netdata agent, dogfooding and iterating as we go. To date, these features have mostly been somewhat reactive and tools to aid once you are already troubleshooting.

Now we feel we are ready to take a first gentle step into some more proactive use cases, starting with a simple node level anomaly rate alert.

Anomaly Rate By Type

Wed, 30 Aug 2023 00:00:00 +0000

We have recently added a more detailed anomaly rate chart to Netdata that breaks out the overall node anomaly rate by type, this lets you more easily see what parts of your infrastructure might be experiencing an uptick in anomalies when you see the overall node anomaly rate increase.

What is `type`?

type is generally the prefix of the chart id in Netdata and controls where charts live within the menu on the overview page, for example the mem.available chart has a type of mem which in part controls why it lives under the “Memory” section of the menu.

Release 1.41: Brand-New UI For Agents & Parents

Thu, 20 Jul 2023 00:00:00 +0000

Netdata Agents and Parents now have a new UI!

Checkout the release meetup video or read on to learn more about the new UI and other features in this release.

Netdata Growth
Release Highlights
Acknowledgements
Contributions
Collectors
Documentation
Packaging/Installation
Health
Exporting
Other Notable Changes
Deprecation notice
- Deprecated in this release
Netdata Release Meetup
Support options

Steady to our schedule, this is another great Netdata release!

Netdata Assistant: Your AI-Powered Troubleshooting Sidekick

Fri, 14 Jul 2023 00:00:00 +0000

Hey there! We’re excited to share a new troubleshooting feature we have added to Netdata, the Netdata Assistant. We’ve built this tool to help you troubleshoot more effectively and with less stress. Let’s dive in.

What’s the Netdata Assistant?

The Netdata Assistant is an AI tool that uses large language models and our community’s knowledge to guide you during troubleshooting.

Here’s a scenario. It’s 3 am and you get an alert. Instead of scrambling to Google what’s going on, you can just click on the assistant button. The Netdata Assistant will give you the lowdown on the alert, why it’s happening, and why you should care. It’ll also guide you on how to troubleshoot it and even offer some handy web links for more info, if you’re interested.

Hidden Costs Of Monitoring: Uncovering Expenses & Solutions

Fri, 07 Jul 2023 00:00:00 +0000

When it comes to monitoring IT infrastructure, the costs you see on the price tag of the tool are often just the tip of the iceberg. Below the waterline, a mass of hidden costs can lurk, which can significantly affect the total cost of ownership.

In this blogpost we will cover the analysis of two traditional monitoring domains, Open Source observability and Commercial Centralized observability solutions, focusing the direct and indirect impacts when implementing these solution. In summary:

Netdata & Ansible example: ML demo room

Fri, 07 Jul 2023 00:00:00 +0000

We are always trying to lower the barrier to entry when it comes to monitoring and observability and one place we have consistently witnessed some pain from users is around adopting and approaching configuration management tools and practices as your infrastructure grows and becomes more complex.

To that end, we have begun recently publishing our own little example ansible project used to maintain and manage the servers used in our public Machine Learning Demo room.

Netdata Parents (Streaming and Replication)

Fri, 30 Jun 2023 00:00:00 +0000

What are they and why do we need them?

A “Parent” is a Netdata Agent, like the ones we install on all our systems, but is configured as a central node that receives, stores and processes metrics data from other Netdata “Child” nodes in our infrastructure.

Netdata Parents are flexible. You can have one big active-active cluster of Netdata Parents, or you can spread a lot of independent Parents across the infrastructure.

Release 1.40: Summary Tiles, Silencing & ML Tweaks

Wed, 14 Jun 2023 00:00:00 +0000

Another release of the Netdata Monitoring solution is here!

Netdata Growth
Release Highlights
- Dashboard Sections’ Summary Tiles
  Added summary tiles to most sections of the fully-automated dashboards, to provide an instant view of the most important metrics for each section.
- Silencing of Cloud Alert Notifications
  Maintenance window coming up? Active issue being checked? Use the Alert notification silencing engine to mute your notifications.
- Machine Learning - Extended Training to 24 Hours
  Netdata now trains multiple models per metric, to learn the behavior of each metric for the last 24 hours. Trained models are persisted on disk and are loaded back on Netdata restart.
- Rewritten SSL Support for the Agent
  Netdata Agent now features a new SSL layer that allows it to reliably use SSL on all its features, including the API and Streaming.
Alerts and Notifications
Visualizations / Charts and Dashboards
Preliminary steps to split native packages
Acknowledgements
Contributions
Deprecation notice
Cloud recommended version
Release meetup
Support options
Running survey

Netdata Growth

🚀 Our community growth is increasing steadily. ❤️ Thank you! Your love and acceptance give us the energy and passion to work harder to simplify and make monitoring easier, more effective and more fun to use.

How Netdata's ML-based Anomaly Detection Works

Tue, 23 May 2023 00:00:00 +0000

How does Netdata’s machine learning (ML) based anomaly detection actually work? Read on to find out!

Design considerations

Lets first start with some of the key design considerations and principles of Netdata’s anomaly detection (and some comments in parenthesis along the way):

We don’t have any labels or examples of previous anomalies. This means we are in an unsupervised setting (best we can try to do is learn what “normal” data looks like assuming the collected data is “mostly” normal).
Needs to be lightweight and run on the agent (or a parent).
- Need to be very careful of impact on CPU overhead when training and scoring (lots of cheap models are better than a few expensive and heavy ones).
- Models themselves need to be small so as to not drastically increase the agents memory footprint (model objects need to be small for storage).
- This has implications for the ML formulation (sorry - no deep learning models yet) and its implementation (we need to be surgical and optimized).
Needs to scale for thousands of metrics and score in realtime every second as metrics are collected.
- Typical Netdata nodes have thousands of metrics and we want to be able to score every metric every second with minimal latency overhead (we need to use sensible approaches to training like spreading the training cost over a wide training window).
Needs to be able to handle a wide variety of metrics.
- There is no single perfect model or approach for all types of metrics so we need a good all rounder that can work well enough across any and all different types of time seres metrics (for any given metric of course you could handcraft a better model but thats not feasible here, we need something like a “weak learners” approach of lots of generally useful models adding up to “more than the sum of their parts”).
Needs to be written in C or C++ as that is the language of the Netdata agent (We are using dlib for the current implementation).
We Need to be very careful about taking big or complex dependencies if using third party libraries.
- We want to be able to easily build and deploy Netdata on any Linux system without having to worry about installing or managing complex dependencies (we need to be careful of more complex algorithms that would have larger dependencies and potentially limit where Netdata can run).

The above considerations are important and useful to keep in mind as we explore the system in more detail.

Revolutionizing Ops Centers With Real-Time Monitoring

Fri, 19 May 2023 00:00:00 +0000

In today’s fast-paced digital landscape, 24-hour operations centers play a crucial role in managing and monitoring large-scale infrastructures. These centers must be equipped with an effective monitoring solution that addresses their unique needs, enabling them to respond quickly to incidents and maintain optimal system performance. Netdata, a comprehensive monitoring solution, has been designed to meet these critical requirements with its advanced capabilities and recent enhancements.

In this article, we will explore how Netdata’s powerful features can transform the way 24-hour operations centers monitor and manage their complex environments, leading to improved incident detection, faster troubleshooting, and better overall system performance.

The Future Of Infrastructure Monitoring: Scalability & AI

Fri, 19 May 2023 00:00:00 +0000

In this blog post, we will explore the importance of scalability, automation, and AI in the evolving landscape of infrastructure monitoring. We will examine how Netdata’s innovative solution aligns with these emerging trends, and how it can empower organizations to effectively manage their modern IT infrastructure.

In today’s increasingly complex IT landscape, the need for efficient and reliable infrastructure monitoring has never been more critical. With the proliferation of microservices, distributed systems, and cloud-native applications, managing and monitoring the performance of these rapidly evolving environments has become a significant challenge. As a result, infrastructure monitoring solutions must adapt to keep pace with these changes and deliver the insights necessary to maintain optimal performance.

Monitoring Multi-Cloud & Hybrid-Cloud Infrastructures

Tue, 16 May 2023 00:00:00 +0000

The advent of multi-cloud and hybrid-cloud architectures has created new opportunities for organizations to leverage best-in-class features from various cloud service providers. However, these complex environments present their own unique challenges, especially when it comes to monitoring and managing performance.

Visibility

The visibility challenge in multi-cloud and hybrid-cloud environments often stems from the use of disparate monitoring tools that are native to each cloud provider. While these native tools (like Amazon CloudWatch, Google Cloud Monitoring, and Azure Monitor) are excellent within their respective ecosystems, they don’t necessarily play well together when it comes to consolidating data and providing a comprehensive, unified view of your entire infrastructure.

Cloud Optimization: Cost, Performance & Resource Strategies

Sun, 14 May 2023 00:00:00 +0000

Cloud optimization is the ongoing process of analyzing, configuring, and refining cloud environments to improve performance, reduce costs, and align resource usage with business needs. As cloud adoption grows, organizations must move beyond cost-cutting alone and treat optimization as a strategic practice.

Cloud Optimization Strategies To Achieve Business Goals

Cloud optimization strategies generally focus on cost control, performance enhancement, and efficient resource utilization. These strategies range from selecting the right cloud service model (IaaS, PaaS, or SaaS), right-sizing your resources, adopting a multi-cloud approach, automating processes, and investing in robust monitoring tools that can reliably reveal resources utilization and help you ensure that services are tailored to meet business objectives.

Migrating To Cloud: Key Challenges & Best Practices

Sun, 14 May 2023 00:00:00 +0000

Embarking on a cloud migration journey? Grasp the obstacles and arm yourself with best practices for a smooth transition. Success lies in understanding, planning, and adapting.

As we continue to advance further into the 21st century, businesses of all sizes are finding themselves in the midst of a digital revolution. At the heart of this transformation lies cloud migration, a process that has become a critical strategic decision for organizations aiming to remain competitive, innovative, and responsive to fluctuating market dynamics.

Transform Monitoring With A Machine Learning Approach

Thu, 11 May 2023 00:00:00 +0000

Unlocking the full potential of monitoring through ML integration, anomaly detection, and innovative scoring engines.

Machine Learning has been making waves in various industries, but its adoption in the monitoring and observability space has been slower than expected. Many “ML” features remain gimmicky and do not provide actual real world value to users that encourages their further use.

At Netdata, we firmly believe that ML is crucial for monitoring, and we’ve taken an ML-first approach to provide users with powerful tools and insights. In this blog post, we’ll discuss the reasons behind our belief in ML, how we’ve integrated ML into our charts and visualizations, our query engines, the scoring engine we’ve built, and how these innovations enable metrics correlations and anomaly advisor.

The Future of Monitoring is Automated and Opinionated

Tue, 09 May 2023 00:00:00 +0000

So, you think you monitor your infra?

As humanity increasingly relies on technology, the need for reliable and efficient infrastructure monitoring solutions has never been greater.

However, most businesses don’t take this seriously. They make poor choices that soon trap their best talent, the people who should be propelling them ahead of their competition.

Consider this: most of the world believes that each company needs to dedicate time, talent, and money to configure and set up the monitoring of their web servers and database servers from scratch!

Release 1.39.0: A new era for monitoring charts.

Mon, 08 May 2023 00:00:00 +0000

Another release of the Netdata Monitoring solution is here!

Netdata open-source growth
Release highlights
- Netdata Charts v3.0 A new era for monitoring charts. Powerful, fast, easy to use. Instantly understand the dataset behind any chart. Slice, dice, filter and pivot the data in any way possible!
- Windows support Windows hosts are now first-class citizens. You can now enjoy out-of-the-box monitoring of over 200 metrics from your Windows systems and the services that run on them.
- Virtual nodes and custom labels You now have access to more monitoring superpowers for managing medium to large infrastructures. With custom labels and virtual hosts, you can easily organize your infrastructure and ensure that troubleshooting is more efficient.
- Major upcoming changes Separate packages for data collection plugins, mandatory zlib, no upgrades of existing installs from versions prior to v1.11.
- Bar charts for functions
- Opsgenie notifications for Business Plan users Business plan users can now seamlessly integrate Netdata with their Atlassian Opsgenie alerting and on call management system.
Data Collection
Alerts and Notifications
- Notifications
- Enhanced Real-Time Alert Synchronization on Netdata Cloud
Visualizations / Charts and Dashboards
- Events Feed
Machine Learning
Installation and Packaging
- Improved Linux compatibility
Administration
- New way to retrieve netdata.conf
Documentation and Demos
Deprecation notice
- Deprecated in this release
Netdata Agent Release Meetup
Support options
Running survey
Acknowledgements

Netdata open-source growth

Over 62,000 GitHub Stars
Over 1.5 million online nodes
Almost 92 million sessions served
Over 600 thousand total nodes in Netdata Cloud

Release highlights

Netdata Charts v3.0

We are excited to announce Netdata Charts v3.0 and the NIDL framework. These are currently available at Netdata Cloud. At the next Netdata release, the agent dashboard will be replaced to also use the same charts.

Infinite Scalability: Monitoring Without Limits

Thu, 04 May 2023 00:00:00 +0000

Scalability is crucial for monitoring systems as it ensures that they can accommodate growth, maintain performance, provide flexibility, optimize costs, enhance fault tolerance, and support informed decision-making, all of which are critical for effective infrastructure management.

Most monitoring solutions struggle with scalability, mainly because of:

High data volume and velocity: Monitoring systems generate vast amounts of data and as the infrastructure grows, so does the volume and velocity of these data.
Resource constraints: Scalability requires efficient resource utilization, leading to bottlenecks and performance issues as the monitored environment grows.
Architectural limitations: Monitoring systems are usually designed with certain architectural constraints that limit their scalability. Most open source solutions rely on monolithic or centralized architectures that can become overwhelmed at scale.

For open source solutions scalability has always been a challenge, increasing their complexity significantly (check for example the scalability issues of Prometheus), while for commercial solutions it usually results in increased data collection to visualization latency and cost.

Monitoring Disks: Workload, Latency & Saturation

Thu, 04 May 2023 00:00:00 +0000

Netdata provides a comprehensive set of charts that can help you understand the workload, performance, utilization, saturation, latency, responsiveness, and maintenance activities of your disks. In this blog we will focus on monitoring disks as block devices, not as filesystems or mount points.

The Disks section in the Overview tab contains all the charts that are mentioned in this blog post.

Disk Workload and Performance

Netdata charts for monitoring the workload and the throughput of your disks:

Understanding Huge Pages

Thu, 04 May 2023 00:00:00 +0000

Memory-intensive applications can benefit from improved performance by using huge pages, as they can reduce TLB pressure and memory fragmentation, and lower the memory management overhead overall. Developers should consider using HugeTLBfs in their mmap() and shmget() calls to take advantage of huge pages.

Transparent Huge Pages (THP) is a Linux kernel feature that provides some of the benefits of huge pages without requiring any development effort. However, THP can cause latency in many applications. Although kernel developers are actively working to address these issues, many system administrators prefer to disable THP altogether.

Unlock the Secrets of Kernel Memory Usage

Thu, 04 May 2023 00:00:00 +0000

The mem.kernel chart in Netdata provides insight into the memory usage of various kernel subsystems and mechanisms. By understanding these dimensions and their technical details, you can monitor your system’s kernel memory usage and identify potential issues or inefficiencies. Monitoring these dimensions can help you ensure that your system is running efficiently and provide valuable insights into the performance of your kernel and memory subsystem.

Slab

The slab allocator is a memory management mechanism introduced by Jeff Bonwick in 1994 to manage memory allocation for kernel objects. The main purpose of the slab allocator is to reduce memory fragmentation and improve the speed of memory allocation/deallocation. The slab allocator groups objects of the same size into “slabs” and caches the objects to speed up future allocations.

Entropy In Cryptography: Key To Security & Randomness

Wed, 03 May 2023 00:00:00 +0000

Entropy is a measure of the randomness or unpredictability of data. In the context of cryptography, entropy is used to generate random numbers or keys that are essential for secure communication and encryption. Without a good source of entropy, cryptographic protocols can become vulnerable to attacks that exploit the predictability of the generated keys.

What is Entropy?

In most operating systems, entropy is generated by collecting random events from various sources, such as hardware interrupts, mouse movements, keyboard presses, and disk activity. These events are fed into a pool of entropy, which is then used to generate random numbers when needed.

Context Switching & Its Impact On System Performance

Tue, 02 May 2023 00:00:00 +0000

Context switching is the process of switching the CPU from one process, task or thread to another. In a multitasking operating system, such as Linux, the CPU has to switch between multiple processes or threads in order to keep the system running smoothly. This is necessary because each CPU core without hyperthreading can only execute one process or thread at a time. If there are many processes or threads running simultaneously, and very few CPU cores available to handle them, the system is forced to make more context switches to balance the CPU resources among them.

Linux CPU Consumption, Load & Pressure Explained

Tue, 02 May 2023 00:00:00 +0000

As a system administrator, understanding how your Linux system’s CPU is being utilized is crucial for identifying bottlenecks and optimizing performance. In this blog post, we’ll dive deep into the world of Linux CPU consumption, load, and pressure, and discuss how to use these metrics effectively to identify issues and improve your system’s performance.

CPU Consumption and Utilization

CPU consumption refers to the amount of processing power being used by applications running on your system. The system.cpu chart in Netdata represents the Total CPU utilization of your Linux system, broken down into different dimensions. Each dimension provides insight into how the CPU is being used by various tasks and processes. Here’s a brief explanation of each dimension:

Server Uptime Monitoring: Core Benefits For High Performance

Tue, 02 May 2023 00:00:00 +0000

Server uptime monitoring tracks the availability and reliability of servers within your infrastructure.

What Is Server Uptime Monitoring?

Server uptime monitoring is the process of continuously tracking the operational status of your servers to ensure optimal performance and availability for users.

With Netdata, you gain access to real-time, high-resolution monitoring that goes beyond basic checks, providing a detailed overview of your entire infrastructure.

Swap Memory: When & How To Use It On Production VMs

Tue, 02 May 2023 00:00:00 +0000

Swap memory, also known as virtual memory, is a space on a hard disk that is used to supplement the physical memory (RAM) of a computer. The swap space is used when the system runs out of physical memory, and it moves less frequently accessed data from RAM to the hard disk, freeing up space in RAM for more frequently accessed data. But should swap memory be enabled on production systems and cloud-provided virtual machines (VMs)? Let’s explore the pros and cons.

Understanding Interrupts, Softirqs, and Softnet in Linux

Tue, 02 May 2023 00:00:00 +0000

Interrupts, softirqs, and softnet are all critical parts of the Linux kernel that can impact system performance. In this blog post, we’ll explore their usefulness, and discuss how to monitor them using Netdata for both bare-metal servers and VMs.

What are Interrupts?

Interrupts are signals generated by hardware devices to indicate that they require attention from the CPU. Hardware devices can generate interrupts for a variety of reasons, including data transmission or reception, input/output operations, and other activities. When an interrupt is generated, the CPU stops what it is doing and handles the interrupt. Interrupts can have a significant impact on system performance, especially if there are a high number of interrupts occurring.

Understanding System Processes States

Tue, 02 May 2023 00:00:00 +0000

The different states of system processes are essential to understanding how a computer system works. Each state represents a specific point in a process’s life cycle and can impact system performance and stability.

Process States

Netdata’s system.processes_state chart provides a view of these states, allowing users to monitor system performance in real-time:

Running: A process is in the Running state when it is actively using the CPU and executing instructions. This state is resource-intensive and can lead to performance issues if there are too many Running processes, causing CPU contention and system slowdowns. Processes in the Running state are prioritized using scheduling algorithms to improve system performance.

Why Scalable Monitoring Matters For Modern Systems

Wed, 26 Apr 2023 00:00:00 +0000

It’s becoming increasingly common to discuss the importance of scalability in monitoring solutions and how it can impact the performance and reliability of distributed systems.

In today’s rapidly evolving technological landscape, organizations are increasingly relying on distributed systems to power their operations. These systems consist of multiple interconnected components that work together to deliver a cohesive experience. They can span across different geographic locations, and often involve a combination of on-premises, cloud, and container-based environments. As such, effectively managing these complex systems is critical to ensuring optimal performance, reliability, and security.

Netdata's AI Insights & Rapid Diagnostics

Wed, 19 Apr 2023 00:00:00 +0000

Introduction to Netdata’s new visualisation providing AI Insights, supporting Rapid Diagnostics.

A New Era in Monitoring Systems Dashboards

We’re thrilled to share an important upgrade to Netdata: AI Insights & Rapid Diagnostics, a technology aiming to redefine what we expect from a monitoring system.

Challenges with Traditional Monitoring Dashboards

Traditional monitoring systems rely on a query language to help engineers create dashboards and alerts. While these languages offer power and flexibility, they come with several challenges that make monitoring and troubleshooting more complex and time-consuming:

Remote UNIX System Monitoring Using Net-SNMP

Wed, 12 Apr 2023 00:00:00 +0000

Need to monitor a UNIX-like system, but can’t install Netdata on it? With our SNMP collector and Net-SNMP, you can get basic system information with just a bit of relatively quick and easy configuration.

What is SNMP?

The Simple Network Management Protocol, commonly known as SNMP, is a relatively lightweight protocol designed for monitoring and configuration management for network appliances like switches, routers or gateways. However, it can also be used for those purposes on almost any UNIX-like system thanks to the Net-SNMP project.

Anomaly Rates in the Menu!

Wed, 29 Mar 2023 00:00:00 +0000

The menu (on the overview or single node tab) now has an anomaly rate button built into it that, for the entire visible window or a highlighted time range, shows the maximum chart anomaly rate within each section.

Read on to learn more about this new feature!

Wait, what is an anomaly rate?

Netdata is the only monitoring agent that natively (for every metric, with zero config and sane defaults) produces anomaly rates in addition to just collecting raw metrics.

Introducing the Netdata demo space

Fri, 24 Mar 2023 00:00:00 +0000

Introducing Netdata’s Demo Space, a quick and easy way to experience monitoring environments before you set them up yourself.

At Netdata, we are always striving to provide the best monitoring experience for our users. We understand that adopting a new monitoring solution can sometimes be challenging, especially when you’re unsure of how it will fit your specific environment. That’s why we’re excited to announce the Netdata Demo Space!

Upcoming Changes to Plugins in Native Packages

Wed, 15 Mar 2023 00:00:00 +0000

At Netdata, we’re committed to trying to make Netdata work as well as possible for our users. Sometimes though, that means changing things in ways that aren’t exactly seamless. Such a change is coming soon for users of our native DEB and RPM packages, and this blog post will explain what’s happening, why we’re doing it, and what it means for our users.

What’s changing?

Starting shortly after the v1.39.0 release of the Netdata Agent, we will be splitting most of our external data collection plugins out to their own individual packages instead of bundling them all in the main netdata package. We already have this type of split for our CUPS and FreeIPMI plugins, and this new change will extend that to also provide separate packages for the following plugins:

Windows Server Monitoring Improvements

Mon, 13 Mar 2023 00:00:00 +0000

Monitor your Windows server and applications running on it with Netdata - simple, powerful and free.

Hey Netdata community,

We have some exciting news for you: we’re launching our new and updated Windows collectors with the goal of making the Windows monitoring experience as seamless as possible 🎉

We know that Windows monitoring has been a long time ask from many of you, and we’ve been working hard to make it easier than ever to monitor your Windows metrics with Netdata.

Anomaly detection on Prometheus metrics

Wed, 01 Mar 2023 00:00:00 +0000

We have recently extended the native machine learning (ML) based anomaly detection capabilities of Netdata to support all metrics, regardless on their collection frequency (update every).

Previously only metrics collected every second were supported, but now Netdata can run anomaly detection out of the box with zero config on metrics with any collection frequency.

This post will illustrate an example of what this means using Prometheus metrics (via the Netdata Prometheus collector) since they typically have a default collection frequency of 10 seconds.

Monitor any SQL metrics with Netdata (and Pandas ❤️)

Wed, 22 Feb 2023 00:00:00 +0000

We recently got this great feedback from a dear user in our Discord:

I would really like to use Netdata to monitor custom internal metrics that come from SQL, not a fan of having 10 diff systems doing essentially the same thing as is, Netdata is pretty much all there in that regard, just needs a few extra features.

This is great and exactly what we want, a clear problem or improvement we could make to help make that users monitoring life a little easier.

Introducing Netdata Functions (↑ Top)

Wed, 15 Feb 2023 00:00:00 +0000

Netdata is committed to making it simpler and easier for everyone to monitor and troubleshoot their infrastructure. With that goal in mind, we’re excited to announce the launch of our new “Functions” feature (↑ Top), which allows Netdata Agent collectors to expose “functions” that can be executed in run-time and on-demand.

What are Netdata functions?

Netdata has always been synonymous with real time monitoring and automated dashboards, with the recent introduction of “functions”, there’s now a new way for users to troubleshoot their infrastructure. A function, in the context of Netdata, is a routine or script that can be invoked to run on a node and retrieve useful information, which is then displayed in the Netdata cloud dashboard.

Introducing Netdata Paid Subscriptions

Fri, 10 Feb 2023 00:00:00 +0000

All Netdata functionality is and will be available for free forever in the Community Plan. Paid tiers include features targeted for businesses and users who would need to customise their monitoring solution with different levels of user access, extra notification mechanisms, customer support and more.

Hello Netdata community,

We are excited to announce that we are introducing new Paid Subscriptions to Netdata as of Wednesday, 22nd of February 2023.

Release 1.38: Dramatic Performance & Stability Gains

Mon, 06 Feb 2023 00:00:00 +0000

Another release of the Netdata Monitoring solution is here!

Release Highlights
- DBENGINE v2 The new open-source database engine for Netdata Agents, offering huge performance, scalability and stability improvements, with a fraction of memory footprint!
- FUNCTION: Processes Netdata beyond metrics! We added the ability for runtime functions, that can be implemented by any data collection plugin, to offer unlimited visibility to anything, even not-metrics, that can be valuable while troubleshooting.

Extending Netdata's anomaly detection training window

Thu, 02 Feb 2023 00:00:00 +0000

We have been busy at work under the hood of the Netdata agent to introduce new capabilities that let you extend the “training window” used by Netdata’s native anomaly detection capabilities.

This blog post will discuss one of these improvements to help you reduce “false positives” by essentially extending the training window by using the new (beautifully named) number of models per dimension configuration parameter.

Background

One of the most important considerations of our native anomaly detection capabilities is the overhead of running the training and scoring computations required to train thousands of models (one per metric) and produce anomaly bits every second based on those trained models.

Release 1.37.1: Patch Release For Security Issues

Mon, 05 Dec 2022 00:00:00 +0000

Netdata v1.37.1 is a patch release to address issues discovered since v1.37.0. Refer to the v.1.37.0 release notes for the full scope of that release.

Release v1.37.1

Netdata v1.37.1 is a patch release to address issues discovered since v1.37.0. Refer to the v.1.37.0 release notes for the full scope of that release.

The v1.37.1 patch release fixes the following issues:

Parent agent crash when many children instances (re)connect at the same time, causing simultaneous SSL re-initialization (PR #14076).
Agent crash during dbengine database file rotation while a page is being read while being deleted (PR #14081).
Agent crash on metrics page alignment when metrics were stopped being collected for a long time and then started again (PR #14086).
Broken Fedora native packages (PR #14082).
Fix dbengine backfilling statistics (PR #14074).

In addition, the release contains the following optimizations and improvements:

Release 1.37: Infinite Scalability & Database Tiering

Wed, 30 Nov 2022 00:00:00 +0000

Another release of the Netdata Monitoring solution is here!

We focused on these key areas:

Infinite scalability of the Netdata Ecosystem

Default Database Tiering, offering months of data retention for typical Netdata Agent installations with default settings and years of data retention for dedicated Netdata Parents.

Overview Dashboards at Netdata Cloud got a ton of improvements to allow slicing and dicing of data directly on the UI and overcome the limitations of the web technology when thousands of charts are presented on one page.

Monitor & Troubleshoot ISP Performance With Netdata

Mon, 28 Nov 2022 00:00:00 +0000

Find out how to monitor your Internet speed and quality and how well your ISP is performing.

What Factors Affect My Internet Speed?

Several factors can influence your internet speed, ranging from your ISP’s infrastructure to your home setup. Here’s a breakdown:

1. ISP Plan & Bandwidth

The speed you experience depends on the plan you choose from your ISP. Higher-tier plans offer more bandwidth, which means faster speeds for downloading, streaming, and gaming.

How to monitor node reboots?

Thu, 17 Nov 2022 00:00:00 +0000

Monitoring the health and status of nodes and servers is a critical part of effective infrastructure monitoring.

How to monitor node reboots?

One of the most critical tasks of monitoring an infrastructure is to check the health of its servers/nodes. In most cases, this results in setting up a “Hardware manager” from the hardware vendor delivering these servers or setting up an SNMP (or similar) agent to continuously monitor the availability of the server and report when there is a reboot / failure.

How To Mute Alerts During Maintenance Windows

Thu, 03 Nov 2022 00:00:00 +0000

The health management APIs in Netdata allows teams to eliminate unnecessary alerting during scheduled maintenance, testing, auto scaling events, and instance reboots.

For all SREs, it is absolutely crucial to filter out expected events during maintenance windows and quickly pinpoint critical issues in your infrastructure. Every minute is crucial while dealing with troubleshooting issues and any distractions that may hijack the troubleshooting process should be subdued. The health management APIs in Netdata allows teams to eliminate unnecessary alerting during scheduled maintenance, testing, auto scaling events, and instance reboots.

Monitor indoor air quality with Airthings and Netdata

Wed, 02 Nov 2022 00:00:00 +0000

Monitoring indoor air quality with Airthings and Netdata. Understanding and measuring common contaminants and pollutants reduces your risk of air quality health concerns.

Indoor air quality and what to monitor

Indoor air quality can be a crucial influence on your health, wellbeing and productivity.

Understanding and measuring common contaminants and pollutants is the first step towards reducing your risk of air quality health concerns.

Airthings is a company that makes great air quality sensors that measure a wide variety of different variables including:

Monitor KSM performance with Netdata

Tue, 01 Nov 2022 00:00:00 +0000

Monitoring KSM (Kernel Same-page Merging) performance at deduping memory shared across VMs.

Kernel Same-page Merging (KSM)

Linux kernels store memory in pages which are moved in and out of memory as a single block. On most Linux architectures pages are 4096 bytes. KSM (Kernel Same-page Merging) is a kernel feature that scans memory looking for pages with identical content, and then de-duplicates them. The most common use-case where such duplicate pages occur is on hosts running multiple virtual machines (VMs).

Monitoring & troubleshooting Cassandra with Netdata

Sat, 29 Oct 2022 00:00:00 +0000

How to monitor and troubleshoot Cassandra with Netdata.

Note: This post is the second part of a Cassandra monitoring series. Be sure to read our first entry here.

Monitoring Cassandra with Netdata

Netdata’s Cassandra collector documentation explains how to set it up to collect metrics automatically.

Once you have followed the instructions in the docs and have installed and configured Netdata on the Cassandra cluster you are ready to start monitoring and troubleshooting. Check out the Cassandra demo room to interact with the charts, metrics and other functionality described here.

How to monitor and fix Database bloats in PostgreSQL?

Fri, 28 Oct 2022 00:00:00 +0000

Database bloat is disk space that was used by a table or index and is available for reuse by the database but has not been reclaimed. Bloat is created when deleting or updating tables and indexes. Here’s how to deal with it!

What is Database bloat?

Database bloat is disk space that was used by a table or index and is available for reuse by the database but has not been reclaimed. Bloat is created when deleting or updating tables and indexes.

Cassandra Monitoring: Key Metrics & Best Practices

Thu, 27 Oct 2022 00:00:00 +0000

What are the important Cassandra metrics to monitor and how to monitor them.

What Is Cassandra & Why Use It

Cassandra is an open-source, distributed, wide-column NoSQL database management system written in Java. Cassandra was originally developed by Avinash Lakshmanan and Prashant Malik at Facebook and then released as open source, eventually becoming part of the Apache project.

Cassandra is a NoSQL database - NoSQL (also known as “not only SQL”) databases do not require data to be stored in tabular format. They provide flexible schemas and scale easily with large amounts of data and high user loads.

How to find out which application is causing server load

Wed, 26 Oct 2022 00:00:00 +0000

We often hear the term load used to describe the state of a server or a device, but we’re here to tell you what it means, precisely, and how to monitor it.

What is server load?

We often hear the term “load” used to describe the state of a server or a device. But what does it really mean?

System load is a measure of the amount of computational work that a system performs. An overloaded system, by definition, isn’t able to complete all its tasks per schedule - this affects the performance and productivity of the system. And while “load” often gets conflated with CPU usage there’s a lot more to it.

How to monitor the disk usage on your infrastructure

Tue, 25 Oct 2022 00:00:00 +0000

The most important part of disk usage monitoring is to check the utilization of each filesystem and each mount point which can reveal existing or impending issues with the storage space on your infrastructure.

What Does Disk Usage (DU) Mean?

Disk usage (DU) refers to the portion or percentage of computer storage that is currently in use. It contrasts with disk space or capacity, which is the total amount of space that a given disk is capable of storing. Disk usage is a crucial metric to any computing system, as it gives the user the information needed not only for storage, but also software requirements and overall operation. Although it usually refers to a computer’s hard disk, it may also refer to external storage, such as a USB drive or compact disc (CD).

7 types of Redis latency and how to fix it

Mon, 24 Oct 2022 00:00:00 +0000

Redis is designed to be fast. In most cases, it is. However, there are times when Redis may be slow, due to network issues, disk latency, or other factors. When this happens, it is important to be able to detect the slow down and investigate the cause of Redis latency.

Redis & latency

Latency is the maximum delay between the time a client issues a command and the time the reply to the command is received by the client. Redis has strict requirements on average and worst case latency.

How to monitor systemd service liveness

Fri, 21 Oct 2022 00:00:00 +0000

The life of a sysadmin or SRE is often difficult, but occasionally very simple things can make a huge difference. Basic monitoring of your systemd services is one of those simple things, which we sometimes overlook. The simplest question one would want to know is if the thing that’s supposed to be running is actually running at all. If you use systemd services, you can guarantee an answer to that question within minutes using Netdata.

Web Servers Monitoring: Key Metrics & Strategies

Thu, 20 Oct 2022 00:00:00 +0000

The Importance Of Monitoring Web Servers

Web servers are among the most important components in modern IT infrastructures. They host the websites, web services, and web applications that we use on a daily basis. Social networking, media streaming, software as a service (SaaS), and other activities wouldn’t be possible without the use of web servers. And with the advent of cloud computing and the movement of more services online, web servers and their monitoring are only becoming more important. Given the extensive usage of Web servers, Sysadmins and SREs should monitor web servers as a key aspect for performance.

Using Pandas In Python: Data Analysis & Performance Insights

Wed, 19 Oct 2022 00:00:00 +0000

Netdata just got a Pandas collector.

Pandas is a de-facto standard in reading and processing most types of structured data in Python so if you have some csv/json/xml data, either locally or via some HTTP endpoint, containing metrics you’d like to monitor, chances are you can now easily do this by leveraging the Pandas collector without having to develop your own custom collector as you might have in the past.

How to monitor HTTP endpoints

Mon, 17 Oct 2022 00:00:00 +0000

The HTTP protocol has become the de facto standard application layer protocol of the internet. From publicly available web sites and APIs to “inter-process” communications in REST based microservice architectures or large Service Oriented Architectures based on SOAP, you find HTTP being used again and again, due to its simplicity and our familiarity with it. How many protocols can you name that have memes for their status codes? Of course, such a popular protocol has endless pages written about how to properly monitor the services that rely on it, with many options specific to every use case. What you will learn here is how to get your basics done in monitoring HTTP endpoints, so you can be up and running in a few minutes, monitoring all HTTP services in your entire infrastructure.

How to monitor DNS query response time

Wed, 12 Oct 2022 00:00:00 +0000

DNS (Domain Name System) servers translate standard language web addresses to their actual IP addresses for network access.

DNS Lookup Journey

DNS response time is the time it takes a Domain Name Server to receive the request for a domain name’s IP address, process it, and return the IP address to the browser or application requesting it. When it comes to DNS response times, the lower the better, and generally values less than 100ms are considered to be in the acceptable range (depending on the application).

Why is data replication important?

Wed, 12 Oct 2022 00:00:00 +0000

High availability. This is what every monitoring tool needs to ensure that you never compromise on IT infrastructure visibility. On top of high availability, do you really want to enable all available features on your production system? It is important for the monitoring tool to have a low footprint on your CPU consumption and memory usage. Let’s dive deeper into the recommended way of configuring Netdata to ensure high availability and a low resource footprint through data replication.

How to monitor host reachability

Mon, 10 Oct 2022 00:00:00 +0000

Most sysadmins and developers have at some point used a few of the popular Linux networking commands or their Windows equivalents to answer the common questions of host reachability - that is, whether a host or service is reachable and how fast it responds.

Common approaches to reachability

One of the simplest, common checks, is to simply ping a host to verify that it’s reachable from where you issue the command, and to see the total time it takes for the host to receive your request.

Introducing the Netdata Source Plugin for Grafana

Fri, 07 Oct 2022 00:00:00 +0000

The open-source community is about to benefit greatly from Netdata’s new Grafana data source plugin, which makes use of a powerful data collection engine.

This new plugin maximizes the troubleshooting capabilities of Netdata in Grafana, making them more widely available. Some of the key capabilities provided to you with this plugin include the following:

Real-time monitoring with single-second granularity.
Installation and out-of-the-box integrations available in seconds from one line of code.
2,000+ metrics from across your whole Infrastructure, with insightful metadata associated with them.
Access to our fresh ML metrics (anomaly rates) - exposing our ML capabilities at the edge!

Why did we decide to do it?

We are huge fans of Open-Source culture. Open-source is deeply rooted in Netdata’s DNA. Because of this, at Netdata, we don’t really buy into the “single pane of glass” or “observability platform” buzzwords. The reality is that things are just more complicated than that in real life.

How to filter metrics by label?

Thu, 06 Oct 2022 00:00:00 +0000

It is sometimes easy to get lost in the mountain of metrics and infinite number of dimensions when working with an infrastructure monitoring tool. Being able to filter metrics by label and visualize only what is relevant to the current scope of monitoring &troubleshooting, becomes absolutely crucial to the success of SREs, Sysadmins and DevOps professionals.

The Netdata chart label filtering feature supports grouping by and filtering each chart based on labels (key/value pairs) applicable to the context and provides fine-grain capability on slicing the data / metrics.

Missing indexes in PostgreSQL? How to quickly identify it

Wed, 05 Oct 2022 00:00:00 +0000

While working on improving the Netdata PostgreSQL collector, we were monitoring our production PostgreSQL instance and something caught our attention immediately. The rows fetched ratio seemed really, really low for one particular database… there were missing indexes in PostgreSQL!

Rows fetched ratio is the percentage of rows that contain data needed to execute the query (rows fetched), out of the total number of rows scanned (rows returned). A low value indicates that the database is performing extra work by scanning a large number of rows that aren’t required to process the query.

Data Collection Strategies For Infrastructure

Tue, 06 Sep 2022 00:00:00 +0000

Monitoring and troubleshooting; unfortunately, these terms are still used interchangeably, which can lead to misunderstandings about data collection strategies.

In this article we aim to clarify some important definitions, processes, and common data collection strategies for monitoring solutions. We will specify the limitations of the described strategies, as well as key benefits which can potentially be also used for troubleshooting needs.

IT infrastructure monitoring is a business process of collecting and analyzing data over a period of time to improve business results.

How Netdata’s Machine Learning works

Thu, 01 Sep 2022 00:00:00 +0000

Following on from the recent launch of our Anomaly Advisor feature, and in keeping with our approach to machine learning, here is a detailed Python notebook outlining exactly how the machine learning powering the Anomaly Advisor actually works under the hood.

Or if you’d rather watch a video walkthrough of the notebook then check out below.

Try it for yourself, get started by signing in to Netdata and connecting a node. Once initial models have been trained (usually after the agent has about one hour of data, zero configuration needed), you’ll be able to start exploring in the Anomaly Advisor tab of Netdata.

Anomaly rate in every chart

Thu, 23 Jun 2022 00:00:00 +0000

A month ago, we introduced unsupervised ML & Anomaly Detection in Netdata, the Anomaly Advisor. Today, we’re happy to announce that we’re bringing anomaly rates to every chart in Netdata Cloud. Anomaly information is no longer limited to the Anomalies tab and will be accessible to you from the Overview and Single Node View tabs as well. This will make your troubleshooting journey easier, as you will have the anomaly rates for any metric available with a single click. Whichever metric or chart you’re exploring will be instant.

Metric Correlations on the Agent

Wed, 15 Jun 2022 00:00:00 +0000

As of v1.35.0 the Netdata Agent can now run Metric Correlations (MC) itself. This means that, for nodes with MC enabled, the Metric Correlations feature just got a whole lot faster!

The Netdata Metric Correlations feature uses a Two Sample Kolmogorov-Smirnov test to look for which metrics have a significant distributional change around a highlighted window of interest. This can be useful when you are interested in short term “change detection” and want to try answer the question “what else changed around this time?”.

Anomaly Advisor: Unsupervised Anomaly Detection

Thu, 26 May 2022 00:00:00 +0000

Today we are excited to launch one of our flagship ML assisted troubleshooting features in Netdata – the Anomaly Advisor.

The Anomaly Advisor builds on earlier work to introduce unsupervised anomaly detection capabilities into the Netdata Agent from v1.32.0 onwards.

Getting Started

Once you enable ML on your nodes, each node will begin producing an “Anomaly Bit” every second in addition to raw metric values. This anomaly bit will be 1 when the trained ML models consider recent raw data for a metric to look anomalous or 0 when things look ’normal’. The Anomaly Advisor leverages this information to enable seamless space or room level anomaly detection out of the box with minimal configuration.

Monitoring without Cooperation: Kubernetes

Fri, 20 May 2022 00:00:00 +0000

Imagine this: You are an engineer at a startup. You are responsible for keeping all the applications running smoothly and safely in production. At first, you have things under control, but soon enough things start getting more complex. The system has grown by hundreds of new server nodes, new services and containers are showing up seemingly every day, and you hear there’s a project to transition everything to Kubernetes (you’ve heard the term “cloud native” so often that the phrase shows up in your dreams).

Kubernetes Throttling Doesn’t Have To Suck. Let Us Help!

Tue, 03 May 2022 00:00:00 +0000

CPU limits are probably the most misunderstood concept in Kubernetes CPU resources allocation and management.

A lot of engineers advise the use of CPU limits on every container as Kubernetes best practice. Unfortunately, as we will prove below, they are wrong: CPU limits should rarely be used, if used at all!

But why? What are the reasons that even senior DevOps engineers with vast experience in the field advise the use of CPU limits?

Troubleshooting Alerts the Right Way: As a Team

Thu, 28 Apr 2022 00:00:00 +0000

At Netdata, we love two things more than anything else:

Troubleshooting and,
Making things easier for you!

Our goal is to make troubleshooting and monitoring as seamless as possible with the open-source Agent. This includes giving you pre-configured alerts so that you get notified immediately when a disruption occurs.

The Netdata Agent comes with over 250 pre-configured and optimized alerts. But we want you to be the master of your infrastructure monitoring by:

CNCF Live: Machine Learning Anomaly Detection

Wed, 27 Apr 2022 00:00:00 +0000

Join Us Live to Talk ML

Join ML Lead Andrew Maguire and Product Manager Shyam Sreevalsan on the 23rd of June at 5pm UTC for the Netdata Machine Learning Meetup, which will be livestreamed on Youtube. In this session, we will demo and preview new and future features, along with a Q&A, and discuss the following topics:

The role of Machine Learning in DevOps Infrastructure Monitoring & Troubleshooting
The Netdata way of approaching Machine Learning
Challenges of building Macvhine learning solutions that are useful and user friendly.

Feel free to ask questions or share ideas on our Community Discord, or RSVP to the event. We look forward to seeing you then!

CNCF Live: Power up your machine learning - Automated anomaly detection

Our Analytics & ML lead Andrew Maguire recently had a chance to share our new Anomaly Advisor feature with the wider CNCF community. In his demonstration he did some light chaos engineering (using Gremlin and stress-ng) to generate some real anomalies on his infrastructure and watch how it all played out in the Anomaly Advisor in Netdata Cloud.

The Netdata Way of Troubleshooting

Mon, 04 Apr 2022 00:00:00 +0000

Together with you, our fabulous community, Netdata is changing the way the world thinks of high fidelity monitoring - and we are gaining momentum.

Our chief troublemaker and CEO, Costa Tsaousis, is the pioneer and architect of this revolution that’s brewing in the monitoring and troubleshooting space.

Watch him explain the Netdata way of troubleshooting:

Why did the world need a new infrastructure monitoring tool? (00:18)

Every great hero needs an origin story. In this section, Costa explains the frustrating conditions in the infrastructure and monitoring space that lead to the conception, development, and subsequent success of the Netdata open-source Agent and Netdata Cloud.

Our Approach to Machine Learning

Fri, 25 Mar 2022 00:00:00 +0000

There is a lot of buzz in the world of machine learning (ML) and as a layperson it can be hard to keep up with it all. Therefore, we decided to write down some of our thoughts and musings on how we are approaching ML at Netdata.

Our Approach to Machine Learning (ML)

We’ll touch on the current state of applied ML in industry in general, and zoom in on ML in the monitoring industry. We’ll discuss how we can leverage “good honest ML” to punch above our weight and add some useful and novel features for our users over the next few years.

Engineering Team Best Practices For Cloud Monitoring

Wed, 02 Mar 2022 00:00:00 +0000

Panel: Engineering Team Best Practices in Working with Cloud

The Netdata Troubleshooting Show| Season 1: Episode 2 | Thursday, March 3rd at 9 am PST (UTC/GMT -8)

[Join Live]For engineering teams, working in the cloud has never been easier, but also more complex. Enter a new world of remote working with distributed global teams, new tech challenges, new business realities, security, monitoring, and more.

Join our dynamic panel of experts live(watch here)as they talk about engineering team best practices for those working in the cloud in 2022.

Netdata Meetup | Install & Monitor From Scratch | Guide

Wed, 02 Mar 2022 00:00:00 +0000

Announcing Netdata Meetups: Virtual How-to Live Show – Join us Friday, March 4th

Join us live this Friday! We are launching the brand new Netdata Meetups! Join us to learn more about Netdata. This how-to virtual series will help you go deeper with Netdata and as a bonus we will add in tips and tricks for monitoring and troubleshooting. We can’t wait to see you on Friday, March 4th at 9 am PST (GMT -8).

Netdata Cloud’s New Architecture

Thu, 10 Feb 2022 00:00:00 +0000

In version v1.32 of Netdata, we announced a remarkable new update that we are extremely proud of; Netdata Cloud now runs on the most reliable and stable backend that we’ve ever built.

Migration to the new Netdata Cloud architecture

To give you the best experience of Netdata Cloud, we started migrating nodes running on the old architecture to the new one. Most users don’t have to take any action on their part. If you need to take action, you will see the pop-up window above in Netdata Cloud.

Meet The Netdata Community: eBPF Hero Thiago

Fri, 04 Feb 2022 00:00:00 +0000

Our ongoing Meet the community series focuses on global Netdata community members. In this installment, learn about Netdata staff member and Software Engineer Thiago Marques, who is hard at work building an eBPF.plugin.

Introduce yourself, what you do, and your current job role at Netdata.

I am Thiago Marques, a C developer who works with the data collector team. My primary responsibility in the company is to develop eBPF.plugin.

Meet The Netdata Community: Rupok Chowdhury Protik

Tue, 07 Dec 2021 00:00:00 +0000

In our ongoing Meet the community series, we focus on global Netdata community members. In this installment, learn about Software Engineer and photographer Rupok Chowdhury Protik, including how they use Netdata.

Introduce yourself, what you do, and your current job role.

Hi, I’m Rupok Chowdhury Protik. I’m a PHP Developer with a Master’s degree in Software Engineering. My stack is mainly LAMP/LEMP. I used to work as the Senior Web Application Developer in a multinational company and then later joined Incsub LLC, a WordPress-focused company with people from more than 50 countries. I am currently working in Incsub LLC as a DevOps Support Engineer.

Netdata Cloud Show: The Observability Panel

Tue, 07 Dec 2021 00:00:00 +0000

“In 2022, 60% of security incidents will involve third parties.”– Forrester

https://www.youtube.com/embed/kILpPCRlVD0

This Thursday, December 9th at 1 PM EST (UTC-5)Join us live this Thursday for the first-ever Netdata Cloud Show, where our all-star panel will be discussing the latest trends around observability.

This will be broadcast live on Netdata YouTube, Twitter, Facebook, and Discord.

Observability Panel:

Boris Zaikin, Senior Software and Cloud Architect at Nordcloud. @boriszzn
Samir Behara, Platform Architect at EBSCO Industries, Inc. @samirbehara
Ralph Meijer, VP of Technology at Netdata. @ralphm

Topics Covered Include:

All-new Netdata Cloud Charts 2.0

Tue, 30 Nov 2021 00:00:00 +0000

Netdata excels in collecting, storing, and organizing metrics in out-of-the-box dashboards for powerful troubleshooting. We are now doubling down on this by transforming data into even more effective visualizations, helping you make the most sense out of all your metrics for increased observability.

The new Netdata Charts provide a ton of useful information and we invite you to further explore our new charts from a design and development perspective. As always, it’s our goal to be as open and transparent as possible with our users on all things Netdata, including the ins and outs of how Netdata is built, why we make certain design decisions (driven by you of course!), and where we are heading as we continue to grow.

Meet The Netdata Community | Bastien, SysAdmin & Dog Lover

Tue, 30 Nov 2021 00:00:00 +0000

Tell people about yourself and what you do.

My name is Bastien. I’m 28, a System and Network Admin, currently SRE in a team of two for my company. We’re working with AWS and Kubernetes to host our web apps.

How are you using Netdata and what do you like so far?

I’m currently in the process of moving to Netdata to monitor our nine Kubernetes clusters and some standalone VMs. What I particularly love about Netdata is the simplicity of installation and configuration and the number of perfect default metrics and graphs.

How to extend the Geth collector

Mon, 13 Sep 2021 00:00:00 +0000

This is the the last of a 2-part blog post series regarding Netdata and Geth. If you missed the first, be sure to check it out here.

Geth is short for Go-Ethereum and is the official implementation of the Ethereum Client in Go. Currently it’s one of the most widely used implementations and a core piece of infrastructure for the Ethereum ecosystem.

With this proof of concept I wanted to showcase how easy it really is to gather data from any Prometheus endpoint and visualize them in Netdata. This has the added benefit of leveraging all the other features of Netdata, namely it’s per-second data collection, automatic deployment and configuration and superb system monitoring.

How to monitor the Geth node in under 5 minutes

Mon, 13 Sep 2021 00:00:00 +0000

This piece is a blog post version of a workshop I gave at EthCC about monitoring an Ethereum Node using Netdata.

Disclaimer: Although we use Netdata, this guide is generic. We talk about metrics that can be surfaced by many other tools, such as Prometheus/Grafana or Datadog.

The contents are as follows:

Introduction to Ethereum Nodes
What is Netdata
How to monitor a system that runs go-ethereum (Geth)
How to monitor go-ethereum (Geth)

Ethereum Nodes

Running a node is no small feat, as it requires increasingly more and more resources to store the state of the blockchain and quickly process new transactions.

Root cause analysis using Metric Correlations

Fri, 03 Sep 2021 00:00:00 +0000

As complexity of systems and applications continue to evolve and change, the number of metrics that need to be monitored grows in parallel. Whether you’re on a DevOps team, an SRE, or a developer building the code yourself, many of these components may be fragmented across your infrastructure, making it increasingly difficult to identify the root cause when experiencing downtime or abnormal behavior. To help solve this challenge, we built the Metric Correlations feature – an automated analysis tool that evaluates all your metrics to identify which have changed the most within a given period of interest.

How To Monitor Disks & Filesystems With eBPF

Mon, 16 Aug 2021 00:00:00 +0000

Introduction to eBPF

Current IT monitoring software lacks the necessary metrics for minimizing downtime for systems and applications. Most provide system and application metrics but there is much more than this required for properly monitoring your infrastructure. With eBPF there is a technological advancement that allows monitoring software to provide rich information from the Linux kernel and present it. eBPF monitoring, specifically, provides a better understanding of what exactly is occurring on internal systems, which helps to identify where performance improvements can be made.

Netdata Is Launching Its Discord Server

Tue, 22 Jun 2021 00:00:00 +0000

It’s been a long time since our last community update, rest assured that we have been hard at work here at Netdata.

Community building is hard, especially when you have such a venerable community like the one here at Netdata, where hundreds of contributors have contributed to creating one of the best monitoring solutions that exist.

Last year we started to concentrate working on consolidating the community by integrating the various platforms where people come together to talk about Netdata. In that effort, we restarted our community forums in November, reworking the categories and creating a support channel, where community members help each other.

Netdata v1.31.0

Wed, 19 May 2021 00:00:00 +0000

Give a warm welcome to Netdata v1.31.0, which features:

Re-packaged and redesigned dashboard: A more informational and feature-rich “frame” for your monitoring and troubleshooting sessions.
eBPF expands into the directory cache: Monitor whether your services or applications are properly using Linux’s memory management for the best performance and minimal disk I/O.
Machine learning-powered collectors: Detect anomalies using only your own data and minimal resource utilization on your monitored nodes.
An improved Netdata learning experience: A timeline of new content, refreshed visuals, and a newly-open sourced repository.

Re-packaged and redesigned dashboard

We re-packaged and redesigned portions of the dashboard to improve the overall experience. Part of this effort is better handling of dashboard code during installation—anyone using third-party packages (such as the Netdata Homebrew formula) will start seeing new features and the new designs starting today.

For those who aren’t using third-party packages (thank you!), your installation process will still get a little bit faster.

Kubernetes monitoring and troubleshooting made simple

Wed, 05 May 2021 00:00:00 +0000

Infrastructure monitoring was difficult enough when entire businesses ran off a few bare metal servers in a dusty, forgotten closet. Other IT infrastructure monitoring tools fell short, unable to provide complete and granular-enough metrics in real time, even when we were only dealing with a handful of systems responsible for running every part of the application stack. They were hard to configure, especially for the non-gurus out there, and didn’t provide the high-resolution metrics the gurus needed to make data-driven troubleshooting decisions.

Netdata 1.30.0

Thu, 01 Apr 2021 00:00:00 +0000

We’re excited to introduce Netdata 1.30.0, which features:

ACLK-NG: A new custom library for streaming metrics data on demand, written entirely in-house, that’s 4x faster than libmosquitto/libwebsockets.
Opt-in product telemetry with PostHog: Goodbye, Google Analytics. Hello, self-hosted instance of PostHog.
Deeper Linux kernel monitoring with eBPF: Expanding our reach into the Linux kernel with page cache and synchronization syscall monitoring.
Smarter preconfigured alarms: Better (and less noisy) defaults, better information.
Developer environment: Contribute to Netdata via a Docker image and VSCode integration.
Documentation improvements & tutorials: Better standards for editing files and restarting Netdata, plus brand-new tutorials.

ACLK-NG

The ACLK-NG is a new, faster method of securely connecting a node running Netdata to Netdata Cloud. In our internal testing, it’s 4x faster than our previous implementation, which uses libmosquitto and libwebsockets.

ACLK-NG		ACLK Legacy
	time (s)	MB/s	time (s)	MB/s
Run 1	1.30	76.92	5.30	18.87
Run 2	1.31	76.34	5.29	18.90
Run 3	1.38	72.46	5.27	18.98
Run 4	1.27	78.74	5.40	18.52
Run 5	1.24	80.65	5.46	18.32
	1.30	77.02	5.34	18.72

With ACLK-NG enabled, you’ll get a snappier experience in Netdata Cloud, as there will be far less latency between requests for metrics and the subsequent response from individual nodes.

To enable ACLK-NG right now, update your nodes with the –aclk-ng option:

Container deployment showdown: Docker or Kubernetes?

Wed, 24 Mar 2021 00:00:00 +0000

Monitoring the current state and performance of applications is critical for IT Ops and DevOps teams alike. Understanding the health of an application is one of the most effective ways of anticipating potential bottlenecks or slowdowns, yet it’s one of the largest challenges faced by many organizations that build and deploy software. This is largely due to applications’ distributed and diversified nature. A single outage has the potential to interrupt entire processes that, at times, can interfere with business as a whole and result in a negative effect on the bottom line.

5 DevOps best practices to reinforce with monitoring tools

Thu, 18 Feb 2021 00:00:00 +0000

As part of a modern software development team, you’re asked to do a lot. You’re supposed to build faster, release more frequently, crush bugs, and integrate testing suites along the way. You’re supposed to implement and practice a strong DevOps culture, read entire novels about SRE best practices, go agile, or add a bunch of Scrum ceremonies to everyone’s calendar. Every week, the industry recommends that you “shift-left” another part of the DevOps pipeline, to the point where you’re supposed to handle everything from unit testing to production deployment optimization from day one.

Introduction to StatsD

Wed, 03 Feb 2021 00:00:00 +0000

StatsD is an industry-standard technology stack for monitoring applications and instrumenting any piece of software to deliver custom metrics. The StatsD architecture is based on delivering the metrics via UDP packets from any application to a central statsD server. Although the original StatsD server was written in Node.js, there are many implementations today, with Netdata being one of them.

StatsD makes it easier for you to instrument your applications, delivering value around three main pillars: open-source, control, and modularity. That’s a real windfall for full-stack developers who need to code quickly, troubleshoot application issues on the fly, and often don’t have the necessary background knowledge to use complex monitoring platforms.

Actionable Alerts With Fewer False Positives

Thu, 21 Jan 2021 00:00:00 +0000

Think about any sport or competitive activity, whether that’s football or a spelling bee. They always feature at least one person who acts as a moderator, referee, or judge. With their domain expertise, this person watches everyone’s behavior and constantly compares that against a set of rules. If someone crosses that threshold, they blow a whistle or throw up a flag. They are, in effect, saying that things have gone from OK to not OK.

Four key metrics for responding to IT incidents and failures

Thu, 07 Jan 2021 00:00:00 +0000

If you’re a veteran in this space, you probably understand the many incident response metrics and concepts, along with the many (at times exasperating) acronyms. For those new to the space, or even those with years of experience, the terminology is often overwhelming.

If you’re one of those people who’s struggling to navigate through the world of DevOps metrics, we’ve created this article for you. In this post, we’ll cover four main incident response metrics: MTTA, MTTR, MTBF, and MTTF. Learn what these acronyms mean, how you can use them, and how you can tie these KPIs into your Netdata experience.

Netdata Year in Review 2020

Fri, 18 Dec 2020 00:00:00 +0000

Looking back at the unprecedented challenges we faced together in 2020, we’d like to extend our thanks to the community of people who have continued to work towards Netdata’s mission of simplifying monitoring and troubleshooting for everyone. Let’s review some of this year’s highlights.

Community	More than 400 contributors have helped us grow.
Nearly 50,000 GitHub stargazers follow our progress.	Our community on GitHub and our forums has grown to 4,000 strong.

Netdata Cloud	Launched in May, now with more than 25,000 users registered.
Continuous delivery of new features included custom dashboards, Metric Correlations, overview page, and centralized alarm notifications.

Netdata Agent	9 dot releases, with support for eBPF, Prometheus metrics, & k8s service discovery.
More than 300 bug fixes, nearly 1,500 closed issues, and almost 3,200 commits.	Nearly 400 new features or improvements.

Company	Raised $14.2M, extending the total raised to $31M.
Recognized in Forbes Cloud 100 Rising Stars 2020, 2020 Stratus Awards as Cloud Disruptor.

Centralize Infrastructure With Alarm Notifications

Thu, 17 Dec 2020 00:00:00 +0000

Netdata is architected on every level, across both the open-source Netdata Agent and Netdata Cloud, to help you own every layer of your monitoring experience. With this design, all metrics data collected by the Netdata Agent stays distributed on your node, but you also leverage Netdata Cloud’s dashboards and multi-node visualizations to view the health and performance of an entire infrastructure from a single application.

Community Update: Discourse, Community Efforts

Wed, 02 Dec 2020 00:00:00 +0000

Open source and community have always been in the DNA of Netdata, with the Agent starting as a very popular open-source project. Since then, a lot has changed, with Netdata maturing into a company, and the Netdata Agent finding its place as an open-source project in a wider offering that redesigns the monitoring experience from the ground up.

While we had a very active GitHub repository, with the majority of the Netdata Agent’s original team actively moderating the discussions and talking with users, we started a more concentrated initiative to manage our community in spring 2020. We launched our first forum using NodeBB, a great open-source project, and the community grew substantially, outgrowing the forum software. At the same time, we were able to identify areas of friction in the community journey. Removing that friction became a centerpiece of our strategy during Q3 2020.

StackPulse: Automated Incident Remediation Workflow

Wed, 02 Dec 2020 00:00:00 +0000

Teams of all types use Netdata to monitor the health of their nodes with preconfigured alarms and real-time interactive visualizations, and when incidents happen, they troubleshoot issues with thousands of per-second metrics on Netdata Cloud. But based on the complexity of the team and the infrastructure they monitor, some parts of their incident management, such as pre-planned communication and escalation processes, or even automated remediation, need to happen outside of the Netdata ecosystem.

What is DevOps?

Thu, 19 Nov 2020 00:00:00 +0000

In software development, it’s important to have a team dedicated to ensuring all systems and applications maintain maximum performance and uptime. Establishing processes that limit system and application slowdowns and outages while expediting the product release process is often done through a developer operations team, also known as dev ops or DevOps.

DevOps teams are responsible for improving communication and collaboration across engineering teams to increase an organization’s ability to efficiently deliver products and services that serve the end-user. In this post, we’ll describe the different elements of DevOps, including what DevOps is, how it works, and how Netdata helps DevOps teams succeed.

Community Repository: Consul, Ansible & ML Recipes

Wed, 11 Nov 2020 00:00:00 +0000

On our journey to democratize monitoring, we are proud to have open source at the core of both our products and our company values. What started as a project out of frustration for lack of existing alternatives (see anger-driven development), quickly became one of the most starred open-source projects on all of GitHub.

Fast-forward a couple of years later, and the Netdata Agent, our open-source monitoring agent, is maturing as the best single-node monitoring experience, offering unparalleled efficiency and thousands of metrics, per-second. At the same time, we have gathered a considerable community on our GitHub repository and new forums.

How Netdata gets you from 0 to monitoring in minutes

Wed, 11 Nov 2020 00:00:00 +0000

Netdata is zero-configuration monitoring. It’s a principle that we’ve stood behind since the project’s beginning, when it was only our CEO Costa trying to solve a “painful, real-world problem,” and it’s one we stand by today. Our insistence on zero-configuration guides every product decision we make, every grooming process, and every React component our frontend teams design.

In fact, zero-configuration is the exact reason why Netdata’s dashboard is open and accessible by default. It’s how Netdata gets you from 0 monitoring to thousands of metrics, collected every second and visualized in real time, in a matter of minutes.

Netdata’s dashboard: open by default and secure by design

Wed, 28 Oct 2020 00:00:00 +0000

Let’s talk through a scenario: You have a Linux-based VM running on DigitalOcean (aka a Droplet), and you install Netdata on it using our recommended kickstart script. As the installation process winds down, the Droplet starts up the Netdata Agent’s web server and serves the local Agent web dashboard on port 19999. You navigate to the dashboard using your browser of choice, check out per-second metrics updating in real time in a few of the hundreds of preconfigured visualizations, then realize…

Real-Time Infrastructure Monitoring Now In Netdata Cloud

Thu, 22 Oct 2020 00:00:00 +0000

The Netdata Agent is well-equipped to solve monitoring and troubleshooting challenges for single nodes. We love that the Agent is so valuable to our users, but Netdata Cloud is designed for infrastructure monitoring. That’s why we’re working so hard to offer even more capabilities and help users monitor and troubleshoot infrastructures of all sizes, entirely for free!

With the new Cloud Overview, you get every real-time chart and metric you need to understand the status of your infrastructure, explore, and troubleshoot, in a single view. We designed the Overview on one existing and beloved feature and another entirely new one that we’re very excited to launch for the first time.

The reality of Netdata’s long-term metrics storage database

Mon, 12 Oct 2020 00:00:00 +0000

The perception that Netdata is only capable of short-term metrics storage is a myth. It’s a pervasive myth we still see in blog posts and through community engagement, despite it being false for more than a year.

However, like all myths, this one on metrics storage began with a kernel of truth. When Netdata first flourished as an open-source project in 2017 and 2018, the default metrics database was RAM-only. You could configure this database’s size, but for many users, that size was limited by the amount of RAM they were willing to allocate for metrics storage. We also kept the default value low to ensure Netdata worked efficiently on all hardware and a variety of operating systems.

Software Extensibility Is Key To Adoption

Fri, 25 Sep 2020 00:00:00 +0000

As with most commercial products, software today is mass produced for reasons of simple economics. But making the same software work for as many people as possible while also meeting the unique requirements different people and organizations have is a challenging task.

The strategies used today to provide extensibility at cost and at the level required by various enterprises are very similar to the ones used in the 80s and 90s, when PCs first took off and captured an enormous amount of the home computing market, and when open source software started gaining popularity.

Investing in Netdata: a growth story

Tue, 22 Sep 2020 00:00:00 +0000

I’m excited to announce an extension to Netdata’s series A funding in the amount of $14.2M, bringing the total amount of funding to $31M. We’re thrilled to share the news; the additional funding will help us continue building the future of health monitoring and performance troubleshooting. In case you missed it, our mission is to redefine infrastructure monitoring. Our unique approach to building the right solution with and for the community is no easy task.

Metric Correlations: Detect Patterns & Anomalies

Wed, 16 Sep 2020 00:00:00 +0000

Today, we are excited to launch our first Netdata Cloud Insights feature, Metric Correlations, developed for discovering underlying issues more quickly and identifying the root cause more efficiently. Read on to learn more about our approach to developing this new feature, how it works, and the many benefits you’ll find incorporating this into your team’s troubleshooting workflow.

Some background

Let’s start with a bit of a disclaimer. It seems machine learning (ML) (or “Artificial Intelligence,” if you are looking for more LinkedIn likes) has gone mainstream in the last few years, and we are probably by now somewhere near the “Peak of Inflated Expectations” on the hype cycle. It is in this context that we want to be clear about what our goals are in this space and our approach to releasing data-driven features that draw on techniques from statistics and ML. In short, we want to be clear, open, realistic, and avoid buzzwords at all costs!

Over the next 12 months, we are hoping to begin building a layer of intelligence¹ throughout Netdata (both Cloud and Agent) to assist with “human in the loop” troubleshooting, mainly to help users more easily surface slowdowns, anomalies, or other issues and lower your cognitive load² as you troubleshoot using Netdata. Simply put, we’re working to streamline your mean time to resolution (MTTR).

Netdata Agent v1.25 & Cloud Enhancements

Wed, 16 Sep 2020 00:00:00 +0000

The v1.25.0 release of the Netdata Agent delivers on our commitment to make our metrics collection, visualization, and troubleshooting platform more stable and usable. We enhanced our recently-added Prometheus collector with user-configurable filtering and grouping, made dramatic improvements to the reliability of the Agent-Cloud link that streams metrics on-demand to your browser when you use Netdata Cloud, and more.

Let’s jump in and look at each improvement.

Netdata named to the Forbes Cloud 100 Rising Stars

Wed, 16 Sep 2020 00:00:00 +0000

We’re excited to announce that we’ve been named to the Forbes 2020 Cloud 100 Rising Stars. This is a list of the top 100 private cloud companies in the world, published by Forbes in partnership with Bessemer Venture Partners and Salesforce Ventures. The 20 Rising Stars represent young, high-growth and category-leading cloud companies who are poised to join the Cloud 100 ranks. We are extremely honored to be recognized amongst our most-promising peers. This is a testament to our momentum we’ve built in partnership with our passionate community of users and contributors worldwide, as well as a validation of our community-first, open-source approach to democratizing infrastructure monitoring and troubleshooting. The major milestones we’ve hit along the way this year include more than 5,000 new users added each day, more than 3 million users total worldwide, and nearly 50,000 GitHub stars, making Netdata the fourth most starred project in the Cloud Native Computing Foundation landscape. We’ve also launched our monitoring service, Netdata Cloud, which has seen very strong growth since its introduction in May, with more than 15,000 users registered so far. We couldn’t have done it without you! The Forbes 2020 Cloud 100 and 20 Rising Stars lists will also appear in the September 2020 issue of Forbes magazine.

The Netdata Community Powered by NodeBB

Thu, 13 Aug 2020 00:00:00 +0000

We recently adopted NodeBB as our software of choice for building the Netdata Community. We have many good reasons for why we wanted to provide our community with a proper home online, but I wanted to cover some of the technical reasons for choosing NodeBB for our platform, and the many parallels between the NodeBB and Netdata projects, which was certainly a driving force behind this decision.

Release 1.24: Prometheus Collector & Multi-Host DB

Mon, 10 Aug 2020 00:00:00 +0000

The v1.24.0 release of the Netdata Agent brings enhancements to the breadth of metrics we collect with a new Prometheus/OpenMetrics collector and enhanced storage and querying with a new multi-host database mode. Let’s take a look at each of these enhancements.

Instantly access thousands of metrics with the new Prometheus/OpenMetrics collector

This release broadens our commitment to open standards, interoperability, and extensibility with a new generic Prometheus collector that works seamlessly with any application that makes its metrics available in the Prometheus/OpenMetrics exposition format, including support for Windows 10 via windows_exporter. Netdata will autodetect over 600 Prometheus endpoints and instantly generate charts with all the exposed metrics, meaningfully visualized.

You can also quickly and easily configure the collector with the names and URLs of additional Prometheus endpoints to instantly view automatically generated charts with all the exposed metrics, meaningfully visualized within Netdata at the same high-granularity, per-second frequency you expect, all in real time. To learn more about how to configure, check out our documentation.

Introducing the all-new Netdata Cloud

Wed, 29 Jul 2020 00:00:00 +0000

In case you missed it, we released an all-new version of Netdata Cloud in May. Netdata Cloud is a free service that can be accessed from any browser and provides you with a consolidated view of your entire infrastructure.

Netdata Cloud works differently from other monitoring solutions. Most solutions limit the number and frequency of metrics because they rely on architectures that aggregate data. Netdata Cloud, however, streams limited metadata from each node running the Netdata Agent, keeping you in control of the data on your systems. The advantage of this architecture is that there is no limit to the number or frequency of metrics, regardless of the scale or complexity of your IT infrastructure. You can truly monitor every metric, from every system and application, across your entire infrastructure, in real time. For free.

Sysadmin Day 2020: IT Heroes and Homelabs

Mon, 27 Jul 2020 00:00:00 +0000

Sysadmin Day 2020 is right around the corner and we’d like to show our appreciation for all the sysadmins out there who keep IT humming along and come to the rescue to resolve critical issues day in and day out. This year, we’re celebrating all week long by hosting an IT Heroes and Homelabs contest. Join the celebration by retweeting our post with the hashtag #SysadminDay #NetdataWin, and we’ll enter you in a drawing to win some Netdata swag!

The Netdata Community

Mon, 27 Jul 2020 00:00:00 +0000

Netdata users and contributors comprise a large, global, but somewhat fragmented community – or a set of communities. You can find us on IRC (#netdata on freenode), on Reddit, on social media, and, of course, on GitHub, where the main open-source Netdata project repo lives. And yes, you can find us on other platforms as well.

GitHub is a great way to get in touch with the Netdata team and project contributors to tell them about bugs or to discuss new features. But we realized that bug reports are not a conversation that works for everyone. A more informal and easier-to-access communication channel would provide a better way for the community to congregate and engage, and would also provide an easier way for everybody to talk to the team. We wanted to provide the community a home.

Why Netdata picked VerneMQ

Tue, 14 Jul 2020 00:00:00 +0000

In 2019, the Netdata team already knew that a Netdata Cloud solution in the form of an online platform would greatly complement Netdata’s distributed monitoring by making it much easier to organize large infrastructures and by enabling new ways for teams to collaborate. The old node registry available at the time wasn’t enough for Netdata’s users.

Building an online platform, even one that does not directly process users’ metrics, is challenging. But less challenging than it was even a few years ago, since the technology stack has improved greatly over the years.

What is Infrastructure Monitoring?

Tue, 30 Jun 2020 00:00:00 +0000

IT is advancing blazingly fast. To keep up with architectural changes and hybrid environments, it’s more important than ever to maintain efficient infrastructure monitoring and troubleshooting. Adding to the complexity is the increase of distributed systems, comprised of many components and services. For IT teams to effectively manage monitoring modern infrastructure, it’s necessary to have the right practices and tools in place that enable teams to do their jobs as quickly as possible with fewer resources.

Release 1.23: Kubernetes & eBPF Observability

Thu, 25 Jun 2020 00:00:00 +0000

As adoption of container infrastructure grows in popularity, we’re continuing to focus on the most effective ways users can quickly and easily deploy container monitoring to instantly get access to deep, real-time insights. Agent release 1.23 introduces service discovery for Kubernetes clusters, monitoring for individual nodes, and eBPF monitoring per application on an event frequency for quickly identifying the root cause. Quickly shed light on your infrastructure performance with these new features!

The role of shift-left testing in an agile environment

Tue, 14 Apr 2020 00:00:00 +0000

With the rapid growth of security threats to infrastructure, it’s more important than ever to proactively address vulnerabilities. As an open-source project, built on the trust of users and contributors, Netdata has security concerns at its core.

Because we’re committed to code security and quality, we apply Agile principles throughout the software development process. A component of this includes regular static analysis. Through continuous, automated testing, we’re able to move quickly to keep up with end-user requests without compromising our source code.

Release 1.21: New Collectors & Faster Exporters

Mon, 06 Apr 2020 00:00:00 +0000

We’re in the middle of a scary, uncertain time, and we hope those of you reading are staying safe and healthy.

Despite the current challenges, the 40+ members of the remote-first Netdata team have been hard at work on the next version of the Netdata Agent: v1.21.0.

This release is foundational: While we do have fantastic new collectors and three new ways to export your metrics for long-term storage, many of the most significant changes aren’t even those you’ll notice. While they may be beneath the hood, they’re going to power some amazing new features, UX improvements, and design overhauls.

Creating A Thriving, Agile, Remote Team

Wed, 25 Mar 2020 00:00:00 +0000

The coronavirus (COVID-19) pandemic has forced many organizations to take unprecedented steps towards remote working. As a fully distributed team, we’ve faced the common challenges of remote work. Based on our experience from our very beginning in 2018, all but a few of these organizations new to remote working will face hurdles to overcome and may try to revert to colocation as soon as possible. Remote working is hard, even when it’s carefully planned and executed. When the transition is rushed and seen as a necessary, temporary inconvenience, challenges are all but inevitable.

The Netdata Culture and People

Mon, 23 Mar 2020 00:00:00 +0000

There are many things I absolutely love about Netdata, but I’m most proud of our people and culture. Some words about this unique experience are long overdue.

In a career that spans over two decades and six other companies of various sizes, nothing compares to the satisfaction of working in a company like ours. My answer to the canned interview question, “Where do you see yourself in 5 years”, was always the same: I don’t care; I just want to be solving problems and working with good people, real professionals, who I can trust and respect. In retrospect, I was missing another huge part of the equation, which is to mention the kind of company I wanted to work for. “Culture eats strategy for breakfast” is a cliche. More importantly, bad culture devours people’s souls; it sucks out any creative energy one may have, reducing engagement and, therefore, productivity. Short-term wins at the expense of company culture guarantee huge losses in the long run.

Contribute to Netdata’s machine learning efforts!

Mon, 16 Mar 2020 00:00:00 +0000

Netdata contributors have greatly influenced the growth of our company and are essential to our success. The time and expertise that contributors volunteer are fundamental to our goal of helping you build extraordinary infrastructures. We highly value end-user feedback during product development, which is why we’re looking to involve you in progressing our machine learning (ML) efforts! As we are continually looking for ways to improve and enhance Netdata, we are starting to explore how we can leverage machine learning to introduce new product features. Our main focus at the moment is around automated anomaly detection. This is a really interesting and challenging problem (high volume, high dimensional data, lack of ground truth labels, and so on), but we should be able to use some of the metrics monitored by Netdata to deliver new, awesome product features and user experiences (AI is the new electricity, after all 😃). However, developing ML-driven product features is quite different than traditional software development (see steps 1 to 7 in the picture above). Mainly, this is because you never really know what specific data transformations, problem formulation, and sets of algorithms will work best in advance. (Here is a good article explaining things, and if you really want to go down a rabbit hole, check out this Stack Overflow question and this Quora thread). Ideally, you first need to prototype your solution “in the lab” on some data you have already collected and do a few iterations of data → problem formulation → prototype. This process gives you a level of confidence in what you are doing (and some data to back it up) to move on to the even-more-complicated step of going from prototype to production. At Netdata, we are currently trying to get to step 4, where we can first prototype some solutions on real-world data and come up with ways to measure progress.

Linux eBPF monitoring with Netdata

Fri, 21 Feb 2020 00:00:00 +0000

Your application isn’t finished when you’ve closed the last if block and you lined up all the brackets. There’s a whole other world of testing, debugging, and optimization that you haven’t even touched yet.

To help you more safely step into that complex phase of making your application even better, we’ve just released a brand-new eBPF collector in v1.20 of Netdata. With this collector enabled, you can monitor real-time metrics of Linux kernel functions and actions from the very same monitoring and troubleshooting dashboard you use for watching entire systems, or even entire infrastructures.

Release 1.20: Kernel Monitoring & Infra-Wide Labels

Fri, 21 Feb 2020 00:00:00 +0000

In Netdata’s first major release of 2020, we’re introducing two new features on the opposite ends of the monitoring spectrum.

On one hand, we’re releasing an eBPF collector, which lets you collect, monitor, and visualize incredibly precise metrics straight from the Linux kernel. On the other, we added the ability to label agents to help you organize entire infrastructures and see every important piece of information about streaming nodes in one place.

Redefining monitoring with Netdata (and how it came to be)

Thu, 19 Dec 2019 00:00:00 +0000

How Netdata was born

In 2013, I worked for a company that relied on financial transactions. We had a very simple SLA: complete all financial transactions within 3 seconds.

We were migrating the infrastructure from colocated (physical servers) to the cloud (VMs). The transition was not smooth. We had a lot of issues on the cloud side, which we couldn’t even detect. Business metrics were randomly reporting significant loss of volume and a very bad SLA, but at the operational level we saw no issues—everything seemed to be working perfectly. Traces were showing a large delay in several transactions, but there were no failures.

Release 1.19: Web Log Parsing & Unit Testing

Wed, 27 Nov 2019 00:00:00 +0000

Network monitoring is complex, which is why we’re developing a monitoring tool that will drastically increase DevOps productivity. This release is all about improving Netdata’s day-in, day-out performance. We’re working hard to make deploy enhancements that help engineers make faster, smarter decisions about their systems.

v1.19 of Netdata delivers a vastly improved way to collect, parse, and understand the health and performance of any service or application that runs through an Apache or Nginx web server.

Agile Team Safety Harness With cmocka & FOSS

Tue, 26 Nov 2019 00:00:00 +0000

Netdata is made up from agile teams who are deeply committed to improving the usability of our product. We want to respond to our users and introduce in-demand features. Working directly with our community is the best way to make Netdata better.

But we face the same the dilemma as all agile teams: How do we do this safely?

Safety means that we can move quickly without compromising the quality of our code. Because we want to move quickly, engage with our users’ desires, and keep quality high, we’re becoming very serious about adopting unit testing in our work.

Release 1.18: What’s new with the database engine?

Sat, 19 Oct 2019 00:00:00 +0000

As your infrastructure grows more complex, storing long-term metrics becomes difficult and costly to retain. Your team stars to limit the amount of historical data they archive, causing gaps in coverage. Anomalies start to slip through the cracks.

Version 1.18 of Netdata aims to solve the monitoring metrics storage problem once and for all.

Aside from 5 new collectors, 16 bug fixes, 27 improvements, and 20 documentation updates, here’s what you need to know.

Release 1.17: Collection frequency gets flexible

Mon, 09 Sep 2019 00:00:00 +0000

The next version of Netdata has arrived! Aside from dozens of quality-of-life and papercut fixes, we’ve launched some new features we know you’ll be excited to use straight away.

Let’s dive in.

What’s new?

Release v1.17.0 contains 38 bug fixes, 33 improvements, and 20 documentation updates.

You can, of course, view the full list at the v1.17.0 release notes on GitHub. But, let’s talk details on a few of the improvements and changes most requested by the Netdata community.

How and why we’re bringing long-term storage to Netdata

Wed, 07 Aug 2019 00:00:00 +0000

We’ve built a lot of amazing things into the open-source Netdata monitoring system. But, no matter how far we’ve come, we’ll always be proud of how little RAM it uses.

Right now, Netdata stores metrics in your system’s RAM using a ridiculously efficient database. It only saves or loads historical metrics from disk when you restart it. With this system, Netdata can be both low-resource and exhaustive in its collection of real-time metrics.

Release 1.16.0: Smarter binaries and built-in TLS

Fri, 19 Jul 2019 00:00:00 +0000

We’re excited to launch release v1.16.0 of the open-source Netdata monitoring agent, which delivers real-time health monitoring and performance troubleshooting to nearly any system or application.

This release also contains 40 bug fixes, 31 improvements, and 20 documentation updates—if you’d like to see the full list, check out the full release notes.

Details aside, I know people are going to be most curious about the big changes we’ve just delivered to Netdata—let’s dive in.

Open Source Contributions: Supporting The Community

Tue, 02 Jul 2019 00:00:00 +0000

Netdata must be doing something right when it comes to inspiring contributions. Our open-source, distributed monitoring agent has on GitHub and has seen contributions from hundreds of people: . We’ve even hired a handful of our contributors to work full-time on making the Netdata ecosystem even more powerful.

The community is passionate about what we’re building, and they’re actively interested in making it work better for their particular needs.

Mon, 01 Jan 0001 00:00:00 +0000

// ============================================ // NETDATA ROI CALCULATOR - LOGIC & VARIABLES // ============================================

// INPUT VARIABLES // ============================================ hosts = integer (10 to 10,000+) // Number of monitored hosts/nodes mttr = integer (5 to 240 minutes) // Current average MTTR for critical incidents incidents = integer (1 to 50+) // Number of critical incidents per month currentCosts = integer ($0 to $1M+) // Annual monitoring tool costs teamSize = integer (5 to 200+) // Size of engineering/DevOps team engineerCost = integer ($50 to $200+) // Average hourly cost per engineer impactGroup = enum [“engineering”, “customers”, “both”] // Who is impacted by downtime revenueImpact = integer ($0 to $1M+) // Revenue loss per hour of downtime

.NET Framework Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

.NET Framework

.NET Framework is a software development platform that runs on Windows. It provides a common set of libraries and tools that developers can use to build different types of applications, such as web apps, desktop apps, mobile apps, games, and more.

.NET Framework Monitoring

One of the challenges of developing and maintaining .NET applications is to ensure their performance, reliability, and security. To do that, developers need to monitor various aspects of their applications, such as code execution, memory usage, exceptions, requests, errors, and more. Monitoring .NET applications can help developers identify and troubleshoot issues before they affect the end users.

1-Wire Sensors

Mon, 01 Jan 0001 00:00:00 +0000

1-Wire Sensors Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What are a 1-Wire Sensors?

These sensors monitor temperature. On Linux these are supported by the wire, w1_gpio, and w1_therm modules. Currently temperature sensors are supported and automatically detected.

Monitoring 1-Wire Sensors with Netdata

Netdata auto discovers hundreds of services, and for those it doesn’t turning on manual discovery is a one line configuration. For more information on configuring Netdata for 1-Wiresensors monitoring please read the collector documentation.

Netdata has a public demo space (no login required) where you can explore different monitoring use-cases and get a feel for Netdata.

1-Wire Sensors Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

1-Wire Sensors Monitoring

What Is 1-Wire Sensors?

1-Wire Sensors are a type of sensor technology used primarily for temperature monitoring through a simple communication protocol. These sensors are commonly employed in environments where real-time temperature tracking is crucial, such as server rooms, laboratories, or manufacturing lines.

Monitoring 1-Wire Sensors With Netdata

Netdata offers an advanced and intuitive platform for monitoring 1-Wire Sensors. By leveraging the Netdata 1-Wire Sensors monitoring tool, you can achieve real-time visibility into your environmental conditions, enabling you to respond swiftly to changes that could affect your infrastructure.

2023 Use Case Survey and Giveaway

Mon, 01 Jan 0001 00:00:00 +0000

2Wcom GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

2Wire Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

3Com

Mon, 01 Jan 0001 00:00:00 +0000

3Com Huawei

Mon, 01 Jan 0001 00:00:00 +0000

3Com SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

3Par Data SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

4D Server

Mon, 01 Jan 0001 00:00:00 +0000

4D Server Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

4D Server Monitoring

What Is 4D Server?

4D Server is a database server that seamlessly combines a powerful SQL engine with innovative application development capabilities. It is renowned for its flexibility and efficiency in managing complex applications and databases. Whether you are building applications, handling large databases, or performing detailed data analysis, 4D Server provides the necessary infrastructure to manage your needs effectively.

Monitoring 4D Server With Netdata

Monitoring 4D Server is essential for maintaining its health and performance. With Netdata, you can easily monitor 4D Server using the integrated capabilities of the go.d.plugin. This monitoring tool leverages an openmetrics (Prometheus) exporter, allowing you to gather essential metrics seamlessly. The Netdata platform can ingest data from any Prometheus exporter, providing users with automated dashboards, real-time alerts, and comprehensive insights—all without the need for a standalone Prometheus server or Grafana setup. Explore how robust 4D Server monitoring can enhance your system management.

4Rf Communications Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

8430FT modem

Mon, 01 Jan 0001 00:00:00 +0000

8430FT Modem Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

8430FT Modem Monitoring

What Is 8430FT Modem?

The 8430FT modem is a crucial component within network stacks, providing connectivity and data transfer capabilities in various technical environments. Understanding how this modem operates allows IT professionals, network engineers, and developers to ensure optimal performance and reliability.

Monitoring 8430FT Modem With Netdata

For monitoring the 8430FT modem, Netdata utilizes an advanced openmetrics (Prometheus) exporter. By leveraging the 8430FT Exporter, Netdata can efficiently collect and visualize essential metrics from the modem, aiding in network performance diagnostics. Unlike traditional setups, Netdata can ingest data from any Prometheus exporter, which means that users do not need an entire Prometheus server or Grafana to obtain actionable insights. With Netdata, these metrics are instantly transformed into automated dashboards and alerts, expediting the troubleshooting process.

A10

Mon, 01 Jan 0001 00:00:00 +0000

A10 Networks Previously Raksha Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

A10 Thunder

Mon, 01 Jan 0001 00:00:00 +0000

Abb Power Protection S.A. SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ablerex Electronic Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Acc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Accedian Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Accelerated Concepts Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Access Points

Mon, 01 Jan 0001 00:00:00 +0000

Access Points Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Access Points Monitoring

What Is Access Points?

Access Points, commonly referred to as APs, are pivotal devices in wireless networks, serving as gateways for clients to connect to larger networks. They facilitate seamless communication by acting as a bridge between wireless clients and the wired network infrastructure.

Monitoring Access Points With Netdata

Monitoring Access Points effectively is crucial for ensuring optimal network performance and reliability. With Netdata’s Access Points monitoring tool, you can gain deep insights into the operations of your wireless network infrastructure. Netdata’s platform allows for real-time monitoring and troubleshooting, offering a comprehensive view of various performance metrics. You can explore our live demo to see how it works in real time.

Accton Technology SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Accuenergy Canada Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Acer SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Acksys SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Acme Packet SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Actidata Company SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Active Directory

Mon, 01 Jan 0001 00:00:00 +0000

Active Directory Certificate Service

Mon, 01 Jan 0001 00:00:00 +0000

Active Directory Federation Service

Mon, 01 Jan 0001 00:00:00 +0000

Active Directory Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Active Directory

Active Directory (AD) is a tool from Microsoft that helps you manage Windows networks. It comes with most Windows Server systems and does many things. At first, Active Directory only helped you control domains. But now, it also helps you with many other things related to identity.

It stores information about objects on the network, such as users, computers, groups, printers, and policies. It also provides methods for authenticating and authorizing users and devices. Active Directory can be deployed on-premises or in the cloud with Azure Active Directory (Azure AD).

ActiveMQ

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ advisory topics: hidden destinations consuming resources

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ advisory topics: hidden destinations consuming resources

List every topic on a busy ActiveMQ Classic broker and count how many start with ActiveMQ.Advisory.. On a broker with a thousand queues, you can easily find a thousand or more advisory topics, each a real destination with its own JMX MBeans, each receiving non-persistent messages for broker events.

This is not a bug. Advisory topics are how the broker publishes internal events: connections opening and closing, consumers and producers starting and stopping, destinations hitting their memory limit, messages expiring. Tools and clients can subscribe to observe the broker. The problem is that the default behavior scales with destination count, not with how much of this information anyone consumes, and the cost is paid in MBean count, heap, GC pressure, and inflated throughput counters.

ActiveMQ authentication failures: credential rotation fallout and brute force

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ authentication failures: credential rotation fallout and brute force

Your broker log starts filling with SecurityException: User name [X] or password is invalid or Authentication failed for user. Sometimes it is a slow drip from a single client. Sometimes it is a flood from dozens of source IPs. Both look similar in the log, but they are very different incidents: one is usually a forgotten service after a credential rotation, the other is either a production outage (real consumers and producers dropped) or an active attack.

ActiveMQ authorization denied: authenticated users hitting Not authorized

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ authorization denied: authenticated users hitting Not authorized

Your ActiveMQ broker log is filling with lines like User orders-svc is not authorized to write to: queue://PAYMENTS.INBOUND or Not authorized to create: topic://ActiveMQ.Advisory.Connection. The client connected fine. Authentication passed. But every send, consume, or destination creation attempt is rejected.

This is an authorization failure, not an authentication failure. The user proved who they are; the broker is saying they cannot do the specific thing they attempted. That distinction drives the whole diagnosis: you are not chasing bad passwords, you are chasing a mismatch between the user the client authenticated as, the operation it attempted, and the authorization entries the broker loaded.

ActiveMQ broker down: telling a crashed broker from a hung one

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ broker down: telling a crashed broker from a hung one

Your pager says the ActiveMQ broker is down. The first question is not “how do I restart it” but “what kind of down is this.” A crashed broker (process gone, port closed) and a hung broker (process alive, port possibly open, nothing moving) have different causes, different evidence, and different safe responses. Restarting a hung broker without capturing diagnostics destroys the evidence. Assuming a crashed broker will come straight back up ignores the KahaDB recovery window, where the process runs and the port may even bind, but no client gets served for minutes to hours.

ActiveMQ broker won't start: port conflicts, store recovery, and lock contention

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ broker won’t start: port conflicts, store recovery, and lock contention

The broker process is running, or maybe it is not, and clients cannot connect. You restart the service, and it still does not come up. Or worse: the process starts, the log goes quiet, and ten minutes later you are still staring at a broker that has not opened its transport connectors.

“Won’t start” in ActiveMQ Classic covers four distinct failure modes that look identical from the outside: something else is holding port 61616 or 8161, KahaDB is replaying a large journal and simply is not done yet, the store is corrupted after an unclean shutdown and the broker is refusing to start, or a store lock in a shared-storage HA pair is held by the wrong broker and this one is waiting forever. Each has a different fix, and applying the wrong one (especially deleting store files or force-clearing locks) can turn a slow start into permanent message loss.

ActiveMQ connection and session leak: clients that never close

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ connection and session leak: clients that never close

The broker’s connection count used to sit around 400. Last month it was 600. This week it is 1,100, and nobody deployed anything new. The JVM thread count and open file descriptor count are climbing in lockstep. Eventually, usually at the worst time, the broker hits its FD limit and starts rejecting every new connection, including the healthy clients.

This is the classic ActiveMQ connection and session leak: clients that open JMS Connections (or Sessions) and never close them. Each leaked connection holds a socket, a file descriptor, one or more transport threads on the default TCP transport, and broker-side state. One JMS Connection can hold many Sessions, and each Session can hold many consumers and producers, so the leak compounds at each layer.

ActiveMQ consumers connected but not acknowledging: the zombie consumer

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ consumers connected but not acknowledging: the zombie consumer

The queue has consumers. ConsumerCount is exactly where it should be. But dequeue rate has collapsed to near zero, InFlightCount is pinned at the prefetch limit, and the backlog keeps growing. From the broker’s perspective everything is subscribed; from the business’s perspective nothing is being processed.

This is the zombie consumer: a consumer that holds a live connection and a full prefetch buffer but never acknowledges. ActiveMQ dispatched the prefetch window of messages to it, and the broker will not send that consumer more until acks come back. If it is the only consumer, the queue stalls while looking fully staffed.

ActiveMQ CVE-2023-46604: the OpenWire deserialization RCE and how to detect exposure

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ CVE-2023-46604: the OpenWire deserialization RCE and how to detect exposure

CVE-2023-46604 is a CVSS 10.0 unauthenticated remote code execution vulnerability in the ActiveMQ Classic OpenWire protocol marshaller. If an attacker can open a TCP connection to your OpenWire transport connector (default port 61616), they can send a crafted packet that causes the broker to instantiate an arbitrary class on the classpath. No credentials are required. It was exploited in the wild before and immediately after disclosure in October 2023, and unpatched, internet-reachable brokers are still being found.

ActiveMQ default credentials and exposed web console: admin/admin on port 8161

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ default credentials and exposed web console: admin/admin on port 8161

ActiveMQ Classic ships with a web console on port 8161 and well-known default credentials: admin/admin. Combined with a management interface reachable from the network, that equals full broker control for anyone who finds the port: browse and purge queues, move messages, read message bodies, create destinations, and in some versions reach the Jolokia JMX REST API for much worse.

ActiveMQ deserialization errors: ClassNotFoundException and the serializable packages whitelist

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ deserialization errors: ClassNotFoundException and the serializable packages whitelist

ClassNotFoundException, InvalidClassException, or StreamCorruptedException in the ActiveMQ broker log means messages are failing to process. Sometimes this is mundane: a producer and consumer compiled against different versions of a class, or an ObjectMessage payload class that is not on the broker’s classpath. Sometimes it is the signature of a Java deserialization exploit attempt, and the correct response is a security incident, not a config tweak.

ActiveMQ destination explosion: dynamic destinations, MBean bloat, and GC pressure

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ destination explosion: dynamic destinations, MBean bloat, and GC pressure

An ActiveMQ Classic broker that has been running for months shows a specific signature: heap usage climbs steadily even when message volume is flat, GC pauses get longer and more frequent, the web console and JMX queries feel sluggish, and eventually the broker OOMs or slides into a GC death spiral. Queue depths look normal. Enqueue and dequeue rates look normal. Nothing about the messaging workload explains it.

ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ disk full on the KahaDB partition: write failures and store corruption risk

The KahaDB partition is at 100%. The broker log shows journal write failures, persistent producers have stopped, and you are in the worst ActiveMQ failure mode: the one where freeing space may not be enough, because an in-flight write at the moment the disk filled can leave the journal or index corrupt.

This is not the same incident as StorePercentUsage hitting 100%. StorePercentUsage measures KahaDB against the configured storeUsage limit in activemq.xml. Disk full measures the partition against physical capacity with df. They are independent limits, and either can fire first. If storeUsage is larger than the partition, the OS runs out of space while the broker still thinks it has headroom, and the failure arrives with no warning from any ActiveMQ metric. This guide covers the OS-level case; for the configured-limit case, see ActiveMQ store is full.

ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ DLQ never expires: setting TTL so the dead-letter queue stops leaking storage

Your ActiveMQ.DLQ has been sitting at some non-zero depth for months. Nobody looks at it because “it’s just the DLQ.” Meanwhile StorePercentUsage creeps up a fraction of a percent a day, the db-*.log journal files keep accumulating, and one morning the broker hits 100% store usage and blocks every persistent producer on the bus.

This is the default behavior of ActiveMQ Classic, not a bug. Messages sent to the dead-letter queue have no TTL. They accumulate forever, and every one of them is a live reference that pins KahaDB journal files against garbage collection. The fix has two parts: configure expiration on the DLQ so the leak stops, and drain-and-investigate the existing backlog instead of just deleting it. Every DLQ message is a failed business transaction; purging blind throws away the evidence.

ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ enqueue outpacing dequeue: reading the rate imbalance before the backlog

On a healthy ActiveMQ Classic broker, enqueue rate and dequeue rate track each other within normal burst variability. When enqueue outpaces dequeue for more than a few minutes, the broker is accumulating a backlog whether or not QueueSize has moved enough to alarm yet. The rate delta is the leading indicator; QueueSize, MemoryPercentUsage, and StorePercentUsage are the lagging confirmations. If you wait for the backlog gauges to fire, you have already lost your runway.

ActiveMQ expired message count climbing: TTL expiry and silent correctness loss

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ expired message count climbing: TTL expiry and silent correctness loss

A queue looks fine. QueueSize is stable, dequeue rate is non-zero, no alerts on memory or store. Then someone asks why a customer order never processed, and you find ExpiredCount on the destination has been climbing for days. Every tick of that counter is a message the broker threw away or shuffled into the DLQ because its TTL ran out before a consumer got to it. Nothing crashed. Nothing paged. The data is just gone.

ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ GC pause death spiral: long pauses, heartbeat timeouts, and reconnect storms

The broker is up. The port is listening. But clients keep dropping, reconnecting, dropping again, and every cycle makes things worse. Connection count charts show a sawtooth: a sudden drop, a spike, another drop. In the JVM, heap usage sits above 90% after every major GC and full GC pauses are climbing into the multi-second range.

This is the GC pause death spiral, a positive-feedback loop and one of the characteristic failure archetypes of ActiveMQ Classic. Heap pressure produces long GC pauses. A long pause freezes every broker thread, including the ones answering client keepalives. Clients exceed wireFormat.maxInactivityDuration (default 30000 ms) and disconnect. After the pause ends, every client reconnects at once, and each reconnect allocates new connection, session, consumer, and subscription objects. That allocation spike raises heap pressure further, the next GC pause is longer, and more clients time out. The broker oscillates between being frozen in GC and being hammered by reconnect storms until it is effectively down.

ActiveMQ HA failover stuck: standby cannot acquire the store lock

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ HA failover stuck: standby cannot acquire the store lock

The active broker died. The standby broker’s process is running, the JVM looks healthy, and yet no client can connect anywhere. ss shows nothing listening on 61616 on either node. The failover you designed for is not happening.

This is the shared-storage master/slave stall: the standby is blocked waiting to acquire the lock file in the KahaDB directory, and until it gets that lock it will not start its transport connectors. That behavior is by design. What is not by design is the lock never becoming available. The usual suspects are a stale lock file left behind by a crashed active, an unhealthy NFS lock daemon, or a SAN mount that is unavailable on the standby.

ActiveMQ InactivityIOException: Channel was inactive for too long

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ InactivityIOException: Channel was inactive for too long

Your producers or consumers are dropping JMS connections with this in the stack trace:

org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long

Sometimes it is one client. Sometimes every client on the broker disconnects within the same second and reconnects in a burst. The exception points at the connection, but the connection is almost never the root cause. Something on one side of the wire stopped producing traffic for longer than the OpenWire inactivity timeout, and the other side declared it dead.

ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ InFlightCount high: prefetch full, acks stalled, and zombie consumers

A queue shows QueueSize near zero, consumers are connected, and nothing is being processed. Dequeue rate is flat, message age is rising upstream, and the only number that looks wrong is InFlightCount, sitting at a suspiciously round value. This is the zombie consumer pattern: the broker dispatched messages into consumer prefetch buffers, and those messages are never coming back as acknowledgments.

ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ JVM heap exhaustion: OutOfMemoryError and the OOM kill

The broker was fine an hour ago. Now the process is gone, or it is alive but frozen: clients disconnected, the web console hangs, and the log shows java.lang.OutOfMemoryError or, if it runs in a container, nothing at all because the kernel killed the JVM with SIGKILL (exit code 137, i.e. 128+9).

This is ActiveMQ JVM heap exhaustion. It is distinct from the broker’s internal memoryUsage hitting 100%. That limit triggers producer flow control and blocks send() calls; this one kills or freezes the JVM itself. The confusing part is that the two are independent counters. You can have MemoryPercentUsage at 50% with plenty of headroom while JVM heap is at 97% and the next full GC never finishes. Non-message allocations (destination metadata, MBeans, connection and session state, cursor overhead, transport buffers) live on the heap too, and ActiveMQ’s flow control accounting does not protect them.

ActiveMQ JVM thread count climbing: transport threads, NIO, and thread leaks

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ JVM thread count climbing: transport threads, NIO, and thread leaks

The broker’s JVM thread count has been climbing for hours or days. Nothing has broken yet, but the trend only goes one direction, and it ends with context-switch overhead, native memory pressure from thread stacks, and eventually a broker that cannot accept new connections or gets OOM-killed despite a healthy heap.

A rising thread count on ActiveMQ Classic is not one problem. With the default blocking TCP transport, thread count is a proxy for connection count, so growth usually means connections are growing. When thread count climbs without connection growth, you have an actual thread leak, and the diagnostic path is completely different. Telling the two apart takes about two minutes with JMX.

ActiveMQ KahaDB corruption: the broker won't start after an unclean shutdown

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ KahaDB corruption: the broker won’t start after an unclean shutdown

The broker was killed (kill -9, power loss, OOM killer, container limit hit) and now it will not come back. Either the process exits during startup, or it sits there with the log stuck partway through KahaDB recovery and port 61616 never accepts a client. The log mentions db.data, a db-*.log journal file, or an IOException about a missing data file.

ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ KahaDB db.data index bloat: slow lookups and slow startup recovery

You restart the broker after an unclean shutdown and it sits there for 20, 30, 40 minutes before it accepts a single client connection. Or the broker is running, but persistent message dispatch feels sluggish and disk reads on the KahaDB partition are elevated for no obvious reason. You look at the KahaDB directory and the journal files are not the problem. The problem is db.data, the B-tree index, sitting at multiple gigabytes.

ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ KahaDB journal files not deleted: one unacked message pinning a 32MB log

Your application queues are empty or nearly empty. Consumers are connected and processing. Yet the KahaDB directory keeps growing, db-*.log files pile up, StorePercentUsage creeps upward, and disk free space trends toward zero. This is the classic KahaDB journal pinning problem, and it confuses operators precisely because the visible queue state looks healthy.

The mechanism is simple and unforgiving: a KahaDB journal file (default 32MB) is only reclaimable when every message stored in that file has been acknowledged. One unacknowledged message anywhere in the file pins the entire file on disk. If that one message is sitting in the DLQ, held by an offline durable subscriber, or stuck in a dead consumer’s prefetch buffer, the file stays, and new files keep being written behind it.

ActiveMQ KahaDB journal write latency: fsync as the persistent-throughput ceiling

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ KahaDB journal write latency: fsync as the persistent-throughput ceiling

A producer is sending persistent messages to ActiveMQ Classic and throughput is far below what the network, CPU, and broker configuration suggest should be possible. Sends are not failing. There are no exceptions. Each send() just takes a few milliseconds, or tens of milliseconds, and the aggregate rate caps out no matter how many producers you add.

This is almost always the KahaDB journal fsync. For persistent messages, ActiveMQ Classic writes the message to the KahaDB write-ahead journal and acknowledges the producer only after the fsync completes. That fsync is the critical path for every persistent send, and its latency sets a hard ceiling on persistent throughput. When the underlying storage gets slow, persistent messaging gets slower in direct proportion to the fsync latency.

ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ memory limit reached: MemoryPercentUsage at 100% and the flow-control cliff

Your producers have stopped sending. No exceptions, no timeouts, no errors on the producer side. The send() calls just hang, and upstream services pile up threads waiting on the broker. In the broker log you find the line operators search for at 3 a.m.:

Usage Manager Memory Usage ... reached memory limit

MemoryPercentUsage on the Broker MBean is at 100%. This is ActiveMQ Classic’s own memory accounting, not JVM heap, and 100% is not a slowdown. It is a cliff edge. At 99% everything works. At 100% every producer is flow-controlled: the broker stops reading from producer sockets, TCP backpressure builds, and sends block silently until memory frees up.

ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ MemoryPercentUsage climbing: reading the flow-control leading indicator

Your ActiveMQ broker’s MemoryPercentUsage gauge is at 65% and climbing. Nothing is broken yet. Producers are sending, consumers are consuming, queue depths look tolerable. But the trend line only goes one direction, and you know what happens at 100%: producer flow control activates, every producer’s send() call blocks silently, and upstream services start hanging with no error message anywhere.

This is the right moment to act, and the gauge gives you more information than most operators extract from it. The absolute value tells you where you are. The rate of climb tells you how much time you have. The surrounding signals tell you why. This article is about reading all three before the cliff edge.

ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ memoryUsage vs JVM heap: the two memory budgets teams confuse

An ActiveMQ Classic broker can hit 100% MemoryPercentUsage and silently block every producer while the JVM heap sits at 40%. The same broker, on a different day, can die from OutOfMemoryError while MemoryPercentUsage shows 50% and every queue-depth dashboard looks calm. Both incidents look contradictory until you understand that ActiveMQ tracks two separate memory budgets, and only one of them is the JVM’s.

ActiveMQ Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ Monitoring

What Is ActiveMQ?

ActiveMQ is a popular open-source message broker designed for message-oriented middleware. It is known for its versatility in handling various messaging protocols, supporting the Java Messaging Service (JMS) API, and its ability to run as a standalone broker or in a fully distributed enterprise setup. For technical users like DevOps, SREs, and developers, ActiveMQ allows seamless integration of numerous applications, facilitating reliable and asynchronous message exchanges. Learn more about ActiveMQ.

ActiveMQ monitoring checklist: the signals every production broker needs

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ monitoring checklist: the signals every production broker needs

Most ActiveMQ monitoring setups fail in one of two ways. Either they only watch process liveness and queue depth, and discover producer flow control from angry users. Or they export every JMX attribute into a dashboard nobody reads, and the one signal that mattered is buried on page four.

This checklist organizes the signals that matter into four maturity levels, from “is the broker alive” to “can you prove end-to-end message flow works.” Each level lists the signal, where it comes from, and the threshold that makes it actionable. Scope is ActiveMQ Classic (5.x and 6.x Classic stream). Artemis has a different store, memory model, and MBean tree; do not apply these thresholds to it.

ActiveMQ monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ monitoring maturity model: from survival to expert

Most ActiveMQ outages are not caused by exotic failure modes. They are caused by signals nobody was watching: a DLQ that grew for three weeks, a memory limit that hit 100% and silently blocked every producer, an offline durable subscription that pinned journal files until the disk filled. The signals were available the whole time. The team had not instrumented them yet.

ActiveMQ network bridge connected but not forwarding: broken demand forwarding

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ network bridge connected but not forwarding: broken demand forwarding

The bridge is up. JMX shows the network bridge MBeans, the broker log shows a successful connection to the remote broker, and yet messages produced on broker A sit in the queue while consumers on broker B wait idle. No errors on either side. Each broker’s local dashboard looks healthy.

This is broken demand forwarding in an ActiveMQ Classic Network of Brokers. A network bridge is not a pipe. It forwards messages only when it knows a consumer exists on the remote side, and it learns that through advisory messages. When the bridge is connected but the demand signal is not flowing, you get a silent stall: backlog on one broker, starvation on the other, nothing alarming on a per-broker view.

ActiveMQ network bridge down: a Network of Brokers partition and store-and-forward backlog

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ network bridge down: a Network of Brokers partition and store-and-forward backlog

In a Network of Brokers, messages cross between ActiveMQ Classic brokers over network bridges. When a bridge drops, the failure is asymmetric: the origin broker keeps accepting messages for destinations whose consumers live on the remote broker, and those messages pile up locally. The remote broker keeps running with consumers connected and nothing to consume. Both brokers look healthy in isolation. Only cross-broker correlation reveals the partition.

ActiveMQ Network of Brokers replay storm: burst forwarding after a bridge reconnect

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ Network of Brokers replay storm: burst forwarding after a bridge reconnect

A network bridge between two ActiveMQ brokers drops. Producers keep sending to the origin broker, and because ActiveMQ networks do reliable store-and-forward, the messages pile up locally. When the bridge reconnects, the accumulated backlog is replayed across the bridge in a burst. The receiving broker, idle a second ago, is now ingesting the backlog at wire speed. Its memory usage spikes, and if the spike reaches 100%, producer flow control activates on the receiver and producers there block silently.

ActiveMQ offline durable subscriber pending messages: the silent storage leak

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ offline durable subscriber pending messages: the silent storage leak

StorePercentUsage has been climbing for three weeks. Your queues are draining, consumer counts look right, dequeue rates track enqueue rates, and yet the KahaDB directory keeps growing and journal files keep stacking up. Nothing in the queue-depth metrics explains it.

The usual suspect is not a queue at all. It is a durable topic subscription whose subscriber went away and never came back. Every persistent message published to that topic is still being written to the store on behalf of that subscription, and it will keep happening forever: ActiveMQ Classic has no automatic expiration for offline durable subscribers by default.

ActiveMQ oldest message age: the queue latency depth alone cannot show

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ oldest message age: the queue latency depth alone cannot show

Every ActiveMQ dashboard has queue depth on it. Almost none show the age of the oldest pending message, which is the number that actually maps to your SLA. A queue holding 50 messages whose head has been waiting 45 minutes is a much bigger problem than a queue holding 50,000 messages whose head is 2 seconds old and draining fast. Depth alone cannot tell you which of those two situations you are looking at.

ActiveMQ per-destination DLQ: IndividualDeadLetterStrategy vs the shared ActiveMQ.DLQ

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ per-destination DLQ: IndividualDeadLetterStrategy vs the shared ActiveMQ.DLQ

Out of the box, ActiveMQ Classic routes every undeliverable message from every queue and topic into a single queue: ActiveMQ.DLQ. One noisy destination with a poison-message problem mixes its failures with everyone else’s, and the only way to attribute a message back to its source is to browse the DLQ and inspect the JMSDestination property on each entry. On a busy broker that is slow, expensive, and usually done too late.

ActiveMQ per-destination memory usage: one noisy queue blocking every producer

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ per-destination memory usage: one noisy queue blocking every producer

Every producer on the broker is stuck in send(). Broker-level MemoryPercentUsage reads 100, the broker log shows the usage manager hitting its memory limit, and upstream services are timing out. But when you list queue depths, almost every queue is empty. One queue holds nearly all the pending messages.

That one queue has consumed the broker’s shared memory pool. Every pending message is charged against its destination’s memory accounting and against the broker-wide system memory limit. If either limit is reached, flow control activates. With no per-destination memory limit set, ActiveMQ Classic does not throttle just that queue’s producers: it throttles everyone’s. This is the default behavior, not a bug.

ActiveMQ poison message loop: redelivery storms and DLQ growth

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ poison message loop: redelivery storms and DLQ growth

A queue looks busy. Consumers are connected, dequeue counters are moving, and yet the business workflow has stalled: orders are not completing, messages are getting older, and somewhere in the broker a queue called ActiveMQ.DLQ is quietly filling up. This is the poison message loop, and it is one of the most misread failure modes in ActiveMQ Classic because the headline metrics look healthy while useful work has stopped.

ActiveMQ prefetch limit: unacked hoarding versus idle consumers

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ prefetch limit: unacked hoarding versus idle consumers

You have four consumers on a queue, producers are sending steadily, and yet throughput is a quarter of what you expect. ConsumerCount says 4. QueueSize looks small. DequeueCount is crawling. The broker looks healthy and the consumers look connected, but three of them are doing nothing.

This is the classic prefetch mismatch. The ActiveMQ prefetch limit controls how many messages the broker pushes to a consumer before it requires acknowledgments back. It is a client-side buffer the broker fills eagerly. With the default of 1000 for queues, the first consumer to connect can absorb the entire visible backlog into its prefetch buffer, where those messages are invisible to load balancing, no longer read as pending depth, and pinned against broker memory until they are acked.

ActiveMQ producer flow control: why send() hangs and producers block silently

Mon, 01 Jan 0001 00:00:00 +0000

Your application is healthy. No exceptions, no error logs, no timeouts. But requests are piling up and every thread that touches the message producer is stuck inside send(). Thread dumps show all your producer threads parked in the JMS client, waiting on a socket write that never completes. Nothing is wrong on the client. Nothing looks wrong on the broker either, until you check one number: MemoryPercentUsage is at 100.

ActiveMQ queue with zero consumers: ConsumerCount at zero and a growing backlog

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ queue with zero consumers: ConsumerCount at zero and a growing backlog

A production queue shows ConsumerCount=0, QueueSize is climbing, and EnqueueCount keeps ticking up while DequeueCount is flat. Nobody is draining the queue. Every message that arrives stays in the broker, charged against destination memory and, for persistent messages, written into the KahaDB journal. Left alone, this ends in one of two places: broker memory hits 100% and producer flow control silently blocks every producer, or the store fills and persistent messaging halts entirely.

ActiveMQ QueueSize growing: reading the broker's backlog gauge correctly

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ QueueSize growing: reading the broker’s backlog gauge correctly

QueueSize is the first number every operator looks at, and the one most often misread. The common mental model is “messages waiting in the queue,” but that is not what the broker reports. QueueSize is a broker-maintained gauge, not enqueued-minus-dequeued, and it includes messages already dispatched to consumers but not yet acknowledged. Two queues showing QueueSize=5000 can be in completely different states, and a queue showing QueueSize=0 can still be in trouble.

ActiveMQ reconnection storm: transport accept spikes and CPU on handshakes

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ reconnection storm: transport accept spikes and CPU on handshakes

The broker is up. Queues are draining. But the accept rate on the OpenWire connector is running at ten times baseline, JVM thread count and open file descriptors are climbing in a sawtooth, and CPU is pinned even though message throughput is flat or down. Clients are connecting, disconnecting, and reconnecting in a tight loop, and every one of those connections costs the broker real work: a TCP accept, an optional TLS handshake, wire-format negotiation, and a new transport thread.

ActiveMQ redelivery rate climbing: rollbacks, nacks, and retry storms

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ redelivery rate climbing: rollbacks, nacks, and retry storms

Your dequeue rate looks busy, but useful work is not getting done. Messages are dispatched, rolled back or nacked, and dispatched again. The broker spends its cycles retrying instead of making progress, and the redelivery counters climb. This is the poison message replay storm pattern in its early stage, and it leads two things you do not want: dead letter queue growth and silent correctness loss.

ActiveMQ shared-storage HA split-brain: two brokers holding the store lock

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ shared-storage HA split-brain: two brokers holding the store lock

In a shared-storage HA pair, exactly one ActiveMQ Classic broker is supposed to hold the KahaDB store lock and run active. The second broker sits in a polling loop, waiting for the lock, with its transport connectors down. Split-brain is the state where that invariant breaks: both brokers believe they are active, both accept client connections, and both write to the same KahaDB store on shared storage.

ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ store is full: StorePercentUsage at 100% and persistent messaging halted

StorePercentUsage has hit 100% on the broker MBean and persistent producers have stopped. Their send() calls are either blocked in flow control or being rejected, depending on your destination policy. Non-persistent traffic may still flow, which makes the outage look partial from the outside.

This is the disk-side equivalent of the memory wall. Where MemoryPercentUsage at 100% blocks producers against the configured memory limit, StorePercentUsage at 100% blocks persistent producers against the configured storeUsage limit in activemq.xml. The broker log shows a “Persistent store is Full” message, and producers stall silently unless you configured sendFailIfNoSpaceAfterTimeout.

ActiveMQ store usage climbing: the store exhaustion spiral

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ store usage climbing: the store exhaustion spiral

StorePercentUsage was 62% last month. Last week it was 74%. This morning it crossed 80% and your alert fired. Queue depths look normal, consumers are connected, dequeue rates look healthy. Nothing is obviously on fire, but the persistent store keeps growing and nobody knows why.

This is the store exhaustion spiral, one of the most common slow-burn failure patterns in ActiveMQ Classic. Unlike the memory-pressure cascade, which develops in minutes, the store spiral develops over days or weeks. Messages accumulate in the KahaDB journal faster than they are acknowledged, journal files pile up on disk, and the store limit creeps closer. When StorePercentUsage reaches 100%, the broker stops accepting persistent messages entirely and every persistent producer blocks. The warning signs were visible for weeks.

ActiveMQ StorePercentUsage vs actual disk space: the limit that fires after the disk is already full

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ StorePercentUsage vs actual disk space: the limit that fires after the disk is already full

Your broker just stopped accepting persistent messages. Producers are blocked or erroring, the KahaDB partition is at 100% disk usage, and yet your dashboard shows StorePercentUsage at 62%. The alert you set on store usage never fired. This is one of the most common ActiveMQ monitoring traps: StorePercentUsage and actual disk usage are two different counters measured against two different denominators, and if you have not reconciled them, one of them is lying to you.

ActiveMQ temp store full: non-persistent overflow and silent message drops

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ temp store full: non-persistent overflow and silent message drops

Non-persistent messages in ActiveMQ Classic live in broker memory. When memory fills, the broker spills them to an on-disk overflow area called the temp store (default data/tmp_storage). When the temp store fills, non-persistent messaging breaks. Depending on your destination policies, it can break in the worst possible way: messages are silently discarded while producers see no error and consumers just see less traffic.

ActiveMQ temporary destination leak: request-reply queues that never close

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ temporary destination leak: request-reply queues that never close

You open the broker’s JMX view and the TemporaryQueues attribute lists hundreds or thousands of entries. The count only goes up. Nothing in the application logs looks wrong, request-reply calls mostly work, but the destination count climbs with traffic and never comes back down. That is a temporary destination leak.

Temporary destinations back the classic JMS request-reply pattern: a client creates a TemporaryQueue, sends a request with the temp queue as the reply-to, and waits for the response. In a healthy system these destinations cycle constantly: created, used, deleted. When the count grows monotonically with request volume, something in that lifecycle is broken, and the broker is accumulating MBeans, metadata, and advisory traffic for destinations that will never be used again.

ActiveMQ Too many open files: file descriptor exhaustion and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ Too many open files: file descriptor exhaustion and refused connections

The broker log shows Too many open files on the accept path, new clients cannot connect, and existing clients may start timing out. In the worst case the broker also fails to open a new KahaDB journal file during rotation, and now you have a store integrity problem that looks, at first glance, like a disk failure. It is not. The broker has run out of file descriptors.

ActiveMQ transport connector not accepting: a live broker that refuses one protocol

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ transport connector not accepting: a live broker that refuses one protocol

The broker process is running. The JVM is up, OpenWire clients on 61616 are producing and consuming, and your process-level health check is green. But the AMQP clients on 5672 cannot connect, and their reconnect loops are filling your application logs.

This partial failure is worse than a full outage in one specific way: most monitoring does not see it. A TCP check against one port, or a process-alive check, tells you nothing about the other four connectors. ActiveMQ Classic runs each transport connector (OpenWire 61616, AMQP 5672, STOMP 61613, MQTT 1883, WebSocket 61614) as its own accept path, and each one can fail independently while the rest of the broker looks healthy.

ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages

Mon, 01 Jan 0001 00:00:00 +0000

ActiveMQ.DLQ growing: dead letter queue accumulation and poison messages

The queue named ActiveMQ.DLQ is growing and nobody is consuming from it. That is the default dead letter queue: it has no consumers, no TTL, and no automatic cleanup, so every message that lands there stays there forever.

Every message in the DLQ is two problems at once. It is a failed business transaction that some consumer gave up on after exhausting redeliveries. And it is a storage leak: DLQ messages are never acknowledged, and in KahaDB a single unacknowledged message pins its entire journal file. A DLQ that grows quietly for weeks is a common root cause behind store exhaustion, which eventually halts persistent messaging on the whole broker.

Actona Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Adaptec Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Adaptec RAID

Mon, 01 Jan 0001 00:00:00 +0000

Adaptec RAID Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Adaptec RAID Monitoring

What Is Adaptec RAID?

Adaptec RAID is a trusted storage solution designed to manage and protect your data using Redundant Array of Independent Disks (RAID) technology. These controllers are widely used for enhancing storage reliability and performance by spreading data across multiple disks.

Monitoring Adaptec RAID With Netdata

To monitor Adaptec RAID efficiently, Netdata offers a comprehensive monitoring tool that integrates seamlessly with your system. The $name monitoring tool helps you gain insights into the status and health of your RAID arrays in real time, enabling proactive maintenance and swift troubleshooting.

Adtran SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Adva AG Optical Networking SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Advanced Fibre Communications Afc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Advantech Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aerohive Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aethra SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Affirmed Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Agent SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Agere Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Agfeo GmbH Co KG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aginode Germany GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Airespace Inc Formerly Black Storm Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Alamos FE2 server

Mon, 01 Jan 0001 00:00:00 +0000

Alamos FE2 server Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Alamos FE2 server Monitoring

What Is Alamos FE2 server?

The Alamos FE2 server is a comprehensive platform that demands diligent monitoring to ensure optimal performance and health. This server acts as a vital component in various infrastructures requiring consistent oversight and data collection for efficient operations.

Monitoring Alamos FE2 server With Netdata

Monitoring the Alamos FE2 server is streamlined with Netdata, which employs an openmetrics (Prometheus) exporter. Netdata is capable of ingesting data from any Prometheus exporter, a significant advantage over traditional setups requiring a dedicated Prometheus server or Grafana for visualization. By leveraging Netdata, users obtain automated dashboards, intelligent alerts, and real-time insights designed to keep the Alamos FE2 server running smoothly.

Alaxala Networks Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Albal Ingenieros S A SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Albentia Systems S A SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Albis Technologies Ltd Formerly Siemens Switzerland Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Alcatel Lucent

Mon, 01 Jan 0001 00:00:00 +0000

Alcatel Lucent ENT

Mon, 01 Jan 0001 00:00:00 +0000

Alcatel Lucent Enterprise SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Alcatel Lucent IND

Mon, 01 Jan 0001 00:00:00 +0000

Alcatel Lucent Omni Access WLC

Mon, 01 Jan 0001 00:00:00 +0000

Alcatel-Lucent BGP

Mon, 01 Jan 0001 00:00:00 +0000

Ale USA Inc Omniswitch SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Alebra Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Alerta

Mon, 01 Jan 0001 00:00:00 +0000

Allaire Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Allied Telesis Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Allot Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Alpine Linux

Mon, 01 Jan 0001 00:00:00 +0000

Alpine Optoelectronics Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Alteon Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Altergy Systems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Alvarion Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

AM2320

Mon, 01 Jan 0001 00:00:00 +0000

AM2320 Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is AM2320?

AM2320 is a temperature and humidity sensor that can be used to measure environment conditions. It features a range of features including a high accuracy of ± 0.5°C and low power consumption. It is suitable for a variety of applications, making it a versatile choice for your project.

Monitoring AM2320 with Netdata

The prerequisites for monitoring AM2320 with Netdata are to have AM2320 and Netdata installed on your system.

Amazon CloudWatch

Mon, 01 Jan 0001 00:00:00 +0000

Amazon Linux

Mon, 01 Jan 0001 00:00:00 +0000

Amazon SNS

Mon, 01 Jan 0001 00:00:00 +0000

AMD CPU & GPU

Mon, 01 Jan 0001 00:00:00 +0000

AMD CPU & GPU Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

AMD CPU & GPU Monitoring

What Is AMD CPU & GPU?

AMD CPUs and GPUs form the core of many computational systems, driving performance in everything from gaming computers to data centers that power cloud applications. The AMD System Management Interface (SMI) is crucial for monitoring the performance and health of these processors, helping users optimize hardware performance through precise metrics.

Monitoring AMD CPU & GPU With Netdata

Monitoring AMD CPU & GPU is essential for ensuring that your systems operate efficiently and without interruption. Netdata leverages the AMD SMI Exporter to monitor these devices. By using an openmetrics (Prometheus) exporter, Netdata can seamlessly ingest metrics without the need for a Prometheus server or Grafana. Once connected, users can enjoy automated dashboards and alerts, making it a comprehensive AMD CPU & GPU monitoring tool.

AMD GPU

Mon, 01 Jan 0001 00:00:00 +0000

American Power Conversion Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Amperion Incorporated SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

An D Cz SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ancor Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Andover Controls Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Anue

Mon, 01 Jan 0001 00:00:00 +0000

Anuesystems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aol Netscape Communications Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ap Nederland B.V. SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Apache

Mon, 01 Jan 0001 00:00:00 +0000

Apache /server-status exposed: the reconnaissance leak hiding in mod_status

Mon, 01 Jan 0001 00:00:00 +0000

Apache /server-status exposed: the reconnaissance leak hiding in mod_status

A scanner, pentest, or audit just flagged your Apache server: /server-status is reachable from the internet. mod_status renders a live view of your server’s internals, and anyone who can load it can watch your traffic in near real time.

The operational problem is that mod_status is also the best monitoring source Apache has. BusyWorkers, the scoreboard, requests per second: all of it comes from /server-status?auto. So the fix is not “disable mod_status.” The fix is to restrict who can reach it, verify the restriction holds on every vhost, and keep scraping it locally.

Apache %D includes client transfer time: the latency alert that cries wolf

Mon, 01 Jan 0001 00:00:00 +0000

Apache %D includes client transfer time: the latency alert that cries wolf

Your p99 latency alert fired at 03:00. You pull the access log, sort by %D, and find requests with durations of 400, 600, even 800 seconds. The server looks fine: CPU normal, workers mostly idle, backend healthy. The “slow” requests are large file downloads to clients on slow connections. Apache served the response instantly; the client took thirteen minutes to read it.

Apache %T vs %D vs %{ms}T: stop rounding latency to whole seconds

Mon, 01 Jan 0001 00:00:00 +0000

Apache %T vs %D vs %{ms}T: stop rounding latency to whole seconds

Open your Apache access log and look at the request duration field. If every line shows 0, your monitoring pipeline is probably fine. Your LogFormat is the problem. The %T directive logs time to serve the request in whole seconds, so any request that completes in under one second logs as 0. On a fleet where most requests finish in well under 100ms, %T collapses the entire latency distribution into a single useless value.

Apache 401/403 floods: credential stuffing, scanner probes, and access denials

Mon, 01 Jan 0001 00:00:00 +0000

Apache 401/403 floods: credential stuffing, scanner probes, and access denials

Your access log is filling with 401s against a login endpoint, or 403s across paths like /.env and /wp-admin. Some of this is the normal background radiation of the public internet. Some of it is an active credential stuffing run against your users. The status code alone does not tell you which; the shape of the traffic does.

The three patterns look different once you group the log lines:

Apache 500 Internal Server Error: modules, handlers, and misconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Apache 500 Internal Server Error: modules, handlers, and misconfiguration

Unlike the 502/503/504 family, which Apache generates while proxying to a broken or exhausted backend, a 500 is generated inside Apache itself or by the handler Apache invoked: a module crashed, a CGI script failed, a rewrite rule looped, an .htaccess directive is invalid, or the server hit a permission problem reaching the content.

That distinction is the diagnostic strategy. For a 502 you look at the backend. For a 500 you look at Apache’s own error log, because every 500 Apache emits has a corresponding error-log line with the real cause. The access log tells you that it happened and which URL triggered it. The error log tells you why.

Apache 502 Bad Gateway: a backend that returned an invalid response

Mon, 01 Jan 0001 00:00:00 +0000

Apache 502 Bad Gateway: a backend that returned an invalid response

You are seeing 502 Bad Gateway responses from Apache, usually in bursts, often on some requests and not others. Users report intermittent failures. Apache is running, the port is open, static content may even work. The proxied application path is the thing failing.

A 502 from Apache is almost never an Apache bug. In a mod_proxy deployment, 502 means Apache acted as a reverse proxy, forwarded the request to a backend, and got back something it could not use: a garbled response, an empty response, a connection closed mid-reply, or a connection that refused to carry the request at all. Apache is surfacing a backend signal.

Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion

Your access log is filling with 503s and users are reporting the site is down. The status code tells you almost nothing: Apache returns 503 for two root causes that look identical in the access log but need completely different fixes.

The first is frontend worker exhaustion. Every worker slot in the scoreboard is busy, Apache has hit MaxRequestWorkers, and new connections queue in the kernel backlog until they time out. The second is a mod_proxy failure: a backend is down, marked errored, or the per-child proxy connection pool is too small, so Apache refuses to forward requests even though its own workers are mostly idle.

Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up

Mon, 01 Jan 0001 00:00:00 +0000

Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up

A 504 on an Apache-proxied path means a gateway in the request chain timed out. There are two cases:

Apache generated the 504: mod_proxy waited longer than the effective proxy timeout for the backend.
The backend generated the 504: Apache received that status and passed it through, often because the backend was proxying to a dead dependency of its own.

The access log alone does not distinguish those cases. Correlate the 504 with Apache’s error log.

Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means

Mon, 01 Jan 0001 00:00:00 +0000

Your Apache 5xx error rate just spiked. The status code is the first fork in the diagnostic path, and it is a sharp one: a 500 means the failure happened inside Apache or a module, while 502, 503, and 504 are mod_proxy telling you something about the relationship between Apache and a backend. Treating them as one bucket of “server errors” wastes the best lead you have.

There is a second trap. A sustained 5xx rate almost always means something real is broken, because 5xx is server-side failure by definition. But the reverse is not true: a proxied application that returns error pages with HTTP 200 is invisible to status-based monitoring. Low 5xx is necessary, not sufficient, evidence of health.

Apache accepting deprecated TLS: TLS 1.0/1.1 and weak-cipher exposure

Mon, 01 Jan 0001 00:00:00 +0000

Apache accepting deprecated TLS: TLS 1.0/1.1 and weak-cipher exposure

If a compliance scan flags “TLS 1.0 enabled” or “weak cipher suites supported” on an Apache vhost, the finding is almost always real. Apache’s default SSLProtocol in the 2.4 branch is all -SSLv3, so TLS 1.0 and TLS 1.1 are offered unless someone explicitly removed them. A vhost that was correct three years ago may still be accepting protocol versions that PCI DSS and modern browser policy treat as deprecated.

Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted

Mon, 01 Jan 0001 00:00:00 +0000

Apache AH00484: server reached MaxRequestWorkers setting - worker pool exhausted

You found this line in the Apache error log:

AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

This is Apache explicitly telling you that every worker slot in its pool was occupied and it had nowhere to put a new connection. From that moment, new connections queue in the kernel’s TCP listen backlog, and once the backlog fills, clients get connection refused. Users experience timeouts or a dead site while the Apache process looks perfectly healthy in ps.

Apache AH00558: Could not reliably determine the server's fully qualified domain name

Mon, 01 Jan 0001 00:00:00 +0000

Apache AH00558: Could not reliably determine the server’s fully qualified domain name

You restart Apache, run apachectl configtest, or check the error log after a deploy, and you see it again:

AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message

The short answer: Apache is still serving traffic. This warning does not stop startup, does not drop requests, and is not a security issue. It means no global ServerName directive is set, so Apache guessed a name for itself and is telling you what it guessed.

Apache and nf_conntrack: table full and the invisible packet drops

Mon, 01 Jan 0001 00:00:00 +0000

Apache and nf_conntrack: table full and the invisible packet drops

Users report timeouts and intermittent connection failures, the load balancer is flapping the backend out of rotation, and yet everything on the Apache host looks fine. BusyWorkers are normal. The 5xx rate is flat. The listen backlog is empty. CPU and memory are unremarkable. Apache is not logging errors, because from Apache’s point of view, nothing is wrong.

The failure is one layer below Apache, in the kernel’s netfilter connection tracking subsystem. When the nf_conntrack table fills up, the kernel emits nf_conntrack: table full, dropping packet and starts silently discarding packets. New SYN packets never reach the TCP stack, so they never reach the listen backlog, so they never reach an Apache worker. Clients see their SYNs dropped and retransmit until they give up: a timeout, not a refusal, which makes it look like a network problem.

Apache backend response time: telling 'Apache is slow' from 'the backend is slow'

Mon, 01 Jan 0001 00:00:00 +0000

Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’

Users report the site is slow. Apache is up, requests eventually complete. Someone says “Apache is slow,” someone else says “the app is slow,” and both are guessing, because from the outside the two are indistinguishable: the client just sees a slow response.

When Apache proxies to a backend, the %D value in your access log bundles three things together: Apache’s own processing, time waiting for the backend, and time transferring the response to the client. A slow backend, a saturated Apache, and a client on a bad connection all produce the same inflated %D. Without separate instrumentation per component, you cannot attribute the latency, and teams routinely burn hours tuning Apache when the database is the problem.

Apache balancer member in error state: reading balancer-manager and failover

Mon, 01 Jan 0001 00:00:00 +0000

Apache balancer member in error state: reading balancer-manager and failover

You opened balancer-manager, or someone pasted a screenshot into the incident channel, and one of your BalancerMembers shows Err in the status column. Traffic is still flowing, but the pool is quietly running on fewer backends than you think. If enough members flip to error state, Apache stops proxying entirely and returns 503s even though httpd itself is healthy.

This state is mod_proxy_balancer doing its job: it detected failures against a backend and pulled that member out of rotation so requests stop dying on it. The problem is that the balancer tells you almost nothing about why, and it keeps the member sidelined on its own retry schedule regardless of whether the backend has recovered.

Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status

Mon, 01 Jan 0001 00:00:00 +0000

Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status

BusyWorkers and IdleWorkers are the two most-quoted numbers from Apache’s mod_status output, and the two most frequently misread. Together they are the primary saturation gauge for the server: how much of the worker pool is currently occupied. Read them wrong and you either miss the onset of worker exhaustion or page someone for healthy autoscaling churn.

This guide covers what the two counters actually count, how to turn them into a utilization ratio you can alert on, why Apache degrades at a cliff edge rather than gradually, and how to distinguish the two very different situations that both show IdleWorkers: 0.

Apache capped by systemd: TasksMax, LimitNOFILE, and MemoryMax override your config

Mon, 01 Jan 0001 00:00:00 +0000

Apache capped by systemd: TasksMax, LimitNOFILE, and MemoryMax override your config

You raised MaxRequestWorkers, restarted Apache, and it still refuses to spawn more workers. The error log shows no AH00484, the host has free CPU and RAM, and connections queue and time out. Or Apache was OOM-killed even though free showed gigabytes available. Or you hit “Too many open files” at a fraction of the limit you set in /etc/security/limits.conf.

Apache child pid exit signal Segmentation fault: crashing workers

Mon, 01 Jan 0001 00:00:00 +0000

Apache child pid exit signal Segmentation fault: crashing workers

You found this line in the error log:

[core:notice] [pid 1234] AH00052: child pid 5678 exit signal Segmentation fault (11)

A child process died on a memory access violation. The parent logged the exit status and spawned a replacement. That is Apache’s designed recovery path, and a single occurrence with no user impact is that mechanism working. But a segfault is never “normal noise”: a bug executed in Apache, a loaded module, or a shared library, and any segfault in production deserves root-cause analysis. Multiple per minute means the service is actively degrading.

Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn

Mon, 01 Jan 0001 00:00:00 +0000

Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn

You ran ss -tan on an Apache host and saw thousands of connections that are not ESTABLISHED. Some are in TIME_WAIT, some in CLOSE_WAIT, and the numbers look alarming. The first question is not “how do I get rid of them” but “which of these is actually a problem”.

The two states look similar in a socket listing but mean opposite things. TIME_WAIT is the kernel doing its job after a connection closes cleanly; it is normal churn on any busy web server. CLOSE_WAIT means the remote peer closed the connection and Apache never finished closing its side; a sustained, growing CLOSE_WAIT count is an application-side leak that consumes file descriptors and, eventually, workers.

Apache configtest and Include wildcards: catching bad config before it bites

Mon, 01 Jan 0001 00:00:00 +0000

Apache configtest and Include wildcards: catching bad config before it bites

apachectl configtest (equivalent to httpd -t) parses your Apache configuration and reports Syntax OK or a specific error. It is the cheapest safety check in your toolchain, and it is the main thing standing between a bad edit and the stale-config trap: the state where you believe a new configuration is live, but Apache rejected it at reload time and is still running the old one.

Apache CPU saturation: TLS handshakes, mod_deflate, mod_rewrite, and mod_security

Mon, 01 Jan 0001 00:00:00 +0000

Apache CPU saturation: TLS handshakes, mod_deflate, mod_rewrite, and mod_security

Apache serving static content is almost never CPU-bound. The request path (accept, read, sendfile, log) is cheap. When httpd processes start eating cores, the cause is nearly always something doing per-request or per-connection computation: TLS handshakes, mod_deflate compression, mod_rewrite regex evaluation, or mod_security rule processing.

The symptom is gradual, not a cliff. Latency creeps up as CPU saturates. Unlike worker exhaustion, which fails hard at 100% utilization, CPU saturation degrades service progressively, which makes it easy to ignore until p99 latency is unacceptable.

Apache error log monitoring: severity levels, AH codes, and what to alert on

Mon, 01 Jan 0001 00:00:00 +0000

Metrics tell you that something is wrong with Apache. The error log tells you what. When BusyWorkers pegs at MaxRequestWorkers, the scoreboard shows saturation, but the error log hands you AH00484: “server reached MaxRequestWorkers setting.” When a child dies, the process table shows a respawn; the error log shows “Segmentation fault.”

This guide covers how to read the log efficiently: the severity hierarchy, the AH code scheme introduced in 2.4, the patterns worth alerting on, and the configuration gotchas that silently suppress the messages you need.

Apache graceful reload ran the old config: the silent stale-configuration trap

Mon, 01 Jan 0001 00:00:00 +0000

Apache graceful reload ran the old config: the silent stale-configuration trap

You edited a vhost, ran apachectl graceful, saw no error, and moved on. Hours later someone notices the new TLS certificate is not being served, the new redirect is missing, or the old ProxyPass target is still receiving traffic. Apache never went down and no alert fired. The reload failed and the old configuration kept running.

This is one of Apache’s worst failure modes because nothing looks broken. The server keeps serving and request rates stay normal. The only thing wrong is that the running configuration is not the configuration on disk, which quietly invalidates every assumption you make while debugging the next incident.

Apache graceful restart pile-up: G states, stacked generations, and doubled memory

Mon, 01 Jan 0001 00:00:00 +0000

Apache graceful restart pile-up: G states, stacked generations, and doubled memory

Memory on an Apache host climbs in steps, each step landing a few minutes after a deploy, a config push, or a log rotation. The scoreboard shows an unusual number of G (gracefully finishing) workers. ps shows far more httpd children than MaxRequestWorkers should allow. Nothing is erroring yet, but the host is drifting toward swap, and the next restart will make it worse.

Apache HTTPD monitoring checklist: the signals every production web server needs

Mon, 01 Jan 0001 00:00:00 +0000

Apache HTTPD monitoring checklist: the signals every production web server needs

This is a working checklist for engineers running Apache HTTPD in production. It organizes the signals worth collecting into four maturity levels, from “is the process alive” through “which worker is stuck and why.” Use it to audit an existing setup for gaps or to build one without over-instrumenting on day one.

Two things before the list. First, every saturation signal in Apache is interpreted through the active Multi-Processing Module (MPM), so the checklist starts there. Second, the levels are cumulative: Level 2 assumes Level 1 is in place. Do not skip ahead. The scoreboard state distribution is useless if nobody is watching the error log for AH00484.

Apache HTTPD monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Apache HTTPD monitoring maturity model: from survival to expert

Most teams monitor Apache at whatever level their last outage forced them to reach. A disk fills up, the site goes down, and “check log disk space” appears in the runbook. A backend hangs, workers exhaust, and “watch BusyWorkers” gets added. This model organizes that organic growth into four deliberate levels so you can see what you have, what you are missing, and which blind spot will produce your next incident.

Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice

Mon, 01 Jan 0001 00:00:00 +0000

Apache keepalive consuming workers: KeepAliveTimeout, the K state, and MPM choice

The symptom looks like a capacity problem: Apache logs AH00484: server reached MaxRequestWorkers setting, new connections start queuing, and users see slow responses or 503s. But the request rate is low, and nothing in CPU, memory, or bandwidth explains it. Then you open the scoreboard and see it: a wall of K states. Most of your workers are not serving requests. They are parked on idle keepalive connections, waiting for a next request that may never come.

Apache killed by the OOM killer: the memory exhaustion cascade

Mon, 01 Jan 0001 00:00:00 +0000

Apache killed by the OOM killer: the memory exhaustion cascade

Your monitoring says Apache is up. The parent process exists, the port is listening, but the site is down or intermittently dead. When you finally check the kernel log, there it is: Out of memory: Killed process ... (httpd). Not once. Dozens of times, at regular intervals, going back hours.

This is the Apache OOM death spiral. The kernel OOM killer terminates httpd child processes because total memory demand exceeded RAM. The Apache parent survives, respawns the children, the new children immediately start serving queued requests and allocating memory, and the OOM killer kills them again. The service looks “running” while serving little or nothing.

Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

Apache listen queue overflow: Recv-Q growth, ListenBacklog, and refused connections

Users report “connection refused” or intermittent timeouts, but Apache is running, the port is open, and a TCP connect from localhost sometimes works. The load balancer flaps the backend in and out of rotation. Nothing in the access log explains it, because the failing connections never got far enough to be logged.

This is the signature of listen queue overflow. Connections complete the TCP handshake in the kernel and sit in the accept queue waiting for an Apache worker to call accept(). When workers cannot keep up, the queue fills, and the kernel starts ignoring or resetting new connection attempts. The server looks up. It is effectively down for a slice of arriving connections.

Apache ListenBacklog vs net.core.somaxconn: the silently truncated accept queue

Mon, 01 Jan 0001 00:00:00 +0000

Apache ListenBacklog vs net.core.somaxconn: the silently truncated accept queue

You tuned Apache for burst absorption. You set ListenBacklog 2048 (or left the default 511, reasoning it was generous). Then a traffic spike arrived, workers saturated, and connections were refused far earlier than your capacity model predicted. The scoreboard and MaxRequestWorkers get the blame, but the real culprit is often one layer down: the kernel quietly rewrote your backlog at listen() time and never told anyone.

Apache log rotation losing lines: logrotate copytruncate vs graceful reopen

Mon, 01 Jan 0001 00:00:00 +0000

Apache log rotation losing lines: logrotate copytruncate vs graceful reopen

You notice it after the fact: the access log has a gap. Requests you know happened, because the application processed them and the client got a response, never appear in any log file. Not in the current log, not in the rotated one. The gap lines up with the moment logrotate ran.

Or the opposite symptom: after rotation, Apache keeps writing to the old file. access.log.1 grows for days while access.log stays empty. Or every night at rotation time you see a blip of dropped connections and a spike of 499s or client resets, because the postrotate script does a hard restart instead of a graceful one.

Apache log stall deadlock: workers stuck in the L state, throughput at zero

Mon, 01 Jan 0001 00:00:00 +0000

Apache log stall deadlock: workers stuck in the L state, throughput at zero

The Apache parent process is running. The port is open. A TCP connect succeeds. But nothing is served: requests hang, throughput is at or near zero, and the scoreboard is a wall of L characters. Every worker has finished its request and is now blocked writing the log line for it.

Workers write access log entries synchronously at the end of the request cycle, before returning to the idle pool. If that write blocks, because the log filesystem is full or a piped logging program has stopped reading, the worker never becomes available again. New workers accept new connections, finish those requests, and block on the same write. Within minutes the entire worker pool is frozen in the Logging state.

Apache MaxConnectionsPerChild: bounding leaky modules by recycling children

Mon, 01 Jan 0001 00:00:00 +0000

Apache MaxConnectionsPerChild: bounding leaky modules by recycling children

The symptom usually arrives as a slow memory problem, not a clean Apache failure. Per-child RSS climbs for hours or days, total Apache memory grows linearly, swap starts to move, and then the kernel OOM killer begins shooting httpd children. The parent respawns them, the new children leak too, and the host enters a respawn and OOM loop.

When that pattern is present, MaxConnectionsPerChild 0 is often part of the story. The default value of 0 means children never recycle. A leaky module, or plain APR pool fragmentation in a long-lived prefork child, gets unlimited time to grow. Setting a finite value, commonly 5000 to 10000, forces each child to exit after handling that many connections so the parent can replace it with a fresh, smaller process.

Apache MaxRequestWorkers tuning: sizing the worker pool against memory

Mon, 01 Jan 0001 00:00:00 +0000

Apache MaxRequestWorkers tuning: sizing the worker pool against memory

MaxRequestWorkers is the single most misconfigured directive in Apache HTTPD. The failure pattern is always the same: someone picks a round number like 1000, deploys it on a 4GB server running mod_php children at 50MB RSS each, and the arithmetic (1000 x 50MB = 50GB of potential demand on 4GB of RAM) guarantees an OOM cascade the first time traffic reaches the limit. The setting feels like a performance knob. It is a memory budget.

Apache Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Apache Monitoring

What Is Apache?

Apache is one of the most widely used web servers in the world. Officially known as the Apache HTTP Server, it was developed and is maintained by an open community under the auspices of the Apache Software Foundation. Since its inception, Apache has grown into an essential component of the web infrastructure. For more details, visit the official Apache website.

Monitoring Apache With Netdata

Monitoring Apache is crucial to ensuring optimal web server performance, resource utilization, and improving user experiences. Netdata’s real-time monitoring solution provides instant insights into the performance and status of your Apache server. The Netdata Agent, when deployed, offers a wide array of metrics and visualizations to help you keep track of connections, request rates, bandwidth usage, and more. For a full demonstration, check out our Live Demo.

Apache No space left on device: a full log disk that stops the server serving

Mon, 01 Jan 0001 00:00:00 +0000

Apache No space left on device: a full log disk that stops the server serving

The error string operators usually search for is (28)No space left on device in the Apache error log. Sometimes it appears at startup and Apache refuses to run. The nastier version appears at runtime: the site goes dark, but the process is alive, the port is open, and TCP connections are accepted. Health checks that only test the socket say everything is fine.

Apache OCSP stapling failure: the silent handshake that adds client latency

Mon, 01 Jan 0001 00:00:00 +0000

Apache OCSP stapling failure: the silent handshake that adds client latency

OCSP stapling failure is a textbook “silently catastrophic” signal: nothing in the error log at default levels, no 5xx, no worker pile-up, and yet every new TLS handshake is slower than it should be. When Apache cannot fetch or attach a stapled OCSP response, each client that wants revocation information makes its own OCSP request to the CA’s responder, adding hundreds of milliseconds before the handshake completes. The server looks healthy from the inside. The slowness only exists on the client side.

Apache open-file limits: raising ulimit and systemd LimitNOFILE correctly

Mon, 01 Jan 0001 00:00:00 +0000

Apache open-file limits: raising ulimit and systemd LimitNOFILE correctly

The default per-process open-file limit on most Linux distributions is 1024, far too low for production Apache. Every client connection, every backend proxy connection, every log file, and every pipe consumes a file descriptor in every child process, and 1024 runs out long before MaxRequestWorkers does. When a child hits the limit, the failure is a cliff edge: “Too many open files” in the error log, failed accepts, failed backend connections, and intermittent 5xx responses, often in only some children at first, which makes the symptom look random.

Apache per-child RSS climbing: the memory-leak slow death

Mon, 01 Jan 0001 00:00:00 +0000

Apache per-child RSS climbing: the memory-leak slow death

Your Apache server has been up for three weeks. Nothing is erroring. Request rate is normal, latency is fine, the scoreboard looks healthy. But free memory keeps shrinking, swap usage is creeping up, and last night the OOM killer took out two httpd children. The parent respawned them, and the cycle started over.

This is the memory-leak slow death: a module loaded into Apache (most commonly mod_php, mod_perl, or a custom module) leaks a small amount of memory per request. Each child’s RSS grows monotonically over hours or days. Nothing looks broken until the system runs out of memory, the OOM killer starts shooting children, and the parent respawns fresh ones that also leak. You are now in a kill-and-respawn spiral.

Apache piped logging failures: when rotatelogs dies and workers SIGPIPE

Mon, 01 Jan 0001 00:00:00 +0000

Apache piped logging failures: when rotatelogs dies and workers SIGPIPE

Piped logging, where ErrorLog or CustomLog points at a program instead of a file (CustomLog "| /usr/bin/rotatelogs /var/log/httpd/access_log.%Y-%m-%d 86400" combined), adds a new failure mode to Apache: the log program itself. When rotatelogs (or whatever sits at the end of the pipe) dies, wedges, or cannot write because its target disk is full, the pipe breaks. Log writes then fail silently, or worse, the Apache children writing to the broken pipe receive SIGPIPE and can crash.

Apache proxy connection refused: AH01114 / (111) Connection refused to backend

Mon, 01 Jan 0001 00:00:00 +0000

Apache proxy connection refused: AH01114 / (111) Connection refused to backend

Your error log is filling with lines like these:

AH00957: HTTP: attempt to connect to 127.0.0.1:8080 (localhost) failed
AH01114: HTTP: failed to make connection to backend: localhost

Clients see 502 or 503 responses. Apache itself is running fine. The failure is on the outbound leg: mod_proxy tried to open a TCP connection to the backend and the kernel told it no.

Apache Pulsar

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar abandoned subscriptions: cursor leaks that pin storage forever

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar abandoned subscriptions: cursor leaks that pin storage forever

Bookie disk usage climbs steadily across the cluster, but publish rates are flat and there is no traffic spike. Per-subscription backlogs look fine for the subscriptions you know about. When you try to delete an old topic to reclaim space, the operation fails with a message about active subscriptions. The topic has not had a consumer in weeks.

This is the signature of a silent subscription cursor leak. Applications create dynamically-named durable subscriptions and then disconnect without unsubscribing. Each abandoned subscription leaves behind a cursor pinned at its last acknowledged position. Pulsar cannot delete any message after that cursor’s position until the subscription acknowledges past it, which never happens because no consumer is connected. Storage grows monotonically, and the growth is invisible in per-subscription backlog dashboards because those dashboards only track subscriptions the team knows about.

Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar active connections climbing: connection leaks and file descriptor exhaustion

pulsar_active_connections has been drifting upward for weeks. Not spiking, not crashing, just rising a few connections a day. Then one morning new producers start failing to connect, or the broker dies with OutOfDirectMemoryError, and the postmortem shows the leak was visible the whole time.

Every TCP connection costs the broker one file descriptor and a slice of Netty direct memory. Connections that are established but never closed accumulate silently until one of those two resources hits its ceiling, at which point the failure is abrupt: refused connections or a crash, with no graceful degradation in between.

Apache Pulsar authentication failures: expired tokens, wrong credentials, and brute force

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar authentication failures: expired tokens, wrong credentials, and brute force

Authentication failures in Apache Pulsar surface through pulsar_authentication_failures_total and AuthenticationException entries in broker logs. In a stable environment, this counter sits near zero. When it spikes, the temporal pattern matters more than the absolute volume: a single burst from a known client after a credential rotation is normal; sporadic low-rate failures from production IPs point to misconfiguration; a sustained flood from unknown sources signals brute force or credential compromise.

Apache Pulsar authorization failures: the AuthorizationException only the logs will show

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar authorization failures: the AuthorizationException only the logs will show

A producer or consumer authenticates successfully but cannot produce, consume, or manage a topic. The client receives an AuthorizationException or a generic permission error. Your dashboard shows nothing unusual because Pulsar exposes no authorization failure counter in its Prometheus metrics. The only evidence is in broker logs.

This is a common blind spot. Authentication failures have a dedicated metric (pulsar_authentication_failures_total), but authorization failures do not. A client with valid credentials but missing role permissions fails silently from a metrics perspective. You find out when an application team reports errors or when someone greps the broker log.

Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar AutoRecovery stalled: under-replicated ledgers that never heal

Under-replicated ledgers are the durability canary for your Pulsar cluster. When a bookie fails, ledger fragments on it lose redundancy and the under-replicated count spikes. AutoRecovery is supposed to drive that count back to zero. When the count stays flat or grows instead of trending down, recovery has stalled.

A stall is not slow recovery. Slow recovery means the count is decreasing, just not fast enough. A stall means it is frozen or climbing. The diagnostic paths and fixes are completely different.

Apache Pulsar backlog age vs size: the latency depth alone cannot show

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar backlog age vs size: the latency depth alone cannot show

Backlog size tells you how much data is waiting. Backlog age tells you how long it has been waiting. Monitoring only one leaves you blind to entire classes of consumer health failures.

A stable backlog of 1,000 entries is unremarkable if those entries are 5 seconds old. The same 1,000 entries from 5 hours ago means a cursor is stuck, a consumer is down, or an application has silently stopped acknowledging. Size alone cannot distinguish these scenarios. Age can.

Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar backlog quota exceeded: producers held or rejected when consumers stall

Producers are timing out or receiving errors, but the root cause is not on the producer side. A subscription backlog has crossed a configured quota, and the backlog policy has kicked in. Depending on the policy, producers are now blocked, rejected, or the oldest messages are being silently deleted. The symptom is a producer outage or data loss, but the root cause is a consumer that stopped keeping up.

Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar bookie add-entry queue not draining: writes arriving faster than the disk can commit

When bookkeeper_server_ADD_ENTRY_IN_PROGRESS grows and does not drain to near-zero within seconds of a traffic burst, the bookie write path is saturated. Writes are arriving faster than the journal disk can fsync them, or the write thread is blocked by GC. Left unaddressed, this causes broker-side timeouts, producer failures, and throughput collapse.

This metric is a gauge, not a counter. Absolute value matters less than trend. Spikes during traffic bursts are normal; sustained positive growth is not. The queue fills and stays filled because the bookie cannot commit entries as fast as they arrive. By the time broker publish latency spikes, the bookie has already been saturated for seconds or minutes.

Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar bookie disk filling: runway to read-only and how to reclaim space

Bookie disk usage decides whether your cluster can accept writes. When a bookie’s ledger directories fill past the configured threshold (default 95%), the bookie transitions to read-only mode and stops accepting new entries. If enough bookies go read-only, the write quorum for affected ledgers cannot be satisfied, and producers start seeing errors.

BookKeeper needs disk headroom to compact entry logs and reclaim space from deleted ledgers. At 95% (diskUsageThreshold), the bookie goes read-only and suspends GC entirely. At that point, shortening retention or deleting topics will not reclaim space because the compaction that rewrites entry logs is itself suspended.

Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar bookie failure cascade: recovery I/O that topples surviving bookies

Bookies are failing one at a time. Each time one drops, AutoRecovery starts replicating its ledgers to the survivors. The recovery reads and writes add I/O load to bookies already handling foreground traffic. Journal sync latency climbs, publish latency follows, then the next bookie starts timing out. It fails, and the cycle accelerates.

This is the bookie failure cascade. The root cause is not the initial bookie failure; it is the recovery mechanism competing with production traffic on bookies near their I/O ceiling. With default settings, AutoRecovery triggers immediately (lostBookieRecoveryDelay=0) and reads entries in batches of 100 (rereplicationEntryBatchSize=100) from surviving bookies, then writes them locally.

Apache Pulsar bookie journal and ledger storage on one disk: the #1 architecture mistake

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar bookie journal and ledger storage on one disk: the #1 architecture mistake

Every BookKeeper bookie has two storage responsibilities with fundamentally different I/O profiles. The journal is a write-ahead log: sequential writes, fsync’d per entry or batch, on the critical path of every producer acknowledgment. Entry logs and ledger indexes (managed by DbLedgerStorage, the default storage backend) store message payloads and serve random reads whenever consumers catch up on historical data.

Apache Pulsar bookie read latency high: catch-up reads competing with the write path

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar bookie read latency high: catch-up reads competing with the write path

When bookkeeper_server_READ_ENTRY_REQUEST or bookie_BOOKIE_READ_ENTRY P99 latency rises well above baseline, the cause is usually not an isolated disk problem. It is consumers draining backlog (catch-up reads) generating a high volume of cold reads that bypass both the broker’s managed ledger cache and the bookie’s read cache, hitting ledger storage disks directly. If journal and ledger directories share a physical device, those read I/O operations compete with journal fsyncs and the write path degrades too.

Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar bookie read-only: disk full and bookie_SERVER_STATUS at zero

bookie_SERVER_STATUS == 0 means the bookie has transitioned to read-only mode and is no longer accepting writes. In most cases this is a self-protective response to ledger disk usage crossing diskUsageThreshold (default 0.95). The bookie keeps serving reads, but any topic whose ensemble includes this bookie may fail to meet its write quorum, causing broker-side write errors and producer timeouts.

Apache Pulsar broker down: telling a dead broker from a fenced one

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar broker down: telling a dead broker from a fenced one

Your alerting fired: the broker’s HTTP admin endpoint on :8080 has been unreachable for more than two minutes, and the broker was previously running, so this is not a fresh deploy or a rolling restart. Pager says “broker down.” That phrase hides two very different incidents.

The first is a hard death: the JVM crashed, got OOM-killed, or the host failed. The process is gone and the OS will tell you so in seconds. The second is a fenced broker: the process is alive, maybe even accepting TCP connections, but it has lost its ZooKeeper session and with it ownership of every namespace bundle it served. From the client’s perspective the broker is down. From the process table’s perspective it is fine. The fix, the blast radius, and the forensics are completely different for each.

Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar broker GC death spiral: heap pressure, stop-the-world pauses, and lost topic ownership

A broker is losing topic ownership, recovering, and losing it again. Clients reconnect in bursts. Lookup failures climb. Publish latency is erratic even though traffic is flat and CPU looks busy for no obvious reason. The broker process never actually dies, which is why restarts and instance health checks keep “fixing” it for ten minutes at a time.

Apache Pulsar broker hotspot: one broker owning far more topics than the rest

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar broker hotspot: one broker owning far more topics than the rest

You have a multi-broker Pulsar cluster where one broker carries a disproportionate share of topic ownership. Cluster-wide averages look acceptable, but that single broker shows elevated GC pressure, higher publish latency, more active connections, or a larger heap footprint than its peers. The imbalance may have built for hours or days without triggering an alert because aggregate metrics hid it.

Apache Pulsar broker lookup failures: new clients cannot find their topic

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar broker lookup failures: new clients cannot find their topic

Your dashboards look fine. Messages are flowing, publish latency is normal, consumers are draining backlog. But the new service you just deployed cannot start: its producer hangs or fails trying to connect to its topic. Restarting it does not help. Existing clients are unaffected.

This is the classic Pulsar grey failure. Topic lookup is how every producer and consumer discovers which broker owns its topic. Existing connections keep working because they already resolved ownership. New traffic fails when lookups cannot resolve. Process health checks, message rates, and bookie metrics can all look healthy until a deploy, restart, scale event, or broker bounce forces mass re-lookup.

Apache Pulsar bundle unload thrashing: the load balancer that never converges

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar bundle unload thrashing: the load balancer that never converges

The primary signal is pulsar_lb_unload_bundle_total, a counter exposed on the broker Prometheus metrics endpoint. In steady state it barely moves: bundle ownership changes are rare, ideally less than once per hour. When this counter climbs past one unload per minute and there is no rolling upgrade, no broker restart, and no planned maintenance, the load balancer is thrashing.

Each unload transfers a namespace bundle (a hash-range slice of topics) from one broker to another. Each transfer drops the TCP connections for every producer and consumer on those topics. Clients reconnect automatically, but reconnection generates metadata operations against ZooKeeper, new ledger creation in BookKeeper, and a latency blip measured in tens of milliseconds per affected topic. When unloads happen every few seconds, those blips stack into sustained degradation: elevated publish latency, client reconnection storms, and metadata store pressure that feeds back into more instability.

Apache Pulsar consumers connected but not acknowledging: the silent consumer stall

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar consumers connected but not acknowledging: the silent consumer stall

Your Pulsar cluster looks healthy. Brokers are up, bookies are writable, producers are publishing at normal rates. But for one topic or subscription, pulsar_rate_out has gone flat. Messages are flowing in but nothing is coming out. The backlog is climbing. Consumers show as connected, but the acknowledgment rate is zero.

No error surfaces in broker logs. The broker serves connections and bookie writes are fast. The problem lives in the consumer application or in the interaction between consumer configuration and broker dispatch policy.

Apache Pulsar dead letter queue filling: maxRedeliveryCount and the DLQ topic

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar dead letter queue filling: maxRedeliveryCount and the DLQ topic

The dead letter topic is where Pulsar parks messages that exhausted their retry budget. When you set DeadLetterPolicy.builder().maxRedeliverCount(N).build() on a Shared or Key_Shared subscription, the broker stops redelivering a message after N failed attempts, routes it to {topic}-{subscription}-DLQ, and auto-acks it in the origin subscription so backlog clears.

A growing DLQ arrival rate is a ledger of application processing failures. A few poison messages parked for inspection is the DLQ working as designed. A steady stream of thousands of messages per second is a systemic failure. Unmonitored, the DLQ topic itself becomes the incident: messages accumulate, storage grows, and nobody notices until disk fills or a downstream team asks why their data is missing.

Apache Pulsar ensemble, write quorum, and ack quorum: what E, Qw, and Qa actually guarantee

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar ensemble, write quorum, and ack quorum: what E, Qw, and Qa actually guarantee

Every persistent message in Apache Pulsar passes through BookKeeper’s quorum system, governed by three parameters: ensemble size (E), write quorum (Qw), and ack quorum (Qa). Configured per namespace and inherited by topics, they determine how many bookies receive each entry, how many must confirm a durable write before the producer is acknowledged, and how the cluster behaves when bookies fail, slow down, or are taken offline.

Apache Pulsar entry log GC falling behind: reclaimed space that never comes back

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar entry log GC falling behind: reclaimed space that never comes back

A bookie’s ledger disk is filling. Retention policies have deleted ledgers, TTL has expired messages, and cursors have advanced past the data. By every logical measure, the space should be free. But disk usage keeps climbing, and the bookie is heading toward read-only.

BookKeeper does not store each ledger in a separate file. Entry logs are large append-only files that interleave entries from many ledgers, written sequentially as they arrive. When a ledger is deleted, its entries remain physically embedded in the entry log alongside data from other ledgers that are still active. The only way to reclaim that space is compaction: the garbage collector reads the live entries from an old entry log, writes them into a new file, then deletes the old one. If compaction stalls, throttles to a crawl, or never triggers, dead data accumulates indefinitely.

Apache Pulsar geo-replication backlog: replication lag and your real RPO window

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar geo-replication backlog: replication lag and your real RPO window

Geo-replication in Pulsar is asynchronous by design. Messages are persisted locally and acknowledged to producers before they reach the remote cluster. Every message sitting in the replication backlog is a message that would be lost if you failed over right now. The replication backlog is your recovery point objective (RPO) window, measured in real time.

The common operational mistake is checking whether replication is “working” (connected, producing to the remote cluster) without checking how far behind it is. A replicator can be fully connected and actively shipping messages while being hours behind. The dashboard shows green. The failover plan assumes near-zero data loss. The reality is data loss measured in hours.

Apache Pulsar geo-replication disconnected: a remote cluster that stopped receiving

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar geo-replication disconnected: a remote cluster that stopped receiving

The symptom is unmistakable: pulsar_replication_disconnected_count is non-zero for all replicators to a specific remote cluster, pulsar_replication_connected_count has dropped to zero, and pulsar_replication_backlog is growing at the rate of local publish throughput. The remote cluster has stopped receiving messages. Local producers and consumers continue working normally because Pulsar geo-replication is asynchronous. Messages persist locally first, then replicate. The damage is silent and cumulative.

Apache Pulsar journal force write queue growing: the earliest write-saturation signal

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar journal force write queue growing: the earliest write-saturation signal

bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE measures the depth of pending fsync batches inside a BookKeeper bookie’s journal write path. In a healthy system this gauge sits at or near zero, draining between write groups. When it sustains a non-zero depth, the journal disk cannot commit writes durably fast enough to keep up with incoming traffic.

This signal rises before bookie_journal_JOURNAL_SYNC latency spikes and before bookkeeper_server_ADD_ENTRY_IN_PROGRESS grows. If you are not watching the force write queue, your first indication of write-path saturation will be broker publish latency degradation or producer timeouts, which means you are already two or three steps into the cascade.

Apache Pulsar ledger rollover latency spikes: the periodic blip that is usually normal

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar ledger rollover latency spikes: the periodic blip that is usually normal

You see a periodic sub-second spike in publish latency on your Pulsar brokers. It happens at regular intervals, lasts a fraction of a second, and then latency returns to baseline. No errors, no bookie failures, no GC pauses. The spike is visible in P99 publish latency but barely touches P50. It correlates across topics on the same broker but not across the entire cluster.

Apache Pulsar LedgerFencedException: split ownership and fencing loops

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar LedgerFencedException: split ownership and fencing loops

LedgerFencedException appears when a broker tries to write to a BookKeeper ledger that another broker has already fenced. The error string is explicit: “Ledger has been fenced off. Some other client must have opened it to read.”

In most production environments, this is expected noise during planned failover. It becomes a problem when fencing occurs without a corresponding bundle transfer, or when two brokers fence each other’s ledgers in a loop.

Apache Pulsar managed ledger cache miss rate high: consumer reads falling through to bookies

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar managed ledger cache miss rate high: consumer reads falling through to bookies

The managed ledger cache is the broker’s off-heap buffer of recently written entries. Connected consumers read from it at sub-millisecond latency. On a cache miss, the broker issues a read to the BookKeeper bookie ensemble holding that ledger segment, which adds disk I/O, network bandwidth consumption between broker and bookie, and consumer read latency.

A sustained cache miss rate above 20% after warmup means the broker is serving a meaningful fraction of consumer reads from storage instead of memory. The extra bookie read traffic loads storage disks that also serve journal writes. If bookie read latency rises enough, publish latency follows. This is the leading edge of the Backlog Cascade: consumer reads and producer writes converging on the same physical disks.

Apache Pulsar message redelivery storm: poison messages and consumers making no progress

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar message redelivery storm: poison messages and consumers making no progress

Subscription redelivery rate is climbing. Consumers are connected, the dispatch rate (pulsar_rate_out) looks healthy, but messages keep cycling back without being acknowledged. The backlog may even appear stable or zero, because pulsar_subscription_back_log measures messages not yet dispatched, not messages dispatched but unacked. The definitive signal is pulsar_subscription_msg_rate_redeliver trending upward relative to dispatch. When redelivery exceeds 10% of your dispatch rate, something is wrong. When it approaches 100%, you have zero forward progress: the broker burns memory and network re-sending messages that will never succeed.

Apache Pulsar messages expiring before consumers read them: TTL and silent loss

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar messages expiring before consumers read them: TTL and silent loss

When pulsar_subscription_msg_rate_expired is non-zero on a topic where every message matters, messages are being silently deleted before consumers can read them. No errors appear in consumer logs. No producer failures occur. The broker’s TTL mechanism acknowledges messages on behalf of the subscription without ever delivering them to the consumer application.

The backlog can hide the problem. TTL expiry trims unacked messages from subscription cursors, so backlog can appear stable or declining while real consumer lag grows underneath. Operators monitoring backlog size alone see a healthy-looking number. The data loss is invisible unless you specifically monitor the expiry rate.

Apache Pulsar metadata store latency: the leading indicator before every cluster-wide failure

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar metadata store latency: the leading indicator before every cluster-wide failure

Metadata store latency is the earliest signal that a Pulsar cluster is heading toward a widespread outage. In most 3.x deployments the metadata store is ZooKeeper. Pulsar 3.3.0 introduced experimental Oxia support as an eventual replacement. Every broker, bookie, and load balancer operation that touches cluster topology, bundle ownership, schema lookups, ledger metadata, or cursor persistence routes through this store. When round-trip latency for those operations rises, the symptoms appear downstream in brokers and bookies, but the root cause is upstream.

Apache Pulsar Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar Monitoring

What Is Apache Pulsar?

Apache Pulsar is an open-source distributed messaging and streaming platform. It provides a unified messaging model and is designed to handle high-throughput, low-latency workloads. With its multi-tenancy, geo-replication, and seamless scalability features, Pulsar is a comprehensive solution tailored for both messaging and streaming use cases.

Monitoring Apache Pulsar With Netdata

Monitoring Apache Pulsar is crucial for ensuring the system’s health and performance. Netdata offers a powerful and intuitive way to monitor Pulsar, leveraging its robust capabilities to deliver real-time insights. The Netdata Pulsar monitoring tool taps into Pulsar’s Prometheus endpoint to collect a wide range of metrics that help you keep your messaging system running smoothly.

Apache Pulsar monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar monitoring checklist: the signals every production cluster needs

Pulsar fails differently from most systems you operate. A cluster can report every process “up” while the write path stalls on a bookie journal disk, while a broker GC-pauses its ZooKeeper session away, or while a subscription silently freezes because its unacked message count hit a limit. Monitoring that only checks “is the broker running” misses almost every real Pulsar incident.

Apache Pulsar monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar monitoring maturity model: from survival to expert

Most Pulsar outages do not arrive without warning. The journal force write queue grows before publish latency spikes. ZooKeeper latency drifts upward for days before brokers lose session ownership. Direct memory climbs for weeks before the broker dies with an OutOfDirectMemoryError while heap dashboards look fine. The signals were there. The team just was not collecting them yet.

This is a four-level maturity model for Pulsar monitoring, from the bare minimum that tells you the cluster is alive to the deep signals operators add after their second or third incident. Use it to audit your current coverage and decide what to instrument next. The levels are cumulative: each one assumes everything below it is already in place.

Apache Pulsar NotEnoughBookiesException: new ledgers cannot be created and writes fail

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar NotEnoughBookiesException: new ledgers cannot be created and writes fail

NotEnoughBookiesException (BookKeeper error code -6, surfaced as BKNotEnoughBookiesException) fires when the ensemble placement policy cannot satisfy the ensemble size (E) and write quorum (Qw) requirements for a new ledger. The broker requests a new ledger on size or time rollover, during topic recovery, and during compaction. If the policy cannot form a valid ensemble, ledger creation fails, the managed ledger has nowhere to write, and messages cannot be persisted durably.

Apache Pulsar OutOfDirectMemoryError: the off-heap crash JVM heap dashboards never show

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar OutOfDirectMemoryError: the off-heap crash JVM heap dashboards never show

The broker is down. Your JVM dashboard shows heap at 55%, GC pauses normal, no heap OOM anywhere. Then someone opens the broker log and finds the real cause:

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: ..., max: ...)

or the plainer JDK variant, java.lang.OutOfMemoryError: Direct buffer memory. In recent Pulsar versions the process may have exited deliberately: PulsarByteBufAllocator catches the allocation failure and, with the default -Dpulsar.allocator.exit_on_oom=true, kills the JVM rather than limp along half-broken. Either way, the broker is dead in the part your heap monitoring was not watching.

Apache Pulsar publish latency high: reading pulsar_broker_publish_latency P99

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar publish latency high: reading pulsar_broker_publish_latency P99

Your alert fired on sustained P99 elevation of pulsar_broker_publish_latency above 2x the rolling baseline. Producers are seeing slow acknowledgements.

The metric pulsar_broker_publish_latency is a Summary metric exposed on the broker Prometheus endpoint. It measures the time from when the broker receives a message from a producer through the BookKeeper write path (write quorum Qw, ack quorum Qa) and back to the client callback. It is broker-side only: it excludes client-to-broker network time and producer-side batching delay. The Summary type provides quantiles at 0.5, 0.95, 0.99, 0.999, 0.9999, and 1.0. Alert on P99. P50 can look healthy while P99 is spiking, and it is the tail that triggers producer timeouts.

Apache Pulsar subscription backlog growing: consumers falling behind producers

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar subscription backlog growing: consumers falling behind producers

A growing subscription backlog means producers are publishing faster than consumers can acknowledge. The absolute backlog size matters less than its trajectory. A high but stable backlog is normal for lagged or replay consumers. A monotonically increasing backlog is a problem regardless of size: it consumes bookie disk until the bookie goes read-only, the backlog quota trips and throttles producers, or retention and TTL silently delete messages the consumer never saw.

Apache Pulsar throttled connections: the broker shedding load under pressure

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar throttled connections: the broker shedding load under pressure

Your dashboard shows pulsar_broker_throttled_connections climbing from zero, producers are complaining about elevated send latency, and some clients are timing out on new operations. The broker process is up, the health endpoint returns 200, and heap looks fine. The broker is not broken. It is defending itself.

Connection throttling is a protective mechanism. When the broker’s internal send queues on a connection back up beyond a configured ceiling, it stops reading new requests from that TCP connection (it disables auto-read on the Netty channel) until the backlog drains. The throttled connection count tells you how many connections are currently in that state. In a healthy cluster this gauge is zero. Any sustained non-zero value means the broker is at or near its capacity for the work being pushed through it.

Apache Pulsar TLS certificate expiry: the silent, total outage no metric warns you about

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar TLS certificate expiry: the silent, total outage no metric warns you about

When every component in a Pulsar cluster uses TLS (brokers, bookies, ZooKeeper, proxies, clients), a single expired certificate causes immediate handshake failures. Existing connections may persist briefly, but every new connection attempt fails the TLS handshake. Producers cannot publish. Consumers cannot subscribe. Brokers cannot reach bookies. Brokers cannot reach ZooKeeper.

Pulsar does not expose a metric for certificate expiry. There is no pulsar_cert_days_until_expiry gauge. The first signals you see are indirect: authentication failures spike, connections drop, and SSL handshake exceptions fill the logs. By that point, the outage is already happening. The only reliable defense is external certificate expiry monitoring that alerts you days or weeks before the cert becomes invalid.

Apache Pulsar topic ownership oscillation: brokers fighting over the same bundle

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar topic ownership oscillation: brokers fighting over the same bundle

Broker A takes ownership of a namespace bundle, opens managed ledgers, and begins serving clients. The load balancer sees A as overloaded and unloads the bundle. Broker B picks it up, fences the ledgers, creates new ones, and starts serving. The balancer now sees B as overloaded and moves it back. The cycle repeats indefinitely.

Each iteration forces ledger fencing, new ledger creation, ZK metadata writes, and client disconnection and reconnection storms. Producers see intermittent publish latency spikes. New clients fail lookups during the brief ownership gap between fencing and re-acquisition. The cluster looks healthy in aggregate because no single broker is down, but the affected topics are in a constant state of transition.

Apache Pulsar unacked messages at the limit: the silent dispatch freeze

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar unacked messages at the limit: the silent dispatch freeze

Consumers are connected. Producers are publishing. Backlog looks flat or zero. But no messages are being processed.

Pulsar brokers enforce a per-subscription limit on unacknowledged messages (maxUnackedMessagesPerSubscription, default 200,000) and a per-consumer limit (maxUnackedMessagesPerConsumer, default 50,000). When unacked messages hit either ceiling, the broker stops dispatching new messages to that subscription or consumer. No error is returned to the client. No exception is thrown. Dispatch pauses silently.

Apache Pulsar under-replicated ledgers: data at risk after a bookie failure

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar under-replicated ledgers: data at risk after a bookie failure

You lost a bookie. The broker layer kept serving traffic because your write quorum absorbed the failure. But now auditor_NUM_UNDER_REPLICATED_LEDGERS is climbing, and it is not coming back down. Every minute it stays elevated is a minute your data exists on fewer copies than your replication factor demands. If another bookie fails before AutoRecovery finishes rereplicating, those ledgers are gone permanently.

Apache Pulsar write stall: bookie journal fsync latency and the blocked write path

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar write stall: bookie journal fsync latency and the blocked write path

The write stall is the most common performance failure in Pulsar. It starts at the bookie journal disk and cascades upward through the write path until producers are blocked or timing out.

Every persistent message write in Pulsar must survive a journal fsync before the bookie acknowledges it. The broker waits for ack quorum (Qa) acknowledgments before acknowledging the producer. When the journal disk cannot sync fast enough, pending fsync operations accumulate, add-entry operations queue up, brokers hold connections open waiting for acks, client buffers fill, and throughput collapses.

Apache Pulsar ZooKeeper session cascade: reconnect storms and the thundering herd

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar ZooKeeper session cascade: reconnect storms and the thundering herd

Multiple brokers lose their ZooKeeper sessions within seconds of each other. Bundle ownership churns across the cluster as surviving brokers acquire orphaned bundles. Every connected producer and consumer simultaneously discovers its topic has moved or its connection has dropped, triggering a mass reconnection event. Each reconnection generates topic lookups, ephemeral node registrations, and watch re-establishments that hit the already overloaded ZK ensemble. Latency rises further, more sessions expire, and the cascade tightens.

Apache Pulsar ZooKeeper session expired: brokers fenced and topics reassigned

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar ZooKeeper session expired: brokers fenced and topics reassigned

A broker’s ZooKeeper session expires when it fails to send heartbeats within the session timeout window. The expiry triggers immediate fencing of the broker, loss of all namespace bundle ownership, and reassignment of those bundles to surviving brokers. Clients connected to the fenced broker see errors like “Topic is temporarily unavailable” or “Attempting to add producer to a fenced topic” until the topics are picked up elsewhere.

Apache Pulsar ZooKeeper watch explosion: thousands of watches killing metadata latency

Mon, 01 Jan 0001 00:00:00 +0000

Apache Pulsar ZooKeeper watch explosion: thousands of watches killing metadata latency

Topic lookups slow down. Bundle ownership transfers stall. Broker sessions flicker between connected and disconnected. ZooKeeper reports thousands, sometimes tens of thousands, of registered watches. This is a watch explosion, and the root cause is rarely in ZooKeeper itself. It is in client connection churn.

Watches accumulate when consumers disconnect and reconnect in waves. Each reconnection registers metadata watches for topic ownership, subscription state, and policy changes. During a mass reconnection event (broker restart, network blip, load balancer cycle), thousands of ephemeral znodes are created and deleted in rapid succession. Each creation and deletion triggers watch registration and notification. ZK processes requests sequentially, so as notification load grows, request latency climbs. Once latency exceeds session timeout, brokers lose sessions, triggering bundle unloads and another round of client reconnections. The feedback loop tightens until the cluster thrashes.

Apache request latency: p50/p95/p99 from the access log %D field

Mon, 01 Jan 0001 00:00:00 +0000

Apache request latency: p50/p95/p99 from the access log %D field

“Apache is slow” is one of the least actionable statements in operations. The follow-up questions are always: slow for whom, on which URLs, and how slow at the tail? Averages cannot answer any of those. A server where every request takes 200ms and a server where half the requests take 5ms and half take 400ms have the same mean latency and completely different operational problems.

Apache request rate dropped to near zero: silent load shedding and its causes

Mon, 01 Jan 0001 00:00:00 +0000

Apache request rate dropped to near zero: silent load shedding and its causes

Your dashboard shows requests per second falling off a cliff, but nobody changed anything, the load balancer still reports healthy demand, and there is no error spike to point at. The most natural reading, “traffic went away,” is usually wrong.

The trap is in how the number is produced. The request rate most operators watch comes from mod_status Total Accesses, which counts completed requests. It is not a measure of arriving demand. When Apache cannot complete requests, because every worker is stuck or a backend has stopped answering, the completed-request rate collapses even while clients are still hammering the front door. The server has not lost its traffic. It is silently shedding load, and the metric you trust is the last place that shows up.

Apache reverse proxy worker busy: proxy connection pool exhaustion (AH01136)

Mon, 01 Jan 0001 00:00:00 +0000

Apache reverse proxy worker busy: proxy connection pool exhaustion (AH01136)

You are seeing intermittent 503s on proxied endpoints under what looks like moderate load. The error log shows AH01136: Reverse proxy worker busy. BusyWorkers is nowhere near MaxRequestWorkers, CPU and memory are fine, and requests keep failing. Raising MaxRequestWorkers changed nothing.

That is the signature of proxy connection pool exhaustion. The bottleneck is not Apache’s worker pool. It is the smaller, less visible pool of backend connections that mod_proxy maintains per child process, and its default size is too small for production load.

Apache scoreboard states explained: what _ S R W K D C L G tell you

Mon, 01 Jan 0001 00:00:00 +0000

Apache scoreboard states explained: what _ S R W K D C L G tell you

When someone asks “why is Apache slow?”, the first place to look is not CPU, not memory, not the access log. It is the scoreboard. The scoreboard is a shared-memory segment where every worker slot records what it is doing right now, one character per slot. It is what mod_status reads, and it is the most diagnostic structure Apache exposes.

Apache slow backend cascade: how one slow upstream starves the whole worker pool

Mon, 01 Jan 0001 00:00:00 +0000

Apache slow backend cascade: how one slow upstream starves the whole worker pool

Users report the site down. The load balancer has pulled the node from rotation. You SSH in expecting to find Apache melting, and instead find a perfectly calm process: normal CPU, normal memory, no crash dumps in the error log. The port is listening, but new connections hang or get refused.

This is the slow backend cascade: the classic “Apache outage” in reverse-proxy deployments. Apache is not broken. Every worker is blocked waiting on a backend that has gone slow, and the frontend has run out of execution slots as a consequence.

Apache Slowloris: R-state workers, slow-read attacks, and mod_reqtimeout

Mon, 01 Jan 0001 00:00:00 +0000

Apache Slowloris: R-state workers, slow-read attacks, and mod_reqtimeout

Your Apache server stopped serving requests, but nothing looks broken. CPU is normal. Memory is normal. The backend is healthy when you test it directly. Request rate is flat or dropping. Yet clients time out, and the load balancer is pulling the node from rotation.

This is the Slowloris signature: a slow-read denial of service where an attacker opens many connections and drips request data byte by byte, holding each worker in the Reading (R) state indefinitely. With enough slow connections, the entire worker pool is consumed reading requests that never complete. The server is not overloaded. It is held hostage.

Apache SSL certificate expired: the total, preventable HTTPS outage

Mon, 01 Jan 0001 00:00:00 +0000

Apache SSL certificate expired: the total, preventable HTTPS outage

Every HTTPS client is failing at once. Browsers show NET::ERR_CERT_DATE_INVALID, API clients throw certificate validation errors, monitoring probes time out, and your access logs have gone quiet because no TLS handshake ever completes. Apache itself is running fine: the process is up, workers are idle, and port 443 accepts TCP connections. The failure is at the TLS layer, before any HTTP request exists.

Apache SSL session cache: shmcb sizing, sharing, and silent eviction

Mon, 01 Jan 0001 00:00:00 +0000

A full TLS handshake is the most CPU-intensive thing Apache does per connection. Session resumption exists to avoid paying that cost on every reconnect: a client that recently completed a handshake can resume instead of redoing the asymmetric cryptography. When resumption silently stops working, every connection pays full price, and the only symptom is rising CPU with no error in the log.

Apache swap thrashing: the latency cliff before the OOM kill

Mon, 01 Jan 0001 00:00:00 +0000

Apache swap thrashing: the latency cliff before the OOM kill

The symptom arrives before the failure. P95 latency doubles, then triples, across every endpoint at once, including static files that should be served from page cache. No 5xx spike yet. No OOM kills in dmesg yet. The Apache parent process is fine, the scoreboard shows workers in normal states, and every request is suddenly slow. This is the swap window: the period between “memory is tight” and “the kernel starts killing children,” where Apache is technically up but effectively degraded.

Apache TLS handshake CPU: broken session resumption and the HTTPS CPU wall

Mon, 01 Jan 0001 00:00:00 +0000

Apache TLS handshake CPU: broken session resumption and the HTTPS CPU wall

The symptom looks like a capacity problem: Apache processes pinned near 100% CPU, request latency climbing, and no obvious cause in the access log. Traffic is up, but not absurdly. The workers are not stuck on a backend. The scoreboard is busy but not exhausted. And yet the box is melting.

On HTTPS-heavy servers, the usual suspect is the TLS handshake. A full TLS handshake with an RSA-2048 certificate costs on the order of 15 ms of CPU time per connection. Session resumption cuts that cost by roughly 10x. When resumption works, repeat clients skip the expensive public-key operation entirely. When resumption is broken, every connection pays the full price, and at a few hundred new connections per second the math stops working.

Apache Too many open files: file descriptor exhaustion (EMFILE)

Mon, 01 Jan 0001 00:00:00 +0000

Apache Too many open files: file descriptor exhaustion (EMFILE)

Your error log starts showing lines like:

(24)Too many open files: AH00035: access to /some/path failed

or proxy requests begin failing with AH01114 connection errors, and users see intermittent 5xx responses. Sometimes only some requests fail. Sometimes the site looks fine for hours and then breaks at peak. The process is running, the port is open, and nothing in CPU or memory looks wrong.

Apache unexpected restarts: SIGTERM vs SIGUSR1, crashes, and OOM respawns

Mon, 01 Jan 0001 00:00:00 +0000

Apache unexpected restarts: SIGTERM vs SIGUSR1, crashes, and OOM respawns

mod_status shows ServerUptimeSeconds at 340 again, and it was 340 an hour ago too. Something is restarting Apache, and unless you deployed at that exact moment, it is not you. The error log says resuming normal operations every few minutes, or caught SIGTERM, or nothing at all between restarts. Each signature points at a different failure mode.

Apache restarts fall into three buckets: intentional graceful reloads (SIGUSR1), hard restarts (SIGTERM or SIGHUP, often from logrotate or config management), and crash respawns (child segfaults, OOM kills, or the parent dying and being restarted by systemd or a wrapper script). The first is usually harmless. The second drops every in-flight connection. The third means something is actively wrong and will keep happening until you find it.

Apache with mod_php: why per-child memory explodes and when to move to PHP-FPM

Mon, 01 Jan 0001 00:00:00 +0000

Apache with mod_php: why per-child memory explodes and when to move to PHP-FPM

Each apache2 or httpd child sits at 80, 120, sometimes 200MB of RSS, the sum of all children creeps toward total RAM, and the OOM killer starts picking off workers at peak traffic. MaxRequestWorkers is already set conservatively, but it does not matter: per-child memory is so large that any worker count high enough to serve your traffic exceeds what the machine can hold.

APC

Mon, 01 Jan 0001 00:00:00 +0000

APC Netbotz

Mon, 01 Jan 0001 00:00:00 +0000

APC PDU

Mon, 01 Jan 0001 00:00:00 +0000

APC UPS

Mon, 01 Jan 0001 00:00:00 +0000

APC UPS

Mon, 01 Jan 0001 00:00:00 +0000

APC UPS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

APC UPS Monitoring

What Is APC UPS?

APC UPS, or Uninterruptible Power Supply systems from APC, are crucial components in ensuring the continuity and protection of power to critical infrastructure. These devices ensure that your servers, data centers, and critical systems stay powered even during a power outage, safeguarding your operations against data loss and downtime.

Monitoring APC UPS With Netdata

Monitoring platforms like Netdata allow you to keep an eye on the health, performance, and capacity of your APC UPS units in real-time. Utilizing the Apcupsd daemon, Netdata provides comprehensive visibility into UPS performance and alerts you to any issues so they can be addressed promptly.

Aperto Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

APIcast

Mon, 01 Jan 0001 00:00:00 +0000

APIcast Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

APIcast Monitoring

What Is APIcast?

APIcast is a powerful API management solution designed for API traffic across various web and cloud applications. It acts as a gateway, controlling how APIs are exposed, secured, and consumed. Built by 3scale, APIcast allows organizations to manage and monetize their APIs effectively, ensuring robust security and operational efficiency. Discover more about APIcast here.

Monitoring APIcast With Netdata

To effectively monitor APIcast, Netdata leverages an openmetrics (Prometheus) exporter. This allows Netdata to ingest data from any Prometheus exporter and provide automated dashboards and alerts without necessitating a Prometheus server or Grafana setup. This seamless integration helps in gaining real-time insights into APIcast’s performance, ensuring optimal operation. By using the Netdata infrastructure monitoring tool, APIcast users can experience detailed, user-friendly dashboards and benefit from real-time alerts, all designed to keep your API services performing at their peak.

Apple Computer Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Applications

Mon, 01 Jan 0001 00:00:00 +0000

Applied Innovation Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Apply to Become a Netdata Partner

Mon, 01 Jan 0001 00:00:00 +0000

AppOptics

Mon, 01 Jan 0001 00:00:00 +0000

Aptis Communications Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Arbor Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Arch Linux

Mon, 01 Jan 0001 00:00:00 +0000

Areca Technology Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Argus Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aricent Communication Holdings Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Arista

Mon, 01 Jan 0001 00:00:00 +0000

Arista BGP

Mon, 01 Jan 0001 00:00:00 +0000

Arista Networks Inc Formerly Arastra Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Arista Switch

Mon, 01 Jan 0001 00:00:00 +0000

Armillaire Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

ARP / IP Neighbor Topology

Mon, 01 Jan 0001 00:00:00 +0000

ARP cache staleness: when IP-to-MAC mapping goes bad

Mon, 01 Jan 0001 00:00:00 +0000

ARP cache staleness: when IP-to-MAC mapping goes bad

Hosts on the same subnet stop reaching each other after a VM live-migrates, a container gets rescheduled, or a firewall fails over. ICMP works from some hosts but not others. TCP sessions hang or reset. The data plane is healthy, but the ARP cache on one or more hosts holds a stale IP-to-MAC mapping.

ARP cache staleness is the gap between when a MAC address changes and when every interested host learns about the change. On Linux, this gap is governed by the neighbor (NUD) state machine and its timing parameters. On Windows Vista and later, the neighbor cache follows the same RFC 4861 model. Both platforms default to roughly the same reachable time window: about 15 to 45 seconds before an entry transitions to a stale state, followed by a probe sequence that adds several more seconds before resolution or eviction.

Arris Interactive LLC SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Arrowpoint Communications Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Artel Video Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Artem Gmbhmichael Marsanu Catrinel Catrinescu SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aruba

Mon, 01 Jan 0001 00:00:00 +0000

Aruba A Hewlett Packard Enterprise Company SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aruba Access Point

Mon, 01 Jan 0001 00:00:00 +0000

Aruba Clearpass

Mon, 01 Jan 0001 00:00:00 +0000

Aruba CX Switch

Mon, 01 Jan 0001 00:00:00 +0000

Aruba Mobility Controller

Mon, 01 Jan 0001 00:00:00 +0000

Aruba Switch

Mon, 01 Jan 0001 00:00:00 +0000

Aruba Wireless Controller

Mon, 01 Jan 0001 00:00:00 +0000

Asante Technology SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ascend Communications Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ascom Sweden AB SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Asentria Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Asetek SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Askey Computer Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

ASP.NET

Mon, 01 Jan 0001 00:00:00 +0000

Astaro AG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Asymmetric routing: why your path and latency measurements lie

Mon, 01 Jan 0001 00:00:00 +0000

Asymmetric routing: why your path and latency measurements lie

Your monitoring says the path is fine. Ping latency is normal, traceroute shows a clean route, and interface counters look healthy. But applications are slow, TCP sessions stall or reset, and users are complaining. Your tools are measuring only half the path.

In asymmetric routing, traffic from host A to host B takes one path (P1) while return traffic from B to A takes a different path (P2). When P2 is degraded, congested, or broken, your measurements average the healthy forward path with the impaired return path. Every acknowledgment and response is fighting through a bad route while the aggregate looks acceptable.

At T SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ateme SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aten International Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Atlas Computer Equipment Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Atm Forum SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Atmel Hellas S A N SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Atto Technology Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Audiocodes Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Audiocodes Mediant SBC

Mon, 01 Jan 0001 00:00:00 +0000

Audit log gaps: detecting syslog/trap tampering or loss

Mon, 01 Jan 0001 00:00:00 +0000

Audit log gaps: detecting syslog/trap tampering or loss

An audit log gap is any period where expected syslog messages or SNMP traps from a network device fail to arrive at the collector. UDP syslog on port 514 and SNMP traps on port 162 are fire-and-forget transports with no delivery guarantee. The kernel silently drops datagrams when socket buffers fill, and the application layer never sees the loss. TCP syslog can stall under collector backpressure. Most gaps are operational: network loss, device-side buffer overflow, or logging subsystem failure. The difficulty is distinguishing those from deliberate log suppression after compromise.

Auditec S.A. SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Auspex Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Austin Hughes Electronics Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

AuthLog

Mon, 01 Jan 0001 00:00:00 +0000

AuthLog Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

AuthLog Monitoring

What Is AuthLog?

AuthLog is a tool designed for monitoring authentication logs to gain security insights and facilitate efficient access management. Its primary function is to track and analyze authentication attempts, providing crucial data for assessing security posture and identifying unauthorized access attempts.

Monitoring AuthLog With Netdata

To monitor AuthLog effectively, Netdata uses an openmetrics (Prometheus) exporter, the AuthLog Exporter. Netdata’s robust monitoring capabilities allow it to ingest data from any Prometheus exporter. This means you can set up automated dashboards, alerts, and more without the need for a Prometheus server or Grafana. By utilizing Netdata, users can leverage real-time monitoring of their AuthLog metrics to ensure comprehensive security coverage and quick response to potential threats.

Availant SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Avamar SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Avaya

Mon, 01 Jan 0001 00:00:00 +0000

Avaya Aura Media Server

Mon, 01 Jan 0001 00:00:00 +0000

Avaya Cajun Switch

Mon, 01 Jan 0001 00:00:00 +0000

Avaya Communication SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Avaya Media Gateway

Mon, 01 Jan 0001 00:00:00 +0000

Avaya Nortel Ethernet Routing Switch

Mon, 01 Jan 0001 00:00:00 +0000

Aventail Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aviat Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Avici Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Avista Labs Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Avocent ACS

Mon, 01 Jan 0001 00:00:00 +0000

Avocent Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Avtech

Mon, 01 Jan 0001 00:00:00 +0000

Avtech Roomalert 32S

Mon, 01 Jan 0001 00:00:00 +0000

Avtech Roomalert 3E

Mon, 01 Jan 0001 00:00:00 +0000

Avtech Roomalert 3S

Mon, 01 Jan 0001 00:00:00 +0000

Avtech Software Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Aware Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

AWS EC2 Compute instances

Mon, 01 Jan 0001 00:00:00 +0000

AWS EC2 Compute Instances Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

AWS EC2 Compute Instances Monitoring

What Is AWS EC2 Compute Instances?

AWS EC2 (Elastic Compute Cloud) instances are a key element of Amazon Web Services, providing scalable computing capacity in the cloud. They eliminate the need to invest in hardware, allowing companies to focus on their applications while benefiting from the flexibility of cloud infrastructure.

Monitoring AWS EC2 Compute Instances With Netdata

Netdata offers a comprehensive platform for monitoring AWS EC2 Compute Instances, highlighting the critical metrics that can significantly impact performance and cost management. Utilizing an openmetrics (Prometheus) exporter, Netdata can seamlessly ingest data from any Prometheus exporter, delivering automated dashboards, alerts, and more, all without the need for a Prometheus server or Grafana. This powerful capability allows technical users to monitor AWS EC2 with precision and minimum friction.

AWS ECS Containers

Mon, 01 Jan 0001 00:00:00 +0000

AWS IP Ranges

Mon, 01 Jan 0001 00:00:00 +0000

AWS Kinesis

Mon, 01 Jan 0001 00:00:00 +0000

AWS Quota

Mon, 01 Jan 0001 00:00:00 +0000

AWS Quota Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

AWS Quota Monitoring

What Is AWS Quota?

AWS Quota refers to the limits set on the number of resources and API requests you can use in Amazon Web Services. Effective management of AWS Quotas ensures optimal resource usage, cost management, and prevents disruption in services due to resource limitations.

Monitoring AWS Quota With Netdata

To monitor AWS Quota with Netdata, you can leverage the aws_quota_exporter, an openmetrics (prometheus) exporter. Netdata is designed to seamlessly ingest data from any Prometheus exporter, providing you with powerful, automated dashboards and alerts without the need for setting up a Prometheus server or Grafana. Netdata’s out-of-the-box integration simplifies the complexity associated with setting up AWS Quota monitoring, enabling you to focus on resource management and optimization.

AWS RDS

Mon, 01 Jan 0001 00:00:00 +0000

AWS RDS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

AWS RDS Monitoring

What Is AWS RDS?

Amazon RDS (Relational Database Service) is a managed cloud database service that simplifies database administration tasks such as hardware provisioning, database setup, patching, and backups. It enables organizations to efficiently scale databases while managing performance, availability, and security. AWS RDS supports various database engines, including MySQL, PostgreSQL, MariaDB, Oracle, and SQL Server, making it a versatile choice for cloud-based database solutions.

Monitoring AWS RDS With Netdata

Monitoring AWS RDS is critical for ensuring the performance, availability, and reliability of your cloud databases. Netdata provides a powerful tool for monitoring AWS RDS using an openmetrics (Prometheus) exporter. Netdata can ingest real-time metrics from any Prometheus exporter, including the rds_exporter, without needing a separate Prometheus server or Grafana setup.

AWS Secrets Manager

Mon, 01 Jan 0001 00:00:00 +0000

AWS SNS

Mon, 01 Jan 0001 00:00:00 +0000

Axis Communications AB SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Azure API Management

Mon, 01 Jan 0001 00:00:00 +0000

Azure App Service

Mon, 01 Jan 0001 00:00:00 +0000

Azure Application Gateway

Mon, 01 Jan 0001 00:00:00 +0000

Azure Application Insights

Mon, 01 Jan 0001 00:00:00 +0000

Azure Cache for Redis

Mon, 01 Jan 0001 00:00:00 +0000

Azure Cognitive Services

Mon, 01 Jan 0001 00:00:00 +0000

Azure Container Apps

Mon, 01 Jan 0001 00:00:00 +0000

Azure Container Instances

Mon, 01 Jan 0001 00:00:00 +0000

Azure Container Registry

Mon, 01 Jan 0001 00:00:00 +0000

Azure Cosmos DB Account

Mon, 01 Jan 0001 00:00:00 +0000

Azure Data Explorer

Mon, 01 Jan 0001 00:00:00 +0000

Azure Data Explorer Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Azure Data Factory

Mon, 01 Jan 0001 00:00:00 +0000

Azure Event Grid Topic

Mon, 01 Jan 0001 00:00:00 +0000

Azure Event Hub

Mon, 01 Jan 0001 00:00:00 +0000

Azure Event Hubs Namespace

Mon, 01 Jan 0001 00:00:00 +0000

Azure ExpressRoute Circuit

Mon, 01 Jan 0001 00:00:00 +0000

Azure ExpressRoute Gateway

Mon, 01 Jan 0001 00:00:00 +0000

Azure Firewall

Mon, 01 Jan 0001 00:00:00 +0000

Azure Front Door

Mon, 01 Jan 0001 00:00:00 +0000

Azure Functions

Mon, 01 Jan 0001 00:00:00 +0000

Azure IoT Hub

Mon, 01 Jan 0001 00:00:00 +0000

Azure IP Ranges

Mon, 01 Jan 0001 00:00:00 +0000

Azure Key Vault

Mon, 01 Jan 0001 00:00:00 +0000

Azure Key Vault

Mon, 01 Jan 0001 00:00:00 +0000

Azure Kubernetes Service Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Azure Load Balancer

Mon, 01 Jan 0001 00:00:00 +0000

Azure Log Analytics Workspace

Mon, 01 Jan 0001 00:00:00 +0000

Azure Logic Apps Workflow

Mon, 01 Jan 0001 00:00:00 +0000

Azure Machine Learning Workspace

Mon, 01 Jan 0001 00:00:00 +0000

Azure Monitor

Mon, 01 Jan 0001 00:00:00 +0000

Azure MySQL Flexible Server

Mon, 01 Jan 0001 00:00:00 +0000

Azure NAT Gateway

Mon, 01 Jan 0001 00:00:00 +0000

Azure PostgreSQL Flexible Server

Mon, 01 Jan 0001 00:00:00 +0000

Azure Service Bus Namespace

Mon, 01 Jan 0001 00:00:00 +0000

Azure SQL Database

Mon, 01 Jan 0001 00:00:00 +0000

Azure SQL Elastic Pool

Mon, 01 Jan 0001 00:00:00 +0000

Azure SQL Managed Instance

Mon, 01 Jan 0001 00:00:00 +0000

Azure Storage Account

Mon, 01 Jan 0001 00:00:00 +0000

Azure Stream Analytics Job

Mon, 01 Jan 0001 00:00:00 +0000

Azure Synapse Analytics Workspace

Mon, 01 Jan 0001 00:00:00 +0000

Azure Virtual Machine

Mon, 01 Jan 0001 00:00:00 +0000

Azure Virtual Machine Scale Set

Mon, 01 Jan 0001 00:00:00 +0000

Azure VPN Gateway

Mon, 01 Jan 0001 00:00:00 +0000

B A T M Advance Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bachmann GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bancomm SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Barco Control Rooms SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Barix AG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Barracuda

Mon, 01 Jan 0001 00:00:00 +0000

Barracuda Cloudgen

Mon, 01 Jan 0001 00:00:00 +0000

Barracuda Networks AG Previous Was Phion Information Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Barracuda Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bay Technical Associates SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

BCache

Mon, 01 Jan 0001 00:00:00 +0000

Bdt GmbH Co KG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Beanstalk

Mon, 01 Jan 0001 00:00:00 +0000

Beanstalk Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Beanstalk Monitoring

What Is Beanstalk?

Beanstalk is a simple and fast work queue. It is designed to reduce the complexity of creating distributed applications and focuses on speed and reliability. Beanstalk helps developers manage background jobs easily, allowing processes to be executed asynchronously.

Monitoring Beanstalk With Netdata

Monitor Beanstalk effortlessly using Netdata’s beanstalk monitoring tool. Netdata provides real-time performance metrics and analytics, enabling you to keep a close eye on your Beanstalk servers and ensure they are running smoothly. Whether you are dealing with data collection, message brokers, or distributed systems, optimizing performance and preventing resource bottlenecks has never been easier with Netdata.

Beijing Raisecom Scientific Technology Development Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bekarts International SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bellcore SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Benu Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Beronet GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Best Power A Division Of General Signal Power Systems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Better Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

BGP flapping: why a peer keeps resetting and how to find the cause

Mon, 01 Jan 0001 00:00:00 +0000

BGP flapping: why a peer keeps resetting and how to find the cause

A BGP peer cycling between Established and Idle is sending a specific signal. The session tears down because one side sent a NOTIFICATION message, and that message carries an error code and subcode that pinpoints the cause. Most monitoring watches only the FSM state (up or down) and ignores the NOTIFICATION payload, so the operator sees flapping without knowing why.

BGP NOTIFICATION and Cease messages: what each subcode is telling you

Mon, 01 Jan 0001 00:00:00 +0000

BGP NOTIFICATION and Cease messages: what each subcode is telling you

A BGP NOTIFICATION in your router log is a peer telling you why it tore down the session. The message carries an error code and an error subcode. Those two numbers tell you whether you are looking at a maintenance window, a route leak, a prefix-limit hit, a CPU-starved control plane, or a BFD-triggered teardown.

Cease (code 6) is the most common NOTIFICATION. Its subcodes, defined in RFC 4486 and extended by RFC 8538 and RFC 9384, hold most of the diagnostic value. Codes 2 through 5 appear less often but point to distinct failure classes: parameter mismatch, malformed updates, hold-timer expiry, and FSM errors.

BGP Peering Topology

Mon, 01 Jan 0001 00:00:00 +0000

BGP RIB and FIB growth: monitoring route-table size before it bites

Mon, 01 Jan 0001 00:00:00 +0000

BGP RIB and FIB growth: monitoring route-table size before it bites

The global BGP routing table grows every year. In 2026, the IPv4 default-free zone sits at approximately 940,000 prefixes, with IPv6 adding roughly 190,000 more. These numbers increase steadily, and the hardware that programs forwarding decisions from them has finite capacity. When that capacity runs out, new routes do not get installed in the forwarding plane. Traffic to affected destinations blackholes. The BGP session stays Established the entire time.

BGP route leak and hijack: the detection signals and alerts that matter

Mon, 01 Jan 0001 00:00:00 +0000

BGP route leak and hijack: the detection signals and alerts that matter

A BGP route leak or hijack does not tear down your session. The peering stays Established, keepalives flow, and the session “up” indicator stays green. What changes is which prefixes your network believes are reachable, through which origin AS, and via what path. Traffic is silently misrouted or blackholed while the session looks healthy.

BGP has no built-in authentication of route ownership. Any AS can announce any prefix. Whether other networks accept the announcement depends on their filtering, and filtering is inconsistently deployed. Approximately 50% of routable IP prefixes carry a Route Origin Authorization (ROA), and only about 6.5% of Internet users sit behind networks that actively reject RPKI-invalid routes. That gap is where leaks and hijacks propagate globally before anyone notices.

BGP session Established but stale: detecting silent route loss

Mon, 01 Jan 0001 00:00:00 +0000

BGP session Established but stale: detecting silent route loss

Your BGP session to a transit provider or iBGP peer is Established, but destinations are unreachable. The RIB is missing prefixes from that peer, or the routes it has are stale. No NOTIFICATION was sent, no session flap occurred, and your monitoring trusts the FSM state.

This is the “Established but stale” pattern. KEEPALIVEs are still exchanged at the TCP level, but the UPDATE exchange has stopped. The peer stopped sending routes, a middlebox is silently dropping UPDATE packets, or Graceful Restart is holding the session open after the remote side went down.

Bharti Telesoft International Pvt Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

BIND 'no more recursive clients: quota reached': the recursive-clients circuit breaker

Mon, 01 Jan 0001 00:00:00 +0000

BIND “no more recursive clients: quota reached”: the recursive-clients circuit breaker

The log message no more recursive clients (N/M): quota reached means BIND has run out of slots for in-flight recursive queries, and new queries are failing. Resolution for your clients is already degraded or broken.

The recursive-clients option (default 1000) caps the number of concurrent upstream fetches the resolver can have outstanding. A soft quota at 90 percent (default 900) starts shedding load before the hard limit. Once the hard limit is reached, every new recursive query returns SERVFAIL, including queries for domains whose upstream nameservers are healthy.

BIND 'too many open files': file descriptor exhaustion and silently dropped queries

Mon, 01 Jan 0001 00:00:00 +0000

BIND ’too many open files’: file descriptor exhaustion and silently dropped queries

BIND logs “too many open files” or “socket: file descriptor exceeds limit” and starts dropping queries. UDP health checks may still pass. Zone transfers fail intermittently. The rndc control channel becomes sluggish or unresponsive. Clients see random timeouts that look like upstream nameserver problems. This is file descriptor exhaustion, and the symptoms masquerade as network or disk issues.

BIND 9 Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is BIND 9?

BIND 9 is a flexible, full-featured open source DNS system.

Monitoring BIND 9 with Netdata

The prerequisites for monitoring BIND 9 with Netdata are to have BIND and Netdata installed on your system.

Netdata auto discovers hundreds of services, and for those it doesn’t turning on manual discovery is a one line configuration. For more information on configuring Netdata for BIND 9 monitoring please read the collector documentation.

You should now see the bind section on the Overview tab in Netdata Cloud already populated with charts about all the metrics you care about.

BIND cache eviction storms: DeleteLRU, an undersized max-cache-size, and the pressure spiral

Mon, 01 Jan 0001 00:00:00 +0000

BIND cache eviction storms: DeleteLRU, an undersized max-cache-size, and the pressure spiral

DeleteLRU is rising fast while DeleteTTL barely moves. Cache hit ratio is dropping. Outbound recursive queries are climbing. RecursClients is trending toward its limit. This is a cache eviction storm: a self-reinforcing loop that can end in SERVFAIL for any query requiring recursion.

The cache is too small for the working set. BIND evicts entries by LRU before their TTLs expire. Each eviction forces a cache miss on the next query for that name, triggering an outbound recursive fetch. More fetches mean more concurrent recursive clients, higher latency per resolution, and less cache room as new entries from upstream compete with entries still under eviction pressure.

BIND cache hit ratio dropping: the leading edge of recursive pain

Mon, 01 Jan 0001 00:00:00 +0000

BIND cache hit ratio dropping: the leading edge of recursive pain

A dropping cache hit ratio is rarely the problem itself. It is the leading indicator that something is about to get worse. When fewer queries are answered from cache, each miss consumes a recursive-client slot, adds latency, and increases exposure to upstream slowness. On a busy resolver, a sustained hit-ratio decline from 95% to 80% can roughly triple outbound query volume and push recursive-client utilization into the danger zone.

BIND clients-per-query and max-clients-per-query: duplicate recursion for popular names

Mon, 01 Jan 0001 00:00:00 +0000

BIND clients-per-query and max-clients-per-query: duplicate recursion for popular names

When hundreds of clients query the same domain at the same instant, BIND does not send hundreds of identical recursive queries upstream. It sends one fetch and attaches the waiting clients to it. Two configuration knobs control how many waiters can attach before BIND starts dropping the overflow: clients-per-query (the soft limit, default 10) and max-clients-per-query (the hard ceiling, default 100).

BIND cold cache after restart: the warming storm and elevated upstream load

Mon, 01 Jan 0001 00:00:00 +0000

After a named restart, the recursive cache is empty. Every incoming query is a cache miss. On a resolver doing 50,000 qps, that means at least 50,000 outbound recursive queries per second to upstream authoritative servers – a 10x increase over steady-state outbound volume. This is the cache-warming storm, and it lasts 30 to 60 minutes.

Three things happen simultaneously: upstream load spikes, the recursive-clients table fills toward its limit, and cache hit ratio starts at zero and climbs. For the first 30 to 60 seconds, some queries may return SERVFAIL while root priming completes and authoritative zones finish loading. All of this is expected behavior.

BIND CPU saturation: single-core bottlenecks, DNSSEC crypto, and per-thread contention

Mon, 01 Jan 0001 00:00:00 +0000

BIND CPU saturation: single-core bottlenecks, DNSSEC crypto, and per-thread contention

BIND CPU saturation often hides from aggregate monitoring. Because named runs one worker thread per CPU core through netmgr, aggregate process CPU can read 25% on a 4-core host while a single core is pinned at 100%. Standard monitoring that checks total CPU utilization sees a healthy daemon. Clients see latency spikes and intermittent timeouts.

The core diagnostic challenge: BIND’s statistics channel reports no per-thread CPU data. You must measure at the OS level with mpstat, pidstat, or direct /proc inspection. Without per-core visibility, the symptom presents as unexplained latency with no obvious cause.

BIND DNSSEC failing from clock drift: NTP, RRSIG inception/expiry windows, and SERVFAIL

Mon, 01 Jan 0001 00:00:00 +0000

BIND DNSSEC failing from clock drift: NTP, RRSIG inception/expiry windows, and SERVFAIL

Signed domains start returning SERVFAIL while unsigned domains resolve normally. The named process is running, CPU is moderate, and the cache hit ratio has not collapsed. There was no recent rndc reload or configuration change. The failure appeared gradually over minutes or hours, not all at once.

This is the fingerprint of a DNSSEC validation failure caused by clock drift. Every RRSIG record carries an inception timestamp (when the signature becomes valid) and an expiry timestamp (when it stops being valid). BIND compares these against the local system clock. If the clock drifts far enough outside the RRSIG validity window, valid signatures appear expired or not-yet-valid, and BIND returns SERVFAIL for the entire domain.

BIND DNSSEC validation failing: 'broken trust chain', ValFail, and SERVFAIL for signed domains

Mon, 01 Jan 0001 00:00:00 +0000

BIND DNSSEC validation failing: ‘broken trust chain’, ValFail, and SERVFAIL for signed domains

Your resolver is returning SERVFAIL for major signed domains. Logs show broken trust chain resolving 'example.com' across dozens of unrelated domains. The per-view ValFail counter is climbing. Unsigned domains resolve normally, and authoritative zones you serve locally still answer.

This is a DNSSEC validation failure on the resolver side. BIND is rejecting signed answers because something in the chain of trust is broken locally. When the breakage is local (clock drift, stale trust anchors, corrupted managed-keys), every signed domain fails simultaneously. When it is upstream (an individual zone with expired signatures), only that zone is affected.

BIND dnssec-validation disabled: the security regression that 'fixes' SERVFAIL

Mon, 01 Jan 0001 00:00:00 +0000

BIND dnssec-validation disabled: the security regression that ‘fixes’ SERVFAIL

SERVFAIL on signed domains is a real production problem. Users cannot reach major services because .com, .org, google.com, and countless other zones are DNSSEC-signed. When validation fails, those names stop resolving. The pressure to restore service is immediate.

The fastest way to make the SERVFAIL disappear is one line in named.conf:

dnssec-validation no;

After a reload, every signed domain resolves again. The monitoring dashboard goes green. The ticket closes. Everything works, including cache-poisoning attacks. The resolver now accepts forged responses for every query it processes.

BIND dynamic update failures: UpdateFail, denied updates, and TSIG drift

Mon, 01 Jan 0001 00:00:00 +0000

BIND dynamic update failures: UpdateFail, denied updates, and TSIG drift

A rising UpdateFail counter on a BIND authoritative server means the dynamic update pipeline is broken. The counter does not tell you why. You need to correlate it with the security log, the zone serial, and the journal file state to narrow the cause.

Failures cluster into three families:

Authentication failures: TSIG key drift, wrong key names, clock skew past the 300-second fudge window.
Authorization failures: the zone’s allow-update or update-policy does not grant the presented key permission to modify the zone.
Journal and filesystem failures: disk exhaustion, permission errors, .jnl corruption from manual zone edits.

Each produces a distinct signal pattern in the logs and counters. Dynamic updates are denied by default. A zone must explicitly include allow-update or update-policy to accept updates. These two directives are mutually exclusive; configuring both is a configuration error and named will refuse to load the zone.

BIND forwarding loops: recursion that never terminates and burns recursive slots

Mon, 01 Jan 0001 00:00:00 +0000

BIND forwarding loops: recursion that never terminates and burns recursive slots

Recursive-clients is climbing toward its limit. SERVFAIL responses are rising across unrelated domains. QueryTimeout counters are ticking up in the per-view resolver stats. It looks like a textbook recursive resolution cascade: an upstream nameserver is slow or unreachable, in-flight queries are piling up, and BIND’s circuit breaker is about to trip.

But when you run rndc recursing to identify the culprit, the upstream IP addresses are not external nameservers. They are your own infrastructure. Another BIND resolver you control, or this very server.

BIND inline signing silently failed: missing keys and a zone served unsigned

Mon, 01 Jan 0001 00:00:00 +0000

BIND inline signing silently failed: missing keys and a zone served unsigned

Your BIND authoritative server is running. Queries return answers. The zone loads without error. But validating resolvers worldwide are returning SERVFAIL for your domain, and there is no error line in your logs. The zone is configured for inline signing, but it is being served unsigned.

When a zone has inline-signing yes; with auto-dnssec maintain; or dnssec-policy default;, BIND should sign the zone transparently and maintain valid RRSIG records. If the signing key files are missing from the key-directory, BIND loads the unsigned zone and serves it as-is. No startup error. No log message. No statistics counter.

BIND journal (.jnl) corruption: dynamic-update and IXFR failures that block zone load

Mon, 01 Jan 0001 00:00:00 +0000

BIND journal (.jnl) corruption: dynamic-update and IXFR failures that block zone load

A zone fails to load after a restart or rndc reload. The named process is running, other zones respond normally, and named-checkzone reports the zone file is syntactically valid. The problem is the journal file (.jnl) alongside the zone file, which records pending dynamic updates and drives IXFR between primary and secondary servers.

Journal corruption has two operational faces. The acute case: named starts but refuses to serve the affected zone, logging “journal rollforward failed: journal out of sync with zone.” The subtle case: IXFR transfers keep failing and falling back to full AXFR. The zone stays current, but each refresh pulls the entire zone over TCP instead of just the diff, wasting bandwidth and CPU.

BIND lame delegations: 'lame server resolving' and nameservers that are not authoritative

Mon, 01 Jan 0001 00:00:00 +0000

BIND lame delegations: ’lame server resolving’ and nameservers that are not authoritative

The lame server resolving message in BIND’s lame-servers log category means the resolver contacted a delegated nameserver that answered but was not authoritative for the zone it was supposed to serve. Each lame encounter wastes a fetch cycle and adds latency. On busy resolvers with high query volume for affected zones, the cost compounds. The per-view Lame counter in resolver statistics tracks how often this happens.

BIND managed-keys and trust anchors: KSK rollover, RFC 5011, and a stale root key

Mon, 01 Jan 0001 00:00:00 +0000

BIND managed-keys and trust anchors: KSK rollover, RFC 5011, and a stale root key

DNSSEC validation depends on a chain of trust anchored at the root zone’s Key Signing Key (KSK). BIND maintains that anchor automatically using RFC 5011 trust anchor management, stored in a managed-keys database. When it breaks, every signed domain on the internet fails validation simultaneously, and the resolver returns SERVFAIL for the majority of real-world queries.

BIND max-cache-size: sizing the resolver cache without triggering the OOM killer

Mon, 01 Jan 0001 00:00:00 +0000

BIND max-cache-size: sizing the resolver cache without triggering the OOM killer

named disappears from the process table. dmesg shows the OOM killer selected it. systemd restarts it, the cache is cold, every query triggers recursion, upstream load spikes, and RSS climbs again. Within hours the cycle repeats. The root cause is often not a memory leak. It is the default max-cache-size.

Since BIND 9.11, the default max-cache-size for views with recursion yes is 90% of physical memory. On a dedicated resolver with 16 GB of RAM, that is 14.4 GB for the cache alone. On a shared box or a mixed-role server that also serves authoritative zones, that default guarantees the OOM killer will eventually visit.

BIND monitoring checklist: the signals every production resolver and authoritative server needs

Mon, 01 Jan 0001 00:00:00 +0000

BIND monitoring checklist: the signals every production resolver and authoritative server needs

BIND (named) processes queries through a pipeline: receive packet, parse, ACL/view check, cache lookup (if recursive), zone lookup (if authoritative), recursive fetch on cache miss, apply RPZ/DNSSEC, serialize response, send. Every signal below maps to a stage in that pipeline or a resource it competes for: CPU, memory, file descriptors, network buffers, source ports.

The levels are cumulative. Most signals come from the statistics channel (JSON at /json/v1/server, explicitly configured in named.conf). The rndc stats file is an alternative but appends indefinitely and can fill disk if not rotated. Commands below assume a single named process (standard deployment; BIND is multithreaded, not multiprocess).

BIND monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

BIND monitoring maturity model: from survival to expert

Most BIND deployments catch complete outages but miss the slow-burn failures that cause real incidents: cache pressure spirals, recursive client exhaustion, DNSSEC signature expiry, and kernel-level UDP drops that BIND itself never sees.

This article maps four monitoring maturity levels, from Survival to Expert. Each level adds signals that catch failure patterns invisible to the previous one. Use this as an inventory checklist: identify your current level, then decide which signals to add next based on whether you run a recursive resolver, an authoritative-only server, or a mixed-role deployment.

BIND named killed by the OOM killer: memory exhaustion and the cold-restart storm

Mon, 01 Jan 0001 00:00:00 +0000

BIND named killed by the OOM killer: memory exhaustion and the cold-restart storm

named vanished from the process table. systemctl status named shows inactive (dead) or a restart timestamp from minutes ago. The likely cause: the Linux OOM killer terminated named when system memory was exhausted.

The cycle is self-reinforcing. BIND’s RSS grows steadily (excessive cache allocation, oversized RPZ, or a version-specific leak) until the kernel OOM killer selects named as the victim. All DNS resolution fails instantly. If systemd restarts the service, the cold cache forces every query through recursion, creating a warming storm that spikes upstream load. If the memory condition persists, the cycle repeats: start, grow, OOM, kill, restart.

BIND named RSS climbing: cache growth, allocator fragmentation, and real leaks

Mon, 01 Jan 0001 00:00:00 +0000

BIND named RSS climbing: cache growth, allocator fragmentation, and real leaks

named RSS trending upward after warmup is the leading indicator before an OOM kill. The hard part is distinguishing three conditions that look identical on an RSS chart: cache-driven growth that will plateau under max-cache-size, allocator fragmentation that inflates RSS without a real leak, and genuine unbounded growth from a bug or misconfiguration.

RSS that climbs during a traffic burst or cache warmup and stays high is normal allocator behavior. BIND’s allocator (jemalloc on most builds) holds freed blocks for reuse rather than returning them to the OS via munmap. The operational question is not “is RSS high?” but “is RSS still growing, and how much runway remains?”

BIND NOTIFY not propagating: secondaries not refreshing when the primary changes

Mon, 01 Jan 0001 00:00:00 +0000

BIND NOTIFY not propagating: secondaries not refreshing when the primary changes

You updated a zone on the primary, reloaded with rndc reload, and confirmed the primary is serving the new serial. The secondaries still show the old one. Queries against them return stale data for minutes or hours. The zone is not broken, just delayed.

This is NOTIFY not reaching the secondaries. NOTIFY (RFC 1996) is a UDP push from the primary telling secondaries to check for a new SOA serial immediately, rather than waiting for the refresh timer. When NOTIFY is lost, blocked, or misconfigured, the secondary learns about changes only when its SOA refresh timer fires (commonly 1 hour for typical SOA values). The data looks delayed, not broken. The refresh timer eventually catches up, which is why this rarely pages, but it can mask more serious transfer problems if the refresh timer is the only thing keeping secondaries current.

BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists

Mon, 01 Jan 0001 00:00:00 +0000

BIND NXDOMAIN spike: DGA malware, water torture, and Windows suffix search lists

The QryNXDOMAIN counter jumping above 3x baseline is a common alert trigger, but the raw rate tells you almost nothing. Two resolvers with identical NXDOMAIN rates can be in completely different states: one healthy, one under active attack.

The signal that matters is query-name cardinality and entropy. If the same names repeat, the spike is benign (Windows suffix search lists, new client rollouts). If nearly every query name is unique and random, you are looking at a water torture attack or DGA malware beaconing. This distinction determines whether you page someone at 3 a.m. or close the alert.

BIND open recursive resolver: DNS amplification abuse and allow-recursion posture

Mon, 01 Jan 0001 00:00:00 +0000

BIND open recursive resolver: DNS amplification abuse and allow-recursion posture

An open recursive resolver answers recursive DNS queries from any source address. BIND configured with allow-recursion { any; } serves legitimate clients, but it also serves attackers: spoofed source IPs turn your resolver into a DDoS amplification relay, where small queries produce large responses directed at victims you have never heard of.

The abuse can be invisible. If the attack volume is small relative to your legitimate traffic, query rates look normal, cache hit ratios look healthy, and SERVFAIL rates stay flat. The only evidence is in the configuration itself and in subtle traffic pattern shifts: elevated ANY or TXT query shares, high source-IP diversity, or responses going to networks that have no business querying your resolver.

BIND query logging in production: the gradual performance bottleneck teams forget to turn off

Mon, 01 Jan 0001 00:00:00 +0000

BIND query logging in production: the gradual performance bottleneck teams forget to turn off

BIND query logging is a debugging tool, not a monitoring strategy. Teams enable it during an incident or security investigation, then forget to disable it. The result is a performance degradation so gradual that it gets attributed to traffic growth, hardware aging, or “BIND being slow.” By the time someone connects the dots, the resolver has been losing throughput for weeks.

BIND random subdomain (water torture) attack: an NXDOMAIN flood that bypasses the cache

Mon, 01 Jan 0001 00:00:00 +0000

BIND random subdomain (water torture) attack: an NXDOMAIN flood that bypasses the cache

Your BIND resolver is up, the process is running, UDP and TCP both answer on port 53. But clients across the network are experiencing slow DNS or outright SERVFAIL. NXDOMAIN rate has spiked to several times baseline. Cache hit ratio is collapsing. Recursive clients are climbing toward the hard limit. Upstream query rate has ballooned to approach or exceed the inbound rate.

BIND RecursClients climbing toward the limit: reading the recursive saturation gauge

Mon, 01 Jan 0001 00:00:00 +0000

BIND RecursClients climbing toward the limit: reading the recursive saturation gauge

RecursClients is the number of queries currently in flight, each waiting for an upstream nameserver to respond. When this gauge climbs toward the configured recursive-clients limit, BIND is running out of slots to start new recursive lookups. Past the hard limit, every new recursive query receives SERVFAIL.

The metric is a gauge, not a rate. The absolute number means little without the configured ceiling next to it: 300 is comfortable against a limit of 1000 and dangerous against a limit of 350. Track RecursClients as a percentage of recursive-clients, not as a raw count, and watch the daily peak for runway estimation.

BIND recursive resolution cascade: one slow upstream taking down all resolution

Mon, 01 Jan 0001 00:00:00 +0000

BIND recursive resolution cascade: one slow upstream taking down all resolution

SERVFAIL responses are flooding your recursive resolver. Users cannot resolve dozens of unrelated domains. But named is running, CPU looks moderate, and the authoritative zones on the same instance are still answering fine. This is the recursive resolution cascade: a single slow or unreachable upstream authoritative server consumes all available recursive-clients slots, and BIND returns SERVFAIL for queries that have nothing to do with the failing upstream.

BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out

Mon, 01 Jan 0001 00:00:00 +0000

BIND REFUSED responses: ACL denials, recursion policy, and clients that get locked out

REFUSED (DNS rcode 5) is a deliberate policy decision: BIND received the query, parsed it, and chose not to answer. The server is alive, listening, and processing queries. It is configured to reject this particular query from this particular source.

The actionable scenario: legitimate client subnets that previously resolved names or queried your zones suddenly start receiving REFUSED, typically after a configuration change. The fix is almost always an ACL or view mismatch, not a restart.

BIND resolver NumFetch per view: per-view recursive pressure in split-horizon setups

Mon, 01 Jan 0001 00:00:00 +0000

BIND resolver NumFetch per view: per-view recursive pressure in split-horizon setups

RecursClients is a global NSStats counter that reports the total number of recursive clients awaiting resolution across all views. It does not break down which view is consuming the pool. When RecursClients climbs to 847 out of 1000, you know the pool is stressed but not whether the pressure is evenly distributed or whether one view is responsible for most of it.

BIND resolver RTT distribution shifting high: upstream nameserver degradation

Mon, 01 Jan 0001 00:00:00 +0000

BIND resolver RTT distribution shifting high: upstream nameserver degradation

The QryRTT histogram in BIND’s per-view resolver statistics is shifting toward higher millisecond buckets. What was once dominated by RTT10 and RTT100 (sub-100ms responses from upstream authoritative servers) is now accumulating in RTT500, RTT800, and RTT1600. In severe cases, most outbound recursive queries land in the overflow RTT1600+ bucket.

This metric is not client-perceived latency. The QryRTT counters measure the round-trip time of BIND’s outbound recursive queries to upstream authoritative nameservers. A rightward shift means the servers BIND depends on for cache misses are taking longer to respond. BIND has no native inbound client latency histogram; if you need end-to-end latency from the client perspective, use dnstap or external measurement.

BIND Response Rate Limiting (RRL): RateDropped, RateSlipped, and throttled legitimate clients

Mon, 01 Jan 0001 00:00:00 +0000

BIND Response Rate Limiting (RRL): RateDropped, RateSlipped, and throttled legitimate clients

You see sustained non-zero RateDropped or RateSlipped counters in BIND’s statistics channel. Either RRL is absorbing a real DNS amplification or flood attack, or the configuration is too aggressive and silently dropping or truncating responses to legitimate clients. BIND’s RRL counters do not distinguish attacker from legitimate client. A dropped response is a dropped response, whether the source was a spoofed botnet node or a real resolver. Telling the difference requires correlating the counters with traffic patterns, source IP distribution, and TCP reachability.

BIND RPZ rewrites: threat-interception counters and reading a malware outbreak

Mon, 01 Jan 0001 00:00:00 +0000

BIND RPZ rewrites: threat-interception counters and reading a malware outbreak

Response Policy Zones (RPZ) let BIND rewrite DNS responses that match policy rules: blocked, dropped, redirected, or passed through with an exception marker. The RPZRewrites counter in BIND’s Name Server Statistics tracks how often this happens. When that counter spikes, it is usually the first telemetry signal that something inside your network is reaching known-bad infrastructure.

This article covers what RPZRewrites actually counts, what it silently excludes, how to distinguish a real malware outbreak from background noise, and what RPZ costs in query performance and memory. It assumes RPZ is already configured. For the broader BIND monitoring framework, see How BIND actually works in production: a mental model for operators.

BIND RRSIG expiry on authoritative zones: the silent signing time bomb

Mon, 01 Jan 0001 00:00:00 +0000

BIND RRSIG expiry on authoritative zones: the silent signing time bomb

Your zone is down. Validating resolvers worldwide return SERVFAIL for every query to your domain. But the authoritative server looks fine: named is running, port 53 is listening, the statistics channel shows normal traffic. No errors in the logs. No alerts fired.

The problem is expired RRSIG signatures. BIND’s authoritative server does not validate its own signatures at serve time. It will serve expired RRSIGs indefinitely, and every validating resolver that queries your zone will reject the response. The outage is invisible from the server itself.

BIND secondary zone expired: the SOA expire timer runs out and the zone returns SERVFAIL

Mon, 01 Jan 0001 00:00:00 +0000

BIND secondary zone expired: the SOA expire timer runs out and the zone returns SERVFAIL

A single zone on your BIND secondary starts returning SERVFAIL. Other zones on the same server answer normally. The named process is up, CPU and memory look fine, and port 53 responds to health checks. Your monitoring shows green because it checks process liveness or queries a different zone. Only clients asking for that one zone are failing.

BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause

Mon, 01 Jan 0001 00:00:00 +0000

BIND SERVFAIL responses: what a DNS SERVFAIL actually means and how to trace the cause

SERVFAIL (RCODE 2) means BIND attempted resolution and could not return a usable answer. It is not NXDOMAIN (name does not exist), REFUSED (policy denial), or FORMERR (malformed query). The process is alive, port 53 is open, the query was received, and the answer is failure.

This makes SERVFAIL invisible to binary health checks. A BIND server returning 100% SERVFAIL to every client still passes process-liveness and port-check probes. SERVFAIL is a symptom with many possible causes: upstream nameserver timeouts, DNSSEC validation failures, recursive-clients exhaustion, broken delegation, a zone that failed to load, or a configuration error. Tracing it requires correlating multiple BIND signals.

BIND SOA expire runway: the countdown that predicts a silent secondary-zone outage

Mon, 01 Jan 0001 00:00:00 +0000

BIND SOA expire runway: the countdown that predicts a silent secondary-zone outage

A BIND secondary serving stale data is not broken. It answers queries with old but functional records, and from the outside it looks healthy. The danger is the countdown underneath: the SOA expire timer, ticking from the last successful zone transfer. When it reaches zero, the secondary stops serving the zone and returns SERVFAIL or REFUSED for every query against it.

BIND SOA serial mismatch: a secondary serving stale data behind the primary

Mon, 01 Jan 0001 00:00:00 +0000

BIND SOA serial mismatch: a secondary serving stale data behind the primary

The secondary’s SOA serial is behind the primary. Clients querying the secondary get stale A values, missing new entries, deleted hosts still resolving. The server responds NOERROR because its zone data is internally consistent. It is not current.

This is the precursor to zone expiry, and it is silent. The secondary serves stale but functional answers for the entire SOA expire period, typically 1 to 4 weeks. Monitoring that checks “can I resolve this zone?” passes. Health checks on port 53 pass. Everything looks fine except the data is wrong. The only signal is the serial number gap between primary and secondary, and BIND’s statistics channel does not expose it. You must probe externally.

BIND tcp-clients exhaustion: the TCP connection limit, transfers, and truncation fallback

Mon, 01 Jan 0001 00:00:00 +0000

BIND tcp-clients exhaustion: the TCP connection limit, transfers, and truncation fallback

BIND is up. rndc status shows “running.” Your UDP health check returns NOERROR in under 2ms. But zone transfers are failing, some DNSSEC-validated queries time out, and clients receiving large responses report intermittent connection refused on TCP/53. If your monitoring only probes UDP, you will not know anything is wrong until a secondary’s zone expires or a downstream resolver escalates a ticket.

BIND TSIG failure on zone transfer: BADKEY, BADTIME, and refused transfers

Mon, 01 Jan 0001 00:00:00 +0000

BIND TSIG failure on zone transfer: BADKEY, BADTIME, and refused transfers

Zone transfers between your BIND primary and secondary have stopped. The secondary is falling behind on SOA serial, and the logs show TSIG verification failures: BADKEY, BADTIME, or BADSIG. The transfer is refused, and the secondary silently drifts toward zone expiry.

The failure hides in the security and xfer-in logging categories, not in the main query path. A secondary can serve stale data for days or weeks until the SOA expire timer runs out, at which point it stops serving the zone entirely and returns SERVFAIL. Nothing alerts until the zone disappears.

BIND UDP packet-rate saturation: softirq, single-core bottlenecks, and pps limits

Mon, 01 Jan 0001 00:00:00 +0000

DNS queries are timing out for some clients. BIND statistics look clean: query rate is steady, SERVFAIL is low, cache hit ratio is normal. The link is not saturated. BIND logs show nothing unusual. But clients keep reporting intermittent failures with no apparent correlation to domain, client subnet, or time of day.

If you have ruled out upstream issues, cache pressure, and DNSSEC failures, the problem may be in the kernel. When the host cannot process UDP packets fast enough, the kernel drops them before BIND reads them from the socket. BIND has no visibility into these drops: no log entry, no statistics counter, no error. The only evidence lives in kernel-level UDP counters that most DNS monitoring setups never collect.

BIND UDP receive buffer errors: the invisible query loss named never logs

Mon, 01 Jan 0001 00:00:00 +0000

BIND UDP receive buffer errors: the invisible query loss named never logs

Clients report intermittent DNS timeouts. Retries are up. Your monitoring says named is healthy: the process is running, port 53 is open, functional queries succeed, and BIND’s statistics channel shows reasonable response rates with no elevated SERVFAIL. You suspect upstream nameserver problems, network path issues, or client-side misconfiguration. None of those investigations turn up anything.

The problem may be happening between the kernel and BIND, in a layer where named has zero visibility. When the kernel’s UDP receive buffer overflows, packets are silently dropped before BIND ever reads them from the socket. There is no log entry, no statistics counter increment, no error of any kind inside named. The only evidence lives in kernel-level counters that most BIND monitoring setups never collect.

BIND UDP vs TCP query ratio: truncation, EDNS negotiation, and TCP fallback spikes

Mon, 01 Jan 0001 00:00:00 +0000

BIND UDP vs TCP query ratio: truncation, EDNS negotiation, and TCP fallback spikes

A rising TCP share of DNS queries is one of the most informative early warning signals in BIND. TCP normally carries under 5% of query volume on a recursive resolver. When that ratio shifts upward, something has changed in the resolution path: responses are being truncated, EDNS negotiation is failing, zone transfers are spiking, or a firewall is interfering with DNS traffic.

BIND unauthorized zone transfer attempts: AXFR/IXFR from sources not on allow-transfer

Mon, 01 Jan 0001 00:00:00 +0000

BIND unauthorized zone transfer attempts: AXFR/IXFR from sources not on allow-transfer

Your BIND logs show denied AXFR or IXFR requests from unknown source IPs. The entries appear under the security category at error severity:

client 198.51.100.42#53124: zone transfer 'example.com/AXFR/IN' denied

Denied attempts mean your ACL is working. The critical question is whether any unauthorized transfer succeeded, because a successful AXFR exfiltrates the entire zone: every record, internal hostnames, SRV targets, TXT metadata, and the full infrastructure topology.

BIND upstream query timeouts: QueryTimeout, retries, and the 30-second resolution stall

Mon, 01 Jan 0001 00:00:00 +0000

BIND upstream query timeouts: QueryTimeout, retries, and the 30-second resolution stall

When a BIND resolver sends a recursive query upstream and gets no response, it waits. The default wait is 10 seconds, and BIND may retry the query up to 3 times before giving up. A single failed resolution can occupy a recursive-client slot for 30 seconds or more. If the upstream failure is broad enough, those slots fill, the recursive-clients limit is reached, and every new recursive query starts returning SERVFAIL.

BIND zone not loaded after reload: 'loading from master file failed' and the silent zone outage

Mon, 01 Jan 0001 00:00:00 +0000

BIND zone not loaded after reload: ’loading from master file failed’ and the silent zone outage

You ran rndc reload after editing a zone file. rndc status reports the server is running. systemd says the service is active. Your monitoring confirms the process is alive and port 53 is open. But something is wrong with one specific zone.

named does not crash when a single zone fails to load. It logs the error and continues serving the zones that loaded successfully. Two outcomes are possible for the affected zone:

BIND zone transfer failed (AXFR/IXFR): reading xfer-in failures before the zone goes stale

Mon, 01 Jan 0001 00:00:00 +0000

BIND zone transfer failed (AXFR/IXFR): reading xfer-in failures before the zone goes stale

A secondary BIND server’s zone transfers are failing. The zone is still being served, clients are still getting answers, and your health checks are still green. The secondary is serving stale data from the last successful transfer, and the SOA expire timer is counting down. When it reaches zero, the secondary stops serving the zone and returns SERVFAIL or REFUSED for every query.

Bintec Communications GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

bio-rd / RIPE RIS

Mon, 01 Jan 0001 00:00:00 +0000

Bird Routing Daemon

Mon, 01 Jan 0001 00:00:00 +0000

Bird Routing Daemon Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Bird Routing Daemon Monitoring

What Is Bird Routing Daemon?

The Bird Routing Daemon is a comprehensive software for managing BGP and other routing protocols. It’s favored in the networking community for its scalability and flexibility. With Bird, administrators can maintain robust and efficient network routing essential for data flow across various network architectures.

Monitoring Bird Routing Daemon With Netdata

Netdata provides an efficient Bird Routing Daemon monitoring tool that leverages an openmetrics (Prometheus) exporter for data collection. This integration allows seamless ingestion of Bird Routing Daemon metrics without needing an additional Prometheus server or Grafana setup. Netdata offers automated dashboards and alerts out-of-the-box, ensuring you have real-time insight into your network’s performance and reliability.

Bird Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bke A S SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Blackbox

Mon, 01 Jan 0001 00:00:00 +0000

Blackbox Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Blackbox Monitoring

What Is Blackbox?

Blackbox is a versatile monitoring tool designed to test endpoints via HTTP, DNS, TCP and ICMP protocols. As a member of the Prometheus ecosystem, Blackbox enables the continuous evaluation of uptime and response times, providing critical insights into external service availability. By simulating requests to your key infrastructure components, Blackbox ensures you are the first to know when issues occur.

Monitoring Blackbox With Netdata

To effectively monitor Blackbox, Netdata integrates with the Prometheus ecosystem via an openmetrics (Prometheus) exporter. This means you can leverage the Blackbox exporter to gather metrics seamlessly. With Netdata, you can ingest data from any Prometheus exporter, streamlining your monitoring setup without needing a full Prometheus server or Grafana dashboard stack. Netdata automatically provides dashboards and alerts, ensuring you have real-time insights into your monitored services.

Blade Network Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log

Mon, 01 Jan 0001 00:00:00 +0000

blk_update_request: I/O error, dev nvme0n1: reading NVMe I/O errors in the kernel log

dmesg shows lines like this:

blk_update_request: I/O error, dev nvme0n1, sector 12345678

Maybe one line. Maybe thousands. Maybe interleaved with nvme nvme0: Resetting controller or nvme nvme0: Removing. The operational question is always the same: is this a single bad block the drive surfaced on a read, or is the device, the controller, or the PCIe link underneath it failing?

Blue Coat Licensing

Mon, 01 Jan 0001 00:00:00 +0000

Blue Coat Systems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bluecat Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bluecat Server

Mon, 01 Jan 0001 00:00:00 +0000

Bluecoat Proxysg

Mon, 01 Jan 0001 00:00:00 +0000

Blueflood

Mon, 01 Jan 0001 00:00:00 +0000

Bluesocket Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bmc Software SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

BMP (BGP Monitoring Protocol)

Mon, 01 Jan 0001 00:00:00 +0000

BOINC

Mon, 01 Jan 0001 00:00:00 +0000

BOINC Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

BOINC Monitoring

What Is BOINC?

The Berkeley Open Infrastructure for Network Computing (BOINC) is a platform for volunteer computing and is used by various scientific projects. It allows users to donate their computing resources to assist in complex calculations and data analysis. Understanding the underlying metrics of BOINC is crucial for maintaining optimal performance and ensuring that resources are effectively utilized.

Monitoring BOINC With Netdata

Netdata provides an intuitive and powerful way to monitor your BOINC instances. Our BOINC monitoring tool collects real-time data and allows you to visualize important metrics to identify trends and potential issues. With Netdata, you can gain insights into task counts and manage distributed computing efforts more effectively.

Borderware Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

BOSH

Mon, 01 Jan 0001 00:00:00 +0000

BOSH Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

BOSH Monitoring

What Is BOSH?

BOSH is an open-source tool used for release engineering, deployment, lifecycle management, and monitoring of distributed systems. As a pivotal component in cloud orchestration, BOSH simplifies the deployment of cloud infrastructure, ensuring scalability and resilience. It’s an essential tool for DevOps, SRE, developers, IT admins, and IT engineers who manage complex application environments.

Monitoring BOSH With Netdata

To monitor BOSH effectively, Netdata employs a powerful openmetrics (Prometheus) exporter called the BOSH exporter. This integration allows Netdata to ingest data directly from any Prometheus exporter. With Netdata, you get automated dashboards and real-time alerts, all without needing a separate Prometheus server or Grafana setup. This makes Netdata a highly efficient BOSH monitoring tool, allowing for a seamless and comprehensive monitoring experience.

Brand Communications Limited SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Bridgewave Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Broadband Access Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Broadcom Limited SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Broadsoft Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Brocade

Mon, 01 Jan 0001 00:00:00 +0000

Brocade Communication Systems Inc Formerly Foundry Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Brocade Communications Systems Inc Formerly Mcdata Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Brocade Communications Systems Inc Formerly Nuview Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Brocade Communications Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Brocade FC Switch

Mon, 01 Jan 0001 00:00:00 +0000

Brother

Mon, 01 Jan 0001 00:00:00 +0000

Brother NET Printer

Mon, 01 Jan 0001 00:00:00 +0000

Bti Photonic Systems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

BTRFS

Mon, 01 Jan 0001 00:00:00 +0000

BungeeCord

Mon, 01 Jan 0001 00:00:00 +0000

BungeeCord Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

BungeeCord Monitoring

What Is BungeeCord?

BungeeCord is a popular proxy server for the Minecraft game, used to connect multiple servers together for seamless gameplay. It allows players to switch between different Minecraft servers without needing to relog, making it an essential tool for managing complex Minecraft networks.

Monitoring BungeeCord With Netdata

To monitor BungeeCord, Netdata employs an openmetrics (Prometheus) exporter. This approach leverages the BungeeCord Prometheus Exporter, allowing Netdata to track important server metrics seamlessly. With Netdata, you can ingest data from any Prometheus exporter, offering automated dashboards, alerting, and more. Remarkably, this doesn’t require a Prometheus server or Grafana, making it simple and efficient.

Bytesphere LLC SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

C C Power Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cable Television Laboratories Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cacheflow Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cacti SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cadant Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

CAIDA Routeviews Prefix-to-AS

Mon, 01 Jan 0001 00:00:00 +0000

Calix Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cambium Networks Limited Formerly Pipinghot Networks Limited SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Carel SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cascade Communications Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra adding and removing nodes safely: vnodes, tokens, and cleanup

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra adding and removing nodes safely: vnodes, tokens, and cleanup

Expanding or contracting a Cassandra cluster redistributes token ownership, triggers bulk streaming, and leaves stale data that cleanup must reclaim. Skip cleanup after a bootstrap, run it while a node is joining, or misconfigure token allocation, and you waste disk space, create hot spots, and risk silent inconsistency.

This guide covers adding and removing nodes with virtual nodes (vnodes) enabled, the default for Cassandra 3.x through 5.x. Commands and paths assume standard packaged Cassandra on Linux. Perform topology changes one node at a time.

Cassandra Batch too large warning: oversized batches and coordinator OOM

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra Batch too large warning: oversized batches and coordinator OOM

Oversized BEGIN BATCH statements cause Batch for [ks.table] is of size N, exceeding specified threshold of M by ... warnings and coordinator OutOfMemoryError. Unlike single-partition batches, which provide atomicity within one partition, multi-partition batches force the coordinator to hold mutation buffers for every affected partition until all replicas acknowledge. When the buffer grows large enough, it triggers heap pressure, long GC pauses, and eventual OOM.

Cassandra clock skew: how NTP drift silently corrupts data

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra clock skew: how NTP drift silently corrupts data

Cassandra’s last-write-wins conflict resolution is simple, fast, and unforgiving. Every write carries a timestamp; when replicas disagree, the highest timestamp wins. The database assumes larger timestamps correspond to later wall-clock events. When node clocks drift, that assumption collapses. A write that happened first can carry a later timestamp, or a later write an earlier one. The result is not a timeout, an UnavailableException, or an ERROR log entry. It is silent data loss, permanent shadowing of valid writes, or the sudden return of deleted rows.

Cassandra commitlog disk full: segment exhaustion and forced flushes

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra commitlog disk full: segment exhaustion and forced flushes

WriteTimeoutException from client drivers, a node still UP in gossip, and a commitlog volume nearing 100% with climbing pending tasks in nodetool info mean commitlog segment exhaustion. Unlike data disk exhaustion, which slows compaction, commitlog pressure blocks the write path directly: every mutation must be durably appended to the WAL before acknowledgment.

Cassandra recycles commitlog segments only after all memtables they reference are flushed to SSTables. When the flush pipeline cannot keep pace, segments accumulate until the total size exceeds commitlog_total_space (or commitlog_total_space_in_mb on older versions) or the filesystem fills. Cassandra then forces flushes of every dirty column family referenced in the oldest segment to free space. If flushes are already backed up, this cascades into blocked segment allocation, dropped mutations, and depending on commit_failure_policy, a node that stops accepting writes entirely.

Cassandra commitlog pending tasks: write-path I/O pressure

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra commitlog pending tasks: write-path I/O pressure

Sustained non-zero CommitLog PendingTasks means a Cassandra node’s write path is backing up. Every write must be appended to the commitlog and synced to disk before the coordinator acknowledges it. When the fsync thread cannot keep up, mutations queue. This starts as elevated write latency; if the queue persists, it forces emergency memtable flushes, overwhelms the flush and compaction pipeline, and produces dropped mutations.

Cassandra compaction death spiral: when writes outrun compaction throughput

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra compaction death spiral: when writes outrun compaction throughput

P99 read latency climbs while disk utilisation on the data volume pins near 100%. nodetool compactionstats shows pending tasks rising hour over hour, and nodetool tablestats reports a growing SSTable count. Writes stay fast; reads slow down. This is the compaction death spiral: writes exceed compaction throughput, SSTables accumulate, and read amplification rises.

Unlike a sudden node crash, this failure is gradual. A background queue grows a little each day. Once disk I/O saturates, the cycle self-reinforces: compaction falls further behind, reads consult more files, latency spikes, and the backlog deepens. By the time client SLAs breach, recovery can take hours.

Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra compaction strategies: STCS vs LCS vs TWCS vs UCS

Compaction merges immutable SSTables, discards tombstones, and reclaims disk space. The strategy assigned to a table controls the tradeoff between write amplification and read amplification, and it determines how much temporary disk headroom you must preserve. A fit strategy keeps SSTable counts low and latency predictable. A mismatch creates compaction debt: creeping P99 read latency first, then disk space exhaustion, and finally write rejections when compaction cannot reclaim space fast enough to keep up with flushes.

Cassandra compaction stuck: large partitions blocking a compaction thread

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra compaction stuck: large partitions blocking a compaction thread

nodetool compactionstats shows a compaction on one table that has not moved past the same byte offset for hours. The progress percentage is frozen, the pending queue behind it is growing, and read latency on that table is creeping up. This is not a slow disk. A single large partition has monopolized a compaction thread, turning background maintenance into a bottleneck that threatens node stability.

Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra consistency levels explained: QUORUM, ONE, LOCAL_QUORUM, and EACH_QUORUM

Consistency level (CL) balances latency, availability, and correctness. It does not control how many replicas store data; replication factor (RF) does. CL controls how many replicas must acknowledge a read or write before the coordinator returns to the client. The wrong choice produces UnavailableException during rolling restarts that should be safe, or leaves replicas inconsistent for hours after a write is acknowledged at CL=ONE. The four CLs that define most production topologies are ONE, QUORUM, LOCAL_QUORUM, and EACH_QUORUM.

Cassandra CorruptSSTableException and FSError: disk failure and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra CorruptSSTableException and FSError: disk failure and recovery

A Cassandra node stops serving traffic or refuses to start. In system.log you see org.apache.cassandra.io.sstable.CorruptSSTableException, FSError, or a JVM shutdown triggered by a filesystem exception. These indicate disk failure, filesystem corruption, or irreversible SSTable damage, not retryable application bugs.

Because SSTables are immutable, a corrupt file cannot be patched. The node either stops serving the data or shuts down, depending on disk_failure_policy. Recovery requires at least one healthy replica. Without that, corruption is data loss.

Cassandra disk space exhaustion: emergency recovery when the data volume fills

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra disk space exhaustion: emergency recovery when the data volume fills

A Cassandra node that runs out of disk space does not degrade gracefully. Compaction halts because it cannot allocate temporary space to merge SSTables. Old SSTables are never deleted. Writes append to the commitlog until segment allocation blocks. At that point the node rejects mutations and CommitLog.WaitingOnSegmentAllocation climbs. You may see No space left on device errors while the data volume still reports a few percent free, because Cassandra’s internal headroom requirements are stricter than the filesystem.

Cassandra dropped mutations: silent write loss and load shedding

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra dropped mutations: silent write loss and load shedding

Your application logs successful writes, but reads return stale or missing data. An alert fires on DroppedMessage rate for MUTATION scope. The client never received an error, yet a replica discarded the write after it sat in the MutationStage queue past timeout. This is Cassandra load shedding. Silent write loss occurs whenever not enough other replicas succeed to meet the consistency level.

Cassandra dropped reads and other messages: reading nodetool tpstats Dropped

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra dropped reads and other messages: reading nodetool tpstats Dropped

When nodetool tpstats reports non-zero values in the Dropped section, the node discarded internal messages that exceeded their stage timeout. These counters are cumulative since JVM startup, not rates. A non-zero value warrants investigation: the timeout is defined in cassandra.yaml by settings such as read_request_timeout_in_ms and write_request_timeout_in_ms, so a drop means the message sat in the queue for seconds.

Dropped messages are a lagging indicator. Correlate the drop type with the matching thread pool pending count, disk I/O latency, and GC pause duration to find the root cause.

Cassandra frequent memtable flushes: small SSTables and compaction burden

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra frequent memtable flushes: small SSTables and compaction burden

When MemtableFlushWriter pending tasks climb on Cassandra nodes, SSTables often land on disk at only tens of megabytes and multiply quickly. Within hours, PendingCompactions rises, read latency drifts, and disk I/O stays pinned despite flat write throughput. Frequent memtable flushing under memory pressure creates compaction debt and read amplification. Once started, the cluster enters a feedback loop that is hard to unwind without targeted tuning.

Cassandra GC death spiral: long pauses, gossip flapping, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra GC death spiral: long pauses, gossip flapping, and recovery

You are paged because a Cassandra node is flapping between UP and DOWN in nodetool status, client timeouts are rising, and system logs show GCInspector warnings. The node has not crashed. It is stuck in a GC death spiral: heap pressure produces long pauses, gossip marks the node DOWN, and the resulting retry and hint traffic creates even more heap pressure when the node recovers. It can start with a single large partition read, a misconfigured cache, or an oversized batch statement, and escalates until the node is effectively useless. Catch it early by watching the GC floor and gossip stability together, not just process uptime.

Cassandra GC pauses too long: diagnosing G1 stop-the-world pauses

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra GC pauses too long: diagnosing G1 stop-the-world pauses

ReadTimeoutException and WriteTimeoutException from clients, GCInspector warnings in system.log, and nodes flapping between UP and DOWN in nodetool status without a JVM restart mean G1 is producing long stop-the-world pauses. Root causes include promotion pressure, humongous objects, or allocation bursts. Left unchecked, one node’s pauses trigger gossip failures, retries, and hint replay that drive cluster-wide degradation.

What this means

G1GC is the default collector for Cassandra 4.x on JDK 11+. During a stop-the-world pause, every thread freezes, including gossip, native transport, and compaction. Cassandra logs GCInspector warnings when a pause exceeds the configured threshold, commonly 500 ms . Pauses longer than ~2 seconds cause gossip rounds to be missed; under the default phi accrual failure detector threshold of 8, sustained pauses of tens of seconds result in the node being marked DOWN . While the JVM is paused, mutations queue, reads stall, hints accumulate on peers, and clients retry. On recovery, hint replay and retry bursts raise allocation pressure, creating a self-reinforcing spiral.

Cassandra gossip flapping: nodes bouncing UP and DOWN

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra gossip flapping: nodes bouncing UP and DOWN

nodetool status shows a node flipping between UN and DN, or multiple nodes doing it in sequence. Each transition forces the cluster to replay hints, recalculate read repair, and propagate gossip state. When a node transitions more than three times in thirty minutes, you are dealing with gossip flapping. It is almost always JVM heap pressure or GC pauses misdiagnosed as a network problem.

Cassandra heap pressure: sizing the JVM heap and tuning G1GC

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra heap pressure: sizing the JVM heap and tuning G1GC

Cassandra runs as a single JVM process per node. Every write path allocation, memtable mutation, read merge buffer, and cache entry lives on the heap. When the heap is undersized or GC is left at JVM defaults, stop-the-world pauses freeze gossip, client requests, and compaction. A pause longer than roughly 18 seconds (the default phi accrual threshold is 8) causes peers to mark the node DOWN, which triggers hinted handoff, replay storms, and client retries that worsen memory pressure: the GC death spiral.

Cassandra hint overflow: max_hint_window expiry and silent data divergence

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra hint overflow: max_hint_window expiry and silent data divergence

You restart a node after a four-hour outage. Gossip converges, nodetool status shows UN, and clients reconnect. Reads at consistency level ONE return stale data. The node has never been repaired.

The problem is hint overflow: the outage lasted longer than max_hint_window_in_ms (default three hours), so coordinators stopped saving hints after the window expired. Writes accepted during the final hour of the outage are missing from that replica. Coordinator logs show no errors; write acknowledgments succeeded because other replicas responded. Only anti-entropy repair closes the gap. Without it, the missing data sits on that replica indefinitely, surfacing as inconsistent reads or resurrected deletes.

Cassandra hints accumulating: hinted handoff backlog and replay storms

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra hints accumulating: hinted handoff backlog and replay storms

A coordinator node triggers a disk alert for /var/lib/cassandra/hints, or a replica returning from maintenance flaps UP/DOWN under a write burst it never requested. Hints let writes succeed when a replica is temporarily unreachable, but a large backlog turns that safety net into a secondary failure. Hints consume coordinator disk, expire after max_hint_window_in_ms, and can synchronize into a replay storm that overwhelms a recovering node.

Cassandra hot partition: when one key saturates a replica set

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra hot partition: when one key saturates a replica set

One or two nodes run hot while the rest of the cluster idles. Client P99 latency doubles or triples, but the average looks fine. Timeouts cluster on a subset of hosts, and nodetool status shows uneven load that does not match token ring expectations. When you trace requests, a single partition key consumes a disproportionate share of reads or writes. This is a hot partition: the partitioner mapped one key to a narrow token range, and the replicas owning that range are saturated.

Cassandra java.lang.OutOfMemoryError: Java heap space - causes and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra java.lang.OutOfMemoryError: Java heap space - causes and recovery

Your Cassandra log shows java.lang.OutOfMemoryError: Java heap space. The node stops responding to client requests, gossip marks it DOWN, and the JVM may crash. Because Cassandra runs as a single JVM process, heap exhaustion freezes every in-memory subsystem: memtables, prepared statement caches, bloom filter summaries, and in-flight request buffers.

The heap may climb toward its limit for hours, or a single massive allocation from a large partition read or oversized batch can push it over immediately. Recovery depends on whether the root cause is capacity, data modeling, or a traffic flood.

Cassandra killed by the Linux OOM killer: off-heap memory and RSS

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra killed by the Linux OOM killer: off-heap memory and RSS

The JVM heap chart shows 50% utilization and a flat line. There is no OutOfMemoryError. Then the Cassandra process vanishes. dmesg shows the OOM killer terminated the JVM: Killed process 12345 (java). The JVM heap metric does not include native allocations: bloom filters, compression metadata, index summaries, direct buffers, and chunk cache. When heap plus off-heap RSS exceeds available RAM, the kernel kills the process. This guide covers how to confirm that pattern, reduce off-heap footprint, and prevent recurrence.

Cassandra large partition pathology: Compacting large partition warnings and reads

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra large partition pathology: Compacting large partition warnings and reads

If system.log prints Compacting large partition ks/table:key (N bytes) and reads to the affected table are spiking, a single partition has crossed compaction_large_partition_warning_threshold_mb (default 100 MB). Oversized partitions force the node to deserialize a large in-memory index on reads, rewrite the entire partition during compaction, and move it atomically during streaming or repair. Left alone, the partition grows until it triggers GC pauses, gossip flapping, and cascading retries. Use this guide to confirm the offending key, measure the blast radius, and choose between immediate relief and a data-model fix.

Cassandra lightweight transaction contention: Paxos round-trips and CAS latency

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra lightweight transaction contention: Paxos round-trips and CAS latency

A normal Cassandra write reaches a coordinator, gets appended to the commitlog and memtable on the replicas, and returns. One network round-trip. A lightweight transaction (LWT) using IF NOT EXISTS, IF EXISTS, or IF <condition> triggers Paxos consensus instead: four round-trips of coordination overhead, plus serialization of contending requests against the same partition. If your monitoring only tracks aggregate read and write latency, LWT tail latency is invisible until clients time out.

Cassandra Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra Monitoring

Apache Cassandra, an open-source NoSQL database management system renowned for its scalability and high availability, is pivotal to many organizations running large-scale data infrastructures. Efficiently monitor Cassandra to ensure its optimal performance and reliability.

What Is Cassandra?

Cassandra is a highly scalable, distributed NoSQL database system designed to manage large amounts of structured data across many commodity servers, providing high availability with no single point of failure. It’s specifically used for critical applications needing reliability at large scale. Learn more about Cassandra.

Cassandra monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra monitoring checklist: the signals every production cluster needs

The four maturity levels below are cumulative. Do not instrument level 2 until level 1 is visible and alerted.

Cassandra’s peer-to-peer architecture and LSM storage engine create failure modes generic infrastructure monitoring misses. A node can be UP in gossip and accepting CQL connections while dropping mutations or accumulating compaction debt that only surfaces hours later. These signals catch liveness, performance, saturation, and consistency failures: GC death spirals, compaction avalanches, tombstone storms, and silent data divergence.

Cassandra monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra monitoring maturity model: from survival to expert

Cassandra exposes JMX MBeans, virtual tables, and log signals. Without priority, teams miss compaction debt or drown in noise. This model structures production monitoring into four cumulative levels. Each level adds signals that reduce mean time to detection and catch the failures that dominate Cassandra incidents: data resurrection from missed repair, compaction death spirals, and GC-induced gossip flapping.

Audit your current instrumentation against these levels. See the Cassandra monitoring checklist for a condensed signal inventory.

Cassandra native transport not running: node UP in gossip but refusing CQL clients

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra native transport not running: node UP in gossip but refusing CQL clients

A node reports UN in nodetool status but rejects CQL connections on port 9042. Gossip and replication are healthy; the failure is isolated to the native transport layer.

Because the node remains in the token ring, it continues to handle internode replication, gossip, and streaming. Applications see it as down; the cluster sees it as up. The JMX attribute NativeTransportRunning on org.apache.cassandra.db:type=StorageService is false while gossip heartbeats continue. The usual triggers are nodetool disablebinary left active after maintenance, or a firewall blocking TCP 9042.

Cassandra network partition and split brain: detection and reconciliation

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra network partition and split brain: detection and reconciliation

A partial network partition does not always stop the cluster. Each isolated subset often continues to serve traffic, accept writes, and report itself healthy while marking the other side as DOWN. By the time you notice contradictory gossip views or resurrected data, the two sides have diverged for hours. A healed partition is not self-resolving. If both sides accepted writes, last-write-wins semantics combined with even modest clock skew can silently overwrite valid data. Cross-DC deployments are most vulnerable because WAN latency already strains the phi accrual failure detector and inter-DC gossip paths have more single points of failure.

Cassandra node showing DN in nodetool status: gossip, phi, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra node showing DN in nodetool status: gossip, phi, and recovery

You run nodetool status and one of your nodes shows DN (Down/Normal). Before you restart anything, understand that this output is the local node’s opinion, not global truth. In Cassandra’s peer-to-peer architecture, every node runs its own phi accrual failure detector over gossip heartbeats. A DN mark means this specific observer has not heard from the target for roughly 18 seconds at default settings. Another node in the same cluster may still show the same target as UN.

Cassandra node stuck in joining (UJ): bootstrap diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra node stuck in joining (UJ): bootstrap diagnosis

You add a node to the ring, run nodetool status, and see it stuck in UJ (Up/Joining) for hours. The cluster sees it in gossip, but it never transitions to UN (Up/Normal). Client drivers do not route traffic to it, so the expansion has not added usable capacity. Until the state changes, the node is a ghost member: visible to the ring but unable to serve reads or writes for its assigned token ranges.

Cassandra Not enough space for compaction: STCS space amplification and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra Not enough space for compaction: STCS space amplification and recovery

Not enough space for compaction in system.log means STCS has hit a structural space-amplification limit. Disk usage may already be above 50%. Cassandra aborts the compaction, skips the tier, and leaves tombstones and old versions unmerged. SSTable count rises, read amplification increases, and free space stops being reclaimed. Left unchecked, this enters a compaction death spiral that ends in write rejection or disk exhaustion. Recovery requires immediate free space, targeted cleanup, and a headroom plan that accounts for transient STCS amplification.

Cassandra OperationTimedOutException: client-side timeouts vs server timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra OperationTimedOutException: client-side timeouts vs server timeouts

OperationTimedOutException (driver 3.x) or DriverTimeoutException (driver 4.x) means the driver gave up before the coordinator responded. This is distinct from a server-side ReadTimeout or WriteTimeout, where the coordinator explicitly errors because replicas were too slow. Confusing the two leads to tuning the wrong timeout, masking a server capacity problem or adding unnecessary client latency. Use server-side signals and driver behavior to tell them apart.

Cassandra pending compactions growing: the compaction backlog runbook

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra pending compactions growing: the compaction backlog runbook

Pending tasks climbing in nodetool compactionstats is normal after a bulk load under STCS, but when the number trends upward for hours it signals that your node is producing SSTables faster than compaction can merge them.

This is the leading indicator of the compaction death spiral. Left unchecked, the backlog drives read amplification up, saturates disk I/O, and eventually exhausts disk space as temporary compaction files accumulate. Writes often stay fast while reads degrade, masking the problem.

Cassandra quorum loss: when too many replicas are down to satisfy the CL

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra quorum loss: when too many replicas are down to satisfy the CL

Your application is throwing UnavailableException. Writes and reads fail immediately, not timing out. The coordinator checked the token ring, counted live replicas, and rejected the request because it cannot satisfy the consistency level. No client retry will fix this server-side. The cluster is in quorum loss and will not recover until enough replicas are restored.

When the count of down nodes for a token range exceeds floor(RF/2), every request at QUORUM or stronger fails instantly. Surviving nodes may still serve CL=ONE requests for ranges that have a live replica, but any operation requiring a majority is an outage for those partitions. This is a PAGE-level composite failure pattern that demands immediate node recovery followed by repair.

Cassandra read latency spikes: P99 vs P50 and proxyhistograms

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra read latency spikes: P99 vs P50 and proxyhistograms

Your application is timing out on Cassandra reads, but a quick glance at average latency looks acceptable. This is the percentile trap. In Cassandra, read latency follows a long-tail distribution: most requests are fast, but a fraction hit slow replicas, large partitions, or GC pauses and become orders of magnitude slower. The critical first step is knowing whether the spike lives at the coordinator or the replica, and whether it is systemic or isolated to the tail. nodetool proxyhistograms and nodetool tablehistograms answer exactly that.

Cassandra ReadTimeoutException: diagnosing coordinator read timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra ReadTimeoutException: diagnosing coordinator read timeouts

The driver throws ReadTimeoutException when the coordinator fails to gather enough replica responses within read_request_timeout_in_ms (default 5000 ms). This is a server-side timeout. It maps directly to the JMX metric ClientRequest,scope=Read,name=Timeouts.

This is not OperationTimedOutException, which fires on the driver’s socket timeout. It is also not UnavailableException, which means not enough replicas were alive to attempt the read. Here, replicas are alive but too slow. The read may have executed partially on some replicas, yet the coordinator could not assemble a response that met the consistency level within the window.

Cassandra repair failing or stuck: partial repairs and how to verify completion

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra repair failing or stuck: partial repairs and how to verify completion

nodetool repair returning to the prompt, or hanging at 99 percent, are both dangerous because the worst outcome is silent: a partial repair. A partial repair anti-compacts some token ranges and skips others, leaving inconsistency while looking complete. Because incremental repair marks SSTables as repaired during anti-compaction, a session that fails mid-range leaves later ranges unrepaired while earlier ones are already marked done. No built-in alert fires when only forty percent of ranges were covered. If unrepaired ranges contain tombstones, deleted data resurrects once gc_grace_seconds passes. This guide shows how to verify completion, diagnose stuck sessions, and recover without causing a cascading I/O incident.

Cassandra repair not running: the silent gap that resurrects deleted data

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra repair not running: the silent gap that resurrects deleted data

Deleted rows reappear in application queries, or system_distributed.repair_history shows a last success older than your gc_grace_seconds window. In Cassandra, a repair that is not running, not completing, or not scheduled is an active data integrity risk.

Cassandra uses tombstones to track deletions and TTL expirations. Those tombstones must survive on every replica until anti-entropy repair propagates the delete. Once gc_grace_seconds passes, compaction drops tombstones. If repair has not finished for a table within that window, nodes that missed the original delete retain live data while the rest have discarded the tombstone. The result is zombie data resurrection.

Cassandra repair overload: when anti-entropy repair causes the outage it prevents

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra repair overload: when anti-entropy repair causes the outage it prevents

A full nodetool repair started during peak traffic can spike P99 read latency from milliseconds to hundreds of milliseconds, trigger write timeouts, and cause nodes to flap between UP and DOWN in gossip. The repair job meant to prevent inconsistency becomes the cause of the outage.

Anti-entropy repair is a heavy distributed scan, not a background task. It reads all local data to build Merkle trees, exchanges hashes with replicas, and streams differing ranges. On a multi-terabyte node this means terabytes of sequential disk reads, heavy CPU hashing, and gigabits of network traffic. Without dedicated headroom, repair competes with the commitlog, memtable flushes, compaction, and client requests for disk bandwidth, CPU, and network. The result is thread pool backpressure, dropped messages, GC pressure from Merkle tree construction, and eventually gossip failure as the node becomes unresponsive.

Cassandra Scanned over N tombstones warning: finding the offending query

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra Scanned over N tombstones warning: finding the offending query

Your application latency may still look acceptable at the median, but system.log is filling with warnings like Read 500 live rows and 12000 tombstone cells for query SELECT ... or Scanned over 5000 tombstones. These messages mean a single read is sifting through thousands of delete markers to return a small result set. Each tombstone consumes CPU, disk I/O, and heap memory during the merge phase. Left alone, the same query will eventually cross tombstone_failure_threshold (default 100000) and be aborted by the coordinator. The log line gives you the query text, but the real operational work is deciding whether the root cause is a missing repair, a compaction backlog, or a data model mismatch, and then confirming which partition is the actual source of the dead data.

Cassandra schema disagreement: nodetool describecluster shows multiple versions

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra schema disagreement: nodetool describecluster shows multiple versions

nodetool describecluster should report exactly one schema version UUID cluster-wide. When the Schema versions section lists more than one UUID, the cluster is in schema disagreement. DDL (CREATE, ALTER, DROP) fails or hangs until every node converges. DML and reads against existing tables are unaffected.

Transient disagreement lasting less than five minutes during or immediately after a DDL change is normal. Cassandra propagates schema mutations asynchronously via gossip. If multiple versions persist beyond five minutes, you have a stuck node, a partitioned peer, or a migration stage backlog.

Cassandra secondary index pitfalls: when 2i scatters across the ring

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra secondary index pitfalls: when 2i scatters across the ring

A query on an indexed column behaves differently in Cassandra than in a relational database. Without a partition key to anchor the request, the coordinator cannot hash the value to a token range and choose the right replicas. It fans out to every node in the cluster, waits for each local index scan to complete, and collates the results. Adding nodes can make indexed queries slower because scatter is linear with cluster size.

Cassandra snapshots silently consuming disk: hard links and clearsnapshot

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra snapshots silently consuming disk: hard links and clearsnapshot

df shows usage climbing toward 90%, but nodetool info Load does not explain the gap. If commitlog and hints are normal and compaction backlog is small, the missing space is likely a snapshot taken days or weeks ago. Cassandra snapshots use hard links, so they allocate no additional blocks at creation. After compaction deletes live SSTables, the snapshot links become the sole owners of old blocks and silently consume gigabytes.

Cassandra streaming failures: stalled bootstrap, decommission, and rebuild

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra streaming failures: stalled bootstrap, decommission, and rebuild

Bootstrapping stuck in JOINING for six hours. Decommission streaming with zero byte progress after eight hours. Rebuild failed, leaving incomplete token ranges. These are streaming failures.

Streaming moves SSTables between nodes during topology changes and repair. When a session fails, the topology is left incomplete. When it stalls with no progress for more than thirty minutes, the node stays in a transitional state and may not handle traffic correctly. Root causes are usually network timeouts, source node bottlenecks, or configuration mismatches. Unlike client request timeouts, streaming failures do not always surface as explicit errors; a session can hang silently while the control channel stays open.

Cassandra thread pool pending and blocked tasks: SEDA backpressure

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra thread pool pending and blocked tasks: SEDA backpressure

You run nodetool tpstats and see non-zero values in the Pending or Blocked columns. On a healthy Cassandra node, request-stage pools like MUTATION and READ should show zero pending tasks in steady state. When pending climbs and stays above zero, the SEDA pipeline is backing up. If Blocked also rises, the node has moved from queuing to rejecting work.

This is not a transient spike you can ignore. Sustained pending tasks on the MUTATION or READ stages add latency to every client request. Blocked tasks mean the queue is full and the node is actively shedding load. In the GOSSIP stage, even a small pending backlog is an emergency: it means gossip is falling behind, which leads to false DOWN marking across the cluster.

Cassandra tombstone storm: delete-heavy tables and read latency collapse

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra tombstone storm: delete-heavy tables and read latency collapse

DELETEs and TTL expirations write tombstones. Read latency bifurcates: P50 stays flat while P99 spikes. Logs show queries scanning thousands of tombstones; eventually clients see queries abort after crossing tombstone_failure_threshold. Disk space does not shrink despite deletions. This is a tombstone storm.

Cassandra does not remove deleted data immediately. A DELETE inserts a tombstone marker that persists until compaction purges it after gc_grace_seconds elapses and all replicas have been repaired. When tombstones scatter across many SSTables, every read must scan and merge them, generating temporary heap objects and escalating GC pressure. Only a subset of partitions may be affected, which is why P50 stays flat while tail latency explodes.

Cassandra TombstoneOverwhelmingException: reads aborted by tombstone_failure_threshold

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra TombstoneOverwhelmingException: reads aborted by tombstone_failure_threshold

TombstoneOverwhelmingException is a hard stop. Cassandra aborts the read after scanning more tombstones than tombstone_failure_threshold allows. The default is 100000 tombstones per query. When this exception hits a production table, client reads fail outright.

Without the threshold, a tombstone-heavy read pins cores, saturates disk I/O, and triggers GC pauses that cascade into gossip flapping or OOM. The exception is a circuit breaker. It is also a signal that tombstones are accumulating faster than compaction can purge them, or that the data model is generating them faster than the storage engine can remove them.

Cassandra Too many open files: file descriptor exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra Too many open files: file descriptor exhaustion

Your application starts logging connection timeouts or Unable to connect errors, yet nodetool status still marks the node as UP. Inside the Cassandra system log, you see java.io.IOException: Too many open files or compaction tasks failing with Cannot open enough files. This is file descriptor exhaustion. The node operates normally until it hits the process ulimit, then it cannot open new SSTables, accept client sockets, or maintain internode connections. The result is a node that looks alive in gossip but is functionally unable to serve reads, writes, or background compaction.

Cassandra too many SSTables per table: read amplification and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra too many SSTables per table: read amplification and how to fix it

Your Cassandra read P99 latency is climbing. Queries that used to take milliseconds now time out. The application reports intermittent ReadTimeoutException. You check nodetool tablestats and see one table sitting at 200 SSTables. For LCS, that is a catastrophe. For STCS, anything sustained above 50 means compaction has fallen behind.

Each extra SSTable adds a bloom filter check and a potential disk seek to every read. The read path must merge fragments from memtables plus every SSTable that might contain the partition. When SSTable counts balloon, the node spends more time checking filters and seeking than returning data. This is read amplification.

Cassandra TTL tombstone accumulation: time-series tables and TWCS

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra TTL tombstone accumulation: time-series tables and TWCS

You set a TTL on every row expecting old data to vanish. Instead, disk usage climbs, read latency spikes, and your logs fill with tombstone warnings.

An expired TTL does not delete data immediately. Cassandra writes a tombstone that survives until compaction runs and the tombstone exceeds gc_grace_seconds (default 864000 seconds, or 10 days). In time-series workloads ingesting millions of TTL’d rows per hour, the wrong compaction strategy turns this bookkeeping into a production incident.

Cassandra UnavailableException: not enough replicas for the consistency level

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra UnavailableException: not enough replicas for the consistency level

Cassandra threw UnavailableException. The coordinator rejected the request immediately without contacting replicas. No mutation occurred, and retrying at the same consistency level against the same coordinator fails until enough replicas are UN in the coordinator’s gossip view. This is a topology problem, not a performance problem.

This exception is qualitatively different from TimeoutException. A timeout means enough replicas were alive but responded too slowly. An unavailable means the coordinator never sent the request because the topology could not satisfy the consistency level.

Cassandra WriteTimeoutException: coordinator write timeouts and writeType

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra WriteTimeoutException: coordinator write timeouts and writeType

A WriteTimeoutException means the coordinator did not receive enough replica acknowledgments before write_request_timeout_in_ms expired. It does not mean the write failed; one or more replicas may have already persisted the mutation, so the outcome is ambiguous. The writeType field determines whether a client-side retry is safe.

Distinguish between a slow replica and an unavailable one. Understand the idempotency contract for the write type, and correlate coordinator timeouts with replica-side saturation. The default write_request_timeout_in_ms is 2000 ms. If replicas cannot append to the commitlog and update the memtable within that window, the coordinator throws this exception.

Cassandra zombie data resurrection: gc_grace_seconds and unrepaired tombstones

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra zombie data resurrection: gc_grace_seconds and unrepaired tombstones

A query returns rows that were deleted weeks ago. Application logs show no errors. All nodes report UP in nodetool status. Compaction is running and disk usage looks normal. The data is back because a tombstone was compacted away before anti-entropy repair verified that every replica saw the delete. Once the tombstone is gone, live data on unrepaired replicas is treated as authoritative. The next repair or read repair streams it back to the tombstone-less nodes.

Castle Rock Computing SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cato Networks

Mon, 01 Jan 0001 00:00:00 +0000

Cato Networks Topology

Mon, 01 Jan 0001 00:00:00 +0000

CDP Topology

Mon, 01 Jan 0001 00:00:00 +0000

Ce T SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Celery

Mon, 01 Jan 0001 00:00:00 +0000

Celery Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Celery Monitoring

What Is Celery?

Celery is an open-source distributed task queue system designed to handle asynchronous tasks effectively. It’s built on a robust, flexible architecture and is commonly used in web development to manage scheduled tasks and ease resource consumption. It functions using a producer/consumer pattern, with workers executing jobs and a message broker dispatching them. Celery is praised for its simplicity, scalability, and language-agnostic nature, making it a favorite among developers seeking reliable task management and distributed computing frameworks.

Centec Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Centillion Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

CentOS

Mon, 01 Jan 0001 00:00:00 +0000

CentOS Stream

Mon, 01 Jan 0001 00:00:00 +0000

Centrum Communications Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ceph

Mon, 01 Jan 0001 00:00:00 +0000

Ceph backfill_toofull: recovery blocked because target OSDs are full

Mon, 01 Jan 0001 00:00:00 +0000

Ceph backfill_toofull: recovery blocked because target OSDs are full

A PG in backfill_toofull is not making progress toward active+clean. The source OSD has the data, CRUSH has chosen the target, but the target refuses the reservation because it is above backfillfull_ratio. Client reads and writes still succeed; the redundancy gap is not closing.

The cluster is usually not at the hard full ratio (0.95 default). It is at backfillfull (0.90 default) on one or more individual OSDs, and that is enough to halt recovery for every PG whose up set includes them. Symptoms: HEALTH_WARN (or HEALTH_ERR on older releases), a flat degraded-object count, and recovery bytes/sec near zero.

Ceph blocked ops: client I/O stuck behind a single slow OSD

Mon, 01 Jan 0001 00:00:00 +0000

Ceph blocked ops: client I/O stuck behind a single slow OSD

A client writes an object. The primary OSD forwards sub-ops to its replicas, then waits for every replica to ack before it acks the client. If any one OSD in the acting set stalls, the whole op stalls. After osd_op_complaint_time (default 30 seconds) the OSD logs a slow op and the monitor raises SLOW_OPS. Past that boundary the op is effectively blocked, not merely slow. Clients time out and retry, which puts more ops in flight against the same stuck OSD.

Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device

Mon, 01 Jan 0001 00:00:00 +0000

Ceph BLUEFS_SPILLOVER: RocksDB metadata spilling onto the slow device

A small subset of OSDs shows periodic commit-latency spikes and slow ops while the rest of the cluster looks healthy. Capacity metrics are normal. SMART is clean. ceph -s reports HEALTH_WARN, and ceph health detail returns something like:

BLUEFS_SPILLOVER
 3 OSDs spilled over ~18 GiB metadata from 'db' device
 (e.g. osd.12 spilled over 6.1 GiB metadata from 'db' device)

BlueStore’s RocksDB metadata has outgrown its dedicated fast DB partition (SSD/NVMe) and is spilling onto the slow HDD data partition. Compaction that took milliseconds on flash now takes seconds on spinning disk. Between compaction cycles the OSD looks fine; during compaction it stalls. This is a cliff edge, not gradual degradation: the moment slow_used_bytes goes nonzero, latency steps up by one to two orders of magnitude on the affected OSDs.

Ceph BlueStore allocator fragmentation: rising latency at moderate fullness

Mon, 01 Jan 0001 00:00:00 +0000

Ceph BlueStore allocator fragmentation: rising latency at moderate fullness

Write latency climbs on OSDs that are only moderately full. Capacity dashboards look healthy, the cluster sits well below nearfull, and SMART is clean. But a subset of OSDs shows rising commit and apply latency, slow ops begin accumulating, and the affected OSDs are the long-lived ones carrying heavy overwrite workloads. This is BlueStore allocator fragmentation: the free-space structure BlueStore walks on every allocation has become shredded into small, non-contiguous free extents, so each new write needs a longer search before a suitable run of blocks is found.

Ceph BlueStore RocksDB compaction stalls: periodic latency spikes

Mon, 01 Jan 0001 00:00:00 +0000

Ceph BlueStore RocksDB compaction stalls: periodic latency spikes

Periodic latency spikes on Ceph OSDs that appear and clear on their own schedule are a signature of BlueStore RocksDB compaction. The cluster reports brief bursts of slow ops on a small set of OSDs, commit latency climbs for seconds to minutes, then returns to baseline without intervention. Health checks may briefly show SLOW_OPS before clearing.

Ceph capacity death spiral: an OSD fails and recovery has nowhere to go

Mon, 01 Jan 0001 00:00:00 +0000

Ceph capacity death spiral: an OSD fails and recovery has nowhere to go

An OSD fails on a cluster you have been running at 80% or higher. ceph -s shows recovery starting, then ceph health detail lists OSD_NEARFULL and BACKFILL_TOOFULL. The degraded PG count climbs, then flatlines. Recovery bytes per second sits near zero. Nothing is healing, and you are one more failure away from data loss.

This is the Ceph capacity death spiral. It is not a single fault: it is the intersection of tight capacity, CRUSH imbalance, and the hard thresholds Ceph uses to protect itself. Recovery needs spare space on the surviving OSDs. When those OSDs are already past the backfillfull ratio (default 0.90), Ceph refuses to push more data onto them, and backfill stalls for every PG that would target them.

Ceph CephFS client eviction (EBLACKLISTED): blocklisted clients and failed I/O

Mon, 01 Jan 0001 00:00:00 +0000

Ceph CephFS client eviction (EBLACKLISTED): blocklisted clients and failed I/O

CephFS clients suddenly see EBLACKLISTED errors on open file handles. Reads and writes that worked seconds ago fail; new operations against the same mount return the same error code. The MDS has forcibly evicted the client and added its address to the cluster-wide OSD blocklist. Every OSD now refuses traffic from that client address, and the mount is effectively dead until the blocklist is cleared or expires.

Ceph client latency vs OSD latency: fast disks, slow clients

Mon, 01 Jan 0001 00:00:00 +0000

Ceph client latency vs OSD latency: fast disks, slow clients

You ran ceph osd perf, the commit_latency and apply_latency columns all look healthy, and you declared the cluster fast. Then a client team opens a ticket: writes are taking 50ms, 100ms, sometimes timing out. You re-check ceph osd perf. Still fast. The OSDs are not the bottleneck, but the clients are still slow.

ceph osd perf, the ceph_osd_*_latency_ms Prometheus metrics, and rbd perf image iostat all measure time inside the OSD data path. They do not measure the round trip the client experiences. Client-visible latency is the sum of OSD processing, public network RTT, replication round-trips to peer OSDs, CRUSH map recalculation during flapping, and queuing delay on the OSD’s public-facing messenger. A cluster can show 2ms commit latency on every OSD while clients see 50ms response times because the public network is saturated, or because public and cluster traffic share one NIC.

Ceph deep scrub performance impact: I/O saturation that mimics an incident

Mon, 01 Jan 0001 00:00:00 +0000

Ceph deep scrub performance impact: I/O saturation that mimics an incident

A Ceph cluster suddenly shows elevated apply latency across many OSDs. Slow ops tick up. Client write latency degrades. The dashboard looks like the start of a real incident. But cluster health is HEALTH_OK and all PGs are active+clean. Before you start chasing a failing disk or a network partition, check whether a deep scrub is running.

Deep scrub reads and checksums every byte of every object in a PG. On HDD-backed OSDs, this is a sequential read pass across the entire device. With default scheduling, it happens to every PG roughly once a week. If the schedule concentrates scrubs onto the same OSDs at the same time, or if your scrub window is narrow, deep scrub will saturate disk I/O and produce symptoms that look identical to a real performance incident.

Ceph degraded objects: reduced redundancy and the race against a second failure

Mon, 01 Jan 0001 00:00:00 +0000

Ceph degraded objects: reduced redundancy and the race against a second failure

Degraded objects in Ceph are the most direct measure of reduced redundancy: somewhere in the cluster, at least one object has fewer live replicas (or fewer erasure-coded fragments) than its pool requires. The cluster is still serving I/O, but the protection against a follow-on failure has thinned for those objects.

ceph_num_objects_degraded is not alarming on its own. Every OSD failure, restart, and CRUSH reweight produces a transient spike that recovery is designed to heal. The real signal is the slope: is the count falling back toward zero, or has it plateaued while the cluster is still degraded?

Ceph failed_repair: when ceph pg repair cannot fix the inconsistency

Mon, 01 Jan 0001 00:00:00 +0000

Ceph failed_repair: when ceph pg repair cannot fix the inconsistency

A failed_repair PG state means Ceph attempted to repair a data inconsistency that scrub or deep-scrub detected, and could not. The PG is now active+clean+inconsistent+failed_repair, the cluster is at HEALTH_WARN (typically via the OSD_SCRUB_ERRORS health check), and the inconsistency will not heal on its own. Reads still succeed from consistent replicas, but the cluster has stopped trying to fix the divergence until you intervene.

Ceph FS_DEGRADED: standby MDS failed to take over a rank

Mon, 01 Jan 0001 00:00:00 +0000

Ceph FS_DEGRADED: standby MDS failed to take over a rank

FS_DEGRADED fires when at least one CephFS rank is failed or damaged and a standby did not promote. Clients can usually still reach the filesystem through surviving ranks, but you are running without the failover reserve the MDS cluster was sized to provide.

FS_DEGRADED is the precursor to MDS_ALL_DOWN. If the last active rank fails before you restore a healthy standby, CephFS becomes fully unavailable. The playbook classifies FS_DEGRADED active for more than 120 seconds as a TICKET. If CephFS is a primary storage interface, treat 120 seconds as the upper bound on response time, not a soft target.

Ceph health detail: mapping ceph_health_detail checks to a cause

Mon, 01 Jan 0001 00:00:00 +0000

Ceph health detail: mapping ceph_health_detail checks to a cause

ceph_health_detail is the bridge between the umbrella status an alert fires on (ceph_health_status: 0=HEALTH_OK, 1=HEALTH_WARN, 2=HEALTH_ERR) and the specific fault you have to fix. Each health check is a separate gauge with name, severity, and message labels; value 1 means active, 0 means inactive. The name label is the check code (OSD_FULL, PG_AVAILABILITY, MON_CLOCK_SKEW, etc.) and is what you build alert routing, dashboards, and post-incident timelines around.

Ceph HEALTH_ERR: reading the umbrella status and finding the real fault

Mon, 01 Jan 0001 00:00:00 +0000

Ceph HEALTH_ERR: reading the umbrella status and finding the real fault

ceph_health_status == 2 is the most severe top-level status Ceph reports, and one of the most over-paged signals in production. The reason is structural: HEALTH_ERR is an umbrella aggregation. It tells you that one or more child health checks crossed into ERR severity. It does not tell you which subsystem failed, whether I/O is actually blocked, or whether the condition will self-resolve when PGs finish peering.

Ceph HEALTH_WARN: which warnings are noise and which are structural

Mon, 01 Jan 0001 00:00:00 +0000

Ceph HEALTH_WARN: which warnings are noise and which are structural

HEALTH_WARN is an umbrella status, not a single condition. It fires during normal recovery after an OSD restart, and it also fires when the cluster is one step from data loss. Both look identical if you only watch the top-level status. The common mistake is blanket-silencing WARN because it fires too often during recovery, then missing the structural warnings that signal real danger.

Ceph LARGE_OMAP_OBJECTS: the RGW bucket-index OMAP storm and resharding

Mon, 01 Jan 0001 00:00:00 +0000

Ceph LARGE_OMAP_OBJECTS: the RGW bucket-index OMAP storm and resharding

The LARGE_OMAP_OBJECTS health warning fires when a deep scrub finds a RADOS object carrying more OMAP keys than the configured threshold. On clusters running the RADOS Gateway (RGW), the most common trigger is the bucket-index shard: a single shard that has accumulated too many entries because a bucket holds millions of objects with too few index shards.

Once a shard crosses the threshold, the symptoms cluster around two areas. RGW clients see slow LIST responses and slow multipart coordination on the affected bucket. The OSD hosting the shard shows elevated commit latency, slow ops, and in the worst case BlueStore RocksDB pressure that resembles a compaction stall. The OSD usually remains up+in, so cluster-level health stays at WARN rather than ERR, which makes the problem easy to overlook until a user complains about a stuck ListObjects call.

Ceph MDS_ALL_DOWN: CephFS is completely unavailable

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MDS_ALL_DOWN: CephFS is completely unavailable

MDS_ALL_DOWN means no active Metadata Server (MDS) rank exists for a CephFS filesystem. Every CephFS client depending on that filesystem blocks on metadata operations. This is a hard outage for CephFS, not a degradation. The health check ceph_health_detail{name="MDS_ALL_DOWN"} goes active the moment the Monitor has no active rank to assign for that filesystem.

The metadata itself is journaled in a RADOS pool. Unless MDS_DAMAGED is also firing, the metadata is intact. The outage is availability, not loss. Recovery means restoring at least one active MDS, after which clients reconnect and resume.

Ceph MDS_CACHE_OVERSIZED: cache above limit and the cap recall that follows

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MDS_CACHE_OVERSIZED: cache above limit and the cap recall that follows

MDS_CACHE_OVERSIZED fires when an active Metadata Server daemon’s in-memory inode and capability cache crosses mds_cache_memory_limit * mds_health_cache_threshold. It is almost always workload-driven: a client has touched enough files in a short enough window that the MDS is caching more metadata than its budget allows.

Ceph MDS_DAMAGED: metadata journal or cache corruption

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MDS_DAMAGED: metadata journal or cache corruption

MDS_DAMAGED means the CephFS Metadata Server has found damaged metadata, either in its journal or in on-disk structures read from the metadata pool, and has deliberately refused to continue serving the rank. CephFS may be partially or fully unavailable, and recovery is not the usual automatic failover path. A standby that takes over would replay the same suspect journal, so the cluster parks the rank until a human intervenes.

Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MON_CLOCK_SKEW: clock drift between monitors and election churn

The MON_CLOCK_SKEW health check fires when the leader monitor detects clock drift beyond mon_clock_drift_allowed (default 0.05 seconds, 50 milliseconds) on any monitor in the quorum. It raises HEALTH_WARN, not HEALTH_ERR, and on its own it does not stop client I/O. But it is the precursor to a class of failure that does: repeated Paxos elections, slow map distribution, and, if the skew grows, monitor quorum loss.

Ceph MON_DOWN: a monitor out of quorum and reduced redundancy

Mon, 01 Jan 0001 00:00:00 +0000

Ceph MON_DOWN: a monitor out of quorum and reduced redundancy

MON_DOWN fires when one or more monitor daemons are not part of the active quorum. The common case is a 3-monitor cluster with one monitor gone: quorum still holds with 2 of 3, but the cluster has lost redundancy and tolerates zero further monitor loss before consensus collapses. The Prometheus Ceph mixin surfaces this as CephMonDown (warning) and escalates to CephMonDownQuorumAtRisk (critical) when the number of down monitors equals the minimum quorum count.

Ceph monitor election storm: monitors that cannot hold a stable quorum

Mon, 01 Jan 0001 00:00:00 +0000

Ceph monitor election storm: monitors that cannot hold a stable quorum

A Ceph monitor election storm is what happens when the MON cluster cannot complete and hold an election. Each round of Paxos leader election starts, partially completes, then restarts before the new leader can commit any map updates. The election epoch counter climbs, the leader name changes moment to moment, and ceph -s itself starts taking several seconds to return because even reading the current map requires a responsive leader.

Ceph monitor quorum lost: the cluster can no longer update its maps

Mon, 01 Jan 0001 00:00:00 +0000

Ceph monitor quorum lost: the cluster can no longer update its maps

Ceph monitor quorum loss is a PAGE condition. Without a majority of MONs agreeing through Paxos, the cluster cannot commit any map update: no OSD up/down transitions, no PG state changes, no pool edits, no CRUSH adjustments. Existing clients keep running on cached maps for a while, but new client connections fail immediately, and as soon as those cached maps go stale, in-flight I/O stalls.

Ceph Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Ceph Monitoring

What Is Ceph?

Ceph is a distributed storage system designed to provide excellent performance, reliability, and scalability. It achieves this by distributing data across multiple storage devices and ensuring data redundancy. As an open-source project, Ceph has become a popular choice for managing petabytes of data because of its self-healing and self-managing capabilities.

Monitoring Ceph With Netdata

Monitoring Ceph clusters effectively is crucial to maintaining their optimal performance and ensuring data availability. Netdata offers a comprehensive Ceph monitoring tool that provides real-time insights into your Ceph infrastructure. By using Netdata’s collector for Ceph, you can access granular metrics for your entire Ceph cluster, including individual Pools and OSDs.

Ceph monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Ceph monitoring checklist: the signals every production cluster needs

This checklist defines the signals a production Ceph cluster needs, organized by monitoring maturity. Each level is a superset of the previous one; higher-level signals rely on lower-level context to distinguish real failures from normal churn. The four levels are survival, operational, mature, and expert. You cannot skip levels in practice.

Two caveats apply across all levels. First, ceph_health_status is an umbrella, not a complete picture: HEALTH_OK does not guarantee performance, and HEALTH_WARN covers both expected churn (active recovery) and structural problems (nearfull, noout trap, scrub inconsistency). Drill into ceph_health_detail before treating WARN as noise. Second, per-OSD and per-pool metrics matter more than cluster averages. A cluster at 60% average capacity with one OSD at 85% is closer to trouble than the average suggests.

Ceph monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Ceph monitoring maturity model: from survival to expert

Ceph’s health surface is unusually broad for a storage system. A cluster can report HEALTH_OK while a single OSD runs at 20x the latency of its peers, while the BlueStore DB partition spills to slow media, or while the PG autoscaler splits placement groups under live client load. The gap between “is the cluster up” and “is the cluster observable enough to operate” is where most preventable outages live.

Ceph nearfull: the 85% warning that decides whether the cluster can heal

Mon, 01 Jan 0001 00:00:00 +0000

Ceph nearfull: the 85% warning that decides whether the cluster can heal

OSD_NEARFULL or POOL_NEAR_FULL in ceph health detail means HEALTH_WARN. Clients are still reading and writing, and nothing looks broken yet. The nearfull ratio (default 0.85) is not a polite reminder to plan storage. It is the point where Ceph warns it is running out of the spare space it needs to heal itself.

One OSD failure at 85% can push surviving OSDs past backfillfull (0.90), at which point backfills refuse to start and recovery stalls. If another OSD fails in that window, you are left with degraded PGs that cannot be recovered, and the next stop is OSD_FULL at 95% where all client writes return ENOSPC.

Ceph noout flag left set: the most common preventable outage

Mon, 01 Jan 0001 00:00:00 +0000

Ceph noout flag left set: the most common preventable outage

The noout flag is the single most common source of preventable Ceph outages. An operator sets it before maintenance to stop Ceph from marking down OSDs as out, then forgets to unset it. The cluster keeps serving I/O, health stays at HEALTH_WARN (not ERR), and nothing visible breaks. But recovery is now disabled by policy. The next OSD failure stacks on top of the first, and degraded PGs that should have healed days ago are still degraded.

Ceph norecover and nobackfill: recovery intentionally, or accidentally, stopped

Mon, 01 Jan 0001 00:00:00 +0000

Ceph norecover and nobackfill: recovery intentionally, or accidentally, stopped

The symptom is familiar: PGs are stuck in degraded or undersized states, the cluster is HEALTH_WARN, but recovery throughput is zero. OSD load, network, and capacity all look normal. The cause is often two cluster-wide flags sitting in the OSD map: norecover and nobackfill. These flags are legitimate tools for protecting client I/O during recovery storms or maintenance, and a common source of “forgotten flag” incidents alongside noout.

Ceph OSD commit and apply latency: reading per-OSD latency outliers

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD commit and apply latency: reading per-OSD latency outliers

Two metrics tell you more about per-OSD health than almost anything else in Ceph: ceph_osd_commit_latency_ms and ceph_osd_apply_latency_ms. They are per-OSD gauges exported by the manager’s Prometheus module, labeled by ceph_daemon, and they measure the OSD’s internal I/O time, not the latency your clients experience end-to-end. A cluster can show healthy aggregate latency while one OSD quietly runs 5x slower than its peers of the same device class. The clients whose objects land on that OSD see tail latency spikes; everyone else sees normal performance. Cluster-wide averages hide this. Per-OSD comparison surfaces it.

Ceph OSD down: telling a dead disk apart from a network blip

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD down: telling a dead disk apart from a network blip

OSD_DOWN fires. ceph_osd_up flipped to 0. The 600-second countdown to OUT and recovery has begun.

Your first job: figure out whether this is a dead disk (act now, plan a replacement) or a network blip (wait, verify, let the OSD come back). Acting on the wrong diagnosis wastes disk and network I/O on a needless backfill, or worse, leaves a failing disk in service past the point where SMART was already warning you.

Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD flapping: OSDs cycling up and down and the peering storm that follows

OSD flapping is a failure cascade. One OSD misses heartbeats, peers mark it down, its PGs start peering and recovering elsewhere, the OSD comes back, peering reverses, and the cycle repeats. Each flap mints a new OSD map epoch that every OSD in the cluster must process. The peering overhead from a single flapping OSD can slow dozens of healthy OSDs enough that they also miss heartbeats, and the cascade spreads.

Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD fullness imbalance: one OSD full while the cluster average looks fine

Cluster writes are blocked. ceph status shows HEALTH_ERR with OSD_FULL active. But ceph df reports the cluster at 65% utilized. Both are correct.

The Ceph full ratio (default 0.95) is enforced per-OSD, not cluster-wide. When any single OSD crosses that threshold, Ceph refuses writes for every PG that OSD serves. Because CRUSH distributes PGs across OSDs, one full OSD can block writes to a large fraction of PGs even when ninety-nine other OSDs have ample free space.

Ceph OSD heartbeat timeouts: the osd_heartbeat_grace precursor to flapping

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD heartbeat timeouts: the osd_heartbeat_grace precursor to flapping

Ceph OSDs declare each other down based on a simple rule: if a peer does not respond to heartbeat pings within osd_heartbeat_grace (default 20 seconds), it is reported to the monitors as unresponsive. Two peers from different failure domains must agree before the monitor marks the OSD down. This is the mechanism that turns a transient stall into an OSD state change, and repeated near-misses against this 20-second window are the most reliable leading indicator of OSD flapping.

Ceph OSD up/down vs in/out: the four states and mon_osd_down_out_interval

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD up/down vs in/out: the four states and mon_osd_down_out_interval

Most Ceph operations references describe an OSD as “down” as if it were a single state. It is not. An OSD carries two independent flags: up/down (is the daemon alive?) and in/out (does CRUSH place data on it?). Combined, that produces four states, and the most dangerous one, down+in, is invisible if you only look at one flag.

The two flags are linked by a clock. After an OSD goes down, the monitors wait mon_osd_down_out_interval (default 600 seconds) before automatically flipping it to out. That 10-minute window is the gap between “OSD stopped” and “recovery starts.” That timer, and the flags on either side of it, are the vocabulary every recovery, flapping, and noout runbook assumes.

Ceph OSD_FULL: all writes stopped at the 95% full ratio

Mon, 01 Jan 0001 00:00:00 +0000

Ceph OSD_FULL: all writes stopped at the 95% full ratio

The cluster suddenly stops accepting writes. Client applications report ENOSPC errors. ceph status returns HEALTH_ERR. ceph health detail shows OSD_FULL active. Reads still succeed, but every write, update, and delete fails cluster-wide.

This is a hard stop, not a throttle. Ceph refuses all write operations once any OSD crosses the configured full_ratio (default 0.95). CRUSH spreads every PG across multiple OSDs, so a single full OSD can block writes to hundreds of PGs even when cluster-average utilization looks moderate.

Ceph osd_memory_target: cache eviction, RSS growth, and OSD OOM kills

Mon, 01 Jan 0001 00:00:00 +0000

Ceph osd_memory_target: cache eviction, RSS growth, and OSD OOM kills

An OSD repeatedly killed by the kernel OOM killer, or a host where OSDs flap up and down shortly after systemd restarts them, very often traces back to one knob: osd_memory_target. The same knob set too low for the working set produces a quieter failure: read latency climbs on HDD-backed OSDs as BlueStore evicts onodes and RocksDB block-cache pages the workload actually needs.

Ceph PG degraded: fewer replicas than the pool size, and when it matters

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG degraded: fewer replicas than the pool size, and when it matters

A degraded placement group in Ceph has fewer copies of some objects than the pool’s configured size. When an OSD goes down or is removed, every PG that had a replica on that OSD drops below target and enters the degraded state. The cluster can still serve I/O as long as the surviving replica count is at or above min_size, but redundancy is reduced until recovery rebuilds the missing copies.

Ceph PG down: no surviving replica for reads or writes

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG down: no surviving replica for reads or writes

A placement group in the down state has no replica that can serve I/O. Reads and writes to objects in that PG fail. Clients see EIO or ENODEV, and ceph health detail reports PG_AVAILABILITY with one or more PGs flagged down. Data is unavailable, not just degraded.

A PG can briefly pass through down during peering after an OSD failure, before activating on a surviving replica. The 300-second sustain on sum(ceph_pg_down) > 0 filters that transient window. Once a PG has been down past the sustain, no OSD in the acting set can serve it, and the cluster will not heal it without operator action.

Ceph PG incomplete: placement groups that cannot serve I/O

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG incomplete: placement groups that cannot serve I/O

An incomplete placement group means the PG cannot find enough authoritative data to serve reads or writes. This is a data-availability failure, and it does not self-resolve the way a transient peering or recovering state does.

Alert on sum(ceph_pg_incomplete) > 0 sustained for more than 300 seconds, summed across all pools. The 300 second sustain filters out cold-start peering after a cluster-wide restart, which typically completes within 60-120 seconds. Anything still incomplete after five minutes is genuinely stuck.

Ceph PG inconsistent (OSD_SCRUB_ERRORS): scrub found replica divergence

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG inconsistent (OSD_SCRUB_ERRORS): scrub found replica divergence

A scrub or deep-scrub finished comparing replicas for a placement group and found they disagree. The cluster surfaces this as OSD_SCRUB_ERRORS (often paired with PG_DAMAGED) and the affected PG sits in active+clean+inconsistent. Client reads still succeed because Ceph serves them from a consistent replica, but at least one copy in the acting set is corrupt.

The danger is not the symptom. Reads work, and Ceph did what it was designed to do: detect silent divergence. The danger is that the corruption was found, not fixed, and the window during which an uncorrupted replica survives is your margin of safety. If the OSD holding the good copy fails before you repair, the object becomes unreadable or unwritable.

Ceph PG stale: the monitor has not heard from the PG's OSDs

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG stale: the monitor has not heard from the PG’s OSDs

A stale placement group means the monitor cluster has stopped trusting the last status it received for that PG. The acting primary OSD that should be reporting state has gone silent, so the MON cannot confirm whether the PG is active, recovering, or unavailable. The recorded state is preserved but flagged as untrusted.

This differs from down. A down PG means the cluster has positively confirmed that no replica can serve I/O. A stale PG means the cluster does not know the current state because the reporting chain broke. Both can co-occur, but the response differs: down is confirmed replica loss; stale is a communication or reporting failure.

Ceph PG undersized: fewer copies than the pool wants to place

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG undersized: fewer copies than the pool wants to place

A Ceph cluster reports ceph health detail listing PGs in an undersized state. The cluster serves I/O at reduced redundancy. The warning does not resolve on its own and recovery stalls. A structural block prevents CRUSH from placing the configured number of replicas.

Unlike peering or recovering, undersized is not transient. The acting set has fewer OSDs than the pool’s configured size (the replication factor for replicated pools, or the k+m sum for erasure-coded pools). Ceph runs the PG at reduced redundancy because CRUSH cannot find enough distinct failure domains to satisfy the rule. The fix is rarely to wait. It requires changing capacity, topology, or the CRUSH rule.

Ceph PG_NOT_DEEP_SCRUBBED: scrub verification debt and undetected bit rot

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PG_NOT_DEEP_SCRUBBED: scrub verification debt and undetected bit rot

PG_NOT_DEEP_SCRUBBED means Ceph’s proactive integrity check is falling behind. Deep scrub is the only mechanism that reads every byte of every object on every replica and verifies byte-for-byte consistency. When PGs miss their deep-scrub window repeatedly, silent corruption from bit rot, DRAM errors, firmware bugs, and incomplete writes after power loss accumulates without any signal.

The companion check PG_NOT_SCRUBBED covers the lighter daily scrub, which compares object metadata across replicas. Both checks surface through ceph health detail and the ceph_health_detail Prometheus metric exposed by the MGR module. There is no per-PG “overdue” gauge in the standard metrics pipeline, which is why scrub debt often goes unmonitored until someone runs ceph -s and sees a HEALTH_WARN they do not recognize.

Ceph PGs stuck peering: the slow peering loop after a mass restart

Mon, 01 Jan 0001 00:00:00 +0000

Ceph PGs stuck peering: the slow peering loop after a mass restart

After a full cluster power cycle, mass OSD restart, or wide maintenance event, peering should finish within a minute or two. When it does not, ceph -s shows a long list of PGs in peering or peering+activating, the count is not decreasing, and client I/O for those PGs is hung. HEALTH_WARN or HEALTH_ERR can sit there for 20, 30, or 60 minutes with no visible progress.

Ceph recovery stalled: degraded PGs that are not healing

Mon, 01 Jan 0001 00:00:00 +0000

Ceph recovery stalled: degraded PGs that are not healing

ceph -s shows HEALTH_WARN or HEALTH_ERR, the cluster reports non-zero degraded or undersized placement groups, and clients still appear served. The degraded PG count is not climbing, which feels like progress. It is not. A flat degraded count with zero recovery rate is one of the most dangerous states a Ceph cluster can sit in: nothing is healing, and the operator assumes the system is working through it.

Ceph recovery storm: rebuild traffic starving client I/O

Mon, 01 Jan 0001 00:00:00 +0000

Ceph recovery storm: rebuild traffic starving client I/O

A Ceph recovery storm occurs when the cluster’s self-healing machinery starves the workloads it is supposed to serve. After an OSD failure, host loss, or bulk OSD addition, recovery and backfill traffic floods the same disks, network links, and OSD CPU that client I/O depends on. Client latency climbs 10x to 100x, applications time out, and retries add more load on top of the recovery stream.

Ceph RGW failed requests: aborted request rate and what it actually counts

Mon, 01 Jan 0001 00:00:00 +0000

Ceph RGW failed requests: aborted request rate and what it actually counts

ceph_rgw_failed_req is one of the most commonly misread RADOS Gateway metrics. Operators see “failed” in the name and assume it counts HTTP 4xx and 5xx responses. It does not. It counts requests where the client connection was aborted before the response completed. That distinction changes who you page, where you look, and which signals you correlate with.

What it is and why it matters

ceph_rgw_failed_req is a per-instance counter exposed by each RGW daemon, scraped from the admin socket and labeled with instance_id (alongside ceph_daemon, instance, job). Its sibling counter is ceph_rgw_req, the total request count. The operational signal is the ratio between the two: a sustained failed-request rate above 5% of total traffic for more than 5 minutes is the playbook’s TICKET condition.

Ceph RGW garbage-collection backlog: deleted data still consuming space

Mon, 01 Jan 0001 00:00:00 +0000

Ceph RGW garbage-collection backlog: deleted data still consuming space

You deleted several terabytes of S3 objects, but ceph df shows raw usage barely moving. The cluster is approaching nearfull and the write freeze is coming. The deletes returned 204 to clients, so they succeeded from the S3 layer’s perspective, but the underlying RADOS objects are still on disk, queued behind the RADOS Gateway garbage collector.

This is one of the quietest contributors to “the cluster is full but we deleted everything”. RGW does not free object data inline on delete. It marks the head object as deleted, enqueues the data objects (tail segments, multipart parts) for asynchronous garbage collection, and relies on a background GC thread on each gateway to drain the queue. When that thread stops making progress, deleted data keeps consuming capacity indefinitely.

Ceph RGW GET/PUT latency: S3 request latency and queue length

Mon, 01 Jan 0001 00:00:00 +0000

Ceph RGW GET/PUT latency: S3 request latency and queue length

When users report slow S3 GET or PUT responses, the RGW daemon is rarely the root cause. RADOS Gateway is a stateless HTTP frontend that translates REST calls into RADOS object operations. Its observed latency is dominated by the time those underlying operations take, plus whatever queuing happens when the gateway has more in-flight work than it can drain.

The RGW perf counters expose two distinct kinds of signal: per-operation latency accumulators (ceph_rgw_op_get_obj_lat_sum/_count and ceph_rgw_op_put_obj_lat_sum/_count) for the S3 operations themselves, plus queue gauges (ceph_rgw_qlen and ceph_rgw_qactive) that show whether requests are piling up inside the daemon. Treating those signals together is the difference between “S3 is slow” and “S3 is slow because one OSD hosting a bucket index shard is in OMAP collapse.”

Ceph RGW orphaned multipart uploads: space that disappears from view

Mon, 01 Jan 0001 00:00:00 +0000

Ceph RGW orphaned multipart uploads: space that disappears from view

Pool capacity climbs toward nearfull but bucket listings and radosgw-admin bucket stats cannot account for it. The obvious explanations (client growth, snapshots, OMAP bloat) do not fit. This pattern is frequently caused by orphaned multipart upload parts in the RADOS Gateway (RGW) data pool.

S3 multipart uploads create a manifest, upload parts into the bucket’s data pool, then call CompleteMultipartUpload to assemble the final object. When that final step never happens, when the client retries uploads of the same key, or when RGW aborts an upload incompletely, the individual parts remain in RADOS. They occupy raw space but no live bucket index entry references them.

Ceph slow requests (SLOW_OPS): operations blocked past osd_op_complaint_time

Mon, 01 Jan 0001 00:00:00 +0000

Ceph slow requests (SLOW_OPS): operations blocked past osd_op_complaint_time

ceph_healthcheck_slow_ops greater than zero means operations on the cluster have crossed osd_op_complaint_time (default 30s) and are stuck, not merely slow. The SLOW_OPS health check surfaces them as a warning, and the metric itself is a live gauge pulled from ceph health detail.

Slow ops are a symptom of something downstream blocking I/O: a failing disk, a saturated BlueStore RocksDB DB, network timeouts between OSDs, or heavy deep-scrub on HDD during a maintenance window. The right first move is to read where in the pipeline each slow op is stuck before changing any config.

Ceph too many PGs per OSD: mon_max_pg_per_osd, peering cost, and sizing

Mon, 01 Jan 0001 00:00:00 +0000

Ceph too many PGs per OSD: mon_max_pg_per_osd, peering cost, and sizing

PG count per OSD is one of the few Ceph sizing decisions with direct cost in both directions. Too many PGs per OSD means more memory and CPU spent on peering, recovery scans, and PG log maintenance across more logical units. Too few means CRUSH cannot spread data evenly, producing hotspots on specific OSDs while others sit idle.

mon_max_pg_per_osd (default 250) is the cluster’s failsafe against runaway PG counts. When an OSD approaches this number, Ceph raises TOO_MANY_PGS and blocks new pool creation, pg_num increases, and replication factor changes. This is a guard rail, not a performance target. The existing cluster keeps serving I/O; only topology changes that would add PGs are blocked.

Ceph unfound objects: the cluster cannot locate a surviving copy

Mon, 01 Jan 0001 00:00:00 +0000

Ceph unfound objects: the cluster cannot locate a surviving copy

When a placement group enters recovery_unfound or backfill_unfound, the cluster has objects it knows should exist but cannot locate on any OSD that is currently up and in. ceph health detail surfaces this as the OBJECT_UNFOUND check, and the cluster-wide gauge ceph_num_objects_unfound rises above zero. Every known replica or erasure-coded chunk is on an OSD that is down, destroyed, or has not yet been probed.

Cerent Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Chateau Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Chatsworth PDU

Mon, 01 Jan 0001 00:00:00 +0000

Check Point Software Technologies Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Checkpoint

Mon, 01 Jan 0001 00:00:00 +0000

Checkpoint Licensing

Mon, 01 Jan 0001 00:00:00 +0000

Cherokee International Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cheyenne Software SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Chia

Mon, 01 Jan 0001 00:00:00 +0000

Chia Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Chia Monitoring

What Is Chia?

Chia is a blockchain and smart transaction platform designed to be eco-friendly through its innovative use of proof-of-space-and-time. It enables users to participate in the blockchain by utilizing free disk space instead of relying on energy-intensive proof-of-work.

Monitoring Chia With Netdata

Monitoring your Chia setup is essential for ensuring optimal performance and troubleshooting issues before they escalate. Netdata offers a Chia monitoring tool through a Prometheus exporter available here. Netdata stands out because it can ingest data from any Prometheus exporter, turning this data into automated dashboards and alerts without requiring additional software such as Prometheus servers or Grafana. By using Netdata, you gain powerful insights into your Chia operations with minimal setup.

Chippcom SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Chloride SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Christ Elektronik CLM5IP Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Christ Elektronik CLM5IP Monitoring

What Is Christ Elektronik CLM5IP?

The Christ Elektronik CLM5IP power panel is an advanced instrument designed for monitoring and controlling power systems. It is widely used in industrial and IoT environments for its ability to collect extensive power usage data, ensure efficient energy management, and optimize performance.

Monitoring Christ Elektronik CLM5IP With Netdata

Monitoring the Christ Elektronik CLM5IP power panel can provide invaluable insights into the energy usage and health of your systems. By using Netdata as a monitoring tool, you can leverage OpenMetrics through a Prometheus exporter, such as the Christmann CLM5IP Exporter, to collect metrics effortlessly. Netdata allows you to ingest data from any Prometheus exporter, providing automated dashboards, real-time alerts, and comprehensive analytics, all without the need for a Prometheus server or Grafana. This makes Netdata a versatile and powerful tool for monitoring Christ Elektronik CLM5IP.

Christ Elektronik CLM5IP power panel

Mon, 01 Jan 0001 00:00:00 +0000

Chromatis Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Chronix

Mon, 01 Jan 0001 00:00:00 +0000

Chrony

Mon, 01 Jan 0001 00:00:00 +0000

Chrony Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Chrony Monitoring

What Is Chrony?

Chrony is a versatile software suite for maintaining the system clock accuracy on your Linux distributions. Acting as an implementation of the Network Time Protocol (NTP), Chrony ensures your systems are synchronized to accurate time sources, which is crucial for various server operations. It is highly adaptive, providing swift synchronization capabilities, making it particularly beneficial for systems with sporadic network connectivity or those that endure extreme delays.

Chrysalis

Mon, 01 Jan 0001 00:00:00 +0000

Chrysalis Luna HSM

Mon, 01 Jan 0001 00:00:00 +0000

Ciena Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cilium Agent

Mon, 01 Jan 0001 00:00:00 +0000

Cilium Agent Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Cilium Agent Monitoring

What Is Cilium Agent?

Cilium is an open-source software package that provides networking, security, and load balancing for cloud-native environments like Kubernetes. The Cilium Agent is a key component within this framework, responsible for enforcing policies and facilitating complex networking requirements with simplicity and transparency.

Monitoring Cilium Agent With Netdata

To monitor Cilium Agent efficiently, Netdata employs an OpenMetrics (Prometheus) exporter. This integration allows Netdata to ingest valuable monitoring data directly from any Prometheus exporter. With Netdata, you don’t need a separate Prometheus server or Grafana setup. The tool provides automated dashboards and real-time alerts, streamlining the process of monitoring Cilium Agent metrics like network security and connectivity.

Cilium Operator

Mon, 01 Jan 0001 00:00:00 +0000

Cilium Operator Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Cilium Operator Monitoring

What Is Cilium Operator?

Cilium Operator is a crucial component for Kubernetes network security management. It enhances Kubernetes networking by providing secure and transparent connections within your cluster. Whether you are a DevOps engineer or a Site Reliability Engineer (SRE), understanding the intricacies of the Cilium Operator and monitoring its performance can significantly enhance the security and efficiency of your deployments.

Monitoring Cilium Operator With Netdata

To effectively monitor Cilium Operator, Netdata uses an openmetrics (Prometheus) exporter. This seamless integration allows Netdata to ingest data from any Prometheus exporter. By doing so, you can access automated dashboards, alerts, and more—all without needing a separate Prometheus server or Grafana. This makes Netdata an ideal Cilium Operator monitoring tool, combining ease of use with powerful capabilities.

Cilium Proxy

Mon, 01 Jan 0001 00:00:00 +0000

Cilium Proxy Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Cilium Proxy Monitoring

What Is Cilium Proxy?

Cilium Proxy is a key component in modern network security and performance for Kubernetes environments. It acts as an intermediary that controls and processes requests and responses between different microservices, ensuring secure and efficient communication.

Monitoring Cilium Proxy With Netdata

Monitoring Cilium Proxy is crucial for maintaining optimal network performance and security. Netdata offers an intuitive solution to monitor Cilium Proxy using an openmetrics (Prometheus) exporter. Netdata can seamlessly ingest data from any Prometheus exporter, providing automated dashboards, alerts, and insightful metrics without the need for a Prometheus server or Grafana. This allows DevOps teams to quickly troubleshoot issues and maintain seamless network operations using Netdata’s unparalleled real-time monitoring capabilities.

Cirpack SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cisco

Mon, 01 Jan 0001 00:00:00 +0000

Cisco 3850

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Access Point

Mon, 01 Jan 0001 00:00:00 +0000

Cisco ASA

Mon, 01 Jan 0001 00:00:00 +0000

Cisco ASR

Mon, 01 Jan 0001 00:00:00 +0000

Cisco BGP

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Catalyst

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Catalyst WLC

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Csr1000V

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Firepower

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Firepower ASA

Mon, 01 Jan 0001 00:00:00 +0000

Cisco ICM

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Ironport Email

Mon, 01 Jan 0001 00:00:00 +0000

Cisco ISE

Mon, 01 Jan 0001 00:00:00 +0000

Cisco ISR

Mon, 01 Jan 0001 00:00:00 +0000

Cisco ISR 4431

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Legacy WLC

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Licensing

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Load Balancer

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Meraki (Cloud Controller)

Mon, 01 Jan 0001 00:00:00 +0000

Cisco NCS

Mon, 01 Jan 0001 00:00:00 +0000

Cisco Nexus

Mon, 01 Jan 0001 00:00:00 +0000

Cisco SB

Mon, 01 Jan 0001 00:00:00 +0000

Cisco UC Virtual Machine

Mon, 01 Jan 0001 00:00:00 +0000

Cisco UCS

Mon, 01 Jan 0001 00:00:00 +0000

Cisco WAN Optimizer

Mon, 01 Jan 0001 00:00:00 +0000

Ciscosystems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Citrix

Mon, 01 Jan 0001 00:00:00 +0000

Citrix Netscaler

Mon, 01 Jan 0001 00:00:00 +0000

Citrix Netscaler SDX

Mon, 01 Jan 0001 00:00:00 +0000

Citrix Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

City Com B.V. SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

ClamAV daemon

Mon, 01 Jan 0001 00:00:00 +0000

ClamAV Daemon Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

ClamAV Daemon Monitoring

What Is ClamAV Daemon?

ClamAV Daemon is part of the ClamAV antivirus software toolkit, developed for detecting threats and viruses on various platforms. It plays a crucial role in ensuring the security of IT infrastructures by scanning files and attachments in real-time to prevent malware and other malicious threats from affecting your systems.

Monitoring ClamAV Daemon With Netdata

Monitoring the ClamAV Daemon is seamless with Netdata, as it utilizes an openmetrics (Prometheus) exporter to fetch real-time metrics. With Netdata, you can ingest data from any Prometheus exporter providing an automated setup with dashboards, alerts, and in-depth analytics, all without the need for setting up a Prometheus server or Grafana.

Clamscan results

Mon, 01 Jan 0001 00:00:00 +0000

Clamscan Results Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Clamscan Results Monitoring

What Is Clamscan Results Monitoring?

Clamscan is a command-line utility used in conjunction with ClamAV for detecting malware on a system. Monitoring Clamscan results is crucial to ensure the effectiveness of your anti-malware efforts, track performance metrics, and maintain the security of your network. By keeping an eye on the data provided by Clamscan, IT admins, DevOps, and security personnel can act promptly in response to vulnerabilities or infections.

Clarent Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Clash

Mon, 01 Jan 0001 00:00:00 +0000

Clash Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Clash Monitoring

What Is Clash?

Clash is a versatile and powerful proxy server designed for efficient network traffic management. It serves as an essential component in managing and troubleshooting network performance in today’s complex IT environments. With various features designed to seamlessly handle traffic routing, Clash ensures that data flows are optimized, which is vital for maintaining network health and performance.

Monitoring Clash With Netdata

To effectively monitor Clash, Netdata offers a seamless integration using an openmetrics (prometheus) exporter. This integration allows technical teams—be it DevOps, SREs, Developers, IT admins, or engineers—to gain comprehensive insights into the performance of their Clash instances. Netdata can ingest data from any Prometheus exporter, providing users with automated dashboards, real-time alerts, and more, without the necessity of having a dedicated Prometheus server or Grafana setup. This makes it easier for users looking for a $name monitoring tool to implement and manage network performance monitoring efficiently.

Classifiers

Mon, 01 Jan 0001 00:00:00 +0000

Clavister AB SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Clickarrray Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse active part count growing: reading MaxPartCountForPartition before it pages

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse active part count growing: reading MaxPartCountForPartition before it pages

Rising MaxPartCountForPartition is the leading indicator for the most common ClickHouse production failure: parts accumulating faster than background merges can consolidate them. A single partition crossing 500 active parts means you have hours, not days, before inserts delay and eventually fail with TOO_MANY_PARTS.

The thresholds are per-partition. A table with ten partitions at fifty parts each is healthy; one partition at 950 parts is approaching throttling. Projections create hidden parts inside the same table that count toward the same limits. Materialized views route inserts to separate target tables that can hit their own limits independently.

ClickHouse ALTER UPDATE/DELETE overuse: why mutations are not row updates

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse ALTER UPDATE/DELETE overuse: why mutations are not row updates

Your inserts are slowing down. system.merges shows long-running background tasks with is_mutation = 1, while the active part count climbs toward the parts_to_delay_insert threshold. Write latency rises and DelayedInserts increases even though memory, disk, and CPU are not exhausted. The culprit is usually an application treating ClickHouse as an OLTP store, issuing ALTER TABLE UPDATE or ALTER TABLE DELETE as routine operations.

ClickHouse async inserts: when async_insert fixes too-many-parts and when it hides it

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse async inserts: when async_insert fixes too-many-parts and when it hides it

Async inserts move batching responsibility from the client to the ClickHouse server. The server buffers rows and flushes them as larger blocks instead of persisting every INSERT as a separate part. When the root cause is an unbatchable client, this stops the small-inserts anti-pattern from flooding the merge pool. When the root cause is high-cardinality partitioning or sustained over-ingestion, async inserts relocate the crisis into an in-memory buffer that loses data on crash and hides backpressure from the application.

ClickHouse authentication failures: system.session_log, brute force, and credential drift

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse authentication failures: system.session_log, brute force, and credential drift

You notice a spike in failed connection attempts to ClickHouse: a security scanner flags repeated TCP 9000 probes, or an application logs connection timeouts after a secrets rotation. ClickHouse exposes authentication events through system.session_log, but only if the feature is enabled. Without it, fallback to server error logs and system.query_log exceptions.

The failures split into two patterns. Malicious: brute-force or credential-scanning campaigns against exposed TCP 9000 or HTTP 8123. Operational drift: a rotated password not updated in a client config, or a deployment shipping an old connection string. Distinguish them fast. Block an external attacker at the network layer; fix credential drift on the client.

ClickHouse background pool saturation: when merges and mutations starve

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse background pool saturation: when merges and mutations starve

Insert latency is climbing and DelayedInserts is ticking up. system.merges shows every slot occupied, yet system.parts keeps growing. When the background merge and mutation pool saturates, new merges queue instead of starting, parts accumulate, and the distance to insert rejections shrinks fast. Distinguish true thread starvation from I/O-bound stalls, identify when mutations are the culprit, and relieve pressure before inserts fail.

ClickHouse cannot connect to ZooKeeper/Keeper: diagnosing the coordination layer

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse cannot connect to ZooKeeper/Keeper: diagnosing the coordination layer

If SELECT * FROM system.zookeeper WHERE path = '/' fails, replicated tables flip to readonly, or ON CLUSTER DDL hangs, the coordination layer is broken. The server stays up and GET /ping returns Ok., so liveness checks miss the problem. Partial degradation can escalate to a write outage if replicated tables cannot re-establish sessions.

ClickHouse uses the coordination service for ReplicatedMergeTree leader election, replication log queues, insert deduplication, and distributed DDL. ClickHouse Keeper typically listens on port 9181; external ZooKeeper ensembles listen on 2181. The diagnostic path differs slightly, but the symptom is the same: ClickHouse cannot reliably complete coordination operations.

ClickHouse checksum mismatch and broken parts: detecting data corruption

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse checksum mismatch and broken parts: detecting data corruption

ClickHouse logs showing Checksum doesn't match, Broken part, or similar errors indicate data corruption. Affected parts move to system.detached_parts. Queries may throw exceptions or return partial results. On replicated clusters, a replica with corrupt parts may lag because it cannot validate fetched parts. Corruption does not self-resolve. You must quarantine the bad part, identify the root cause, and rebuild the data from a healthy source or backup.

ClickHouse client connections climbing: TCP 9000, HTTP 8123, and connection leaks

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse client connections climbing: TCP 9000, HTTP 8123, and connection leaks

TCPConnection or HTTPConnection climbing on a ClickHouse node means each active socket consumes a file descriptor. When growth is uncorrelated with query throughput, it is usually a connection leak or pool misconfiguration rather than healthy concurrency. If the count approaches max_connections, the server rejects new client connections. If the Linux nofile limit is reached first, queries and merges fail with “too many open files.”

ClickHouse DB::Exception: Too many parts - causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse DB::Exception: Too many parts - causes and fixes

When logs show DB::Exception: Too many parts (N). Merges are processing significantly slower than inserts (exception code 252), INSERTs that succeeded yesterday are now failing. Data backs up in ingestors, message queues, or client buffers, and every retry adds load to a system already drowning in small files.

At least one partition in a MergeTree table has exceeded the hard limit for active data parts, and background merges cannot consolidate them fast enough. Do not restart the server or raise the limit indefinitely. Reduce the part creation rate, remove blockers from the merge pipeline, and give background threads runway to catch up.

ClickHouse DelayedInserts climbing: the warning before too-many-parts

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse DelayedInserts climbing: the warning before too-many-parts

Insert latency is climbing and system.events.DelayedInserts is no longer flat. ClickHouse is sleeping during INSERT because at least one partition has crossed parts_to_delay_insert. The database still accepts writes, but injects a sleep before each insert commits. This is the warning window before hard failure. If the merge backlog is not resolved, DelayedInserts climbs until RejectedInserts starts ticking and clients receive DB::Exception: Too many parts.

ClickHouse detached parts piling up: reading system.detached_parts and reclaiming space

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse detached parts piling up: reading system.detached_parts and reclaiming space

Disk usage is climbing and system.disks shows unreserved_space shrinking, yet active part counts in system.parts look normal. Queries are not failing, merges appear healthy, and there is no insert rejection storm. The hidden consumer is often a growing pile of detached parts: directories that ClickHouse removed from the active dataset but left on the filesystem. Unlike active parts, detached parts do not participate in queries or merges, and ClickHouse never deletes them automatically. Standard monitoring that only watches system.parts with active = 1 misses them entirely. This article explains how to read system.detached_parts, interpret the reason column, decide whether to reattach or discard the data, and prevent silent recurrence.

ClickHouse disk space collapse: why merges need free space and how the spiral starts

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse disk space collapse: why merges need free space and how the spiral starts

Disk usage climbs from 75% to 85% over a week, then hits 90%. A few hours later ClickHouse rejects inserts, part counts spike, and the volume is at 98%. The system did not simply run out of space. It entered a self-reinforcing spiral where background merges, the mechanism that reclaims space, stalled because they needed free space to work.

ClickHouse disk space monitoring: free_space, unreserved_space, and the 80% target

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse disk space monitoring: free_space, unreserved_space, and the 80% target

ClickHouse disk space is not a simple capacity gauge. Merges write new parts before deleting old ones, so free space is an operational dependency of the storage engine. Insufficient disk does not just mean “running low”: background merges stall, parts accumulate, and the system enters a self-reinforcing death spiral.

This guide covers the system.disks metrics, why unreserved_space matters more than free_space, and operational targets that keep merges alive.

ClickHouse distributed DDL stuck: ON CLUSTER queries that never finish

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse distributed DDL stuck: ON CLUSTER queries that never finish

ON CLUSTER DDL is not atomic. The initiator writes a task into a shared queue in ZooKeeper or ClickHouse Keeper under /clickhouse/task_queue/ddl/. Each node pulls entries in order, executes the DDL locally, and reports status back. One slow, down, or misconfigured node can leave the entry unfinished forever, blocking all later DDL on that node and creating schema drift that is invisible until a replica rejects an insert or a subsequent ALTER fails.

ClickHouse distributed query amplification: one coordinator, many shard subqueries

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse distributed query amplification: one coordinator, many shard subqueries

A SELECT against a distributed table registers one entry in system.processes on the coordinator. That query fans out subqueries to every addressed shard, waits for intermediate results, then merges them locally. True CPU, memory, and network load is multiplied by the shard count, and the slowest participating shard dictates latency.

This amplification is invisible to cluster-average metrics. A dashboard averaging CPU or query latency across all nodes can look healthy while one shard is saturated, because the coordinator and healthy shards mask the straggler. Operators often misdiagnose the slowdown as a generalized cluster problem or a coordinator bottleneck, when the root cause is usually a single overloaded shard, a missing sharding key filter, or an expensive GLOBAL operator that broadcasts data across the network.

ClickHouse full table scan: partition pruning failures and the primary key

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse full table scan: partition pruning failures and the primary key

A query that returned in milliseconds yesterday now takes minutes and saturates CPU. Check whether it is reading every row before you add vCPUs or shards. A missing filter on the partition key or a mismatch with the primary key prefix forces ClickHouse to open every part and scan every granule. Symptoms are runaway read_rows counts, tail latency spikes, and sustained CPU decompression that hardware scaling cannot fix.

ClickHouse insert latency rising: the leading indicator of write-pipeline trouble

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse insert latency rising: the leading indicator of write-pipeline trouble

Your ClickHouse inserts are taking longer. A query that committed in 200 ms last week is now taking 5 seconds, then 15, then 30. In most databases this signals slow disks or lock contention. In ClickHouse, sustained insert latency is the earliest operational signal that the write pipeline is congesting. It precedes DelayedInserts, part-count alerts, and the hard stop of RejectedInserts by minutes to hours. Wait for the error and the merge debt is already severe.

ClickHouse Keeper latency high: the early warning before sessions expire

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Keeper latency high: the early warning before sessions expire

INSERTs to replicated tables slow down, ON CLUSTER DDL hangs, and the replication queue grows on followers. SELECT 1 and HTTP /ping stay healthy, and non-replicated tables are fine. The culprit is usually the coordination service, not ClickHouse itself.

Rising ZooKeeper or ClickHouse Keeper operation latency is a leading indicator. Replicated inserts, replication log updates, and distributed DDL all round-trip through Keeper. Because Keeper writes its transaction log synchronously, disk I/O on the Keeper node is the most common bottleneck. A degraded-but-connected coordination service is worse than a hard partition: it silently slows every replicated operation until sessions start expiring and replicas flip to readonly. This article explains how to read the early signals, isolate the cause, and fix it before sessions expire.

ClickHouse Keeper saturation spiral: too many tables, DDL storms, and cluster freeze

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Keeper saturation spiral: too many tables, DDL storms, and cluster freeze

INSERTs fail. Replicated tables flip read-only. ON CLUSTER DDL hangs. curl http://localhost:8123/ping still returns Ok. and SELECT 1 still works. This is Keeper saturation: the coordination layer is choking while liveness probes give false confidence.

The spiral starts when ZooKeeper or ClickHouse Keeper cannot keep up with metadata load. Every replicated table registers znodes and watches. Every DDL operation adds more. Coordination latency climbs until heartbeat traffic cannot complete within the negotiated session timeout; sessions expire, replicas become read-only, and writes fail. Reconnecting nodes then trigger a thundering herd.

ClickHouse killed by the OOM killer: RSS, max_server_memory_usage, and cgroup limits

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse killed by the OOM killer: RSS, max_server_memory_usage, and cgroup limits

You restart a pod and kubectl describe pod shows Reason: OOMKilled with exit code 137. Inside ClickHouse, MemoryTracking sits well below max_server_memory_usage, and system.text_log shows no warning. The process is gone, merges are dead, and replication queues are backing up.

The Linux OOM killer targets RSS, not ClickHouse’s internal MemoryTracking. Untracked allocations, jemalloc arena fragmentation, and cgroup accounting quirks create a persistent gap between what ClickHouse thinks it is using and what the kernel sees. In containerized environments, the cgroup OOM killer can evict the pod before ClickHouse ever triggers its own server-wide limit.

ClickHouse long-running queries: finding and killing the resource hog

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse long-running queries: finding and killing the resource hog

A query that should finish in seconds is still running after twenty minutes. Memory on the ClickHouse node is climbing, query latency has doubled, and you suspect a single query is holding resources it will never release. In ClickHouse, a long-running query can be a legitimate analytical job crunching terabytes, a Cartesian JOIN exploding in memory, or a GROUP BY that has spilled to disk and slowed to a crawl. Telling the difference determines whether you kill it or let it finish.

ClickHouse mark cache and uncompressed cache: reading low hit rates

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse mark cache and uncompressed cache: reading low hit rates

ClickHouse query latency spikes after a restart, or monitoring shows a sustained drop in mark cache hit rate. Before increasing mark_cache_size, determine whether you are seeing normal warmup or a cache that is too small for the working set.

The mark cache stores primary key index granule positions. The uncompressed cache stores decompressed column blocks. Low hit rates in each produce different symptoms. This guide covers how to read the metrics, distinguish warmup from real problems, and decide when tuning is warranted.

ClickHouse Memory limit (for query) exceeded: per-query limits and GROUP BY/JOIN blowups

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Memory limit (for query) exceeded: per-query limits and GROUP BY/JOIN blowups

Code: 241. DB::Exception: Memory limit (for query) exceeded means a single query’s allocations breached the max_memory_usage ceiling. ClickHouse tracks memory in a hierarchy: server-wide, per-user, and per-query. The server kills the query to protect the rest of the workload.

This differs from Memory limit (total) exceeded (server-level pressure) and Memory limit (for user) exceeded (profile-level pressure). A query-level breach usually stems from one of three patterns: a high-cardinality GROUP BY, an unbounded DISTINCT, or a JOIN that materializes more rows than expected.

ClickHouse Memory limit (total) exceeded - server-wide memory pressure and fixes

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Memory limit (total) exceeded - server-wide memory pressure and fixes

Code: 241. DB::Exception: Memory limit (total) exceeded: would use X bytes, current RSS Y, maximum Z. is the server-level cap, not a per-query limit. When ClickHouse’s MemoryTracking hits max_server_memory_usage (default 90% of physical RAM), the server kills the heaviest running queries to protect the process. New and existing queries fail until memory drops. Find the largest memory consumer and stop it before the OOM killer does.

ClickHouse memory pressure death spiral: runaway queries, retries, and OOM

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse memory pressure death spiral: runaway queries, retries, and OOM

MEMORY_LIMIT_EXCEEDED errors climb in the query log. Queries that normally finish in seconds now take minutes or are killed outright. The ClickHouse process is near its memory limit, but killing the heaviest query only frees capacity for a moment before another query is killed. If the application retries immediately, pressure never drops. With spill-to-disk enabled, the bottleneck shifts to disk I/O, starving background merges and slowing the whole system.

ClickHouse MemoryTracking vs MemoryResident: reading the memory gap correctly

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse MemoryTracking vs MemoryResident: reading the memory gap correctly

You finish a large batch query, open monitoring, and see ClickHouse MemoryTracking drop by 40 GB while MemoryResident barely moves. Or you watch RSS climb for hours after a restart while MemoryTracking tracks the rise steadily. Neither pattern indicates a leak. The gap between ClickHouse’s internal ledger and OS resident set size is normal: the server accounts for memory synchronously while jemalloc retains pages for reuse.

ClickHouse merge death spiral: when parts accumulate faster than merges consolidate

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse merge death spiral: when parts accumulate faster than merges consolidate

Insert latency climbs. Application logs show ClickHouse throttling writes. Eventually inserts fail with Too many parts. Disk usage rises even though ingestion volume is flat. The cluster is up but refusing writes.

This is the merge death spiral: a self-reinforcing loop where parts accumulate faster than background merges consolidate them. Every INSERT creates immutable on-disk parts. Background merge threads combine smaller parts into larger ones to keep query performance healthy and part counts low. When insert pressure exceeds merge throughput, the backlog grows, merge overhead increases, and the system chokes on its own structure.

ClickHouse merge duration climbing: the leading indicator of part explosion

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse merge duration climbing: the leading indicator of part explosion

system.merges shows elapsed times in hours. Your dashboards show P99 merge duration climbing over the past 48 hours. Rising merge duration is the earliest signal that your cluster is heading toward a part-count crisis, typically 1 to 3 days before inserts throttle or fail entirely.

ClickHouse merges not keeping up: diagnosing a stalled or starved merge pool

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse merges not keeping up: diagnosing a stalled or starved merge pool

When insert latency climbs and system.merges is empty while active parts grow, the background merge pool is likely stalled or starved. ClickHouse relies on background merges to consolidate immutable parts after each INSERT. Without merges, parts accumulate: query scans open more files, memory pressure shifts to the mark cache and file descriptor tables, and the system approaches the TOO_MANY_PARTS threshold.

ClickHouse Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Monitoring

What Is ClickHouse?

ClickHouse is a columnar database management system (DBMS) known for its high performance in managing online analytical processing (OLAP) queries. It excels in processing large volumes of data, making it a popular choice for data analytics tasks. Understanding and monitoring the performance of ClickHouse can ensure that your systems are running efficiently and reliably.

Monitoring ClickHouse With Netdata

Netdata provides a comprehensive “ClickHouse monitoring tool” that allows you to monitor ClickHouse instances in real-time, with insightful visuals and metabolism into the system’s performances. You can easily integrate Netdata with ClickHouse by using the go.d.plugin, enabling automatic discovery and effortless setup.

ClickHouse monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse monitoring checklist: the signals every production cluster needs

ClickHouse failures usually begin as storage-structure debt: immutable parts accumulate faster than merges consolidate them, coordination sessions expire, or disk space drops below the threshold merges need to complete. Query latency degrades only after the crisis is hours old.

This checklist groups monitoring signals into four maturity levels. Level 1 is the minimum viable instrumentation to avoid data loss and unavailability. Each subsequent level adds leading indicators that catch part accumulation, replication divergence, and memory pressure while they are still reversible. The signals are drawn from ClickHouse system tables and OS-level metrics. They apply to single-node, sharded, and replicated setups; replicated tables add ZooKeeper/Keeper signals that belong in Level 2.

ClickHouse monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse monitoring maturity model: from survival to expert

Most production ClickHouse incidents are not mysterious. They are predictable storage-structure or coordination failures that better monitoring would have surfaced hours earlier. If you are running ClickHouse at scale, you need to know whether your observability is actually catching the failure modes that matter, or just proving that the process is running.

This maturity model is a diagnostic mirror, not a trophy case. Use it to audit your dashboards, tune alert noise, and decide what to instrument next. Each level builds on the last. Skipping levels leaves predictable gaps: teams with beautiful query-latency dashboards still get surprised by merge death spirals because they never instrumented part counts per partition.

ClickHouse mutation stuck: parts_to_do not decreasing and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse mutation stuck: parts_to_do not decreasing and how to recover

Check system.mutations and find a mutation active for hours. parts_to_do has not moved in thirty minutes and parts are accumulating. Queries return, but insert latency climbs. On replicated tables, each replica processes mutations independently, so a stall on one node creates silent divergence while the rest of the cluster appears healthy.

Mutations rewrite data parts to apply ALTER UPDATE, ALTER DELETE, or projection changes. They run sequentially per table and share the background merge and mutation pool with regular merges. A stalled mutation blocks subsequent mutations for that table and consumes threads needed for merges. The result is merge starvation, insert delays, and eventually Too many parts rejections.

ClickHouse mutations silently blocking merges: the hidden cause of part growth

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse mutations silently blocking merges: the hidden cause of part growth

You are watching part counts climb on a ClickHouse node. system.merges shows active background tasks, health checks return Ok, and the log shows no mutation errors. Yet inserts are slowing and MaxPartCountForPartition is trending toward the delay threshold. The pool looks busy, so merges should be keeping up. They are not: some of those busy slots are mutations, and mutations starve merges silently.

ClickHouse No space left on device: emergency recovery when the data disk fills

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse No space left on device: emergency recovery when the data disk fills

When ClickHouse returns No space left on device to client inserts or the server log fills with write errors, the situation is past a simple capacity alert. ClickHouse does not degrade gradually on a full disk. Background merges halt immediately because they require temporary free space to write combined parts before removing source files. Once merges stop, small insert parts accumulate, metadata overhead grows, and disk usage accelerates. TTL-based expiration also stops because TTL cleanup is executed by merges. ClickHouse system tables such as system.query_log and system.part_log are MergeTree tables that can grow unbounded and consume the remaining space.

ClickHouse projections and hidden parts: the part count you can't see

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse projections and hidden parts: the part count you can’t see

Inserts fail with TOO_MANY_PARTS while system.parts on the base table looks comfortable. If you monitor only the source table, you are missing projection sub-parts inside base directories and independent parts in materialized view target tables. A single insert can spawn parts across projection subdirectories and multiple downstream tables. The number in system.parts is a floor, not a ceiling. The gap between visible and real part count is where merge crises begin.

ClickHouse query error rate high: reading exception codes 241, 252, and 999

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse query error rate high: reading exception codes 241, 252, and 999

What this means

ClickHouse logs failed queries to system.query_log with type = ‘ExceptionWhileProcessing’ and a numeric exception_code. The FailedQuery counter in system.events increments for each failure. Keep the FailedQuery / Query ratio below 1%. A sustained climb above that threshold signals a systemic issue, not a few bad queries.

The codes that dominate production incidents fall into five classes.

ClickHouse query latency P99 spikes: tail latency, hot shards, and cold data

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse query latency P99 spikes: tail latency, hot shards, and cold data

Your ClickHouse cluster shows healthy average query latency, but users report intermittent timeouts. P99 spikes while P50 stays flat. This divergence is tail latency: a subset of queries is dramatically slower. In distributed clusters, the culprit is often a single hot shard or a replica serving stale data. In tiered storage, a query touching cold S3 data can be one to two orders of magnitude slower.

ClickHouse Replica is lost: SYSTEM RESTORE REPLICA and recovering a diverged replica

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Replica is lost: SYSTEM RESTORE REPLICA and recovering a diverged replica

You run SELECT count() on two replicas and get different results. Your application returns inconsistent aggregations depending on which node answers. The replication dashboard looks green: ZooKeeper sessions are active, queue_size is zero, and no replica is readonly. The replica has permanently lost parts, but ZooKeeper does not know because the loss happened outside the replication log. Silent divergence hides behind healthy metrics until queries start returning wrong results.

ClickHouse replicas out of sync: when SELECT count() differs across replicas

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse replicas out of sync: when SELECT count() differs across replicas

You run the same SELECT count() against two replicas and get different numbers. The replica with the lower count still responds instantly, its HTTP ping returns Ok., its replication queue may even be empty, and its CPU and memory look fine. Standard liveness checks give false confidence while the replica serves stale or incomplete data.

ClickHouse replication is asynchronous and coordinated through ZooKeeper or ClickHouse Keeper. A replica can lose its session, miss fetches, or silently drop data, yet continue serving reads from local disk. Distributed queries hide the divergence unless you compare row counts or partition metadata across nodes.

ClickHouse ReplicatedDataLoss > 0: detecting and responding to lost parts

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse ReplicatedDataLoss > 0: detecting and responding to lost parts

ReplicatedDataLoss > 0 is a hard signal in ClickHouse. A nonzero value in system.events means the server has determined that a data part is missing and cannot be retrieved from any available replica. This is not replication lag, a transient fetch failure, or the normal ReplicatedPartFetchesOfMerged optimization.

Queries that touch the affected part can return incomplete results or errors. The immediate risk is silent divergence between replicas, where one replica serves stale or incomplete results without failing the query. Confirm the event, identify the scope, and determine whether a healthy peer still has the part.

ClickHouse replication lag: absolute_delay, queue_size, and catch-up diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse replication lag: absolute_delay, queue_size, and catch-up diagnosis

You notice absolute_delay climbing on one replica. SELECTs there return older rows than on peers, and failover to the lagging node risks losing recently inserted data. In ClickHouse, replication lag is not a single failure; it is a symptom with several distinct causes. A replica can fall behind because it cannot pull entries from the Keeper log, because it cannot fetch parts fast enough, because a mutation is blocking the queue, or because the source replica itself is too slow to serve data.

ClickHouse replication queue stuck: num_tries, last_exception, and dead entries

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse replication queue stuck: num_tries, last_exception, and dead entries

A replica can show low absolute_delay in system.replicas while system.replication_queue contains entries with num_tries in the hundreds and the same last_exception repeating for hours. These dead entries do not self-resolve. They block merges, fetches, or mutations, causing silent divergence and stale reads. This guide shows how to read the queue correctly, identify the failure mode from the entry type, and clear the blockage without making the replica diverge further.

ClickHouse slow queries: diagnosis from query_log to plan to fix

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse slow queries: diagnosis from query_log to plan to fix

P99 latency climbs before averages move. In ClickHouse, tail latency spikes usually mean a subset of queries is hitting cold data, scanning too many parts, or spilling to disk. The goal is to separate plan problems from resource problems fast, then confirm the fix before the next batch of queries arrives.

What this means

Slow queries show up as rising query_duration_ms in system.query_log for QueryFinish events, or rising elapsed in system.processes. ClickHouse latency is sensitive to how many parts a query opens, whether the mark cache is warm, and whether the pipeline stays in memory.

ClickHouse small inserts anti-pattern: why single-row inserts melt the merge pool

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse small inserts anti-pattern: why single-row inserts melt the merge pool

DB::Exception: Too many parts spikes, query latency climbs, and the background merge pool runs flat out while making no progress. Disk, CPU, and memory look healthy. The cause is usually single-row INSERTs from the application.

Every INSERT into a MergeTree table creates at least one immutable data part on disk. ClickHouse is optimized for batch inserts. When clients send single-row or micro-batch inserts, parts are created faster than background merges can consolidate them. This is the number one driver of part accumulation in production clusters.

ClickHouse system log tables eating disk: query_log, part_log, and TTL

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse system log tables eating disk: query_log, part_log, and TTL

You are investigating disk growth on a ClickHouse node. User tables look reasonable, but system.query_log or system.part_log are consuming tens or hundreds of gigabytes. These tables use MergeTree-family engines and are subject to the same part-count and merge pressure limits as production tables. Without a TTL rule, data accumulates forever. During incidents with high query error rates or retry storms, query_log can expand fast enough to threaten disk capacity and stall logging.

ClickHouse Table is in readonly mode: is_readonly on replicated tables and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Table is in readonly mode: is_readonly on replicated tables and how to fix it

Inserts fail with TABLE_IS_READ_ONLY. system.replicas shows is_readonly = 1 for affected tables and the replica rejects writes. For replicated tables, this is a coordination failure, not a disk or memory problem: the replica has lost its session with ClickHouse Keeper or ZooKeeper, or the ensemble is unreachable.

Reads from local parts may still succeed, which hides the failure from load balancers and monitoring probes that rely on SELECT 1 or the HTTP ping endpoint. Brief readonly states lasting seconds are normal during Keeper leader elections. Sustained readonly lasting minutes means the replica is diverging and inserts are being lost or routed elsewhere.

ClickHouse too many open files: file descriptors, part count, and nofile limits

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse too many open files: file descriptors, part count, and nofile limits

ClickHouse aborts queries with “Too many open files” or the server process dies. Logs show errors about failing to open column files or metadata. This is not a traditional leak; it is a capacity cliff. Every active MergeTree part keeps multiple files open, background merges temporarily spike that count, and the Linux nofile limit is usually the bottleneck. The default limit of 1024 is catastrophic for production ClickHouse.

ClickHouse too many partitions: why over-partitioning multiplies your part count

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse too many partitions: why over-partitioning multiplies your part count

You are watching MaxPartCountForPartition climb, or you have already seen DB::Exception: Too many parts. You check the table and see a modest total of active parts. The table looks healthy, but ClickHouse rejects inserts anyway.

The problem is not the table total. It is the partition boundary.

ClickHouse enforces part limits per partition, not per table. A high-cardinality PARTITION BY key like toYYYYMMDD(timestamp) or a business identifier creates thousands of isolated part pools. Parts never merge across those pools. One hundred partitions with thirty parts each produces 3,000 parts, multiplied merge-scheduling overhead, and higher file-descriptor pressure. One partition crossing the hard limit kills inserts for that partition while the rest of the table appears idle.

ClickHouse Too many simultaneous queries: max_concurrent_queries and query storms

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Too many simultaneous queries: max_concurrent_queries and query storms

You run a query and ClickHouse returns “Too many simultaneous queries.” New connections either queue or fail outright. Queries that completed in seconds yesterday now time out. The server is not down, but it is not usable.

ClickHouse is optimized for fewer, heavier analytical queries. As concurrency rises, CPU, memory, and I/O contention increase non-linearly. A small spike can become a storm because each query is greedy.

ClickHouse TTL not deleting data: why expired rows survive and the disk fills

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse TTL not deleting data: why expired rows survive and the disk fills

A MergeTree table with a configured TTL can still accumulate disk usage when expired rows remain visible to SELECT. This is not a syntax or timezone error. ClickHouse enforces TTL only as a side effect of background merges. When merges stop or never start, TTL stops, and storage grows without warning.

What this means

ClickHouse evaluates TTL expressions only during merge operations. When the merge scheduler combines parts, it checks whether rows exceeded their TTL interval. Depending on table settings, it either drops the entire part if every row expired, or rewrites the part to exclude expired rows. Both paths require a merge. If the background merge pool is saturated, disk space is too low for temporary merge output, or a partition has no new inserts to trigger merge selection, expired rows remain indefinitely. The system does not log a warning for TTL skips; the only visible symptom is growing storage.

ClickHouse unauthorized DROP TABLE: auditing DDL and privilege anomalies

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse unauthorized DROP TABLE: auditing DDL and privilege anomalies

A production table disappears. An application returns “table does not exist.” In ClickHouse, DROP TABLE removes the table definition and MergeTree data parts from disk immediately. On replicated tables, a single ON CLUSTER command propagates through the coordination service and can erase the table everywhere before you intervene. There is no native undo.

This guide is for finding out what happened, determining blast radius, and closing the gaps.

ClickHouse ZooKeeper session has expired: causes, recovery, and tuning

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse ZooKeeper session has expired: causes, recovery, and tuning

When system.replicas shows is_session_expired = 1, the replica has lost its ZooKeeper session and stopped participating in replication. It rejects inserts, coordinated merges, and distributed DDL. Depending on quorum and load-balancer configuration, writes may shift silently to other replicas or halt for entire shards.

is_session_expired often appears alongside is_readonly = 1, but the two are distinct. Session expiration means the coordination session is dead. Read-only means the replica refuses writes. Session expiration is the most severe because it breaks the replica’s contract with the cluster.

Cloud Foundry

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Foundry Firehose

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Foundry Firehose Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Foundry Firehose Monitoring

What Is Cloud Foundry Firehose?

Cloud Foundry Firehose is a powerful component of the Cloud Foundry platform providing a single interface to access a variety of operational data streams. It serves as the backbone for collecting metrics, logs, events, and other crucial data about the system, making it indispensable for effective cloud service management and troubleshooting.

Monitoring Cloud Foundry Firehose With Netdata

To monitor Cloud Foundry Firehose effectively, Netdata leverages an openmetrics (Prometheus) exporter. This integration makes it possible to ingest real-time data from any Prometheus exporter, providing users with automated dashboards, alerts, and more, all without the need for a Prometheus server or Grafana. By using Netdata’s Cloud Foundry Firehose monitoring tool, DevOps teams can visualize live metrics and troubleshoot issues much faster, enhancing the overall reliability and efficiency of the Cloud Foundry environment.

Cloud Foundry Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Foundry Monitoring

What Is Cloud Foundry?

Cloud Foundry is an open-source platform as a service (PaaS) providing a choice of cloud service providers, development frameworks, and application services. With Cloud Foundry, developers can efficiently manage application deployment, lifecycle, and scaling. This platform supports a variety of languages and frameworks, including Java, Node.js, Ruby, and Python, making it versatile for many development needs.

Monitoring Cloud Foundry With Netdata

To monitor Cloud Foundry, Netdata utilizes an openmetrics (Prometheus) exporter. This capability allows Netdata to ingest data efficiently from any Prometheus exporter, providing users with automated dashboards and alerting systems without the hassle of managing a separate Prometheus server or Grafana. This seamless integration ensures that monitoring gets technical teams actionable insights promptly, optimizing their operations and infrastructure management.

Cloudgenix SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

CloudWatch Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

CloudWatch Monitoring

What Is CloudWatch?

Amazon CloudWatch is a powerful monitoring and management service offered by AWS that provides data and actionable insights to monitor applications, understand and respond to system-wide performance changes, optimize resource utilization, and gain a unified view of operational health. It enables DevOps engineers, IT admins, and developers to track metrics, monitor log files, and set alarms to quickly react to potential performance issues.

Monitoring CloudWatch With Netdata

To monitor CloudWatch with ease and precision, Netdata employs the openmetrics (prometheus) exporter. The great advantage of using Netdata as a CloudWatch monitoring tool is that it can ingest data from any Prometheus exporter. This setup allows for automated dashboards and alerts, which eliminate the need for a standalone Prometheus server or Grafana. By bridging these functionalities, tools for monitoring CloudWatch become highly intuitive and efficient.

ClusterControl CMON

Mon, 01 Jan 0001 00:00:00 +0000

ClusterControl CMON Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

ClusterControl CMON Monitoring

What Is ClusterControl CMON?

ClusterControl CMON is a robust database management platform designed to streamline the management and monitoring of database clusters. By providing critical insights into database health, performance, and security, it serves as an invaluable tool for IT administrators and developers who need to maintain high availability and performance.

Monitoring ClusterControl CMON With Netdata

Netdata offers a seamless solution to monitor ClusterControl CMON using its advanced metrics collection capabilities. To monitor ClusterControl CMON, Netdata uses an openmetrics (Prometheus) exporter, allowing you to harness the benefits of Prometheus without the complexity of setting up a Prometheus server. By ingesting data from any Prometheus exporter, Netdata provides automated dashboards, alerts, and more. This streamlined setup means you do not need additional services like Grafana, simplifying your monitoring stack while ensuring detailed insights into your ClusterControl CMON environment.

CockroachDB

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB /health?ready=1: load balancer checks, draining, and impaired nodes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB /health?ready=1: load balancer checks, draining, and impaired nodes

CockroachDB exposes a readiness endpoint at GET /health?ready=1 on its HTTP port (default 8080). Load balancers and Kubernetes readiness probes use it to decide whether to route SQL traffic to a node. The node returns HTTP 200 when ready and HTTP 503 when not. This signal separates “process is alive” from “this node should receive client connections.”

The most common operator mistake is using a plain TCP check against the SQL port (26257) or the plain /health endpoint instead of /health?ready=1. A TCP check succeeds as long as the port is listening, telling you nothing about whether the node is draining, write-stalled, or GC-thrashing. The plain /health endpoint returns 200 whenever the process is running, regardless of draining state. Neither is safe for routing decisions.

CockroachDB admission control throttling: queue depth, store-write, and capacity headroom

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB admission control throttling: queue depth, store-write, and capacity headroom

When admission control starts queuing requests, p99 latency climbs while throughput stays flat or degrades. The cluster hasn’t crashed, disks aren’t full, and CPU may not be saturated. An internal flow-control system decided the node or store is at capacity and started holding work back to protect itself.

Admission control (v21.2+, enabled by default since v22.1) regulates work through five queues: kv, sql-kv-response, sql-sql-response, elastic-cpu, and store-write. Each gates a different class of work with distinct triggers. Knowing which queue is deep and why is the difference between a five-minute diagnosis and a multi-hour investigation into “why is the database slow.”

CockroachDB backup job failures: RPO breaches, duration trends, and stuck jobs

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB backup job failures: RPO breaches, duration trends, and stuck jobs

CockroachDB scheduled backups run through the internal jobs system, visible via crdb_internal.jobs (backed by system.jobs). When a scheduled backup fails silently, stalls indefinitely, or grows so slowly that it cannot complete within its interval, your recovery point objective (RPO) is at risk. The failure is insidious: backups often appear healthy until you realize the last successful completion was 36 hours ago.

CockroachDB certificate expired: TLS handshake failures and online rotation

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB certificate expired: TLS handshake failures and online rotation

CockroachDB uses mutual TLS for every connection: node-to-node, client-to-node, and admin UI. There is no plaintext fallback and no grace period. When a certificate expires, affected connections fail immediately.

Node certificate expiry prevents nodes from completing TLS handshakes with each other, which can look like a network partition or quorum loss. Client certificate expiry prevents applications from connecting. CA certificate expiry invalidates the entire trust chain at once, breaking every connection simultaneously.

CockroachDB changefeed lag: changefeed_max_behind_nanos and the GC time bomb

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB changefeed lag: changefeed_max_behind_nanos and the GC time bomb

Changefeed lag in CockroachDB starts as a consumer latency problem and ends as a disk space emergency. The changefeed_max_behind_nanos gauge measures how far behind your CDC feeds have fallen, but the real danger is what follows: a stalled changefeed holds a protected timestamp that prevents MVCC garbage collection, and dead data accumulates silently until the disk fills. The cluster appears healthy from the SQL side. Queries are fast, nodes are up, ranges are available. But every deleted row and every overwritten value stays on disk because GC cannot reclaim the space.

CockroachDB clock skew cascade: how shared NTP drift causes quorum loss

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB clock skew cascade: how shared NTP drift causes quorum loss

Multiple CockroachDB nodes crashed overnight. The logs show “clock synchronization error: this node is more than 500ms away from at least half of the known nodes.” You restart them, and they crash again. Some ranges are now unavailable. The cluster is losing quorum.

A shared NTP failure caused multiple nodes to drift past CockroachDB’s self-termination threshold in quick succession. Single-node clock skew is bad but recoverable. Multi-node skew from a shared NTP source can take down quorum faster than the cluster can heal.

CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB clock synchronization error: this node is more than 500ms away from at least half of the known nodes

You see this fatal log line on a CockroachDB node:

clock synchronization error: this node is more than 500ms away from at least half of the known nodes

The process exits immediately. If the node is managed by systemd, Kubernetes, or a process supervisor, it restarts and crashes again. The crash-loop continues until the clock problem is fixed.

CockroachDB clock_offset_meannanos high: catching clock drift before self-termination

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB clock_offset_meannanos high: catching clock drift before self-termination

clock_offset_meannanos measures the mean clock offset between a CockroachDB node and its peers. When it climbs, you are on a path that ends in silent performance degradation from widened read uncertainty windows, or a node self-terminating to preserve data consistency.

The thresholds are unforgiving. CockroachDB uses a default --max-offset of 500ms. A node self-terminates when its mean offset exceeds 80% of that value (400ms) relative to a majority of peers. But a constant 200ms offset, stable and below any threshold, still doubles the uncertainty interval for every read. Transactions silently restart more often, P99 read latency creeps up, and nobody suspects the clock.

CockroachDB compaction backlog growing: when Pebble can't keep pace with writes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB compaction backlog growing: when Pebble can’t keep pace with writes

Pebble’s background compaction threads sometimes fall behind foreground writes. In CockroachDB, this does not cause immediate failure. SSTable files accumulate in Level 0 and the compaction queue, read amplification rises, and the node drifts toward write stalls. Because the database continues to serve traffic, the backlog is easy to miss until it becomes severe.

The earliest visible sign is a gentle upward trend in Level 0 file counts or marked-for-compaction files over hours. These metrics fluctuate with workload bursts, but a sustained upward slope means the store is consuming headroom. Healthy operation requires compaction throughput at least twice the write ingestion rate. Less than that leaves no margin for bursts, MVCC garbage collection, or rebalancing.

CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB connection storm after failover: reconnect stampedes and surviving-node overload

When a CockroachDB node dies, every client connected to it reconnects at the same time. Without jittered backoff in client connection pools, hundreds or thousands of new connections land on the surviving nodes within seconds. The survivors are not overwhelmed by additional query load. They are overwhelmed by connection overhead: per-connection goroutines, session memory allocations, TLS handshakes, and SQL planner initialization.

CockroachDB consistency check failed: checksum mismatches and confirmed corruption

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB consistency check failed: checksum mismatches and confirmed corruption

When the background consistency checker detects a checksum mismatch between replicas of a range, CockroachDB logs a fatal error and the node terminates. This is not a transient condition. The database is telling you that data integrity is compromised.

The definitive message: “consistency check failed with N inconsistent replicas” or a Pebble-level “checksum mismatch” error. Both originate from CockroachDB’s consistency checker subsystem, not from unrelated log noise. This article covers the failure mechanism, the recovery path, and the signals that help you scope the damage.

CockroachDB context deadline exceeded: timeouts, slow ranges, and overload

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB context deadline exceeded: timeouts, slow ranges, and overload

“context deadline exceeded” tells you something is slow. It does not tell you what. The same error appears whether a leaseholder is overloaded, a network path is congested, L0 sublevels are climbing past write-stall territory, or the application set a 2-second statement timeout on a query that normally takes 50 milliseconds.

Treat the error as a symptom, not a diagnosis. The diagnostic question is: which layer is slow?

CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB CPU saturation: Raft ticking, SQL execution, and the per-node ceiling

CockroachDB is CPU-hungry by design. Every range runs its own Raft state machine. Every SQL statement parses, plans, and executes through Go. Compaction, encryption-at-rest, and checksumming all burn cycles. When CPU saturates, the failure mode is not a clean slowdown: Raft heartbeats get delayed, admission control starts queuing, GC pauses lengthen, and the node risks losing liveness. The cluster average can look healthy while one leaseholder melts.

CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB detecting hot ranges: per-range QPS, CPU asymmetry, and the Hot Ranges page

Hot ranges are the most common performance bottleneck in CockroachDB that does not surface in aggregate metrics. The leaseholder model routes all reads and writes for a range through a single node. When one range receives disproportionate traffic, that node saturates while the rest of the cluster idles. The cluster-wide CPU average looks healthy. The per-node breakdown tells a different story.

CockroachDB disk space running out: capacity_available trends and the 20% rule

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB disk space running out: capacity_available trends and the 20% rule

When a CockroachDB store runs out of disk space, the node cannot accept writes, cannot compact its LSM tree (the operation that would reclaim space), and enters a downward spiral that typically requires operator intervention. The capacity_available metric tracks remaining free space per store, but raw free space alone does not tell the full story. MVCC garbage, protected timestamps, compaction space amplification, and replica rebalancing all consume space faster than headline data growth suggests.

CockroachDB disk stall detected: storage_disk_stalled and node self-termination

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB disk stall detected: storage_disk_stalled and node self-termination

When storage_disk_stalled goes nonzero on a CockroachDB node, the Pebble storage engine has detected that disk I/O is no longer completing within the expected time window. The node is on a countdown to self-termination. This is not a performance degradation warning. It is a safety mechanism preparing to fire.

CockroachDB writes every committed transaction through a write-ahead log (WAL). The WAL fsync is on the critical path for every write: Raft cannot acknowledge a commit until the log entry is persisted to disk. If that fsync blocks for long enough, the node cannot process Raft heartbeats, cannot commit writes, and cannot renew its liveness record. Rather than continue operating in a state that could produce data inconsistency, CockroachDB terminates the process.

CockroachDB error 53200: SQL memory budget exhausted and query rejection

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB error 53200: SQL memory budget exhausted and query rejection

Error 53200 is CockroachDB’s PostgreSQL-compatible signal that the per-node SQL memory budget has run out. Applications see SQLSTATE 53200 (“insufficient resources”) with text such as “memory budget exceeded” and byte counts showing what was requested, what is allocated, and what the budget allows.

The SQL memory budget is enforced per-node, bounded by --max-sql-memory (default 25% of system RAM). When a node’s SQL execution layer exhausts its budget, queries on that node either spill to temporary disk storage (slow) or are rejected outright with 53200. One large analytical query on a single gateway node can starve every other session connected to that node, even if the rest of the cluster has memory to spare.

CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB file descriptor exhaustion: SSTables, connections, and ulimit

When CockroachDB hits its file descriptor limit, multiple subsystems fail at once. New SQL client connections are refused. SSTable file opens fail with I/O errors. Inter-node gRPC connections fail, destabilizing Raft consensus. The node may crash, stall, or refuse to start. In the logs you will see “too many open files” errors and, if the limit is below the startup minimum, the message “open file descriptor limit of X is under the minimum required Y”.

CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB Go GC pauses high: when garbage collection threatens Raft heartbeats

GC pause time spiking, node liveness flapping, lease transfers churning. SQL latency oscillates between acceptable and terrible. The cluster is alive but unstable, and the oscillation pattern is the tell.

CockroachDB runs on the Go runtime, which performs stop-the-world garbage collection pauses. When those pauses grow long enough, they block the Raft heartbeat loop. The node stops responding to heartbeats for the duration of the pause. If the pause is long enough, the cluster declares the node dead and redistributes its leases. Then the pause ends, the node recovers, leases come back, the heap grows again, and the cycle repeats.

CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB hot range bottleneck: one leaseholder saturated while the cluster idles

One node runs hot while the rest of the cluster idles. CPU on that node is 80%+, but others hover around 20-30%. SQL latency is elevated, but only for specific tables. Transaction retry rates climb with writetooold as the dominant restart cause. No range is unavailable, no node has lost liveness, and disk I/O is within normal bounds. The cluster has aggregate capacity, but a single range is funneling all its traffic through one leaseholder.

CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB intent accumulation cascade: abandoned transactions and intentcount growth

When intentcount and intentbytes climb and refuse to come down, your cluster is accumulating unresolved write intents from transactions that never committed or rolled back. Every subsequent transaction touching those keys must stop, resolve the intent, then proceed. At scale, the cluster spends more CPU and I/O resolving old intents than executing new work.

This is the intent accumulation cascade. It is one of CockroachDB’s subtler failure modes because the cluster looks healthy on infrastructure metrics: nodes are live, ranges are available, disk I/O is within bounds. The damage shows up in transaction latency and throughput, not in availability signals.

CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB LSM compaction death spiral: L0 sublevels, read amplification, and write stalls

SQL P99 latency jumps from milliseconds to seconds. KV write latency climbs. Nodes transfer leases. Logs show Pebble write stall messages. This is the LSM compaction death spiral: writes outpace the storage engine’s ability to compact data from Level 0 down the LSM tree. L0 sublevels stack up, read amplification rises, and the node eventually stalls writes to protect itself. By the time write stalls appear, the node is already at risk of losing Raft leases and appearing partially unavailable. This guide shows how to diagnose the spiral, stop it, and prevent it.

CockroachDB memory pressure, GC thrashing, and Raft liveness failure

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB memory pressure, GC thrashing, and Raft liveness failure

A CockroachDB node loses liveness in a repeating pattern: it drops out, the cluster redistributes its leases, it recovers, reacquires leases, then drops again. Each cycle lasts seconds to minutes. Application queries see intermittent timeouts, ambiguous results, and latency spikes that correlate with the node’s oscillation. The DB Console shows the node flapping between live and not-live states.

This is the memory pressure to GC thrashing to Raft liveness failure cascade. The Go runtime heap grows until garbage collection pauses become long enough to prevent the node from renewing its liveness heartbeat. Once the heartbeat interval lapses, the cluster marks the node dead and moves its leases. When GC completes and memory is freed, the node recovers and reacquires leases, restarting the cycle.

CockroachDB Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB Monitoring

CockroachDB is a cloud-native distributed SQL database designed to scale horizontally and survive datacenter failures. Monitoring CockroachDB is crucial for maintaining the health, performance, and reliability of your database infrastructure. Netdata provides a comprehensive CockroachDB monitoring tool that enables real-time visibility and enhanced troubleshooting capabilities to ensure optimal database operations.

What Is CockroachDB?

CockroachDB is a distributed SQL database, similar in functionality to Google Spanner, designed to deliver strong fault tolerance, consistency, and scalability across distributed environments. Its architecture allows for seamless scaling and resilience, making it an ideal choice for modern, high-demand database operations.

CockroachDB monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB monitoring checklist: the signals every production cluster needs

CockroachDB layers SQL execution on a replicated KV store backed by Pebble LSM trees, Raft consensus, and MVCC concurrency control. Each subsystem has distinct failure modes, and interactions between them create cascades that single-signal monitoring cannot catch. A cluster can show healthy CPU, adequate disk space, and sub-millisecond SQL latency while L0 sublevels climb toward write stalls or clock offset drifts toward the self-termination threshold.

CockroachDB monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB monitoring maturity model: from survival to expert

CockroachDB failures rarely announce themselves through a single metric. A slow disk turns into L0 compaction debt, which stalls writes, which drops Raft proposals, which makes ranges unavailable. A drifting clock raises transaction retry rates long before any node self-terminates. To run this database safely, you need layered observability that matches the system’s own layers: storage engine, Raft replication, distributed SQL, and transaction execution.

CockroachDB MVCC garbage growing: tombstones, GC lag, and silent disk growth

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB MVCC garbage growing: tombstones, GC lag, and silent disk growth

MVCC garbage accumulation produces no errors, no latency spikes, and no user-visible symptoms until the disk fills. By then, you may be hours away from a compaction death spiral that takes the node or cluster offline.

CockroachDB retains old MVCC versions of every key until garbage collection removes them after the GC TTL window (default 25 hours). Deletes and updates create tombstones: markers that a key was removed at a specific timestamp. These tombstones persist in the LSM tree until MVCC GC runs and Pebble compaction physically removes them downstream. If anything blocks that cleanup, dead data accumulates silently.

CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB node liveness failure: heartbeats, lease redistribution, and flapping

Lease transfer spikes, briefly unavailable ranges, and client errors such as ambiguous results or connection resets indicate node liveness failure. In the logs, nodes transition to not-live and back within seconds. When the cluster decides a node cannot renew its liveness heartbeat, it redistributes leases. If the node recovers fast enough to renew but not fast enough to stay healthy, it flaps: an oscillating state more destructive than a clean outage because it repeatedly interrupts in-flight work and prevents stable failover.

CockroachDB out of memory: sys_rss, --cache, --max-sql-memory, and the OOM killer

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB out of memory: sys_rss, –cache, –max-sql-memory, and the OOM killer

A CockroachDB node disappears. No graceful shutdown, no drain sequence, no error in the SQL layer. The process is gone, and dmesg shows the kernel OOM killer selected it. Or in Kubernetes, the pod restarts with reason OOMKilled.

The root cause is almost always a mismatch between what CockroachDB thinks it can allocate and what the container or host actually allows. CockroachDB partitions its memory into two manually-sized pools: the Pebble block cache (--cache) and the SQL execution budget (--max-sql-memory). The Go garbage collector only manages the Go heap. CGo allocations, primarily the Pebble block cache and memtables, are manually managed. When the sum of these pools plus runtime overhead exceeds the container or host limit, the OOM killer intervenes.

CockroachDB Pebble write stalls: when the storage engine refuses writes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB Pebble write stalls: when the storage engine refuses writes

When application writes time out or return ambiguous errors, check CockroachDB logs for pebble: write stall and watch storage_write_stalls. Pebble pauses writes to a store when Level 0 compaction debt exceeds safe thresholds. Until compaction drains L0, that store cannot accept new writes.

Write stalls are the most severe storage signal in CockroachDB. A brief stall during bulk loading may be harmless, but sustained stalls at one per second mean the node cannot meet its Raft obligations. The node may still serve reads, but it can lose Raft leadership because it cannot append log entries. That cascades into lease transfers and temporary range unavailability.

CockroachDB protected timestamp GC stall: stalled changefeeds that silently fill the disk

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB protected timestamp GC stall: stalled changefeeds that silently fill the disk

Disk space is growing on your CockroachDB cluster and nothing explains it. capacity_available trends downward hour after hour, but live data size has not changed. No bulk imports, no schema changes, no write spike. The SQL layer looks healthy: latency is fine, throughput is normal. But the disk is filling.

A CDC changefeed, a backup job, or another internal operation has created a protected timestamp (PTS) record that prevents MVCC garbage collection from reclaiming old data versions. The job has stalled, paused, or entered a retry loop, but the PTS record persists. Dead MVCC data accumulates underneath it. Disk usage grows linearly until free space drops below what compaction needs, at which point L0 sublevels spike, write stalls begin, and nodes lose liveness.

CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB Raft liveness failure cascade: slow node, lost leases, rolling unavailability

You notice brief, repeating SQL timeouts or ambiguous result errors across your application. In the CockroachDB DB Console, one node is still marked as running, yet its range lease count is oscillating and lease transfers are spiking. This is not a clean node crash. It is a Raft liveness failure cascade: a node has become slow enough to miss Raft heartbeats, but not dead enough to fail permanently. The cluster keeps trying to move leases away and back, creating rolling windows of unavailability.

CockroachDB Raft snapshot storm: when followers fall too far behind

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB Raft snapshot storm: when followers fall too far behind

Raft snapshots are CockroachDB’s fallback when incremental log replication is no longer possible. A leader sends a full copy of a range (up to 512 MiB by default) to a follower that has fallen so far behind that the log entries it needs have already been truncated. In steady state, this almost never happens. When it starts happening at scale, it creates a resource-draining cycle that can degrade an entire cluster.

CockroachDB range count per node too high: Raft ticking as a scaling dimension

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB range count per node too high: Raft ticking as a scaling dimension

CPU utilization is climbing across one or more CockroachDB nodes, and the usual explanations do not fit. Query throughput is flat. Disk I/O looks healthy. Admission control is not queuing. L0 sublevels are in single digits. CPU keeps trending upward, and the only change is that the database holds more data.

The likely cause is the scaling dimension most teams never instrument: range count per node. Every range in CockroachDB is an independent Raft consensus group. Even idle, each range’s Raft state machine ticks at a fixed interval, consuming CPU separate from query execution, compaction, or any workload-driven process. This baseline overhead grows linearly with range count per node.

CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB range unavailable: diagnosing ranges_unavailable and recovering quorum

The Prometheus gauge ranges_unavailable is the hardest of hard signals in CockroachDB. When it rises above zero and stays there, some slice of your keyspace has no leaseholder or has lost Raft quorum. Reads and writes to those ranges block or fail, and applications see errors, retries, or timeouts. If the unavailable ranges include system keyspaces, the impact is not limited to user tables. The cluster may lose the ability to heartbeat node liveness, execute schema changes, or route requests.

CockroachDB ReadWithinUncertaintyInterval restarts: the near-diagnostic signal of clock skew

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB ReadWithinUncertaintyInterval restarts: the near-diagnostic signal of clock skew

When CockroachDB reports readwithinuncertainty as a transaction restart cause, you have a clock synchronization problem. This restart cause does not appear in meaningful quantities for any other reason. Any sustained nonzero rate, even well below the self-termination threshold, indicates that NTP or the underlying clock source is not keeping node clocks aligned.

The error stems from CockroachDB’s Hybrid Logical Clock (HLC) design. Each transaction receives a timestamp from its gateway node’s HLC, which combines physical wall-clock time with a logical counter. When a read on one node encounters a write from a transaction that started on a different node, and the two nodes’ clocks are not aligned, the database cannot determine which transaction began first. CockroachDB conservatively restarts the reading transaction at a higher timestamp. Under SERIALIZABLE isolation, this restart requires client-side retry logic and adds directly to tail latency.

CockroachDB replica unavailable: lost quorum and stuck Raft groups

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB replica unavailable: lost quorum and stuck Raft groups

ranges_unavailable has gone nonzero. Clients are seeing “replica unavailable” errors. Some portion of your keyspace cannot be read or written. This is an active availability incident.

In CockroachDB, every range (a ~512 MiB slice of the keyspace) is replicated across multiple nodes. A write requires quorum acknowledgment from a majority of replicas before it commits. When too few replicas are reachable, the range loses quorum and cannot serve reads or writes. The ranges_unavailable metric tracks exactly this condition: ranges with no leaseholder or with lost Raft quorum.

CockroachDB result is ambiguous: ambiguous commit errors and how to handle them

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB result is ambiguous: ambiguous commit errors and how to handle them

Your application received result is ambiguous (PostgreSQL error code 40003, statement_completion_unknown) from CockroachDB. The transaction may have committed. It may have been aborted. The database does not know, and neither do you.

This is not a bug. It is a correctness property of CockroachDB’s distributed commit protocol. When the transaction coordinator loses contact with the leaseholder during the final commit step, or during the last statement of an implicit transaction, it cannot determine whether the commit succeeded. Rather than guess and risk a silent double-apply, CockroachDB surfaces the ambiguity.

CockroachDB RETRY_WRITE_TOO_OLD: the most common contention restart and how to kill it

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB RETRY_WRITE_TOO_OLD: the most common contention restart and how to kill it

Your cluster is healthy: nodes are live, ranges are available, disk space is fine, L0 sublevels are low. But transaction commit latency is climbing, P99 is widening, and the txn_restarts metric shows a rising rate tagged writetooold. Whether applications surface errors to users depends on whether their retry logic works.

RETRY_WRITE_TOO_OLD is the most common transaction restart cause under CockroachDB’s default SERIALIZABLE isolation. It is not an infrastructure problem. It is a contention problem rooted in schema design, key patterns, and transaction scope. Adding nodes does not help because the bottleneck is logical serialization, not physical capacity. Once you identify the conflicting keys and the transaction patterns driving them, writetooold restarts are fixable.

CockroachDB sequential primary key hotspot: SERIAL, timestamps, and write skew

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB sequential primary key hotspot: SERIAL, timestamps, and write skew

One node in your CockroachDB cluster runs hot while the others sit near idle. Write latency for specific tables climbs into hundreds of milliseconds. Transaction restarts tagged writetooold increase steadily alongside the insert rate. The cluster has plenty of aggregate CPU, memory, and disk headroom, but the workload cannot use it.

CockroachDB stores data in an ordered keyspace divided into ranges of approximately 512 MiB. Each range has a single leaseholder that serves all reads and coordinates all writes for keys in that range. When primary keys are sequential (SERIAL, auto-incrementing integers, timestamps), every new insert lands near the end of the keyspace, concentrated in the same range. That range’s leaseholder becomes a single-node bottleneck.

CockroachDB SQL error rate: XX000 internal errors and error-code triage

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB SQL error rate: XX000 internal errors and error-code triage

sql_failure_count is a single Prometheus counter with no error-code labels. A serialization conflict (40001) is expected noise in a contended workload. An internal error (XX000) is a database fault that needs immediate escalation. Treating the aggregate counter as one signal guarantees the wrong response.

This article covers how to break that aggregate apart by error code, when XX000 internal errors warrant paging, and how to triage the other common error classes.

CockroachDB SQL latency high: sql_service_latency, per-fingerprint diagnosis, and the layers

Mon, 01 Jan 0001 00:00:00 +0000

P99 sql_service_latency is climbing. Applications are timing out or retrying. The DB Console overview chart shows the spike, but the aggregate histogram cannot tell you which query is slow or which subsystem is adding the latency.

The most common diagnostic mistake is stopping at aggregate P99. A single slow query hiding among thousands of fast ones barely moves aggregate P99 but can take down an endpoint. A system-wide storage problem lifts all fingerprints together. Without per-fingerprint breakdown, you cannot distinguish “one query regressed” from “everything got slower.”

CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB storage_l0_sublevels climbing: the earliest warning of write stalls

storage_l0_sublevels climbing on a single store is the earliest actionable signal for CockroachDB storage distress. The cluster may still serve traffic with slightly elevated SQL latency and no active write stalls. L0 sublevels typically provide 10-30 minutes of warning before write stalls begin, but only when monitored per-store.

As L0 sublevels rise, Pebble consults more SSTables per read, read amplification increases nonlinearly, and compaction falls behind. Admission control shapes regular traffic at 5 sublevels and elastic traffic at 1 sublevel. Past 20 sublevels, write stalls become imminent. Because this metric is per-store, a node with multiple stores can mask a hot disk behind healthy node-level aggregates.

CockroachDB store is full: emergency disk recovery and why deletes don't free space

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB store is full: emergency disk recovery and why deletes don’t free space

A CockroachDB node’s store directory is full or nearly full, and the node may have already shut itself down to protect data integrity.

Deleting data will not help immediately. CockroachDB uses MVCC: a SQL DELETE writes tombstones at new timestamps rather than removing old data. Until those tombstones propagate through Pebble compaction, the old versions stay on disk. A large-scale delete can temporarily increase disk usage because the tombstones are written before old data is reclaimed.

CockroachDB too many connections: sql_conns, pooling, and goroutine pressure

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB too many connections: sql_conns, pooling, and goroutine pressure

CockroachDB has no hard built-in connection limit comparable to PostgreSQL’s max_connections. Connections accumulate until something else breaks: goroutine pressure, memory exhaustion, or file descriptor limits.

The sql_conns gauge tracks active pgwire (SQL) connections per node. With properly configured client-side pooling, this number stays modest. When it climbs past 1000-2000 per node, you almost certainly have a pooling problem upstream.

CockroachDB transaction contention storm: retries, intents, and tail latency

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB transaction contention storm: retries, intents, and tail latency

The cluster looks healthy. Node liveness is stable, ranges_unavailable is zero, disk I/O is within bounds, CPU is moderate across all nodes. But P99 SQL latency is climbing, the transaction abort rate is up, and applications are reporting timeout errors. The database is spending most of its energy waiting, retrying, and resolving conflicts rather than doing useful work.

This is a transaction contention storm. The infrastructure is fine; the workload is the problem. Multiple transactions are contending for the same keys, creating serialized hot paths. Each conflict generates retries under CockroachDB’s default SERIALIZABLE isolation, which in turn leaves write intents that block subsequent transactions. Under sustained load, the retry-intent-latency cycle becomes a positive feedback loop.

CockroachDB transaction retry rate high: breaking down txn_restarts by cause

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB transaction retry rate high: breaking down txn_restarts by cause

When application latency creeps upward without an obvious cause, or you start seeing PostgreSQL error code 40001 (serialization failure) in application logs, the transaction retry rate is likely climbing. CockroachDB exposes txn_restarts as a set of counters, each tagged by cause. The aggregate number tells you retries are happening. It does not tell you why.

The five dominant causes are writetooold, serializable, readwithinuncertainty, txnaborted, and txnpush. Three additional sub-metrics (asyncwritefailure, commitdeadlineexceeded, unknown) cover edge cases. A spike dominated by readwithinuncertainty is a clock infrastructure problem. A spike dominated by writetooold is a schema or application contention problem. The fix for one does nothing for the other.

CockroachDB TransactionRetryWithProtoRefreshError: RETRY_SERIALIZABLE causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB TransactionRetryWithProtoRefreshError: RETRY_SERIALIZABLE causes and fixes

The application logs show TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/.../0). SQLSTATE is 40001. Clients are timing out or failing outright. The cluster looks healthy: nodes are live, ranges are available, disk space is fine. The problem is in the transaction layer.

Under SERIALIZABLE isolation (CockroachDB’s default), the database uses optimistic concurrency. It proceeds as if no conflict exists, then detects conflicts at commit or statement boundary. When a transaction’s preemptive refresh fails because another transaction already committed a write to a key the first transaction read, CockroachDB emits RETRY_SERIALIZABLE. This is expected behavior, not a bug.

CockroachDB under-replicated ranges: ranges_underreplicated and the healing margin

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB under-replicated ranges: ranges_underreplicated and the healing margin

When ranges_underreplicated rises above zero, CockroachDB has ranges with fewer live replicas than the configured replication factor. The replicate queue is already working to heal them by transferring Raft snapshots to new target nodes, but until healing completes, those ranges have reduced or zero fault tolerance margin.

The key operational question is not whether under-replicated ranges exist (they will, transiently, after almost any node event), but whether the cluster is healing fast enough to stay ahead of the next failure. This is the healing margin: the buffer between the current replication state and the point where a range loses quorum and becomes unavailable.

CockroachDB WAL fsync latency high: the most direct write-path health signal

Mon, 01 Jan 0001 00:00:00 +0000

CockroachDB WAL fsync latency high: the most direct write-path health signal

Elevated storage_wal_fsync_latency means the storage device is taking too long to durably persist write-ahead log entries. Every write in CockroachDB must complete a WAL fsync before Raft can acknowledge it. Fsync latency is the gating signal for the entire write path.

On healthy local SSDs, WAL fsync P99 should be under 10ms. Sustained values above 50ms mean the device is struggling. Above 200ms, the node is approaching disk stall territory, where CockroachDB’s built-in stall detection may terminate the process to prevent data inconsistency. Disk stalls cause self-termination before utilization metrics react: a volume reporting 60% utilization can stall on an individual fsync for seconds if the underlying hardware is failing, the I/O scheduler is misbehaving, or a cloud provider is throttling IOPS.

Codan Limited SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Codex SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Codima Technologies Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cohesity Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cold-start topology: why your map is incomplete after a collector restart

Mon, 01 Jan 0001 00:00:00 +0000

Cold-start topology: why your map is incomplete after a collector restart

You restart your flow collector or topology engine for a routine upgrade. The process comes back up cleanly. The dashboard loads. But the topology map is half-empty, endpoint positions are wrong, and within minutes someone pages you asking why a security investigation points to the wrong switch port.

The root cause is not a bug. After a restart, the topology inference engine has no cached neighbor tables, no FDB entries, no ARP data, and potentially no flow templates. It must rebuild all of these from live polling and flow data before it can produce a reliable view. The window between restart and first complete topology ranges from a few minutes to over 30 minutes, depending on poll cadence, template refresh intervals, and which sources your topology engine fuses.

Collectd

Mon, 01 Jan 0001 00:00:00 +0000

Collectd Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Collectd Monitoring

What Is Collectd?

Collectd is a powerful tool for monitoring systems and applications. It collects performance data from your systems and networks, providing detailed insights to help troubleshoot issues and optimize performance. As a long-standing part of the open-source community, Collectd offers a comprehensive and extensible solution for those needing robust data collection capabilities.

Monitoring Collectd With Netdata

To monitor Collectd effectively, Netdata uses an OpenMetrics (Prometheus) exporter. This integration allows Netdata to ingest data directly from any Prometheus exporter without requiring a Prometheus server or Grafana setup. With Netdata, you gain access to automated dashboards, real-time alerts, and insightful metrics visualization, making it an ideal Collectd monitoring tool. Easily connect and start monitoring your systems using Collectd exporter combined with Netdata’s intuitive interface.

Collector CPU and TSDB write-queue saturation: the capacity signals

Mon, 01 Jan 0001 00:00:00 +0000

Collector CPU and TSDB write-queue saturation: the capacity signals

When a network monitoring collector saturates, the first visible symptom is rarely high collector CPU. It is traffic charts showing a decline during a traffic spike, an SNMP poll cycle drifting past its configured interval, or unexplained gaps in flow data. The degradation sits one to three subsystems downstream of the actual bottleneck, which is why collector-side incidents are frequently misdiagnosed.

Colubris Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Com21 SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Comap A S SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Comet System S R O SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Command_Timeout climbing: the drive is taking too long to respond

Mon, 01 Jan 0001 00:00:00 +0000

Command_Timeout climbing: the drive is taking too long to respond

SMART attribute ID 188 (Command_Timeout) is climbing. Two interpretation traps cause false alarms here, and the symptom overlaps with at least four distinct failure modes ranging from benign background maintenance to imminent controller death.

The first trap is vendor-specific raw value encoding. On Seagate drives, the raw value packs three 16-bit counters into a single 48-bit field, producing numbers in the billions that decode to single-event counts. Monitoring tools that read the raw value as one integer will page you for a single timeout during power-on sequencing.

Commend International GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Communication Company Nari Group Corporation Information Technology SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Community

Mon, 01 Jan 0001 00:00:00 +0000

Compaq SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Compellent Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Compex Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Computer Associates International SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Computer Network Technology SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Comtech Efdata Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Concord Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Concourse

Mon, 01 Jan 0001 00:00:00 +0000

Concourse Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Concourse Monitoring

What Is Concourse?

Concourse is a powerful, open-source CI/CD system that automates and streamlines your application development and deployment processes. Its clean and straightforward interface, coupled with its ability to handle complex workflows, makes it a popular choice among DevOps teams looking to enhance their software release cycles.

Monitoring Concourse With Netdata

Monitoring your Concourse environment is crucial for ensuring optimal pipeline performance and detecting issues before they affect application delivery. With Netdata, monitoring Concourse becomes seamless and efficient. Netdata leverages an openmetrics (Prometheus) exporter to collect comprehensive metrics from your Concourse instance. This means that you can ingest data directly from any Prometheus exporter, receiving automated dashboards, alerts, and more, without the need for a separate Prometheus server or Grafana setup.

Confmon Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Connection Technology Systems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Conntrack

Mon, 01 Jan 0001 00:00:00 +0000

Consul

Mon, 01 Jan 0001 00:00:00 +0000

Consul "ACL not found": requests rejected after a token or policy change

Mon, 01 Jan 0001 00:00:00 +0000

Consul “ACL not found”: requests rejected after a token or policy change

“ACL not found” appears as HTTP 403 responses carrying “ACL not found” or “token does not exist: ACL not found”. It surfaces in agent and server logs, sidecar injector output, and as failed service registrations, health check updates, or KV writes. The blast radius depends on which token is missing: a single application token breaks one service; a replication or agent token can break an entire datacenter’s authorization pipeline.

Consul "No cluster leader": every write is failing

Mon, 01 Jan 0001 00:00:00 +0000

Consul “No cluster leader”: every write is failing

When Consul cannot elect a Raft leader, every write fails. The error surfaces to clients as rpc error making call: No cluster leader, sometimes as no known leader. Service registrations hang, KV writes return errors, sessions and ACL tokens cannot be created, and health check state stops reaching the catalog.

Only stale reads continue to work: stale-mode API queries and DNS lookups (which default to allow_stale) return whatever the responding server last committed. Default and consistent reads fail because they require a leader, which can surface differently to callers depending on their read mode.

Consul "too many open files": file descriptor exhaustion on servers

Mon, 01 Jan 0001 00:00:00 +0000

Consul “too many open files”: file descriptor exhaustion on servers

When a Consul server hits its file descriptor ceiling, every consumer that needs a new FD fails immediately: RPC connections are refused, DNS listeners stop accepting queries, xDS streams to Envoy sidecars cannot open, and outbound health-check connections fail. Gossip probes time out and the failure detection protocol marks otherwise healthy peers as suspect or failed. A healthy-looking cluster a minute ago is now generating cascading pages.

Consul ACL resolution latency: token cache thrashing on every request

Mon, 01 Jan 0001 00:00:00 +0000

Consul ACL resolution latency: token cache thrashing on every request

Every authenticated Consul request pays an ACL resolution tax. When that tax is sub-millisecond, nobody notices. When the token cache cannot hold the working set, every request becomes a miss and that tax multiplies across DNS lookups, HTTP API calls, RPC forwarding, and Connect intention evaluation. The cluster keeps answering, but slowly, and the slowness is uniform across every authenticated path.

Consul anti-entropy not syncing: local agent state and the catalog drifting apart

Mon, 01 Jan 0001 00:00:00 +0000

Consul anti-entropy not syncing: local agent state and the catalog drifting apart

A service is registered on the local agent. The health check is passing locally. But dig myservice.service.consul returns nothing, the load balancer has no targets, and the catalog API shows no instances. The servers have a leader, gossip is healthy, Raft metrics look normal. The problem is between the agent and the catalog.

Consul’s catalog is authoritative on the servers. Each agent maintains its own local state (registered services, health checks, node metadata) and periodically reconciles that state with the server catalog through a background process called anti-entropy sync. The agent treats its local view as authoritative and pushes changes to the catalog. When this sync fails, the catalog retains the last-known state it received from that agent. A service registered locally never appears cluster-wide. A service deregistered locally continues to show up in DNS and API queries.

Consul blocking query accumulation: leaked watches that pile up goroutines

Mon, 01 Jan 0001 00:00:00 +0000

Consul blocking query accumulation: leaked watches that pile up goroutines

Goroutine count on your Consul servers is creeping upward. File descriptor usage follows the same trajectory. Neither reverses during quiet periods. Eventually the server hits its FD limit or goroutine scheduling overhead degrades everything: new connections are refused, health checks stop reaching the catalog, anti-entropy sync stalls, and the cluster cascades.

The culprit is leaked blocking queries. Consul’s watch mechanism and any client using the HTTP long-polling API (?index= and ?wait= parameters) holds a goroutine and an FD on the server for up to the wait timeout (5 minutes default, 10 minutes maximum). A fleet of consul-template instances, application-level watchers, or service mesh control planes can open thousands of concurrent blocking queries. This is fine when queries are properly opened and closed. The leak starts when they are not: a client crashes without cleanly closing its connection, a consul-template bug prevents goroutine cleanup, a watch configuration exceeds the server’s tracking capacity, or a go-memdb WatchSet bug causes goroutine proliferation.

Consul catalog bloat: too many services and checks slowing everything down

Mon, 01 Jan 0001 00:00:00 +0000

Consul catalog bloat: too many services and checks slowing everything down

DNS latency is creeping up. API calls to /v1/catalog/services take longer than they used to. Server restarts that used to take 20 seconds now take minutes. The Raft snapshot on disk keeps growing, and so does server RSS. No single failure point, no stack trace, no page. Just a slow, compounding drag on everything Consul does.

This is catalog bloat. Registered service instances and health checks grow the in-memory state store, the on-disk Raft snapshot, anti-entropy reconciliation work, and catalog-scan latency for every DNS and API query. The cost compounds week over week until a snapshot creation tips a leader election, a restart takes long enough to miss a deploy window, or DNS p99 crosses the threshold where applications time out on service discovery.

Consul catalog poisoning: services registered from unknown nodes

Mon, 01 Jan 0001 00:00:00 +0000

Consul catalog poisoning: services registered from unknown nodes

A service instance appears in your catalog on a node you have never deployed. Its ServiceAddress points somewhere you do not control. That is catalog poisoning: a registration that did not come from your infrastructure.

Without strict ACLs, any process that can reach the Consul HTTP API can call the registration endpoints and add an instance for any service name. Consumers resolving discovery through Consul DNS or the catalog and health APIs will route a fraction of their traffic to that address. In a non-Connect deployment there is nothing in the data path that verifies the instance is what it claims to be.

Consul client rpc failed: agents alive but the catalog is going stale

Mon, 01 Jan 0001 00:00:00 +0000

Consul client rpc failed: agents alive but the catalog is going stale

A client agent’s consul.client.rpc.failed counter ticks up. The agent process is running, gossip reports the node as alive, local health checks execute on schedule, and consul members lists the node as healthy. The server cluster looks fine: leadership is stable, Raft commit times are normal, and there is no election noise. Nothing on the standard dashboard is red.

Consul Connect "no healthy upstream": Envoy has no endpoints to route to

Mon, 01 Jan 0001 00:00:00 +0000

Consul Connect “no healthy upstream”: Envoy has no endpoints to route to

When an application in a Consul Connect mesh receives HTTP 503 with response_flags=UH in the Envoy access log, Envoy’s router rejected the request because it had zero eligible endpoints for the upstream cluster. The downstream service is fine, the network is fine, and the request never left the sidecar. Somewhere between the Envoy admin port, the local Consul agent, the Consul servers, and the upstream instances, the endpoint set has gone empty, all-unhealthy, or silently ejected.

Consul Connect CA rotation failure: a root roll that never finished

Mon, 01 Jan 0001 00:00:00 +0000

Consul Connect CA rotation failure: a root roll that never finished

Consul Connect CA root rotation promotes a new root, cross-signs the new intermediate against the old root, lets both roots coexist while leaf certificates roll over, and retires the old root only after the last leaf signed by it expires. When that rollout never completes, the first thing you notice is intermittent mTLS failures between specific service pairs, hours after the rotation was triggered, with no obvious network or config change to blame.

Consul Connect certificate expired: mTLS handshakes failing across the mesh

Mon, 01 Jan 0001 00:00:00 +0000

Consul Connect certificate expired: mTLS handshakes failing across the mesh

Consul Connect relies on short-lived leaf certificates for service-to-service mTLS. When those certificates expire without rotating, the failure is cliff-edged: handshakes that worked seconds ago start failing, Envoy logs fill with TLS errors, and dependent services stop communicating. The blast radius grows over minutes to hours as each leaf certificate reaches its own expiry independently.

The first symptom is usually intermittent: a specific upstream starts returning connection resets or 5xx errors. As more leaves expire, failures fan out across service pairs until the mesh is broadly broken. Expiry alone is a countdown with no early warning. The signal that matters earlier is renewal success: are leaves actually being re-issued, or is the rotation pipeline silently broken while the clock runs out?

Consul critical health checks spiking: real outage or broken checks?

Mon, 01 Jan 0001 00:00:00 +0000

Consul critical health checks spiking: real outage or broken checks?

The alert fires when /v1/health/state/critical jumps from single digits to dozens or hundreds of checks in minutes. The page says “services are unhealthy,” but that is ambiguous: either real services are down, or the checks themselves are broken. Pick the wrong fork and you burn an hour on the wrong dependency.

The first decision is not “what failed.” It is “is this a service-level event, a node-level event, or a monitoring-level event?” The cheapest signal is the check Output text field, which carries the raw error from check execution. It holds strings like connection refused, i/o timeout, certificate has expired. That field usually resolves the diagnosis in seconds, and most teams ignore it until their second or third incident.

Consul cross-datacenter query failure: prepared-query failover masking a DC outage

Mon, 01 Jan 0001 00:00:00 +0000

Consul cross-datacenter query failure: prepared-query failover masking a DC outage

A Consul prepared query with cross-DC failover does exactly what it was configured to do: it silently forwards lookups to a remote DC when local results are empty. Consumers receive valid service discovery results, applications keep connecting, and no error fires. The local DC may have zero healthy instances, and nobody on the consumer side knows.

The DNS response is identical whether nodes came from the local DC or a remote one. There is no EDNS flag, no source record, no indicator in the answer. Receiving results does not mean local health.

Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog

Mon, 01 Jan 0001 00:00:00 +0000

Consul DeregisterCriticalServiceAfter: instances vanishing from the catalog

A service that was merely slow to recover has disappeared from the Consul catalog. Healthy-instance count dropped to zero. DNS returns empty results. Load balancers have no targets. Yet the service process is running, the host is up, and the Consul agent on that host is healthy and gossiping. DCSA did exactly what you configured.

DeregisterCriticalServiceAfter (DCSA) is an agent-side reaper that removes a service instance and its checks from the catalog once the check has been critical for a configured duration. It is deliberate garbage collection for stale entries. Leave it unset and critical checks plus their output strings accumulate unbounded in the state store, growing memory and slowing snapshots.

Consul DNS latency high: slow lookups stalling connections and failovers

Mon, 01 Jan 0001 00:00:00 +0000

Consul DNS latency high: slow lookups stalling connections and failovers

Slow Consul DNS lookups rarely look like a DNS problem at first. Applications see connection timeouts, retries, and slow startup. Load balancers see health check flapping. Service mesh sidecars see upstream resolution failures. The DNS layer is invisible to most teams until consul.dns.domain_query crosses the page threshold.

The first decision in a Consul DNS latency incident is structural: is the slowness in the server cluster (Raft, disk, catalog size) or in the DNS path itself (TTL, agent CPU, query complexity, downstream resolvers)? The signals are all in agent telemetry, but you have to know which ones to correlate.

Consul DNS SERVFAIL: service discovery is broken for your applications

Mon, 01 Jan 0001 00:00:00 +0000

Consul DNS SERVFAIL: service discovery is broken for your applications

SERVFAIL from Consul’s DNS interface on port 8600 means service discovery is broken for every application resolving names through it. Applications see connection refused, timeouts, and cascading failovers. The symptom is broad, but the root cause lives along one of four layers: the downstream resolver (if present), the Consul agent’s built-in DNS server, the agent-to-server RPC channel, or the server’s catalog state.

Consul Go GC pauses: stop-the-world stalls that disturb Raft timing

Mon, 01 Jan 0001 00:00:00 +0000

Consul Go GC pauses: stop-the-world stalls that disturb Raft timing

Consul servers are Go binaries, and Go’s garbage collector still has stop-the-world phases despite doing most of its work concurrently. On a server with a large live heap and a high allocation rate, those pauses can stretch into hundreds of milliseconds. The Raft leader sends heartbeats from a goroutine in the same process. When a pause runs long enough, the leader stops heartbeating; followers see consul.raft.leader.lastContact climbing toward the election timeout and start a new election. Writes block for the duration.

Consul goroutine count climbing: the leak behind slow resource exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

Consul goroutine count climbing: the leak behind slow resource exhaustion

consul.runtime.num_goroutines is a gauge of how many concurrent operations the process is juggling: every blocking query, every gRPC or xDS stream, every health check handler runs inside a goroutine. In a healthy cluster the number is baseline-dependent but flat. When it creeps upward hour over hour, something is spawning goroutines and not cleaning them up.

The absolute number is a distraction. A medium cluster idles between a few hundred and 20,000 goroutines; a large Connect deployment legitimately runs higher. The trend is the signal. Monotonic growth without a matching increase in services, watchers, or sidecar count is a leak, regardless of where the number sits.

Consul gossip encryption key mismatch: a botched keyring rotation splits the pool

Mon, 01 Jan 0001 00:00:00 +0000

Consul gossip encryption key mismatch: a botched keyring rotation splits the pool

You rotated the gossip encryption key. Nodes are dropping from the member list, showing as “failed” or “suspect” to peers, and service discovery is degrading. The network is fine. The agents are running. Raft may even be stable. But the gossip pool has split.

The root cause is at the protocol layer. Consul’s Serf gossip protocol encrypts every message with a symmetric key. When nodes hold different keys, they cannot decrypt each other’s gossip messages. Each side sees the other as unresponsive, and failure detection kicks in. This looks exactly like a network partition, but pings between the affected hosts succeed.

Consul gossip flapping: nodes oscillating between alive, suspect, and failed

Mon, 01 Jan 0001 00:00:00 +0000

Consul gossip flapping: nodes oscillating between alive, suspect, and failed

When Consul nodes rapidly flip between alive, suspect, and failed in the gossip pool, every state transition fires a membership event that downstream consumers react to. Load balancers pull endpoints in and out, consul-template reloads configurations, and Envoy sidecars receive new xDS endpoint pushes. A flapping node generates more operational noise than a cleanly failed one.

The Serf gossip protocol converges under stable network conditions. Under packet loss, CPU starvation, or partial partitions, it oscillates instead. The symptom appears in consul members as nodes cycling between alive, suspect, and failed; in Serf metrics as spikes in consul.serf.lan.member.flaps; and in downstream systems as configuration reload storms that track the flap cadence.

Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes

Mon, 01 Jan 0001 00:00:00 +0000

Consul gossip storm after mass recovery: rejoin floods and anti-entropy spikes

After a mass recovery event (AZ healing, partition recovery, fleet-wide rolling restart), dozens or hundreds of Consul agents rejoin the LAN gossip pool in a compressed window. Each returning agent generates member-join events that every other member processes, and triggers anti-entropy sync to reconcile its local service registrations against the server catalog. The coordinated spike hits Serf message queues, catalog registration writes, and Raft commits simultaneously.

Consul health check flapping: the passing/critical oscillation that churns the catalog

Mon, 01 Jan 0001 00:00:00 +0000

Consul health check flapping: the passing/critical oscillation that churns the catalog

Every health check state transition is a catalog write. The client agent runs the check locally, detects a status change, and pushes the update to the leader. The leader commits a Raft log entry, the FSM applies it, per-service caches invalidate, blocking queries watching that service return, and every downstream consumer (load balancer, Envoy control plane, consul-template, DNS resolver) re-evaluates. One flapping check is an annoyance. A dozen flapping in unison saturates the Raft write pipeline and looks indistinguishable from a registration storm.

Consul intention denied: service-to-service traffic blocked by policy

Mon, 01 Jan 0001 00:00:00 +0000

Consul intention denied: service-to-service traffic blocked by policy

Service A can no longer reach service B. Envoy sidecars log 403 responses or connection resets. The Consul server cluster looks healthy: leader stable, Raft committing, gossip intact, certificates within their lifetime. Traffic that worked an hour ago is now blocked.

This is almost always policy, not infrastructure. Consul Connect intentions are the mesh authorization layer, evaluated at connection establishment and enforced by Envoy RBAC filters. When an intention denies a connection, the data plane is doing exactly what it was configured to do. The diagnostic job is to determine whether the denial is correct (policy working as designed) or a misconfiguration: wrong identity, missing allow, precedence mistake, or stale cache.

Consul KV store saturation: using the KV store as a database

Mon, 01 Jan 0001 00:00:00 +0000

Consul KV store saturation: using the KV store as a database

Consul KV writes are slow. Raft commit times are climbing. Service registration and health check updates lag. DNS queries for service discovery take longer than usual. This combination often points to an application treating Consul’s KV store as a general-purpose database: high-frequency writes, large values, or deep key trees.

The problem is architectural, not a tuning issue. The KV store is part of Consul’s Raft finite state machine. Every KV write is a Raft log entry that must be committed through the leader, replicated to a quorum of servers, and applied to the in-memory state store on every server. Every KV value is included in every Raft snapshot. When an application treats the KV store as a database, the write load backs up the entire Raft pipeline, and every Consul subsystem that depends on Raft degrades with it.

Consul KV value too large: the 512KB limit and why approaching it hurts

Mon, 01 Jan 0001 00:00:00 +0000

Consul KV value too large: the 512KB limit and why approaching it hurts

You pushed a value into Consul KV and got HTTP 413, or your server logs are filling with lines like Request body(524401 bytes) too large, max size: 524288 bytes. That is the obvious failure: a single value crossed the default 512KB ceiling. The less obvious failure is that values well under the limit are still expensive, because every KV byte is replicated through Raft to every server and re-emitted in every snapshot.

Consul KV write latency high: every write is a Raft commit

Mon, 01 Jan 0001 00:00:00 +0000

Consul KV write latency high: every write is a Raft commit

A KV PUT that used to return in 20ms is now taking 500ms, 2s, or timing out entirely. Applications stall on locks, configuration writes queue, and leader election alerts may be firing. Consul’s KV is not a standalone subsystem. Every KV write is a Raft log entry that must be replicated to a quorum of servers, fsynced to disk on each, and applied to the in-memory state machine before the API call returns. KV write latency is a direct reflection of Raft commit health.

Consul leader election storm: repeated elections and rolling write outages

Mon, 01 Jan 0001 00:00:00 +0000

Consul leader election storm: repeated elections and rolling write outages

Your Consul cluster has a leader. Writes are still failing. The leader keeps changing. Applications see intermittent “no cluster leader” errors, watches reconnect, DNS returns stale results, and every few seconds a different server wins an election only to lose it again.

This is a leader election storm. Unlike a clean one-time failover, the cluster never stabilizes. Each election blocks all writes for one to several seconds. When elections recur faster than the recovery window, the cluster is effectively write-unavailable while technically always having “a leader.” A naive alert on “no leader” stays silent. The real signal is recurrence: leadership transitions accumulating over time, each one a brief but real outage.

Consul leader stable but commits stalled: writes silently failing

Mon, 01 Jan 0001 00:00:00 +0000

Consul leader stable but commits stalled: writes silently failing

You are paged for “Consul writes failing.” You check /v1/status/leader: it returns an address. You check consul members: all servers alive. Leadership has been stable for hours. Your “is there a leader?” alert is silent. Yet KV writes time out, service registrations hang, and sessions cannot be created.

The leader is alive, holds leadership, and answers health probes. But its commit index is not advancing. Somewhere between the leader receiving a write and that write becoming visible in the FSM, the pipeline is stalled. The cluster is write-dead.

Consul lost quorum: Raft peers below the majority needed to elect a leader

Mon, 01 Jan 0001 00:00:00 +0000

Consul lost quorum: Raft peers below the majority needed to elect a leader

A Consul cluster has lost quorum when the number of voting Raft peers drops below the majority required to elect a leader. For a 3-server cluster that means fewer than 2 voters. For a 5-server cluster, fewer than 3. With no leader, every write fails: service registrations, health-check state updates, KV writes, session creation, ACL token creation. Reads served in stale mode still return data, but that data is frozen at the moment quorum was lost.

Consul Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Consul Monitoring

What Is Consul?

Consul is a highly scalable and distributed service networking platform that enables users to manage and secure their microservices connections. Developed by HashiCorp, Consul offers key features such as service discovery, health checking, load balancing, and a secure service mesh. Learn more about Consul.

Monitoring Consul With Netdata

Netdata provides a powerful and real-time monitoring solution for Consul, utilizing the go.d.plugin. This tool connects seamlessly to your Consul installation, enabling comprehensive monitoring by leveraging the Consul REST API. By using Netdata, you gain instant access to crucial Consul metrics, helping you ensure optimal performance and quick identification of potential issues. To see Netdata in action, check our Live Demo.

Consul monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Consul monitoring checklist: the signals every production cluster needs

Consul runs three subsystems concurrently: Raft consensus, Serf gossip, and a catalog state machine backed by BoltDB. Each fails independently. A monitoring setup that checks only “is there a leader” and “are all members alive” will miss the most common production incidents: slow disk causing leader churn, client agents losing RPC connectivity while gossip stays green, blocking query leaks, and Connect certificate rotation failures.

Consul monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Consul monitoring maturity model: from survival to expert

Consul runs three independent subsystems that must all be healthy for the cluster to function: Raft consensus, Serf gossip, and the catalog state machine with its anti-entropy sync pipeline. Most teams monitor one or two well and discover the third during an incident.

The model is cumulative. Each level adds signals to the previous one; you cannot skip to Mature without the Operational baseline in place. A team with composite-pattern detection at Expert but no client-agent RPC monitoring at Operational is blind to the most common silent failure mode: the catalog drifting from reality while every server-side metric looks healthy.

Consul on EBS: burst-credit exhaustion and the sudden latency cliff

Mon, 01 Jan 0001 00:00:00 +0000

Consul on EBS: burst-credit exhaustion and the sudden latency cliff

Consul is stable. Raft commit times under 10ms, leadership steady, discovery in single-digit milliseconds. Then, with no deploy, no config change, and no traffic spike, writes start timing out. consul.raft.commitTime jumps from 5ms to 500ms. Followers report lastContact climbing toward the election timeout. Within seconds, the cluster elects a new leader, then another, then another. Every write fails with “no cluster leader.”

Consul Permission denied (403): authorized token, missing permission

Mon, 01 Jan 0001 00:00:00 +0000

Consul Permission denied (403): authorized token, missing permission

A 403 “Permission denied” from the Consul HTTP API means the request reached the server with a token it recognizes, but policy evaluation denied the operation. The token exists, resolves, and is not expired. It lacks the rule covering the resource being touched: a service, a KV prefix, a node, an operator endpoint. This differs from “ACL not found”, where the SecretID is unknown to the server, and the two need different responses.

Consul raft commitTime high: the write pipeline is slowing down

Mon, 01 Jan 0001 00:00:00 +0000

Consul raft commitTime high: the write pipeline is slowing down

consul.raft.commitTime is the single best indicator of Raft health on a Consul server. It is a leader-only timer that folds disk write latency, replication latency, and FSM apply time into one number. When it climbs, every Consul write slows: service registrations queue, KV writes block, health check updates lag, and consumers downstream of the catalog start timing out. The disk-bound leader cannot get log entries committed fast enough, followers lose touch, and the next step is leader elections, during which the cluster cannot commit writes at all.

Consul Raft data directory full: the server that can no longer write

Mon, 01 Jan 0001 00:00:00 +0000

Consul Raft data directory full: the server that can no longer write

The server’s data_dir volume hits 100%. Raft cannot append to raft.db and cannot fsync snapshot files. The server either refuses writes or crashes. If the affected server is the leader, the commit index stalls and followers drift toward an election.

The most common cause is not disk hardware failure. It is a snapshot that keeps failing. Raft truncates its log only after a successful snapshot, so each failed attempt leaves raft.db larger than it should be. Eventually the log fills the volume.

Consul raft lastContact rising: followers drifting toward an election

Mon, 01 Jan 0001 00:00:00 +0000

Consul raft lastContact rising: followers drifting toward an election

consul.raft.leader.lastContact is the predictive signal before a Raft election fires. It measures the elapsed time since the leader last successfully contacted each follower server. When this value rises, a follower is drifting toward the election timeout. Cross that threshold and the follower starts a new election: writes stall, clients see “no cluster leader” errors, and downstream consumers retry in a thundering herd.

Consul Raft log divergence: catching a corrupt follower before it wins an election

Mon, 01 Jan 0001 00:00:00 +0000

Consul Raft log divergence: catching a corrupt follower before it wins an election

A divergent Consul follower can gossip normally, answer Serf probes, and appear in consul members as alive while its persisted Raft state is corrupt or inconsistent. The leader replicates to the healthy majority and clients see correct data. The divergence becomes catastrophic only when that follower wins an election and serves state the cluster never agreed on.

Consul registration storm: catalog churn overwhelming Raft

Mon, 01 Jan 0001 00:00:00 +0000

Consul registration storm: catalog churn overwhelming Raft

The complaint comes in as “Consul is slow.” DNS lookups lag, service discovery returns stale results, HTTP API writes take longer than usual. The cluster has a leader, gossip is healthy, no node is down. The Raft commit time metric is climbing. The root cause is often hiding in the catalog registration rate.

Every service registration and deregistration is a Raft write. The leader appends the entry to its log, persists it (fsync), replicates to followers (who also persist), waits for quorum, and applies the entry to the in-memory state store. This is the same pipeline that handles KV writes, health-check state updates, session creation, and ACL operations. When the registration rate is high enough, it saturates the pipeline and every consumer of the consensus layer pays the cost.

Consul serf queue backlog: an agent falling behind on gossip

Mon, 01 Jan 0001 00:00:00 +0000

Consul serf queue backlog: an agent falling behind on gossip

A Consul agent whose consul.serf.queue.Event, consul.serf.queue.Intent, or consul.serf.queue.Query metric stays above zero is no longer keeping up with gossip. In steady state all three should read zero. Transient spikes during bulk joins, leaves, and rolling restarts are expected and self-drain within a few gossip intervals. The problem begins when the spike does not drain.

Serf’s queue depth is the observable proxy for the node’s internal health score. A sustained non-zero value means the agent is receiving gossip faster than its event loop can process. Downstream effects start subtle: failure detection latency rises, join and leave intents propagate slowly, and the node’s own probe replies arrive late at peers. Then the feedback loop engages.

Consul serfHealth check failing: the node-level check behind mass deregistration

Mon, 01 Jan 0001 00:00:00 +0000

Consul serfHealth check failing: the node-level check behind mass deregistration

Every Consul node carries an automatic health check called serfHealth. It is not user-defined, it cannot be removed, and it reflects the node’s membership in the Serf LAN gossip pool. When this check goes critical, Consul treats the entire node and every service registered on it as unhealthy, regardless of what individual service checks report.

This is the most commonly misdiagnosed failure in Consul. Operators see a service “go down” across all instances on a host, investigate the service checks, and find them still passing. The services are fine. The node’s gossip membership broke, and serfHealth cascaded that failure into every service on the node.

Consul server in failed state: reading consul members during an incident

Mon, 01 Jan 0001 00:00:00 +0000

Consul server in failed state: reading consul members during an incident

You ran consul members during an incident and one of your servers shows failed. Before fixing anything, understand what the status means. Gossip failure and Raft failure are separate subsystems. A server in failed state is not answering gossip probes, but it may still be a voting Raft peer, or its agent process may still be running but starved of resources.

Consul server memory climbing toward OOM: heap, catalog, and snapshots

Mon, 01 Jan 0001 00:00:00 +0000

Consul server memory climbing toward OOM: heap, catalog, and snapshots

A Consul server’s RSS is climbing and OOMKill is close. The cause could be legitimate catalog growth, a goroutine leak, snapshot amplification, or a runtime pathology. Memory growth in Consul usually mixes real state growth, Go runtime behavior, and at least one subsystem leaking.

Go’s garbage collector does not return memory to the OS immediately. RSS of roughly 2x the live heap is normal and is not a leak. What you should investigate is monotonic RSS growth that survives a GC, or live heap (consul.runtime.alloc_bytes) that does not return to a baseline after a transient event.

Consul service has zero healthy instances: discovery returns nothing

Mon, 01 Jan 0001 00:00:00 +0000

Consul service has zero healthy instances: discovery returns nothing

A consumer calls /v1/health/service/<name>?passing=true and gets []. DNS lookups for <name>.service.consul return nothing. Every downstream consumer treats the service as gone. This is total unavailability for that one service, even if the rest of the Consul cluster is healthy.

Two consumer surfaces are affected at once. API consumers see the empty array directly. DNS consumers (port 8600) get a negative response, and clients may cache it. Recovery from zero means both fixing the checks and flushing the negative caches downstream.

Consul session invalidation: distributed locks releasing across the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Consul session invalidation: distributed locks releasing across the cluster

Sessions are Consul’s mechanism for binding distributed locks to node and application health. When a session is invalidated, every KV key it holds is acted on according to its Behavior. With the default release, the lock holder is cleared and the key’s ModifyIndex increments. With delete, the key is removed entirely. Every consumer watching that key reacts.

A spike in session invalidation surfaces as a cluster-wide release of distributed locks. Leader elections fire, caches flush, configuration reloads trigger, and any logic keyed off ModifyIndex change wakes up. The operator’s job is to identify which sessions invalidated, why, and whether the root cause is a single node, a renewal-path failure, or a gossip flap.

Consul slow disk causing Raft timeouts: the number-one cause of leader instability

Mon, 01 Jan 0001 00:00:00 +0000

Consul slow disk causing Raft timeouts: the number-one cause of leader instability

Your Consul cluster is cycling through leaders. Writes fail with “no cluster leader” errors. The cluster technically has a leader at any given moment, but constant re-elections make writes effectively unavailable. The logs show [WARN] raft: heartbeat timeout reached, starting election repeating across all servers.

The root cause is almost always the same: slow disk I/O on the volume backing the Raft data directory. Consul’s Raft log store performs an fsync on every appended entry. When the disk cannot complete those syncs quickly enough, heartbeats and commits slow down. Followers miss their heartbeat window and start new elections. The new leader lands on the same slow storage. The cycle repeats.

Consul snapshot size growing: state bloat, slow restores, and commit spikes

Mon, 01 Jan 0001 00:00:00 +0000

Consul snapshot size growing: state bloat, slow restores, and commit spikes

Consul snapshots grow with the size of the FSM state: every registered service instance, every health check, every KV entry, every ACL token, every session. None of those individual writes looks expensive, so a team that registers a few new services per week will not see a spike in any single metric. What they get instead is a slow, monotonic climb in snapshot size that compounds across months until one day a snapshot save takes long enough to push Raft commit times past the heartbeat timeout, or a rejoined follower sits in snapshot restore long enough to fall out of the replication loop entirely.

Consul stale DNS queries: the agent is answering from cache

Mon, 01 Jan 0001 00:00:00 +0000

Consul stale DNS queries: the agent is answering from cache

When consul.dns.stale_queries climbs, the Consul agent is doing what it was designed to do: keep answering DNS from local state when it cannot get a fresh answer from a server. That behavior has been the DNS default since Consul 0.7. But “stale” means two very different things depending on how stale, and why.

A stale answer two seconds old during a leader handoff is harmless availability. A stale answer five minutes old, still pointing at instances that crashed four minutes ago, is silently routing traffic to dead services. The counter alone cannot tell you which case you are in. Correlate it with agent-to-server RPC health, server health, and your consistency configuration.

Consul stale Raft peer: removing a failed server from the configuration

Mon, 01 Jan 0001 00:00:00 +0000

Consul stale Raft peer: removing a failed server from the configuration

A Consul server has crashed, been decommissioned, or been replaced, but its entry still sits in the Raft configuration as a voting peer. Serf gossip already marks the node failed or left, yet GET /v1/operator/raft/configuration still lists it as a voter. The dead peer keeps counting toward quorum even though it will never cast another vote.

This is not always an emergency. A healthy cluster with autopilot enabled usually self-heals within tens of seconds. It becomes an emergency when autopilot is disabled, when the cleanup window has not elapsed, or when a second failure tips the cluster below the now-inflated quorum threshold.

Consul TLS not fully enforced: verify_incoming, verify_outgoing, and server hostname

Mon, 01 Jan 0001 00:00:00 +0000

You have CA, certificate, and key files configured for Consul. The agent starts without errors, health checks pass, services register, DNS resolves. But the cluster may still accept plaintext or unauthenticated connections because verify_incoming, verify_outgoing, or verify_server_hostname are missing, set to false, or placed in the wrong configuration stanza.

Certificate files (ca_file, cert_file, key_file) do not enforce TLS on their own. Consul requires explicit verification flags to reject plaintext connections and require client certificates. Without them, any network observer between agents and servers can read the catalog, KV values, and health check results, and any process that can reach the RPC port can impersonate a Consul agent.

Consul WAN federation down: a datacenter missing from the WAN pool

Mon, 01 Jan 0001 00:00:00 +0000

Consul WAN federation down: a datacenter missing from the WAN pool

When an entire remote datacenter’s servers vanish from the WAN gossip pool, the symptom is usually downstream: cross-DC service lookups return empty results, prepared queries that failover across DCs stop working, and ACL or Connect CA replication between datacenters silently stalls. The Consul UI on the local DC may still look healthy because the local cluster is fine. The damage is in the federation layer.

Consul WAN link saturation: cross-DC latency triggering gossip suspicion

Mon, 01 Jan 0001 00:00:00 +0000

Consul WAN link saturation: cross-DC latency triggering gossip suspicion

Remote datacenter servers start oscillating between alive and suspect in the WAN gossip pool. Cross-DC RPCs fail with “no path to datacenter” errors. ACL replication lag in secondary DCs climbs into seconds. Cross-DC prepared queries time out. LAN Consul signals look fine, but the federation is broken.

The usual cause is WAN link saturation. Consul’s WAN gossip pool uses a separate, more conservative set of timing defaults than LAN, but it is still vulnerable to high or spiking RTT. When the inter-DC link saturates, gossip probes time out, remote servers are marked suspect then failed, and WAN membership flaps. Each flap disrupts cross-DC RPC routing, which compounds the problem because RPC retries add traffic to the already-saturated link.

Consul xDS stream churn: Envoy sidecars running on stale configuration

Mon, 01 Jan 0001 00:00:00 +0000

Consul xDS stream churn: Envoy sidecars running on stale configuration

Each Envoy sidecar in a Consul Connect mesh holds a long-lived gRPC xDS stream to a Consul server. That stream is the control plane: endpoint lists, intentions, and mTLS certificate rotations all flow across it. When the stream is healthy, the data plane converges within seconds of a catalog change. When it breaks or churns, the sidecar falls back to whatever Envoy cached last.

Contact Netdata

Mon, 01 Jan 0001 00:00:00 +0000

containerd Containers

Mon, 01 Jan 0001 00:00:00 +0000

Containers

Mon, 01 Jan 0001 00:00:00 +0000

Conteg SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Convertronic GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cooler Master Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Copper Mountain Communications Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS /health vs /ready: the readiness-probe mistake that serves SERVFAIL

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS /health vs /ready: the readiness-probe mistake that serves SERVFAIL

Your CoreDNS pods look healthy. Liveness passes, readiness passes, the pods are Running and Ready. Yet for the first seconds or minutes after every pod start, clients get SERVFAIL for cluster.local names. New deployments fail to resolve their dependencies, retries pile up, and then the problem vanishes on its own.

This is the classic symptom of pointing the Kubernetes readiness probe at /health instead of /ready. The two endpoints answer different questions, and confusing them makes your readiness gate meaningless.

CoreDNS 403 from the Kubernetes API: RBAC misconfiguration breaking cluster DNS

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS 403 from the Kubernetes API: RBAC misconfiguration breaking cluster DNS

The symptom usually arrives in one of two forms. Either CoreDNS pods never become ready after an install or upgrade, stuck at 0/1 while kubectl shows no crash and no obvious error, or cluster DNS suddenly returns SERVFAIL for cluster.local names while external resolution keeps working. In both cases, the smoking gun is the same: coredns_kubernetes_rest_client_requests_total{code="403"} is incrementing.

A 403 from the Kubernetes API is not a transient error. It means the API server received the request, authenticated the caller, and refused it because the CoreDNS ServiceAccount lacks the RBAC permissions the kubernetes plugin needs to list and watch Services, Endpoints, and EndpointSlices. Without those watches, the plugin cannot build or maintain its in-memory record set, so it cannot answer queries for the cluster zone.

CoreDNS 5-second DNS timeout: the Kubernetes glibc A+AAAA conntrack race

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS 5-second DNS timeout: the Kubernetes glibc A+AAAA conntrack race

Your application latency histogram has a cluster of samples at almost exactly 5 seconds. Not 4.8, not 5.3: a tight spike right at the glibc default resolver timeout. Tracing the slow calls shows they stall on name resolution. CoreDNS dashboards are green: low latency, no SERVFAIL, healthy cache hit ratio, both replicas fine.

That combination, application-side DNS stalls at exactly the resolver timeout with clean CoreDNS metrics, is the signature of the glibc A+AAAA conntrack race. The loss is in the kernel network path between the application pod and CoreDNS, not inside CoreDNS. No amount of Corefile tuning will touch it.

CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS all upstreams down: the forwarding black hole and healthcheck_broken

Every forwarded query through CoreDNS is suddenly returning SERVFAIL, request latency has dropped instead of rising, and coredns_forward_healthcheck_broken_total is climbing. This is the upstream black hole: every upstream resolver configured in the forward plugin is failing health checks at the same time, and CoreDNS has nowhere to send external queries.

The counter-intuitive part is the latency profile. When upstreams die hard, failures are fast. Queries do not sit in timeouts the way they do with a slow upstream. If you only alert on high latency, this failure mode sails under your dashboards until clients start reporting resolution errors.

CoreDNS ANY query flood: DNS amplification and reflection abuse

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS ANY query flood: DNS amplification and reflection abuse

You are looking at coredns_dns_requests_total and the type="ANY" series has jumped from a flat zero to a sustained rate, or ANY queries have crept past a few percent of total traffic. That pattern is the classic fingerprint of DNS amplification and reflection abuse: an attacker sends small ANY queries with a spoofed source address, and your CoreDNS replies with large responses delivered to the victim’s IP. Your server is the reflector; someone else absorbs the blast.

CoreDNS AXFR zone transfer attempts: reconnaissance of internal service topology

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS AXFR zone transfer attempts: reconnaissance of internal service topology

An AXFR query asks a DNS server for the entire contents of a zone: every record, not one. Against a Kubernetes cluster running CoreDNS, a successful zone transfer hands the requester a complete map of internal service topology: every Service name, every namespace, every headless Service endpoint, every pod hostname you publish. That is exactly the map an attacker wants before lateral movement.

CoreDNS cache collapse: the cold-cache thundering herd after a rollout

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS cache collapse: the cold-cache thundering herd after a rollout

You just rolled out a CoreDNS config change or image bump. Within seconds, DNS latency across the cluster jumps, upstream DNS traffic spikes to several times baseline, and SERVFAILs appear in application logs. One to five minutes later, it all goes away on its own. The dashboard is green again.

That is the cache collapse pattern: every CoreDNS pod restarted at roughly the same time, every in-memory cache emptied at once, and every client in the cluster re-queried the same names simultaneously. The flood hit your upstreams harder than they could absorb. In the worst version, upstreams return SERVFAIL, CoreDNS caches those SERVFAILs for 5 seconds each, and a brief overload amplifies into a visible outage.

CoreDNS cache evictions: the cache is too small for the working set

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS cache evictions: the cache is too small for the working set

coredns_cache_evictions_total climbing steadily, or upstream query load and DNS latency creeping up without any change in traffic volume, both point at the same thing: the cache plugin is evicting entries before their TTL expires because it has run out of room, and queries that should have been sub-millisecond cache hits are being forwarded to upstream resolvers instead.

This is progressive degradation, not a cliff edge. Latency rises, upstream load rises, and the hit ratio slides, slowly enough that it often goes unnoticed until someone asks why DNS got slower this quarter.

CoreDNS cache hit ratio dropping: latency and upstream load climbing together

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS cache hit ratio dropping: latency and upstream load climbing together

The cache hit ratio is coredns_cache_hits_total / coredns_cache_requests_total. When it falls, every query that used to be answered from memory in under a millisecond now goes to the kubernetes plugin or out to an upstream resolver. Two things happen at once: coredns_dns_request_duration_seconds climbs, and forwarded query volume climbs with it. That second effect is the dangerous one, because the upstream flood can degrade the upstreams themselves and turn a cache problem into a resolution outage.

CoreDNS conntrack table full: silent UDP packet drops with a node-wide blast radius

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS conntrack table full: silent UDP packet drops with a node-wide blast radius

Applications across a node start timing out on DNS lookups. Then on database connections. Then on API calls. You check CoreDNS and the dashboards are green: low latency, no SERVFAILs, healthy pods. But the timeouts keep coming, and they are not limited to DNS.

This is conntrack table exhaustion. Every UDP DNS query that traverses Kubernetes iptables DNAT creates a connection tracking entry on the node with a default timeout of 30 seconds. At high DNS QPS, the table fills. When it does, the kernel drops new packets silently: no ICMP error, no reset, no log line the application will ever see. The kernel log says nf_conntrack: table full, dropping packet and nothing else anywhere warns you.

CoreDNS Corefile parse error: startup failures and invalid plugin config

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS Corefile parse error: startup failures and invalid plugin config

You changed the Corefile, and now one of two things has happened. Either the CoreDNS pods are in CrashLoopBackOff and cluster DNS is down, or the pods look fine but your change never took effect and the log is full of parse errors. Both are the same root event: the Corefile parser rejected your configuration. The difference is whether the parser ran at process start (fatal) or during a live reload (rejected, old config kept).

CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS CPU throttling: CFS limits making a green dashboard lie about latency

Applications are reporting intermittent DNS timeouts and multi-hundred-millisecond resolution spikes. You open the CoreDNS dashboard and everything looks fine: coredns_dns_request_duration_seconds shows sub-millisecond cache hits, SERVFAIL is zero, QPS is normal, pods are healthy. Both things are true at the same time, and that is exactly the problem.

When a CoreDNS pod has a tight CPU limit, the Linux CFS scheduler throttles the container: the process is runnable but the kernel refuses to schedule it until the next quota period. That stall is real latency for the client, but it never appears in any CoreDNS metric, because CoreDNS only measures time spent actively processing a query. Time spent waiting for CPU does not exist as far as coredns_dns_request_duration_seconds is concerned.

CoreDNS CrashLoopBackOff: triaging loop, OOM, config, and port-bind causes

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS CrashLoopBackOff: triaging loop, OOM, config, and port-bind causes

A CoreDNS pod in CrashLoopBackOff means the process starts, dies, and gets restarted by Kubernetes in a tightening backoff loop. While this is happening, at least one DNS replica is contributing nothing. If all replicas are crash-looping, the kube-dns Service has no backends and cluster-wide name resolution is down: queries hang at the client until a pod recovers.

CrashLoopBackOff is not one failure. It is a symptom with a small set of common root causes, and they need different fixes. The four you will see most often: the loop plugin detecting a forwarding loop, the OOM killer terminating the container, a Corefile that fails to parse, and a port-bind failure on startup. RBAC and Kubernetes API access failures show up in the same pod state too, usually as fatal list errors or persistent un-readiness.

CoreDNS DNS programming latency: how long a Service takes to become resolvable

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS DNS programming latency: how long a Service takes to become resolvable

You create a Service, the API server accepts it, kubectl get svc shows it, and yet for some period of time nothing in the cluster can resolve its name. That gap, from “the object exists in the API” to “CoreDNS answers for it”, is DNS programming latency, and CoreDNS exposes it as a first-class metric: coredns_kubernetes_dns_programming_duration_seconds.

This is the Kubernetes DNS Programming SLI. It exists because service discovery delay is a real failure mode: a rollout completes, endpoints are ready, but clients still get NXDOMAIN or stale answers for seconds or tens of seconds. From the application’s perspective the new pods are unreachable by name. From CoreDNS’s request metrics everything looks green.

CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS forward max_concurrent rejects: the forward plugin is overwhelmed

coredns_forward_max_concurrent_rejects_total is incrementing and clients are getting REFUSED responses for forwarded queries. Something is piling up between CoreDNS and your upstreams, and the rejects are the valve letting pressure escape.

This counter only increments when the number of in-flight forwarded queries hits the max_concurrent cap you configured. Every rejected query gets a REFUSED response, not SERVFAIL. That distinction matters for triage: REFUSED from this path is a capacity signal, not a resolution failure, and unlike NXDOMAIN it is not subject to negative caching, so it will not linger in client caches after the pressure clears.

CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS GC pauses adding tail latency: go_gc_duration_seconds and heap pressure

Your CoreDNS dashboards look mostly fine: P50 latency is sub-millisecond, SERVFAIL is zero, cache hit ratio is healthy. But P99 is spiking to tens or hundreds of milliseconds at irregular intervals, and a small fraction of clients see DNS timeouts they cannot explain. The process never crashes. Upstream latency is clean. CPU looks acceptable on average.

This is the classic signature of Go garbage collection stop-the-world pauses landing on DNS query latency. CoreDNS is a Go process, and every in-flight query is a goroutine. When the GC stops the world, every one of those goroutines waits. DNS is supposed to be sub-millisecond for cache hits, so a pause that would be invisible in a batch service is user-visible here. The working threshold: GC pauses over 10ms are impactful for DNS and show up as P99 spikes.

CoreDNS goroutine count climbing: blocked upstream calls and leaks

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS goroutine count climbing: blocked upstream calls and leaks

Your go_goroutines graph for CoreDNS is trending up. Maybe it spiked during an incident and never came back down. Maybe it has been climbing slowly for days. Either way, the question is the same: are queries piling up behind a slow upstream, or is something leaking goroutines that will never exit?

The distinction matters because the fixes are completely different. A blocked-upstream spike resolves when the upstream recovers. A leak grows until the OOM killer ends the debate for you.

CoreDNS high request latency: reading P99 by zone to find the cause

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS high request latency: reading P99 by zone to find the cause

P99 on coredns_dns_request_duration_seconds is at 400ms when it used to sit at 15ms. Do not start with the aggregate number: aggregate latency in CoreDNS blends at least two very different workloads. Cache hits should complete in single-digit milliseconds. Forwarded and Kubernetes-backed queries depend on systems outside CoreDNS itself.

The zone label on the duration histogram is the fork in the road. High latency in cluster.local points at the Kubernetes path. High latency in forwarded zones points at upstream resolvers. High latency everywhere, at every percentile, points at the CoreDNS process itself: CPU saturation or GC pressure. Until you split the histogram by zone, you are guessing.

CoreDNS Kubernetes API disconnect: stale records and new services going invisible

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS Kubernetes API disconnect: stale records and new services going invisible

The symptom that usually surfaces first is not a DNS error. It is a developer saying “I deployed the service twenty minutes ago and nothing can reach it,” while every existing workload looks fine. nslookup kubernetes.default works. External domains resolve. CoreDNS dashboards are green. But anything created or changed recently in the cluster does not exist as far as DNS is concerned.

CoreDNS Loop detected and CrashLoopBackOff: the forwarding loop that kills the pod

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS Loop detected and CrashLoopBackOff: the forwarding loop that kills the pod

Your CoreDNS pods are in CrashLoopBackOff. kubectl logs shows a line like [FATAL] plugin/loop: Loop (127.0.0.1:53 -> :53) detected for zone ".", and the process exits a few seconds after every start. There are no CoreDNS metrics to look at, because the process dies before the metrics endpoint is ever scraped.

The crash is intentional. The loop plugin detected that the Corefile’s forward target routes queries back to CoreDNS itself, and it killed the process on purpose to prevent an infinite query amplification storm. Kubernetes restarts the pod, the loop is detected again, and the cycle repeats.

CoreDNS memory climbing: heap growth, post-GC minima, and leak detection

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS memory climbing: heap growth, post-GC minima, and leak detection

CoreDNS memory has been creeping up for hours. The Grafana panel shows a sawtooth that never quite comes back down, and you need to answer one question before the pod gets OOMKilled: is this Go being Go, or is something actually leaking?

The trap is that go_memstats_heap_inuse_bytes is spiky by design. The heap grows, the garbage collector reclaims, the heap grows again. If you react to instantaneous peaks you will chase ghosts all night. The signal that separates normal GC behavior from a leak is the trend of the post-GC minima: the floor each sawtooth returns to. If that floor keeps rising over an hour or more and never returns to baseline, you have a genuine leak heading for the container memory limit.

CoreDNS memory limit and the re-list spike: sizing for the restart peak

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS memory limit and the re-list spike: sizing for the restart peak

A CoreDNS pod that runs fine for weeks gets OOMKilled thirty seconds after a restart. Kubernetes starts it again. It re-syncs, allocates heavily, crosses the limit, and gets killed again. Restart count climbs into the hundreds, and cluster DNS capacity drops by half or more while the crash loop runs. kubectl describe pod shows Last State: Terminated, Reason: OOMKilled, and nothing in steady-state monitoring predicted it.

CoreDNS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS Monitoring

What Is CoreDNS?

CoreDNS is a flexible and extensible DNS server that can serve as a cluster DNS inside Kubernetes or provide other DNS services. Built with a focus on flexibility, plugin support, and efficiency, it’s designed to facilitate the addition and combination of functionalities. You can read more about CoreDNS on their official website.

Monitoring CoreDNS With Netdata

Netdata is an ideal tool for monitoring CoreDNS, offering real-time visibility and insightful metrics. With Netdata, you can track DNS queries, responses, and errors to ensure the robustness and performance of your DNS server. Explore our Live Demo to experience Netdata’s capabilities firsthand.

CoreDNS monitoring checklist: the signals every production resolver needs

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS monitoring checklist: the signals every production resolver needs

CoreDNS has one failure mode that defeats most monitoring setups: a pod that passes every health probe while returning SERVFAIL for every query. The /health endpoint on port 8080 checks process liveness. It does not test DNS resolution. The /ready endpoint on port 8181 is plugin-aware, but it does not test resolution either. Teams that alert on pod status and probe results are monitoring the wrong thing.

CoreDNS monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS monitoring maturity model: from survival to expert

Most CoreDNS monitoring setups are either a single “is the pod up” check or a wall of dashboards nobody reads during an incident. Neither works. The right question is not “how many metrics do we collect” but “which failure modes can we actually detect right now.” That is what a maturity model answers: it orders signals by the failures they catch, so you can see exactly which class of outage would currently reach your users before it reaches your alerts.

CoreDNS ndots:5 query amplification: one lookup becoming five

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS ndots:5 query amplification: one lookup becoming five

CoreDNS dashboards show 50,000 QPS, the denial cache is churning, and NXDOMAIN responses are a third of all traffic, but nothing is broken. What you are looking at is probably not real demand. The default ndots:5 in every pod’s /etc/resolv.conf expands each external lookup through the Kubernetes search domain list, so one logical lookup becomes four to six wire queries before the real name is ever tried.

CoreDNS NOERROR with zero answers: the resolution failure that reports success

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS NOERROR with zero answers: the resolution failure that reports success

Your RCODE dashboards are green. NOERROR rate is healthy, SERVFAIL is near zero, health probes pass. Yet an application team reports their service cannot resolve a dependency. dig returns instantly with status: NOERROR and ANSWER SECTION: 0. No address, no error, no timeout. The resolver library treats this as “the name exists but has no records” and the application fails with a connection error that looks nothing like a DNS problem.

CoreDNS not resolving external domains: the missing catch-all forward zone

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS not resolving external domains: the missing catch-all forward zone

The symptom is specific: names inside the cluster resolve, but anything outside the cluster fails. kubernetes.default.svc.cluster.local works. example.com does not. Applications report DNS timeouts, getaddrinfo failures, or intermittent external dependency errors while service discovery inside Kubernetes looks fine.

That split is the clue. CoreDNS is not one global resolver. It matches each query to the most specific server block in the Corefile, then runs that block’s plugin chain. If cluster.local is handled by the kubernetes plugin but there is no catch-all block for ., external names match nothing useful and are refused instead of forwarded.

CoreDNS NXDOMAIN flood from one source: DGA malware and domain enumeration

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS NXDOMAIN flood from one source: DGA malware and domain enumeration

Your CoreDNS dashboards are green: SERVFAIL is zero, latency is fine, cache hit ratio looks normal. But the NXDOMAIN rate has jumped, and when you pull query logs, one source IP is responsible for thousands of NXDOMAIN responses per hour, querying names that do not exist.

This is a security signal, not a CoreDNS health signal. A single source generating a sustained flood of NXDOMAIN responses is doing one of three things: cycling randomly generated domains looking for a command-and-control server (DGA malware in a compromised pod), probing which service names exist in your cluster (reconnaissance), or misbehaving in a way that merely looks like an attack (a config typo hammering a name that was never created).

CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS NXDOMAIN vs SERVFAIL: why alerting on the wrong one buries real incidents

Your CoreDNS alert fired again. The “DNS error rate” panel shows a wall of red, so you open it, see a pile of NXDOMAIN responses, and silence the alert. Meanwhile, a genuine upstream failure is producing SERVFAIL responses somewhere in that same wall of red, and nobody will notice until an application team opens a ticket.

This is the most common CoreDNS alerting mistake: treating all non-NOERROR responses as one bucket. NXDOMAIN and SERVFAIL look similar in a naive error counter, but they mean opposite things. NXDOMAIN is the server correctly reporting that a name does not exist. SERVFAIL is the server admitting it failed to answer at all. Alerting on the sum of both guarantees that the noise (NXDOMAIN, constant and expected in Kubernetes) drowns the signal (SERVFAIL, rare and always actionable).

CoreDNS OOMKilled: the memory cliff and the restart crash loop

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS OOMKilled: the memory cliff and the restart crash loop

You found the CoreDNS pods in kube-system with OOMKilled as the last terminated state. Maybe one pod restarted once and recovered. Maybe both replicas are flapping and cluster DNS is degraded or down. Either way, the pod status operators paste into a search bar is the same: Last State: Terminated, Reason: OOMKilled.

Memory exhaustion in CoreDNS is a cliff edge, not a slowdown. There is no degradation curve where latency rises and errors climb while you watch. RSS approaches the container memory limit, the kernel OOM killer terminates the process instantly, and every client served by that pod loses DNS until a replacement is ready. The warning phase exists only in your metrics, and only if you are watching the right ones.

CoreDNS panics: recovered handler crashes and coredns_panics_total

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS panics: recovered handler crashes and coredns_panics_total

You found coredns_panics_total above zero, or you saw Recovered from panic in the CoreDNS logs. The pods are not restarting, the health endpoint returns 200, and most DNS queries still resolve. That combination is what makes this failure mode easy to ignore and dangerous to dismiss.

A recovered panic means a query handler crashed inside the plugin chain and CoreDNS’s recovery wrapper caught it, keeping the process alive. The server survived. The query that triggered the panic did not: the client that sent it got no answer and waited out its own timeout. Every increment of the counter is a real query that failed, plus evidence of a genuine bug in CoreDNS or one of its plugins.

CoreDNS per-replica divergence: why averaged metrics hide a failing pod

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS per-replica divergence: why averaged metrics hide a failing pod

Your CoreDNS dashboard looks slightly off. Average latency drifted up, SERVFAIL ratio is elevated but under the threshold, and nobody is being paged. Meanwhile, half the cluster’s DNS queries are slow or failing, because one of your two replicas is degraded and kube-proxy is still sending it traffic.

This is the per-replica divergence failure mode. CoreDNS in Kubernetes is almost never a singleton: the default deployment runs two replicas behind the kube-dns ClusterIP Service, and kube-proxy load-balances across them. Any aggregation that averages or sums across pods blends a sick replica’s numbers with healthy ones until the result looks like mild degradation instead of a partial outage.

CoreDNS per-upstream health check failures: degraded redundancy before total loss

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS per-upstream health check failures: degraded redundancy before total loss

One of your upstream DNS resolvers is failing CoreDNS’s health checks. Queries are still being answered because the remaining upstreams are carrying the forwarding load, so nothing is paging yet. This is exactly the window you want: degraded redundancy, not an outage. The next failure in this sequence is coredns_forward_healthcheck_broken_total incrementing, at which point every upstream is marked unhealthy and you are in a real incident.

CoreDNS pod alive but not ready: the /ready endpoint and API sync

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS pod alive but not ready: the /ready endpoint and API sync

The pod is Running. The process is up, the logs show CoreDNS started, maybe it is even answering some queries. But kubectl get pods -n kube-system shows 0/1 READY, the pod is excluded from the kube-dns Service endpoints, and your cluster is running on one fewer DNS replica than you think. If this is your only replica, or the other one is on the same node, you are one hiccup away from a cluster-wide DNS outage.

CoreDNS query rate dropped to zero while the process looks healthy

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS query rate dropped to zero while the process looks healthy

The dashboard shows coredns_dns_requests_total flatlined at zero. You check the pod: Running, zero restarts. /health on port 8080 returns 200. The process is fine. So either nobody in the cluster is resolving names anymore, or the traffic is dying somewhere between your clients and the CoreDNS process, and CoreDNS has no idea.

The second possibility is the trap. CoreDNS only counts queries it actually receives. A UDP packet dropped by the kernel before it reaches the CoreDNS socket is never counted, never logged, and never reflected in any CoreDNS metric. A full conntrack table, an overflowing UDP receive buffer, a network policy isolating the pod, or a readiness failure that pulled the pod out of the Service endpoints all look identical in CoreDNS metrics: nothing.

CoreDNS reload failed: config drift when the new Corefile never took effect

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS reload failed: config drift when the new Corefile never took effect

You pushed a Corefile change, nothing broke, and everyone moved on. Weeks later you discover the cluster is still running the old configuration. The metric that would have told you is coredns_reload_failed_total, and it was nonzero the whole time.

A failed CoreDNS reload is not an outage. The reload plugin polls the Corefile every 30 seconds (with jitter) and triggers a graceful reload when the SHA512 checksum of the file changes. If the new configuration has a syntax error, an invalid plugin directive, or hits a port conflict, CoreDNS logs the error, increments coredns_reload_failed_total, and keeps serving DNS with the old config. Queries keep flowing. The only thing that changed is that reality and your intent have quietly diverged.

CoreDNS response size distribution: large answers, TCP pressure, and amplification

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS response size distribution: large answers, TCP pressure, and amplification

Most CoreDNS dashboards track QPS, latency, and SERVFAIL. Response size rarely makes the cut until something odd happens: TCP/53 traffic climbs for no obvious reason, a workload starts resolving incomplete answers, or a security review asks whether your resolvers could be used as DDoS reflectors. All three questions are answered by the same signal: the distribution of DNS response sizes CoreDNS is sending.

CoreDNS response truncation: the TC bit, EDNS bufsize, and TCP fallback

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS response truncation: the TC bit, EDNS bufsize, and TCP fallback

Most DNS traffic fits in a single UDP datagram. When it does not, the protocol has an escape hatch: the server sets the TC (truncated) bit in the response, and the client retries the same query over TCP. CoreDNS implements this correctly in the common case, but the edges are where production incidents live: firewalls that silently block TCP/53, legacy clients that do not speak EDNS0, and known CoreDNS bugs where the TC flag never reaches the client at all.

CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS returning REFUSED: no matching zone, an ACL, or the forward concurrency limit

CoreDNS is answering queries with REFUSED and clients cannot resolve names. Unlike SERVFAIL, which means “I tried and failed,” REFUSED means “I declined to try.” REFUSED is almost never an upstream or network problem: CoreDNS itself is deciding not to answer, and it does that for exactly three reasons.

Two are configuration problems: no server block matches the query’s zone, or the acl plugin is rejecting the source. One is a capacity problem: the forward plugin’s max_concurrent limit is shedding load. Determine which of the three you are dealing with before touching the Corefile. Editing the Corefile to fix a capacity REFUSED, or scaling replicas to fix a config REFUSED, wastes the incident.

CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS returning SERVFAIL: the resolver is failing queries and what to check first

Applications are failing to resolve names and CoreDNS is answering with SERVFAIL. In Kubernetes this surfaces as connection errors everywhere at once, because nearly all in-cluster communication depends on DNS. SERVFAIL is not one failure. It is CoreDNS saying “a plugin in my chain could not answer this query,” and the cause is usually upstream DNS, the Kubernetes API, or the Corefile itself.

CoreDNS serve_stale: keeping resolution alive while masking upstream failure

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS serve_stale: keeping resolution alive while masking upstream failure

The serve_stale option in the CoreDNS cache plugin is a resilience feature with a monitoring side effect that bites teams during real incidents. When it is enabled, CoreDNS answers queries from expired cache entries instead of failing them when the upstream resolver is unreachable. Clients keep resolving names through an upstream outage. Dashboards stay green. The upstream can be dead for an hour before anyone notices, because the one metric that would tell you is almost never graphed.

CoreDNS SERVFAIL cache amplification: a one-second blip becomes a five-second outage

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS SERVFAIL cache amplification: a one-second blip becomes a five-second outage

Your dashboards show an upstream DNS hiccup that lasted one second. Your users report an outage that lasted five. Both are accurate, and the gap between them is the CoreDNS cache plugin doing what it was designed to do.

CoreDNS caches SERVFAIL responses for 5 seconds by default. The cache plugin keeps separate positive (success) and negative (denial) caches, and SERVFAIL goes into the denial cache alongside NXDOMAIN and NODATA. The intent is sound: RFC 2308 permits caching server-failure responses to shield an already-struggling upstream from a retry storm. The side effect is that a single SERVFAIL answer for a hot record is served to every client that asks during the 5-second window, even after the upstream has fully recovered.

CoreDNS serving stale cluster DNS: why you need a functional freshness test

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS serving stale cluster DNS: why you need a functional freshness test

Your CoreDNS dashboards are green. Query rate normal, latency sub-millisecond, SERVFAIL near zero, cache hit ratio healthy. Meanwhile, a Service deleted an hour ago still resolves, and the new Service your team just deployed returns NXDOMAIN. Applications are failing, but nothing in your monitoring fired.

This is the silent stale data pattern. The kubernetes plugin builds its DNS records from a watch on the API server. If that watch disconnects and cannot re-establish, CoreDNS keeps answering from its in-memory snapshot. The snapshot drifts from reality. No errors go back to clients. No performance metric moves, because the server is genuinely healthy and fast. The data is simply wrong.

CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS slow upstream: per-upstream latency, goroutine pileup, and the to label

Overall DNS latency is climbing. P99 is above 500ms, but P50 looks only mildly elevated. SERVFAIL is nonzero but nowhere near a full outage. The /health endpoint returns 200, every upstream passes its health checks, and yet clients are complaining that resolution is slow. Meanwhile go_goroutines is trending up and heap is following it.

This is the slow upstream drag pattern: one upstream DNS server is responding slowly but still returning valid answers. Because it never actually fails, the forward plugin’s health checks never mark it down, so queries keep getting routed to it. Each of those queries holds a goroutine while it waits. Latency accumulates, goroutines accumulate, and memory follows. Left alone, this ends in either max_concurrent rejects (if you set one) or OOM (if you did not).

CoreDNS too many open files: file descriptor exhaustion and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS too many open files: file descriptor exhaustion and refused connections

CoreDNS pods are logging too many open files and DNS resolution is failing intermittently or completely. Clients see timeouts or refused connections while the process keeps running and /health still returns 200. Restarting the pod fixes it for a while, then it comes back.

This is file descriptor exhaustion, and it is a cliff-edge failure. The moment process_open_fds reaches process_max_fds, CoreDNS cannot open anything new: no upstream connections, no listening sockets, no log files, no Kubernetes API watch streams. There is no queuing and no graceful degradation. New connection attempts fail immediately and DNS breaks.

CoreDNS TTL=0 responses: the anti-pattern that silently bypasses the cache

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS TTL=0 responses: the anti-pattern that silently bypasses the cache

Your cache hit ratio is sliding. Upstream query volume is climbing. P99 latency is drifting up with it. You open the Corefile and the cache plugin is right there, configured the way it has been for months. Nothing changed on your side, but the cache has effectively stopped working.

A common cause is records arriving with TTL=0. When an upstream resolver, or your own zone data, answers with a zero TTL, those answers are uncacheable by contract: every client query for that name has to be forwarded again. The cache plugin is present and correct, but there is nothing for it to hold on to. It looks like a cache failure but is actually a data problem arriving through the response path.

CoreDNS UDP buffer errors: kernel receive-buffer overflow dropping queries

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS UDP buffer errors: kernel receive-buffer overflow dropping queries

Applications are reporting intermittent DNS timeouts. You open the CoreDNS dashboard and everything looks fine: latency is low, SERVFAIL rate is zero, the health endpoint returns 200, and QPS is suspiciously flat. Not spiking, not zero, just lower than the client demand you know exists.

This is the UDP buffer cliff. Incoming UDP traffic is exceeding the kernel socket receive buffer on the node, and the kernel is dropping DNS queries before CoreDNS ever reads them. Because CoreDNS only counts the packets it actually receives, every CoreDNS-level metric looks clean. The failure lives one layer below, in the kernel.

CoreDNS upstream connection cache misses: new connections adding latency per query

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS upstream connection cache misses: new connections adding latency per query

You are here because forwarded DNS queries got slower, or because coredns_proxy_conn_cache_misses_total is climbing and you need to know whether it matters. It does. Every connection cache miss means CoreDNS opens a fresh connection to an upstream resolver instead of reusing one from its pool. Setup cost lands on that query, and each new connection holds a file descriptor until it is closed or reaped.

CoreDNS with NodeLocal DNSCache: what changes about monitoring

Mon, 01 Jan 0001 00:00:00 +0000

CoreDNS with NodeLocal DNSCache: what changes about monitoring

You deployed NodeLocal DNSCache to fix the 5-second timeout race and conntrack pressure, and it worked. DNS latency dropped, the conntrack table stopped filling, and CoreDNS QPS fell off a cliff. Then someone looked at the CoreDNS dashboard, saw near-zero traffic, and concluded CoreDNS was oversized. That conclusion is how the next incident starts.

NodeLocal DNSCache runs a caching DNS proxy as a DaemonSet on every node. It intercepts pod DNS queries before they enter the iptables DNAT and conntrack path, so each query is answered on the local node instead of traversing NAT to a central CoreDNS pod. This eliminates the kernel race condition behind the 5-second glibc timeout and removes DNS as a conntrack consumer.

Coriolis Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Correlating cloud VPC flow logs with on-prem NetFlow

Mon, 01 Jan 0001 00:00:00 +0000

Correlating cloud VPC flow logs with on-prem NetFlow

Cloud flow logs and on-prem flow records share the 5-tuple concept but diverge in nearly every dimension that matters for correlation: transport, latency, sampling, timestamps, topology, and NAT visibility. Cloud providers emit VPC flow logs via push to object storage with implicit sampling and aggregation intervals measured in minutes. On-premises devices export NetFlow v5/v9, IPFIX, or sFlow over UDP with configurable sampling and near-real-time delivery.

Cortex

Mon, 01 Jan 0001 00:00:00 +0000

Cosine Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Couchbase

Mon, 01 Jan 0001 00:00:00 +0000

Couchbase Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Couchbase Monitoring

What Is Couchbase?

Couchbase is a distributed NoSQL cloud database ideal for interactive web and mobile applications. Designed to support massive data volumes and a large number of users, Couchbase provides core database functions while enhancing performance through its advanced memory-first architecture.

Monitoring Couchbase With Netdata

Monitoring Couchbase’s health and performance is essential to maintaining its operations-efficiently. With Netdata, you can seamlessly integrate Couchbase monitoring and get real-time insights into vital metrics. Netdata’s intuitive dashboard provides detailed metrics, enabling you to keep a close watch on Couchbase’s performance.

CouchDB

Mon, 01 Jan 0001 00:00:00 +0000

CouchDB Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

CouchDB Monitoring

What Is CouchDB?

CouchDB is an open-source database developed by the Apache Software Foundation, known for its seamless storage and retrieval of JSON documents. It features a distributed architecture with easy replication, making it a popular choice for web applications and mobile apps that require offline-first functionality. Learn more on the CouchDB official site.

Monitoring CouchDB With Netdata

Monitoring CouchDB is crucial to ensure optimal performance and reliability of your database infrastructure. Netdata provides powerful tools for monitoring CouchDB, offering real-time insights into various metrics such as database activity, HTTP requests, active tasks, and more. Leveraging the Netdata agent, you can efficiently track these metrics and respond swiftly to potential issues.

CPU performance

Mon, 01 Jan 0001 00:00:00 +0000

Cradlepoint

Mon, 01 Jan 0001 00:00:00 +0000

CraftBeerPi

Mon, 01 Jan 0001 00:00:00 +0000

CraftBeerPi Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

CraftBeerPi Monitoring

What Is CraftBeerPi?

CraftBeerPi is an open-source brewing automation software designed to revolutionize the homebrewing experience. By integrating with various sensors and controllers, CraftBeerPi helps manage the entire brewing process, ensuring precision and consistency with every batch.

Monitoring CraftBeerPi With Netdata

When it comes to ensuring the peak performance of your brewing setup, monitoring CraftBeerPi with Netdata is an optimal choice. To monitor CraftBeerPi, Netdata employs an openmetrics (Prometheus) exporter, specifically the CraftBeerPi exporter. This integration allows Netdata to ingest data from any Prometheus exporter seamlessly.

CrateDB

Mon, 01 Jan 0001 00:00:00 +0000

Cray Communications A S SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cray SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Croix Rouge Francaise SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Crossbeam Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Crowdsec

Mon, 01 Jan 0001 00:00:00 +0000

Crowdsec Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Crowdsec Monitoring

What Is Crowdsec?

Crowdsec is an open-source, collaborative security solution designed to detect and respond to threats. It emphasizes community collaboration to build a robust defense against cyber threats by leveraging shared threat intelligence.

Monitoring Crowdsec With Netdata

To effectively monitor Crowdsec, Netdata utilizes an openmetrics (Prometheus) exporter. This approach allows Netdata to assimilate data from any Prometheus exporter, providing seamless integration for users without the necessity of deploying a Prometheus server or Grafana. With automatic dashboards and alerting capabilities, Netdata transforms your Crowdsec monitoring setup into a high-performance, real-time monitoring system.

Cryptowatch

Mon, 01 Jan 0001 00:00:00 +0000

Cryptowatch Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Cryptowatch Monitoring

What Is Cryptowatch?

Cryptowatch is a powerful platform that provides market information for cryptocurrencies, allowing traders and analysts to track price changes, volume, and other critical metrics in real-time. It aggregates data from various exchanges to present a unified view of cryptocurrency markets, making it an indispensable tool for anyone involved in this digital asset space.

Monitoring Cryptowatch With Netdata

Netdata’s advanced monitoring capabilities make it an excellent choice for keeping tabs on Cryptowatch. By using an openmetrics (Prometheus) exporter, specifically the Cryptowatch Exporter, Netdata can seamlessly ingest data from Cryptowatch. This process does not require a full Prometheus server or Grafana setup, providing automated dashboards and alerts out of the box. This simplicity and integration capability mark Netdata as a leading Cryptowatch monitoring tool in the market.

Ctc Union Technologies Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cube Optics AG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

CUDA out of memory with free memory available: GPU memory fragmentation

Mon, 01 Jan 0001 00:00:00 +0000

CUDA out of memory with free memory available: GPU memory fragmentation

Your training run or inference server has been up for hours. Then it dies with torch.cuda.OutOfMemoryError: CUDA out of memory. You check nvidia-smi and the GPU shows gigabytes free. The error message and the driver disagree, and the driver looks right.

Both are right. The GPU has free memory in aggregate, but no single contiguous block large enough for the allocation that just failed. The memory is fragmented: free in total, unusable in practice.

CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion

Your training job or inference server died with RuntimeError: CUDA out of memory. Tried to allocate X MiB. Sometimes the GPU is genuinely full. More often, nvidia-smi shows gigabytes free and the error makes no sense at first glance.

GPU framebuffer is a cliff-edge resource. Unlike CPU memory there is no swap and no graceful degradation: when a cudaMalloc cannot be satisfied, the allocation fails atomically and the process dies. It is also one of the most misdiagnosed GPU failures, because the number nvidia-smi shows is not the number your framework is working with.

Cumulus Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

CUPS

Mon, 01 Jan 0001 00:00:00 +0000

Current_Pending_Sector non-zero: unreadable sectors and I/O latency spikes

Mon, 01 Jan 0001 00:00:00 +0000

Current_Pending_Sector non-zero: unreadable sectors and I/O latency spikes

A non-zero Current_Pending_Sector (SMART attribute ID 197) means the drive has sectors it cannot reliably read. These sectors failed a read and are waiting for either a write to trigger reallocation or an offline scan to confirm the defect. The drive has not yet remapped them to its spare pool.

The operational impact is often misdiagnosed. Every time the filesystem touches one of these sectors, the drive firmware enters a multi-second retry loop. This shows up in iostat as extremely high await with low throughput and idle CPU. Teams chase this as a software freeze, a kernel bug, or a filesystem problem for hours before checking SMART data.

Custom

Mon, 01 Jan 0001 00:00:00 +0000

Custom MMDB Database

Mon, 01 Jan 0001 00:00:00 +0000

Cxr SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cyan SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cyber Ark SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cyber Power System Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cyberguard Corporationdavid Rhein SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Cyberpower PDU

Mon, 01 Jan 0001 00:00:00 +0000

D Link Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dantel Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dantherm Cooling A S SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Danware Data A S SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dasan Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Data Domain Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Data Units Written vs rated TBW: computing SSD endurance runway

Mon, 01 Jan 0001 00:00:00 +0000

Data Units Written vs rated TBW: computing SSD endurance runway

SSD endurance is finite and specified in the datasheet as a TBW (Total Bytes Written) rating. The odometer is a SMART counter, but it uses non-obvious units and counts host writes, not NAND writes. This guide covers how to read the write-volume counter from smartctl, convert it to bytes correctly, compare it against rated TBW, and project when the drive reaches its endurance limit.

Datadirect Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Datalogic S P A SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Datapower Technology Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dataprobe Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

DB-IP IP Intelligence

Mon, 01 Jan 0001 00:00:00 +0000

DCGM nv-hostengine down or hung: GPU telemetry goes dark or stale

Mon, 01 Jan 0001 00:00:00 +0000

DCGM nv-hostengine down or hung: GPU telemetry goes dark or stale

Your GPU dashboards flatline, or worse, they do not. The graphs keep rendering smooth, plausible values for temperature, utilization, and power, but nothing on the node has actually changed in twenty minutes. Both symptoms point at the same component: nv-hostengine, the DCGM daemon that sits between NVML and every monitoring tool you run.

nv-hostengine fails in two distinct ways, and only one of them is obvious. The obvious failure is a dead process: no data, gaps in every time series, alerts firing on absent metrics. The dangerous failure is an alive-but-hung daemon: the process exists, the socket accepts connections, and DCGM serves stale values from its in-memory field value cache because an underlying NVML call is blocked on the driver. Dashboards look normal. Threshold alerts stay quiet because the cached values are inside bounds. You find out when a human notices the numbers have not moved.

DCGM, nvidia-smi, and NVML: which NVIDIA GPU telemetry source to use

Mon, 01 Jan 0001 00:00:00 +0000

DCGM, nvidia-smi, and NVML: which NVIDIA GPU telemetry source to use

Every NVIDIA GPU troubleshooting session starts with a telemetry question: where do I get this number, and can I trust it? Teams routinely mix the three available sources without realizing they sit at different layers of the same stack. They grep nvidia-smi output in cron jobs, run DCGM and nvidia-smi side by side and wonder why both get slow, or alert on a DCGM field that silently returns zeros on their driver version.

Deal Registration

Mon, 01 Jan 0001 00:00:00 +0000

Debian

Mon, 01 Jan 0001 00:00:00 +0000

Debian SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dec SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Decapsulation

Mon, 01 Jan 0001 00:00:00 +0000

Deep Sea Electronics PLC SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Del Mar Solutions Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Deliberant SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dell

Mon, 01 Jan 0001 00:00:00 +0000

Dell BGP

Mon, 01 Jan 0001 00:00:00 +0000

Dell EMC Data Domain

Mon, 01 Jan 0001 00:00:00 +0000

Dell EMC ScaleIO

Mon, 01 Jan 0001 00:00:00 +0000

Dell EMC ScaleIO Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Dell EMC ScaleIO Monitoring

What Is Dell EMC ScaleIO?

Dell EMC ScaleIO, also known as VxFlex OS, is a software-defined storage solution designed to turn local storage into shared block storage. ScaleIO is tailored for modern data centers, providing enhanced flexibility and efficiency in handling storage needs. You can learn more about ScaleIO here.

Monitoring ScaleIO With Netdata

Netdata offers a comprehensive and user-friendly platform to monitor ScaleIO instances effectively. By utilizing Netdata’s ScaleIO monitoring tools, you can collect detailed metrics from ScaleIO components via the VxFlex OS Gateway API. This enables real-time insights and actionable analytics, crucial for maintaining optimal performance.

Dell Force10

Mon, 01 Jan 0001 00:00:00 +0000

Dell Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dell OS10

Mon, 01 Jan 0001 00:00:00 +0000

Dell Powerconnect

Mon, 01 Jan 0001 00:00:00 +0000

Dell Poweredge

Mon, 01 Jan 0001 00:00:00 +0000

Dell PowerStore

Mon, 01 Jan 0001 00:00:00 +0000

Dell PowerVault ME4/ME5

Mon, 01 Jan 0001 00:00:00 +0000

Dell Sonicwall

Mon, 01 Jan 0001 00:00:00 +0000

Delta Electronics Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Delta Electronics Switzerland AG Formerly Delta Energy Systems Sweden AB SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Deltanet AG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Designer Systems Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

dev.cpu.0.freq

Mon, 01 Jan 0001 00:00:00 +0000

dev.cpu.temperature

Mon, 01 Jan 0001 00:00:00 +0000

Deva Broadcast Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Device control-plane CPU saturation: when SNMP polling causes the spike

Mon, 01 Jan 0001 00:00:00 +0000

Device control-plane CPU saturation: when SNMP polling causes the spike

The router’s control-plane CPU is pinned at 95%. SNMP polls are timing out. BGP sessions are approaching hold-time expiry. The instinct is to blame the device or suspect an attack, but the monitoring system itself is frequently the source of the load.

The control-plane CPU handles everything that is not hardware-forwarded packet switching: the SNMP agent, BGP, OSPF, STP, the CLI, syslog, AAA, and management interfaces. When SNMP polling saturates this CPU, every control-plane function degrades at once. The symptoms look like a device problem. The cause is often on the collector side.

Device memory pressure: control-plane memory pools and leaks

Mon, 01 Jan 0001 00:00:00 +0000

Device memory pressure: control-plane memory pools and leaks

A network device alerts at 85% control-plane memory utilization. Or worse: it does not alert, and you discover the problem when BGP sessions start dropping from hold-time expiry, SNMP stops responding, or the device reboots itself. Control-plane memory exhaustion degrades every process on the route processor: routing protocols, CLI, SNMP, management interfaces, logging.

“High memory utilization” on a network device is ambiguous. Some platforms cache aggressively and report 97% used under normal operation. Some pools are expected to sit near zero free. A genuine memory leak may take weeks to manifest, making it easy to dismiss the trend until the device crashes at 3 a.m.

devstat

Mon, 01 Jan 0001 00:00:00 +0000

Dialogic Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dialogic Media Gateway

Mon, 01 Jan 0001 00:00:00 +0000

Didactum Security GmbH Formerly Didactum Ltd Deutschland SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Digipower Manufacturing Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Digital China Shanghai Networks Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Digital Link SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Digital Video Broadcasting Dvb SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Discord

Mon, 01 Jan 0001 00:00:00 +0000

Discord

Mon, 01 Jan 0001 00:00:00 +0000

Discourse

Mon, 01 Jan 0001 00:00:00 +0000

Discourse Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Discourse Monitoring

What Is Discourse?

Discourse is a modern forum software for your community. It reimagines what a discussion platform should be, offering seamless integration with existing web apps, and a mobile-optimized experience.

Monitoring Discourse With Netdata

To monitor Discourse effectively and ensure optimal performance, you can rely on Netdata—a powerful real-time monitoring solution. Netdata leverages the Discourse Prometheus exporter to collect relevant metrics. This allows users to monitor Discourse without the need for setting up a Prometheus server or Grafana. Automated dashboards, alerts, and visual insights are readily available from Netdata’s platform.

Disk space

Mon, 01 Jan 0001 00:00:00 +0000

Disk Statistics

Mon, 01 Jan 0001 00:00:00 +0000

Dismuntel S A L SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Distributed Management Task Force Dmtf SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Distributed Processing Technology SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dlink

Mon, 01 Jan 0001 00:00:00 +0000

Dlink DGS Switch

Mon, 01 Jan 0001 00:00:00 +0000

DMARC

Mon, 01 Jan 0001 00:00:00 +0000

DMARC Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

DMARC Monitoring

What Is DMARC?

DMARC (Domain-based Message Authentication, Reporting & Conformance) is an email authentication policy and reporting protocol. It is designed to give domain owners a way to protect their domain from being used in email spoofing, phishing scams, and other cybercrimes. By implementing DMARC, organizations can significantly improve the security and credibility of their email communications.

Monitoring DMARC With Netdata

Netdata provides a robust DMARC monitoring tool using the openmetrics exporter from Prometheus. Specifically, it uses the dmarc-metrics-exporter to collect detailed metrics that offer insights into email authentication processes. With Netdata, you can ingest data from any Prometheus exporter and instantly gain access to automated dashboards and alerts without necessitating a Prometheus server or Grafana setup. This streamlined setup helps you keep a close watch on DMARC metrics in a user-friendly manner.

DMCache devices

Mon, 01 Jan 0001 00:00:00 +0000

DMCache Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

DMCache Monitoring

What Is DMCache?

DMCache, a critical component of storage systems, acts as a device mapper cache enabling systems to manage disk block caching. By leveraging DMCache, systems can significantly enhance I/O performance by storing frequently accessed disk blocks in a faster storage medium.

Monitoring DMCache With Netdata

Effective DMCache monitoring ensures that your storage systems are performing optimally and that resources are efficiently utilized. The Netdata DMCache monitoring tool offers real-time visibility into your DMCache devices, allowing you to track critical performance metrics and gain actionable insights through comprehensive visualizations.

DNS query

Mon, 01 Jan 0001 00:00:00 +0000

DNS Query Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

DNS Query Monitoring

What Is DNS Query?

DNS Query refers to the process by which an inquiry is made to the Domain Name System to obtain information about a domain, such as the corresponding IP address. It’s a critical function that ensures users can access websites and services smoothly.

Monitoring DNS Query With Netdata

Netdata offers a robust DNS Query monitoring tool that allows users to track the round-trip time (RTT) of DNS queries. Utilizing Netdata, you can efficiently monitor DNS query performance, ensuring that your applications and services maintain optimal connectivity.

DNSBL

Mon, 01 Jan 0001 00:00:00 +0000

DNSBL Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

DNSBL Monitoring

What Is DNSBL?

DNSBL, or Domain Name System-based Blackhole List, is a mechanism that helps identify and block email messages from known spammers by using a real-time database of blacklisted IP addresses often employed by email servers to filter out unwanted messages and reduce spam.

Monitoring DNSBL With Netdata

To monitor DNSBL, Netdata utilizes an openmetrics (Prometheus) exporter. This approach ensures that users can gather comprehensive metrics related to DNSBL effectively. With Netdata, there is no necessity for a dedicated Prometheus server or Grafana setup as Netdata integrates seamlessly with any Prometheus exporter, showcasing automated dashboards, alerts, and in-depth visibility into DNSBL metrics such as domain reputation and security management.

DNSdist

Mon, 01 Jan 0001 00:00:00 +0000

DNSdist Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

DNSdist Monitoring

What Is DNSdist?

DNSdist is a highly flexible and powerful DNS proxy server designed to handle a wide range of queries efficiently. It helps balance and protect your DNS infrastructure with advanced features like query filtering, load balancing, and telemetry.

Monitoring DNSdist With Netdata

Netdata provides unparalleled visibility into DNSdist operations through its comprehensive dashboard. By using the DNSdist monitoring tool, you can quickly understand traffic patterns, identify bottlenecks, and ensure DNS reliability.

Dnsmasq

Mon, 01 Jan 0001 00:00:00 +0000

Dnsmasq DHCP

Mon, 01 Jan 0001 00:00:00 +0000

Dnsmasq DHCP Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is Dnsmasq for DHCP?

Dnsmasq is an open-source, lightweight, DNS caching and forwarding server. It is designed to provide DNS resolution for small and home networks. Dnsmasq provides local DNS caching, forwarding, and recursive lookups, as well as DHCP, TFTP, and other related services. It also has support for DNS and DHCPv6, as well as various other features such as DNS-over-TLS and IPv6 privacy extensions. Dnsmasq is a versatile and highly configurable tool that is simple to use. DNSMasq_DHCP is a feature of DNSMasq that provides a combined server to serve both DNS (Domain Name System) and DHCP (Dynamic Host Configuration Protocol) requests. It is a fast and lightweight DHCP server with support for both IPv4 and IPv6, and can be used to serve IP addresses to hosts on a LAN. DNSMasq_DHCP also offers features such as DNS and DHCP performance tuning, DHCP address range management, and support for multiple DNS domains.

Dnsmasq DHCP Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Dnsmasq DHCP Monitoring

Dnsmasq is a lightweight, easy-to-configure, and highly efficient DHCP server that serves small- to medium-sized networks. Monitoring Dnsmasq DHCP effectively ensures the reliability and performance of your network configuration services.

What Is Dnsmasq DHCP?

Dnsmasq provides network infrastructure for small networks: DNS, DHCP, router advertisement, and network boot. These server functionalities are crucial for network management and troubleshooting.

Monitoring Dnsmasq DHCP With Netdata

Netdata is the perfect solution for monitoring Dnsmasq DHCP as it provides real-time monitoring, enabling swift identification and resolution of potential network issues. With its turnkey installation, Netdata automatically detects and visualizes metrics needed for effective Dnsmasq DHCP monitoring.

Dnsmasq Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Dnsmasq Monitoring

What Is Dnsmasq?

Dnsmasq is a lightweight and robust DNS forwarder and DHCP server, perfect for networking tools used in small to medium scale networks. It’s designed to be easy to configure while maintaining a wide variety of functionalities. Dnsmasq can cache DNS queries for improved performance of local networks. You can learn more about Dnsmasq on its official documentation.

Monitoring Dnsmasq With Netdata

Netdata provides an unparalleled real-time monitoring solution for Dnsmasq. By leveraging Netdata’s Dnsmasq monitoring tool, administrators and developers gain seamless insights into the DNS and DHCP services. Netdata’s easy-to-navigate UI, with real-time visualizations, and live demo, brings valuable metrics to the forefront, aiding proactive diagnosis and trend recognition.

Docker

Mon, 01 Jan 0001 00:00:00 +0000

Docker

Mon, 01 Jan 0001 00:00:00 +0000

Docker

Mon, 01 Jan 0001 00:00:00 +0000

Docker commands hang: docker ps, inspect, and exec freezes

Mon, 01 Jan 0001 00:00:00 +0000

Docker Commands Hang: docker ps, inspect & exec Freezes

docker ps, docker inspect, and docker exec hang while containers continue serving traffic. This is a Docker daemon hang: the management plane is dead while the data plane survives. Standard process monitors show dockerd as alive, yet you cannot manage, inspect, or evacuate workloads.

This guide covers how to distinguish a hang from a crash, identify whether the root cause is storage, a stuck shim, a plugin, or an internal deadlock, and recover without unnecessary host reboots or container kills.

Docker container cannot connect to another container

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container Cannot Connect To Another Container

Connection timeouts or “connection refused” between containers on the same host point to a failure in one of four layers: the bridge network, Docker’s embedded DNS, iptables, or the application bind address. Inter-container networking relies on Linux bridges, veth pairs, iptables rules, and an embedded DNS resolver at 127.0.0.11. This guide isolates the faulty layer and fixes it.

What This Means

Docker isolates each container in its own network namespace. Containers on the same custom bridge communicate through a Linux bridge and resolve each other by name via Docker’s embedded DNS. The default bridge network has no embedded DNS. If containers are on different networks, if the target application binds to 127.0.0.1, or if iptables rules were wiped by an external firewall reload, traffic stops. The symptom appears at the application layer, but the root cause can be Layer 2, Layer 3, or the application itself.

Docker container cannot connect to the internet: diagnosis and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Docker container cannot connect to the internet

The host may be online while Docker’s bridge, iptables rules, DNS proxy, or connection tracking table is in a broken state. Applications fail with connection timeouts, package managers stall, and external health checks return unhealthy. Isolate whether the failure is DNS, routing, packet filtering, or daemon state before fixing it.

What this means

Outbound connectivity from a container traverses the container’s network namespace, a veth pair attached to a bridge (docker0 or user-defined), iptables NAT and filter rules managed by Docker’s libnetwork, the host’s routing table, and the upstream physical interface. On user-defined networks, Docker’s embedded DNS resolver at 127.0.0.11 proxies queries to the host’s configured resolvers. On the default bridge network, there is no embedded DNS; containers inherit the host’s /etc/resolv.conf directly. A failure at any layer produces the same symptom: requests time out.

Docker container exits immediately: how to diagnose it

Mon, 01 Jan 0001 00:00:00 +0000

Docker container exits immediately: how to diagnose it

A container that exits immediately prints its ID during docker run, appears briefly in docker ps, then vanishes. docker ps -a lists it as Exited with an uptime measured in seconds. Unlike a restart loop, it stays stopped.

A Docker container is a Linux process wrapper, not a virtual machine. Its lifecycle is bound to PID 1 in the container’s namespace. When that process completes, crashes, or is killed, the container exits. An immediate exit means the process finished its work, failed to start, or was terminated during initialization. This guide gives you a diagnostic flow to find out why PID 1 terminated, interpret the exit code, and fix the root cause.

Docker container high CPU usage: causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Docker container high CPU usage: causes and fixes

A container alert for high CPU is easy to generate but hard to interpret. Docker reports CPU as a percentage of total host cores, so a value of 340% on an 8-core machine simply means the container is using 3.4 cores. That might be expected for a multi-threaded workload, or it might signal a runaway process, CFS throttling, or a cryptominer. This guide shows how to distinguish legitimate compute from pathological behavior, and how to fix the root cause without restarting the container blindly.

Docker container high memory usage: how to diagnose it

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container High Memory Usage: How To Diagnose It

Your container is sitting at 90% of its memory limit but has not been OOMKilled. Or it is being killed repeatedly and you cannot tell whether the limit is too low or the application is leaking. docker stats shows a single percentage, but that number mixes reclaimable page cache with anonymous memory that the kernel cannot reclaim. To diagnose this correctly, you need to decompose cgroup memory.stat, map it to your runtime’s actual allocations, and decide whether the problem is cache pressure, a runtime mismatch, or a true leak.

Docker container keeps restarting: causes, checks, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container Keeps Restarting: Causes, Checks & Fixes

A container that restarts every few seconds is not self-healing. It is a crash loop that wastes CPU, floods logs, and masks the real failure. Docker’s restart policy can hide whether the application is OOM-killed, segfaulting, or waiting for a dependency that never arrives. This guide shows how to read the signals, map exit codes to causes, and stop the loop before it degrades the host.

Docker container memory leak: how to find one and prove it

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container Memory Leak: How To Find One And Prove It

Memory that only ever climbs is easy to spot. The harder problem is proving whether the growth is a leak, unbounded caching, or a limit set below the working set. During an incident, operators need to decide in minutes whether to page an on-call developer or bump a cgroup limit. This guide shows how to use cgroup memory.stat, process-level RSS, and container restart patterns to build a defensible diagnosis. You will be able to separate anonymous memory growth from reclaimable cache, identify whether the leak lives in application heap or runtime overhead, and present evidence that justifies either a code fix or a capacity change.

Docker container running but unhealthy: how to diagnose health check failures

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container Running But Unhealthy: How To Diagnose Health Check Failures

The container is up. docker ps shows it running. But the status is (unhealthy) and your load balancer or orchestrator has stopped sending traffic. This means Docker’s health check command is returning a non-zero exit code, or timing out, while the container’s main process stays alive. The container is running, but Docker does not consider it ready.

An unhealthy state is not a crash. In Swarm, it triggers replacement. In Compose, depends_on with condition: service_healthy blocks downstream services. Even on a single host, an unhealthy mark often precedes a restart loop that buries the real error in noise. You need to distinguish between a broken application, a broken probe, and a broken runtime configuration.

Docker Containers

Mon, 01 Jan 0001 00:00:00 +0000

Docker CPU throttling: the hidden cause of container latency

Mon, 01 Jan 0001 00:00:00 +0000

Docker CPU Throttling: The Hidden Cause Of Container Latency

Your application latency just spiked. p99 response times doubled or tripled. CPU dashboards show the container at 40% utilization. Memory is fine. Network is quiet. You restart the container, redeploy, or blame the code, but the pattern repeats.

The culprit is often CPU throttling. Docker uses Linux CFS bandwidth control to enforce CPU limits in discrete 100ms periods. A container can exhaust its quota in a burst, spend the rest of each period throttled by the kernel, and still report a modest average CPU over a longer window. This guide shows how to confirm throttling from cgroup metrics, calculate its severity, and fix it without guessing.

Docker daemon not responding: how to troubleshoot a hung dockerd

Mon, 01 Jan 0001 00:00:00 +0000

Docker Daemon Not Responding: How To Troubleshoot A Hung Dockerd

A hung Docker daemon is one of the more disorienting production failures you can face. The process is still running, your containers are still serving traffic, but every docker command hangs. You cannot inspect, stop, or create containers. Deployments stall. Automation times out.

This guide covers the diagnostic ladder from socket probe to storage driver investigation, explains when to wait versus when to restart, and describes what to avoid when you are not sure what is wrong.

Docker disk space full: how to troubleshoot /var/lib/docker

Mon, 01 Jan 0001 00:00:00 +0000

Docker Disk Space Full: How To Troubleshoot /var/lib/docker

You notice deployments failing with “no space left on device,” image pulls hanging, or the Docker daemon becoming sluggish. On a Docker host, everything lives under /var/lib/docker: image layers, container writable layers and logs, named volumes, and build cache. When this filesystem fills, the failure is cascading and abrupt. New containers cannot start, running containers may fail on writes, and daemon operations deadlock.

Docker DNS not working inside containers

Mon, 01 Jan 0001 00:00:00 +0000

Docker DNS not working inside containers

Your application logs show connection timeouts. Health checks against dependency names are failing. curl from inside a container returns “Could not resolve host” while the same name resolves fine on the host. DNS inside Docker is not a simple passthrough to the host resolver. It is a stack of namespace-specific forwarders, embedded resolvers, and inherited configuration that breaks in specific, repeatable ways.

This guide will help you determine whether the failure is a missing embedded DNS, a poisoned resolv.conf, an upstream forwarding issue, or a version regression. You will be able to distinguish between inter-container name resolution failures and external lookup failures, identify the root cause with safe read-only checks, and apply the correct fix without guessing.

Docker Engine

Mon, 01 Jan 0001 00:00:00 +0000

Docker exit code 1: application errors and how to find them

Mon, 01 Jan 0001 00:00:00 +0000

Docker Exit Code 1: Application Errors And How To Find Them

A container exiting immediately with code 1 is a common production incident. Unlike exit code 137 (OOM killer) or code 125 (Docker daemon error), code 1 means the application inside the container called exit(1). Docker is only reporting what PID 1 did.

The challenge is that code 1 is a catch-all. It can mask an unhandled JavaScript exception, a Python import error, a missing configuration file, a shell script failing under set -e, or a Go binary that cannot reach its database. Time spent checking Docker daemon health or host memory is wasted when the real failure is an application-level error written to stdout or stderr.

Docker exit code 137: OOMKilled or SIGKILL?

Mon, 01 Jan 0001 00:00:00 +0000

Docker Exit Code 137: OOMKilled Or SIGKILL?

A container exits with code 137. Docker restarts it, or it stays down, and you need to know why. The number itself only tells you that the process received SIGKILL. What matters for your next step is whether the kernel’s cgroup OOM killer fired because the container exceeded its memory limit, or whether an external actor sent the signal. The remediation for an undersized memory limit is completely different from fixing a misconfigured stop timeout or an orchestrator sending a premature kill. This guide shows how to classify the cause in under a minute using only the Docker CLI and cgroup files.

Docker exit code 143: SIGTERM and graceful shutdown failures

Mon, 01 Jan 0001 00:00:00 +0000

Docker exit code 143: SIGTERM and graceful shutdown failures

You are reviewing container exit codes after a deployment or node drain and see 143. If your monitoring alerts on it, you might think something failed. Exit code 143 is not an error. It is 128 plus signal 15 (SIGTERM), and it means the container’s PID 1 process received SIGTERM and exited voluntarily. This is exactly what docker stop is designed to do.

Docker Hub Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Docker Hub Monitoring

What Is Docker Hub?

Docker Hub is a cloud-based repository where Docker users and partners create, test, store, and distribute container images. It is the centralize hub for approximately 100,000 container images; providing the ability to automate workflows, build images from GitHub or Bitbucket, and distribute them to your users.

Monitoring Docker Hub With Netdata

To effectively monitor Docker Hub, leverage the insightful metrics retrieved by Netdata’s robust monitoring agent—the go-to Docker Hub monitoring tool. Netdata’s Docker Hub collector keeps track of your repository statistics, providing essential visibility into its health and performance.

Docker Hub repository

Mon, 01 Jan 0001 00:00:00 +0000

Docker image cleanup: safe pruning strategies for production hosts

Mon, 01 Jan 0001 00:00:00 +0000

Docker image cleanup: safe pruning strategies for production hosts

When df -h /var/lib/docker shows 87% utilization, the urge to run docker system prune -a is strong. On production hosts, that is a mistake. Cleanup is not about finding the single command that reclaims the most space. It is about knowing exactly what each flag deletes, what it leaves behind, and which filters prevent a 3 a.m. image re-pull because a base layer was removed.

Docker image pull failures: registry, network, and auth diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

Docker Image Pull Failures: Registry, Network & Auth Diagnosis

What This Means

When you run docker pull, the daemon negotiates a TLS connection to the registry, authenticates if required, resolves the manifest for the requested tag and architecture, then downloads missing layers. If any step fails, the pull aborts.

Registry errors surface as HTTP 429 or 401/403 responses. Network errors appear as timeouts, connection resets, or TLS handshake failures. Local problems such as a full disk or a hung daemon can also abort a pull even when the registry is healthy. In orchestrated environments, a single node’s pull failure can trigger the scheduler to retry on other nodes, turning a localized auth error into a cluster-wide rate limit storm. Distinguishing these layers quickly is the difference between a five-minute fix and a prolonged outage.

Docker JVM memory tuning: heap, off-heap, and the cgroup mismatch

Mon, 01 Jan 0001 00:00:00 +0000

Docker JVM Memory Tuning: Heap, Off-Heap, And The cgroup Mismatch

Your Java container is OOMKilled at 02:00. Heap usage is 60%. Docker reports exit code 137 and OOMKilled: true. The JVM never threw an OutOfMemoryError.

In a container, the kernel enforces memory limits through cgroups, but the JVM heap is only one component of process RSS. Off-heap memory, metaspace, thread stacks, direct byte buffers, and GC overhead all count against the same cgroup limit. When total RSS crosses that limit, the kernel kills the container without a JVM-level error. In some JDK and kernel combinations, the JVM fails to detect the cgroup limit entirely and sizes the heap against host RAM, which guarantees an OOM kill.

Docker log rotation: preventing json-file logs from filling disk

Mon, 01 Jan 0001 00:00:00 +0000

Docker Log Rotation: Preventing json-file Logs From Filling Disk

Docker’s default json-file log driver appends every line of container stdout and stderr to a JSON file on the host under /var/lib/docker/containers/<id>/. Without size limits, that file grows monotonically. A single verbose container can consume tens of gigabytes, and because the driver is the default, this often happens silently until /var/lib/docker fills. At that point image pulls fail, container creates are rejected, and the daemon may hang on storage operations. This guide covers how to cap log files with daemon.json and per-container log-opt overrides, verify the caps are working, and choose a different driver when json-file is not appropriate.

Docker logs taking too much disk space: how to fix log growth

Mon, 01 Jan 0001 00:00:00 +0000

Docker Logs Taking Too Much Disk Space: How To Fix Log Growth

The default json-file logging driver captures every line a container writes to stdout and stderr and appends it to a file under /var/lib/docker/containers/. There is no size limit out of the box. A container with verbose logging, a stuck retry loop, or a crash spiral can grow its log file by gigabytes per day until the filesystem is full.

Docker memory limits: how to set them and what happens when they hit

Mon, 01 Jan 0001 00:00:00 +0000

Docker memory limits: how to set them and what happens when they hit

A container without a memory limit can consume all available RAM, force the kernel to reclaim page cache, push the system into swap, and eventually trigger a host-level OOM kill that takes down the Docker daemon or other critical processes. Setting --memory is not enough: limits can be silently ignored, misread by runtimes, or masked by swap behavior that turns a clean failure into a slow crawl.

Docker memory usage explained: anonymous, file, slab, and what counts

Mon, 01 Jan 0001 00:00:00 +0000

Docker memory usage explained: anonymous, file, slab, and what counts

You look at docker stats, see a container sitting at 1.2 GB of a 1.5 GB limit, and assume it is about to explode. It might be fine. That total includes page cache the kernel will reclaim the second another process asks for memory. Meanwhile, a different container reports 600 MB with no limit set, yet its anonymous memory grows 50 MB per hour and will force a host-level OOM kill before lunch.

Docker Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Docker Monitoring

What Is Docker?

Docker is an open platform that packages applications and their dependencies into lightweight, portable units called containers. A container bundles everything the software needs to run — code, runtime, system tools, libraries, and settings — so it behaves the same way on a developer’s laptop, in staging, and in production. Because containers share the host operating system’s kernel instead of shipping a full guest OS, they start in milliseconds and use far fewer resources than virtual machines.

Docker monitoring checklist: the signals every production host needs

Mon, 01 Jan 0001 00:00:00 +0000

Docker Monitoring Checklist: The Signals Every Production Host Needs

Production Docker incidents rarely look like Docker problems at first. They show up as application latency, deployment failures, or hosts that suddenly refuse to schedule containers. By the time you notice, the daemon may be hung, a log file has filled the disk, or a container has been silently throttled into unusable latency. This checklist groups the essential production signals into three priority tiers: must-have alerts that keep the host alive, should-have metrics that expose resource pressure before it becomes an outage, and nice-to-have security and internal signals for mature environments. Every signal includes where to read it from the raw cgroup filesystem or the Docker API so you can instrument hosts without guessing paths.

Docker OOMKilled: causes, detection, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

Docker OOMKilled: causes, detection, and prevention

A container exits with code 137 and restarts. The application loses in-memory state. Dependent services start failing. The restart loop begins. This is the OOMKilled pattern, and it is one of the most common and most misdiagnosed failure modes in Docker environments.

This article covers how to confirm an OOM kill, distinguish it from an external SIGKILL, understand why it happened, and prevent recurrence. It also covers the JVM-in-container memory mismatch, which is responsible for a large share of OOM kills in Java workloads.

Docker port binding: address already in use

Mon, 01 Jan 0001 00:00:00 +0000

Docker port binding: address already in use

A docker run -p 8080:80 or docker compose up fails with bind: address already in use. The error is clear, but the owner is not. ss may show a system service. docker ps may show nothing, yet Docker still refuses. The port can appear free while an orphaned DNAT rule or a split firewall backend blocks the bind.

Distinguish a genuine socket conflict from an orphaned DNAT rule, a rootless Docker regression, and a WSL2 iptables/nftables split brain. Then reclaim the port and prevent recurrence.

Docker published port not reachable: troubleshooting -p and EXPOSE

Mon, 01 Jan 0001 00:00:00 +0000

Docker published port not reachable: troubleshooting -p and EXPOSE

You mapped a port with -p 8080:80, but curl against the host IP returns connection refused. docker ps shows the mapping, the container is running, and the port still appears closed.

A published port depends on three layers: a runtime mapping rule (-p), a host forwarding path (iptables DNAT and FORWARD policy), and an application listener inside the container bound to an interface that receives the forwarded packet. EXPOSE in a Dockerfile is metadata. It does not publish ports, create firewall rules, or set bind addresses.

Docker socket security: why /var/run/docker.sock is root access

Mon, 01 Jan 0001 00:00:00 +0000

Docker socket security: why /var/run/docker.sock is root access

Mounting /var/run/docker.sock into a container grants that container a root-equivalent privilege boundary. This is not a Docker vulnerability. The daemon runs as root, listens on a filesystem socket, and trusts any client that can write to it.

Symptoms of abuse look like container escape or host compromise, but the container never escaped. It asked the root-owned daemon to perform privileged operations on its behalf. This guide explains the mechanism, how to audit exposure, and what constraints you can apply without rebuilding your pipeline.

Docker volume cleanup: finding and removing orphaned volumes

Mon, 01 Jan 0001 00:00:00 +0000

Docker volume cleanup: finding and removing orphaned volumes

Run docker system df and the Local Volumes line keeps growing. Run docker system prune and you reclaim images and build cache, but the volume count barely drops. A few weeks later the disk alert fires again.

Data persists in /var/lib/docker/volumes/ after its consumer is long gone. These volumes waste disk, complicate capacity planning, and can hide sensitive data in forgotten corners of the filesystem.

Domain expiration date

Mon, 01 Jan 0001 00:00:00 +0000

Domain Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Domain Monitoring

What Is Domain Monitoring?

Domain monitoring involves tracking various domain-related metrics such as the time until a domain expires. This ensures that your domain remains operational without unexpected downtime due to expiry. By using tools designed for monitoring domains, businesses can avoid disruptions to their online presence.

Monitoring Domain Expiration With Netdata

Netdata offers a comprehensive domain monitoring tool through its WhoisQuery module. This tool helps in tracking domain expiration dates effectively. By integrating this into your monitoring strategy, you can maintain continuous oversight of your domains’ expiration status and take proactive measures to renew them on time.

Dovecot

Mon, 01 Jan 0001 00:00:00 +0000

Dovecot Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Dovecot Monitoring

What Is Dovecot?

Dovecot is a widely-used open-source IMAP and POP3 server for Unix-like operating systems. Its focus on security, ease of use, and performance makes it a preferred choice for many system administrators when it comes to mail server deployments. For more details on Dovecot, visit the official Dovecot website.

Monitoring Dovecot With Netdata

Netdata provides a comprehensive $name monitoring tool designed to capture and visualize a wide array of metrics from your Dovecot server. By monitoring Dovecot with Netdata, you gain real-time insights into your server’s performance, understand traffic patterns, and detect anomalies that could indicate underlying issues.

Dps Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dragonwave SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Drive disappeared from the bus: sudden controller or electronics death

Mon, 01 Jan 0001 00:00:00 +0000

Drive disappeared from the bus: sudden controller or electronics death

A device node that was present minutes or hours ago is now gone. lsblk no longer shows it. smartctl returns “No such device.” The drive did not warn you through SMART because the component that failed is the one that would have reported the problem. This is one of the few storage failure modes that produces zero SMART telemetry before it happens.

Drive temperature too high: HDD, SATA SSD, and NVMe thresholds

Mon, 01 Jan 0001 00:00:00 +0000

Drive temperature too high: HDD, SATA SSD, and NVMe thresholds

A temperature reading of 68C means very different things depending on what is in the slot. For an enterprise HDD, it is past the danger threshold and the drive is likely sustaining damage. For an NVMe SSD, it may be within normal operating range and the drive is not even throttling yet. Alerting on a single global threshold across mixed drive types produces two failure modes simultaneously: false pages on normal NVMe temperatures and missed alerts on dangerously hot HDDs.

Dutch Electricity Smart Meter

Mon, 01 Jan 0001 00:00:00 +0000

Dutch Electricity Smart Meter Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Dutch Electricity Smart Meter Monitoring

What Is Dutch Electricity Smart Meter?

Dutch Electricity Smart Meters are advanced IoT devices used for tracking energy consumption in households. They provide granular data through the P1 port, which is beneficial for energy management and efficient monitoring. To gather insights, these meters can be hooked up with various monitoring tools, such as the Prometheus P1 Exporter.

Monitoring Dutch Electricity Smart Meters With Netdata

To monitor Dutch Electricity Smart Meters, Netdata utilizes an openmetrics (Prometheus) exporter. This integration enables Netdata to ingest data from any Prometheus exporter. With this setup, users benefit from Netdata’s automated dashboards, alerts, and other features without the need for a Prometheus server or Grafana. This means you can have a comprehensive and real-time view of your smart meter data effortlessly.

Dynatech Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Dynatrace

Mon, 01 Jan 0001 00:00:00 +0000

Dynatrace

Mon, 01 Jan 0001 00:00:00 +0000

Dynatrace Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Dynatrace Monitoring

What Is Dynatrace?

Dynatrace is a leading application performance management (APM) solution that provides comprehensive visibility into your applications, infrastructure, and user experiences. It is designed to help organizations monitor, optimize, and scale their environments effectively. Whether it’s cloud-native applications or more traditional server-client architectures, Dynatrace offers automated and intelligent observability across the stack.

Monitoring Dynatrace With Netdata

Monitoring Dynatrace with Netdata offers seamless integration via the openmetrics (Prometheus) exporter. Netdata can ingest metrics from any Prometheus exporter, allowing it to create automated dashboards, alerts, and more without the need for a Prometheus server or Grafana. By utilizing the Dynatrace Exporter, you can easily monitor Dynatrace metrics in real-time with Netdata, enabling better application performance management.

E Dynamics Org SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

E T A Elektrotechnische Apparate GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Eastern Research Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Eaton Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Eaton Epdu

Mon, 01 Jan 0001 00:00:00 +0000

Eaton UPS

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Cachestat

Mon, 01 Jan 0001 00:00:00 +0000

eBPF DCstat

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Disk

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Filedescriptor

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Filesystem

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Hardirq

Mon, 01 Jan 0001 00:00:00 +0000

eBPF MDflush

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Mount

Mon, 01 Jan 0001 00:00:00 +0000

eBPF OOMkill

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Process

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Processes

Mon, 01 Jan 0001 00:00:00 +0000

eBPF SHM

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Socket

Mon, 01 Jan 0001 00:00:00 +0000

eBPF SoftIRQ

Mon, 01 Jan 0001 00:00:00 +0000

eBPF SWAP

Mon, 01 Jan 0001 00:00:00 +0000

eBPF Sync

Mon, 01 Jan 0001 00:00:00 +0000

eBPF VFS

Mon, 01 Jan 0001 00:00:00 +0000

Ecreso SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Edial Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Egenera Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Egnite GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Eicon SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ekinops Sas SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch

Mon, 01 Jan 0001 00:00:00 +0000

ElasticSearch

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch all shards failed: diagnosing search_phase_execution_exception

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch all shards failed: diagnosing search_phase_execution_exception

You run a search and Elasticsearch returns search_phase_execution_exception with reason all shards failed. Every shard copy involved in the query returned a failure to the coordinating node. The outer error is a container; the actual root cause lives in the per-shard failure reasons inside the response body. Do not assume the cluster is down. This error fires on clusters with green health and stable nodes when a query is malformed, a mapping is incompatible, or a resource limit is breached uniformly across every target shard.

Elasticsearch ALLOCATION_FAILED after max retries: reroute and corrupt shard recovery

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch ALLOCATION_FAILED after max retries: reroute and corrupt shard recovery

A shard that repeatedly fails allocation exhausts index.allocation.max_retries (default 5) and becomes permanently UNASSIGNED. Elasticsearch stops automatic placement. Cluster health is RED if the shard is a primary, YELLOW if it is a replica. Indexing to the affected index is blocked until the primary is assigned.

This state typically follows transient node restarts, disk pressure events, or translog corruption. After the fifth failed attempt, the allocator stops retrying. The shard will not move without explicit operator action: POST /_cluster/reroute?retry_failed=true for transient failures, or forced allocation with accepted data loss when no valid copy remains.

Elasticsearch authentication failures: audit logs, brute force, and credential drift

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch authentication failures: audit logs, brute force, and credential drift

Elasticsearch does not expose authentication failure counts through _nodes/stats or _cluster/health. The security audit log is the only structured source for authentication_failed, access_denied, and run_as_denied events. Without it, brute force attempts, credential stuffing, and expiring service tokens are invisible until they cause an outage.

What this means

When xpack.security.audit.enabled is true, each node writes security events to a local audit log file. The events that matter for auth issues are:

Elasticsearch CircuitBreakingException: [parent] Data too large - causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch CircuitBreakingException: [parent] Data too large - causes and fixes

When a search or indexing request returns HTTP 429 with CircuitBreakingException: [parent] Data too large, data for [<http_request>] would be [X], which is larger than the limit of [Y], the parent circuit breaker has rejected the operation. This is Elasticsearch protecting the JVM from an out-of-memory kill, not a client-side rate limit.

Since version 7.0, the parent breaker tracks real memory usage by default. It can trip even when individual child breakers are within limits. The node is under genuine heap pressure. Determine quickly whether the cause is a single abusive query or structural memory exhaustion.

Elasticsearch cluster health red: unassigned primaries and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch cluster health red: unassigned primaries and how to recover

Cluster health red means at least one primary shard is unassigned. Queries against affected indices return partial results or fail; writes are blocked. yellow only signals missing replicas, but red signals active data unavailability.

Cluster health is a lagging indicator. A red status sustained longer than two minutes after the cluster has formed is a real fault; a brief flash during startup is normal. By the time the status turns red, a node has likely departed, a disk has crossed a watermark, or a shard copy has been rejected as corrupt.

Elasticsearch cluster health yellow: unassigned replicas vs real allocation blocks

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch cluster health yellow: unassigned replicas vs real allocation blocks

GET /_cluster/health returning status: yellow means all primaries are assigned but at least one replica is not. Some clusters are yellow by design. Others are yellow because a disk cascade, stuck allocator, or failed shard is blocking recovery. Benign yellow resolves. Structural yellow persists. If your cluster has been yellow for more than thirty minutes, something is actively blocking allocation. Teams that treat yellow as cosmetic often miss the transition from transient recovery to a real incident, leaving them one node failure away from red.

Elasticsearch cluster state too large: field count, index count, and per-node heap

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch cluster state too large: field count, index count, and per-node heap

Every node holds a copy of the cluster state in heap. When it grows large, the cost is paid everywhere: 200 MB of state consumes 200 MB on every node, and the elected master burns additional CPU and heap serializing and publishing updates. Symptoms show up indirectly: the master feels sluggish, pending tasks queue for minutes, heap pressure climbs on nodes that should be idle, and master elections stall indexing and shard allocation. The usual drivers are too many indices from per-minute or per-hour time-series patterns; a mapping explosion from uncontrolled dynamic fields; excessive aliases; or churn from frequent template and setting changes. Raw size measured via /_cluster/state is a rough proxy. The indicators that matter are field-count growth, cluster state version churn, pending-task age, and master node heap pressure.

Elasticsearch cluster_block_exception: blocked by, the read-only blocks explained

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch cluster_block_exception: blocked by, the read-only blocks explained

Elasticsearch returns cluster_block_exception when a write or metadata operation hits an active index or cluster-level block. The blocked by array contains a FORBIDDEN or TOO_MANY_REQUESTS string, a numeric code, and an API label. Writes stop. Depending on the code, deletes and metadata changes may also fail.

This article maps the four numeric codes most common in production, explains how to confirm which block is active, and gives the exact commands to clear it. Most incidents involve the /12 flood-stage disk watermark, but manual index blocks and cluster-level overrides produce the same exception and require different fixes.

Elasticsearch coordinating node overload: aggregation merge, heap spikes, and 429s

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch coordinating node overload: aggregation merge, heap spikes, and 429s

HTTP 429 or 503 responses appear on search requests while data nodes look healthy. Heap spikes on one node while others stay flat. The slow log shows heavy aggregation queries. That node is the coordinator, and it is running out of heap during the reduce phase.

Every node can act as a coordinating node. For each search, the coordinator broadcasts the query to relevant shards, collects partial results, and merges them. Aggregations compute locally per shard and reduce in memory on the coordinator. High-cardinality terms aggregations, deep pagination, or large fetch sizes force the coordinator to hold massive intermediate structures in heap. If the estimate exceeds the circuit breaker limit, the request is rejected. If the breaker is too slow, the node may suffer long GC pauses or disconnect from the cluster.

Elasticsearch CPU saturation: search, merges, GC, and hot-spotting

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch CPU saturation: search, merges, GC, and hot-spotting

Search latency climbs, bulk indexing slows, and clients see EsRejectedExecutionException or HTTP 429. On data nodes, CPU is pinned above 90% and load average rises. Elasticsearch burns CPU in query execution, text analysis, segment merges, and JVM garbage collection. If saturation persists, search and write thread pool queues grow until the cluster rejects work.

The first trap is assuming all CPU belongs to Elasticsearch. os.cpu.percent includes every process on the host, co-located containers, and kernel work. In containers this diverges sharply from process.cpu.percent, which tracks only the Elasticsearch process. A node may show 89% OS CPU and 67% process CPU, while the container runtime reports a third figure. Attribute CPU correctly before tuning thread pools or adding nodes.

Elasticsearch disk full: emergency recovery and freeing space safely

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch disk full: emergency recovery and freeing space safely

Writes fail with TOO_MANY_REQUESTS/12/index read-only / allow delete (api). Cluster health is red or yellow and data nodes are pinned above 90 percent disk. Indexers buffer or drop data while the cluster attempts shard relocations onto already-full disks. Recover without corrupting metadata or amplifying pressure.

What this means

Elasticsearch uses three disk watermarks. Low (85 percent) stops new shard allocation. High (90 percent) starts relocating shards off the node. Flood stage (95 percent) forces every index with a shard on that node into index.blocks.read_only_allow_delete: true, which stops writes.

Elasticsearch disk I/O saturation: merges, fsync, and page-cache starvation

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch disk I/O saturation: merges, fsync, and page-cache starvation

When Elasticsearch data nodes show climbing I/O wait while indexing and search latency rise, but CPU is not the bottleneck, the cluster stays green while throughput falls and thread pool queues grow. This pattern usually traces to one of three disk pressures: background segment merges rewriting data faster than storage can absorb, translog fsync overhead from durability guarantees, or OS page-cache eviction forcing every search to read from disk. This guide shows how to tell them apart, confirm the bottleneck with safe read-only checks, and relieve pressure.

Elasticsearch disk watermark cascade: from low watermark to cluster-wide read-only

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch disk watermark cascade: from low watermark to cluster-wide read-only

Writes fail with cluster_block_exception or FORBIDDEN/12/index read-only / allow delete (api). Kibana becomes unreachable; Logstash and Beats buffer or drop data. The cluster did not fail at once. It crossed a sequence of thresholds that turned single-node disk pressure into a cluster-wide write outage. With homogeneous disk sizes, every data node likely hit the thresholds within minutes, leaving no relocation target and no relief valve.

Elasticsearch disk watermark tuning: thresholds, max_headroom, and multiple data paths

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch disk watermark tuning: thresholds, max_headroom, and multiple data paths

Elasticsearch uses disk watermarks to decide when a node is too full to accept shards. The defaults (85%, 90%, 95%) were designed for small disks. On modern nodes with multi-terabyte volumes, those percentages leave hundreds of gigabytes free while still blocking allocation and triggering expensive rebalancing. Elasticsearch 8.x introduced max_headroom to cap the free-space requirement on large disks, but the interaction between percentages, absolute bytes, and max_headroom is not obvious. If you run multiple data paths, watermarks apply per path rather than per node, so a partially full disk can cause node-wide allocation restrictions. This article explains how the allocator evaluates disk space, how to tune thresholds without causing relocation storms, and why multiple data paths complicate the picture.

Elasticsearch document indexing failures: index_failed, bulk item errors, and version conflicts

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch document indexing failures: index_failed, bulk item errors, and version conflicts

A bulk request can return HTTP 200 while rejecting individual documents inside it. indices.indexing.index_failed climbs, but pipelines that only check HTTP status miss the rejections and documents disappear. This guide covers three failure classes: mapper parsing and type conflicts, per-item bulk errors hidden in HTTP 200 responses, and version_conflict_engine_exception under concurrent updates. It also distinguishes these from node-level write rejections and circuit breaker trips, which produce different symptoms and require different fixes.

Elasticsearch EsRejectedExecutionException: write thread pool rejections and HTTP 429

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch EsRejectedExecutionException: write thread pool rejections and HTTP 429

HTTP 429 responses from Elasticsearch, or stack traces containing EsRejectedExecutionException, mean the write thread pool queue is full. The write thread pool (named bulk before Elasticsearch 6.3) executes indexing, bulk, update, and delete operations on each data node using a bounded queue. When all threads are busy and the queue fills, the node rejects the operation. This guide covers how to confirm the diagnosis, distinguish it from other rejection paths, and fix the root cause instead of masking the symptom.

Elasticsearch expensive queries: leading wildcards, regex, deep pagination, and scripts

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch expensive queries: leading wildcards, regex, deep pagination, and scripts

Search latency spikes and data-node CPU saturation usually trace to one of four query patterns: leading wildcards or unbounded regex, deep from+size pagination, runtime Painless scripts, and deep aggregations. Elasticsearch uses a scatter-gather read path: the coordinating node broadcasts every query to one copy of every target shard. A single expensive query fans out, and the slowest shard sets overall latency. Coordinating-node heap pressure rises when it merges large intermediate result sets during the fetch phase or aggregation reduce phase.

Elasticsearch exposed without authentication: open clusters and snapshot exfiltration

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch exposed without authentication: open clusters and snapshot exfiltration

TCP/9200 is externally reachable and responds without credentials. When Elasticsearch binds to a public interface with security disabled, anyone who can reach the HTTP port can query indices, modify cluster state, and register snapshot repositories. The fastest exfiltration path is not reading documents individually. It is registering an attacker-controlled snapshot repository and copying entire indices out in a single background operation.

Elasticsearch fielddata circuit breaker tripped: text-field aggregations and the keyword fix

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch fielddata circuit breaker tripped: text-field aggregations and the keyword fix

Queries return CircuitBreakingException: [fielddata] Data too large... and HTTP 429s while JVM heap on one or more data nodes climbs toward the breaker limit. The node rejects queries to protect itself before OOM. This almost always means a query is aggregating, sorting, or scripting against an analyzed text field that lacks a keyword sub-field, forcing Elasticsearch to load an expensive fielddata cache into heap.

Elasticsearch FORBIDDEN/12/index read-only / allow delete (api) — flood stage recovery

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch FORBIDDEN/12/index read-only / allow delete (api) — flood stage recovery

When Elasticsearch returns cluster_block_exception with FORBIDDEN/12/index read-only / allow delete (api), every write, update, and index creation fails with HTTP 403. Search continues to work.

This happens when a data node crosses the flood-stage disk watermark (95% by default). Elasticsearch auto-applies index.blocks.read_only_allow_delete to every index with a shard on that node, preventing writes to avoid Lucene segment corruption from a full disk.

Elasticsearch heap pressure death spiral: GC, node removal, and the cascade

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch heap pressure death spiral: GC, node removal, and the cascade

When a node drops from _cat/nodes and network tests pass, check its JVM GC logs. Stop-the-world pauses over 10 seconds cause the master to remove the node. Survivors then absorb recovery traffic, their heap climbs, and they begin missing fault-detection checks too. This feedback loop is the heap pressure death spiral. It masquerades as network instability because operators check connectivity while the real problem is memory saturation.

Elasticsearch high disk watermark [90%] exceeded: shard relocation and the cascade

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch high disk watermark [90%] exceeded: shard relocation and the cascade

When Elasticsearch logs high disk watermark [90%] exceeded on [node] ... shards will be relocated away, the allocator immediately begins moving shards off the affected node. Relocation generates disk I/O and network traffic on both source and target. If targets were already close to their own watermarks, incoming shards can push them past 90%. This is the disk watermark cascade: a self-reinforcing loop where one full node triggers relocations that make other nodes full, eventually leaving the cluster with no legal allocation target and, if flood stage is hit, read-only indices.

Elasticsearch ILM stuck: indices not rolling over, shrinking, or deleting

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch ILM stuck: indices not rolling over, shrinking, or deleting

Disk usage climbs steadily. Old indices that should have been deleted remain. Shard count grows, and the cluster approaches cluster.max_shards_per_node. In ILM, indices are stuck in one phase for hours or days. This is the ILM stuck pattern: silent accumulation that becomes a disk watermark crisis, heap pressure, or unassigned shard storm when the cluster runs out of room.

ILM polls every ten minutes by default. When an index cannot advance, it sits. Because the failure is gradual, it rarely pages until a secondary limit is breached. Detect the stuck state early and fix the root cause before accumulation triggers cascading failures.

Elasticsearch indexing pressure rejections: memory backpressure before heap failure

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch indexing pressure rejections: memory backpressure before heap failure

Bulk indexing clients report rejections while cluster health is green, disks are below the high watermark, and write thread pool queues are not saturated. Yet the nodes are pushing back. Pull _nodes/stats/indexing_pressure and you will see climbing coordinating_rejections, primary_rejections, or replica_rejections. This is the indexing pressure framework, introduced in Elasticsearch 7.9, enforcing memory-based backpressure. It tracks in-flight indexing bytes at the coordinating, primary, and replica stages. The default limit is 10% of the JVM heap for coordinating and primary work, and 1.5 times that limit for replica operations. It fires before write thread pool rejections, indicating that in-flight write memory is too high rather than disk or CPU.

Elasticsearch indexing rate dropped to zero: where the write path stalls

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch indexing rate dropped to zero: where the write path stalls

Your ingestion pipeline reports healthy connections, but Elasticsearch stopped accepting writes. The index_total counter is flat, upstream queues are building, and documents are erroring or disappearing. Because index_total increments for every document, update, delete, and individual bulk item, a sustained rate of zero means the write path is stalled. The cluster may still report green health, nodes may still respond to pings, and search may still work, but the pipeline is backed up.

Elasticsearch ingest pipeline bottleneck: grok, enrich, and per-processor time

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch ingest pipeline bottleneck: grok, enrich, and per-processor time

Logstash or Beats instances log HTTP 429 errors. Elasticsearch indexing rate drops while upstream volume is flat. The write thread pool queue grows, but cluster health is green and there is no disk pressure or heap pressure. The culprit is often a single slow processor inside an ingest pipeline. A grok pattern backtracking on malformed logs, an enrich processor doing synchronous lookups, or a heavy Painless script throttles every document before it reaches Lucene. Because ingest processing runs on the write path, the backlog overflows the write thread pool queue and returns bulk rejections to upstream clients.

Elasticsearch IOException: Too many open files -- file descriptors, segments, and ulimit

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch IOException: Too many open files – file descriptors, segments, and ulimit

IOException: Too many open files means the Elasticsearch process has reached its per-process RLIMIT_NOFILE. Once open_file_descriptors reaches max_file_descriptors, the kernel returns EMFILE on every new open() call. The node cannot create Lucene segments, accept transport or HTTP connections, or maintain cluster membership. Each shard is a Lucene index composed of multiple segment files, and every network connection consumes a descriptor.

Elasticsearch JVM heap usage high: reading the sawtooth and the post-GC floor

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch JVM heap usage high: reading the sawtooth and the post-GC floor

Your Elasticsearch alert fires: jvm.mem.heap_used_percent has crossed 75 percent and is holding there. You pull up the graph and see a jagged sawtooth climbing toward the ceiling. The first instinct is to add heap or restart the node. Both are usually wrong.

The sawtooth is normal. Elasticsearch runs on the JVM with a young generation that fills with short-lived objects and empties on young garbage collections. The peak of the tooth is noise. The signal that matters is the post-GC floor: the minimum heap used immediately after a collection. In a healthy node, the floor stays between roughly 30 and 50 percent of max heap, and young GC dominates. When the floor trends upward, old generation objects are accumulating. Old GC pauses stop the world, and once a pause exceeds the cluster fault detection timeout, the master removes the node and triggers shard reallocation. That reallocation places more heap pressure on the survivors, beginning a death spiral.

Elasticsearch Limit of total fields [1000] in index has been exceeded — mapping explosion

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch Limit of total fields [1000] in index has been exceeded — mapping explosion

Every write suddenly returns illegal_argument_exception: Limit of total fields [1000] in index [X] has been exceeded. Indexing stops. The temptation is to raise index.mapping.total_fields.limit and move on. Do not. This is a mapping explosion: dynamic mapping creates a new field for every unique key in your documents. The 1000-field limit is a guardrail, not the root cause. Runaway mappings bloat the cluster state, inflate heap on every node, and eventually destabilize the master. Diagnose the source, relieve pressure safely, and fix the data shape so it does not recur.

Elasticsearch long GC pauses: old-generation stop-the-world and node drops

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch long GC pauses: old-generation stop-the-world and node drops

In Elasticsearch 8.x, nodes can drop out of the cluster without logging errors. The master logs node-left with reason disconnected, while the departed node shows no ERROR entries because its JVM was frozen in an old-generation stop-the-world GC pause. A single pause longer than 10 seconds fails a fault-detection check; roughly 30 seconds of total unresponsiveness triggers removal. Once the master reallocates shards, remaining nodes face additional heap pressure and the cascade continues.

Elasticsearch mapper_parsing_exception: type conflicts and failed document indexing

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch mapper_parsing_exception: type conflicts and failed document indexing

Document counts do not match what your pipeline sent. The _bulk endpoint returns HTTP 200, yet documents are missing from queries. In the Elasticsearch logs you see mapper_parsing_exception with messages like failed to parse field [fieldname] of type [typename] in document with id [id]. The indices.indexing.index_failed counter is climbing. This is a per-document schema rejection, not a cluster outage, and it silently drops data.

Elasticsearch mapping explosion: dynamic mapping, cluster state bloat, and master pressure

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch mapping explosion: dynamic mapping, cluster state bloat, and master pressure

Intermittent master elections, climbing heap on every node, and spiking indexing latency are hallmarks of a mapping explosion. Queries slow down. Administrative operations such as index creation or snapshot management crawl. If your data sources include unstructured JSON, such as application logs with variable keys, Kubernetes labels, or user-generated metadata, check mapping growth first.

Dynamic mapping creates a new field for every unique key it encounters. Over hours, an index can accumulate tens of thousands of fields. Every mapping is part of the cluster state, which every node holds in its JVM heap. The master serializes and publishes the updated state on every mapping change. As the state grows, publication slows, pending tasks queue up, and the master becomes unstable. Unchecked, this leads to heap pressure death spirals and cascading node removals.

Elasticsearch master instability: frequent elections and metadata overload

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch master instability: frequent elections and metadata overload

Index creation requests time out. _cluster/health hangs or returns timeouts. The node listed by _cat/master changes every few minutes outside planned maintenance. Shard allocation stalls, and new indices stay red or unassigned even though all data nodes are reachable. These symptoms indicate a master node that cannot keep up with cluster state updates, triggering repeated elections and leaving the cluster without stable coordination.

Elasticsearch master_not_discovered_exception: no elected master and stalled writes

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch master_not_discovered_exception: no elected master and stalled writes

HTTP 503 and master_not_discovered_exception mean bulk indexing, index creation, mapping updates, and shard allocation checks are being rejected. Search requests that do not need fresh cluster state may return cached results briefly, but the cluster cannot process writes or administrative work. The data nodes may be healthy, but without an elected master, the cluster cannot update shard routing, publish state changes, or acknowledge document writes. The root cause is usually one of four problems inside the master-eligible cohort.

Elasticsearch merge storms: segment explosion, I/O saturation, and refresh tuning

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch merge storms: segment explosion, I/O saturation, and refresh tuning

Search latency climbs, indexing slows, and heap usage rises while cluster health stays green. The cause is often a merge storm. Background Lucene segment consolidation has fallen behind, leaving nodes with hundreds or thousands of small segments. Each extra segment adds search overhead, consumes file descriptors, and increases memory pressure. This guide covers how merge storms develop, how to confirm the diagnosis, and how to fix them without making things worse.

Elasticsearch Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch Monitoring

What Is Elasticsearch?

Elasticsearch is a powerful search and analytics engine designed to quickly query large volumes of data, supporting use cases such as log and event data analytics, full-text searches, and more.

Monitoring Elasticsearch With Netdata

To effectively monitor Elasticsearch, leveraging a comprehensive and dynamic monitoring tool is essential. Netdata offers a robust Elasticsearch monitoring tool, providing real-time insights into Elasticsearch’s performance and health. Netdata’s agent continuously collects key metrics, presenting them in interactive dashboards, which helps diagnose issues efficiently.

Elasticsearch monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch monitoring checklist: the signals every production cluster needs

Elasticsearch failures cascade. A long GC pause on one node causes it to miss fault detection checks; the master removes it. Shards relocate to survivors, increasing heap pressure and thread pool load. If disk is near the high watermark, relocation I/O pushes other nodes toward flood stage, which sets indices to read-only and blocks writes. By the time GET /_cluster/health returns red, the leading indicators fired minutes ago.

Elasticsearch monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch monitoring maturity model: from survival to expert

Incidents escalate when teams monitor only the survival layer and miss leading indicators that predict cascades. A rising heap floor, a growing segment count, or a master task backlog surface long before cluster health turns red.

This guide organizes Elasticsearch monitoring into four levels: survival, operational, mature, and expert. Each level adds signals that reduce mean time to detection and prevent composite failure patterns. Build level 1 before going live, level 2 before handling production traffic, and levels 3 and 4 after your first serious incident.

Elasticsearch node left the cluster: fault detection, reallocation, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch node left the cluster: fault detection, reallocation, and recovery

Your cluster health turned yellow and number_of_nodes dropped by one. The master logs a NODE_LEFT event, shards are unassigned, and the remaining nodes absorb extra load. In the next minute, the allocator decides whether to move data. Misread the cause and a transient restart becomes an expensive reallocation storm, or a genuine hardware failure goes unaddressed while replicas rebalance.

This guide covers how Elasticsearch decides a node is gone, what happens to its shards, and how to recover without deepening the incident.

Elasticsearch node OOM-killed: heap ceiling, page cache, and container limits

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch node OOM-killed: heap ceiling, page cache, and container limits

An Elasticsearch node leaves the cluster, restarts seconds later via systemd or a supervisor, and is killed again. Kernel logs show the OOM-killer terminated the Java process. heap.percent often looks reasonable right up until the kill.

The JVM heap is only one component of resident set size. Off-heap allocations, memory-mapped Lucene segments, and co-located processes all compete for the same memory budget. In containers, the cgroup limit is the hard boundary, not the host’s physical RAM.

Elasticsearch pending cluster tasks backlog: the master can't keep up

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch pending cluster tasks backlog: the master can’t keep up

Index creation hangs, shards stop allocating, and administrative settings updates time out. GET /_cluster/pending_tasks shows a queue that grows instead of draining, with some tasks waiting for minutes. The elected master cannot keep up with cluster state updates, and every metadata-dependent operation stalls.

Elasticsearch processes cluster state changes serially on the master. Index creation, mapping updates, shard allocation decisions, and settings changes all enter a single priority-ordered queue. When tasks arrive faster than the master can compute, serialize, and publish the updated state to the rest of the cluster, the backlog grows. A healthy cluster typically holds zero to five pending tasks, each resolving in under a second. Once the queue exceeds a few hundred tasks, or any individual task ages past several minutes, the cluster is approaching an operational outage.

Elasticsearch refresh_interval tuning: search visibility vs indexing throughput

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch refresh_interval tuning: search visibility vs indexing throughput

Elasticsearch makes newly indexed documents searchable within one second by default. Every refresh creates a new Lucene segment, invalidates per-segment query caches, and adds merge debt. During bulk loads, reindexing, or high-throughput indexing, the default interval becomes a throughput bottleneck. Tuning refresh_interval requires understanding its interaction with the write path, the merge scheduler, and the query cache.

In self-managed clusters, the default index.refresh_interval is one second, though recent versions implement lazy refresh: an index refreshes every second only if it received a search request in the last 30 seconds. If idle, auto-refresh stops until the next search arrives. Explicitly setting refresh_interval overrides lazy refresh and forces refreshes on the interval regardless of search traffic. In Elastic Cloud Serverless, the default is five seconds with a hard floor of five seconds (or -1). Setting the interval to -1 disables automatic refresh.

Elasticsearch replica lag: sequence-number gaps and in-sync set removal

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch replica lag: sequence-number gaps and in-sync set removal

A replica shard is falling behind its primary. The local checkpoint on the replica is lower than the global checkpoint on the primary, and the gap is growing across consecutive samples. If the replica cannot acknowledge writes fast enough, Elasticsearch removes it from the in-sync set. At that point, wait_for_active_shards semantics change for the replication group and data redundancy is reduced. The replica must undergo peer recovery to rejoin. Read sequence-number checkpoints, find the bottleneck, and stop the lag before the master drops the replica.

Elasticsearch search latency high: query phase, fetch phase, and the slow shard

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch search latency high: query phase, fetch phase, and the slow shard

Elasticsearch P95 search latency jumped from 50 ms to multiple seconds. Cluster health is green, search thread pool queues are growing, and users report timeouts. Elasticsearch executes every search as a two-phase scatter-gather across targeted shards, so the slowest shard sets the floor for response time. The _nodes/stats API reports cumulative counters, not per-shard or per-query latencies, meaning one bad shard or one expensive query can be invisible to cluster-wide averages while destroying tail latency. Determine whether time is lost in the query phase, the fetch phase, or the coordinating node merge step, then isolate the outlier.

Elasticsearch search thread pool rejections: failed queries and small search queues

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch search thread pool rejections: failed queries and small search queues

Your application logs show HTTP 429 responses from Elasticsearch. The error is es_rejected_execution_exception and the message points to the search thread pool. User-facing queries are failing, not just slowing down.

The search thread pool uses a fixed number of threads and a bounded queue. The default queue size is 1000. Because search is a scatter-gather operation across shards, a single expensive query can hold a thread for seconds while the coordinating node waits for every shard to respond. The queue drains slowly, and under burst or pathological load it fills fast. Once full, Elasticsearch rejects new search requests immediately.

Elasticsearch shard recovery stuck: throttling, translog replay, and concurrent limits

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch shard recovery stuck: throttling, translog replay, and concurrent limits

You restart a data node during a rollout. Ten minutes later the cluster is still yellow. You check _cat/recovery and see a replica pinned at 38 percent for the last hour. The network is not saturated, the source node is healthy, and there are no obvious errors in the logs, yet the shard refuses to reach STARTED state.

Stuck recovery usually comes down to four constraints: a bandwidth throttle capping transfer speed, a large translog that must replay sequentially, a concurrent recovery limit serializing work, or a disk watermark on the target node blocking allocation entirely.

Elasticsearch slow search after restart: cold OS page cache and warmup

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch slow search after restart: cold OS page cache and warmup

Restart an Elasticsearch node and searches that normally return in tens of milliseconds now take seconds. CPU stays low, disk read throughput spikes, and iowait climbs. This is not a failing disk, a runaway query, or a JVM heap problem. It is a cold OS page cache.

Elasticsearch relies on the OS filesystem cache to serve search requests from Lucene segment files. After any restart, that cache is empty. The kernel must read segments from disk into memory on demand. Until the working set is resident, queries incur disk I/O that should have been cache hits. Depending on the ratio of dataset size to available RAM, warmup can last minutes to hours.

Elasticsearch snapshot failed or partial: backups that silently stop working

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch snapshot failed or partial: backups that silently stop working

You check backups and the last successful snapshot is three days old. Or you see PARTIAL snapshots where only some indices were captured. Cluster health is green, indexing and search are fine, but your recovery point is slipping. Elasticsearch does not fail the cluster when snapshots break, so the problem surfaces only when someone asks, “When did we last test a restore?”

Elasticsearch this action would add too many shards: max_shards_per_node limit

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch this action would add too many shards: max_shards_per_node limit

Creating an index or rolling over a data stream returns HTTP 400 validation_exception: “this action would add [N] shards, but this cluster currently has [X]/[Y] maximum normal shards open”. The cluster has hit cluster.max_shards_per_node, which defaults to 1000 open shards per non-frozen data node. Raising the limit via _cluster/settings unblocks writes but postpones the outage. The durable fix is consolidation.

Elasticsearch thread pool queue growing: the precursor to rejection

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch thread pool queue growing: the precursor to rejection

A climbing write or search queue in _cat/thread_pool means a node is receiving work faster than it can complete it. Rejections are the lagging indicator. By the time clients see EsRejectedExecutionException, the cluster is already degraded.

For the write pool, the default queue size is 10000 (ES 7.x+). For search, it is 1000. Sustained write queues above 1000 or search queues above 100 warrant investigation. The management pool is different: even small amounts of sustained queuing mean the master is falling behind on cluster state operations, which blocks allocation, mapping updates, and recovery.

Elasticsearch TLS certificate expiry: cluster fragmentation and client lockout

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch TLS certificate expiry: cluster fragmentation and client lockout

A node reboots and never rejoins the cluster. Kibana shows connection errors while your data pipeline buffers events. curl returns a TLS handshake failure even though the Elasticsearch process is still listening on port 9200. In Elasticsearch 8.x, security is enabled by default: every node and client relies on TLS certificates. When they expire, failure is abrupt and total. Transport-layer expiry fragments the cluster by rejecting inter-node handshakes. HTTP-layer expiry locks out clients while the cluster internals may still operate. There is no built-in grace period. At the expiry timestamp, connections fail immediately, often with no prior warning in the application logs.

Elasticsearch too many segments per shard: search slowdown and force-merge

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch too many segments per shard: search slowdown and force-merge

Searches that used to complete in tens of milliseconds are now taking hundreds. CPU on data nodes climbs without a traffic spike. Cluster health stays green, but query latency degrades and timeouts increase. The problem is invisible at the cluster level: shards have accumulated too many Lucene segments, and every query scans every segment in every relevant shard. When a shard holds more than roughly 100 segments, the background merge policy has fallen behind. Search slows, file descriptor usage rises, and merge I/O competes with indexing and queries. Since Elasticsearch 7.7, most segment metadata moved off-heap, so the node may not run out of JVM heap, but search performance degrades all the same.

Elasticsearch too many shards per node: overallocation and the heap tax

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch too many shards per node: overallocation and the heap tax

Searches slow from milliseconds to seconds. Nodes drop out after long GC pauses, triggering shard relocations that overload the remaining nodes. Cluster health stays green while _cat/nodes shows climbing heap and segments.memory growing steadily. The cause is usually shard overallocation: thousands of open shards with individual nodes carrying more than their heap can sustain.

Each shard is a self-contained Lucene index. Elasticsearch keeps per-segment metadata in heap memory, and that overhead is charged per field, not per segment. A node with 1,000 shards may be spending multiple gigabytes of heap on metadata alone, leaving less room for caches, aggregations, and in-flight requests.

Elasticsearch translog growing: flush problems, durability, and slow recovery

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch translog growing: flush problems, durability, and slow recovery

Shard recovery estimates climb from minutes to hours. A node restart stalls during translog replay. Disk usage creeps up on data nodes, and uncommitted translog size per shard is past the flush threshold. The translog is growing faster than Elasticsearch can flush it to Lucene commit points.

Every indexing operation appends to the per-shard write-ahead log before the document is searchable. A flush commits those operations to a Lucene segment and truncates the log. When flush cannot keep pace with the write path, uncommitted operations accumulate. During recovery, Elasticsearch replays every operation sequentially. A multi-gigabyte translog translates directly into a multi-hour recovery window, extending time-to-recovery and increasing cascade risk if another node fails during replay.

Elasticsearch unassigned shards: reading allocation explain and fixing each reason

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch unassigned shards: reading allocation explain and fixing each reason

Yellow or red cluster health with unassigned_shards > 0 means the allocator cannot place one or more shard copies on any node. Missing primaries block queries and risk data loss; missing replicas only cost redundancy. Do not guess from the cluster color. The allocator already knows why it rejected every node. Ask it.

What this means

Unassigned primaries make their data unreachable. Affected indices return partial results or fail. Unassigned replicas remove redundancy; a second failure on those primaries drops the data. The master allocator evaluates every node through a chain of deciders: disk watermarks, allocation filters, awareness attributes, the same-shard rule, and retry limits. When every node is rejected, the shard stays UNASSIGNED until the blocking condition clears or you intervene.

Electroline Equipment Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Elfiq Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Elgato Key Light devices.

Mon, 01 Jan 0001 00:00:00 +0000

Elgato Key Light Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Elgato Key Light Monitoring

What Is Elgato Key Light?

Elgato Key Light devices are high-quality lighting solutions that offer advanced controls for content creators, videographers, and professionals. These devices ensure optimal lighting conditions and are integral to achieving the perfect visual setup with ease. By intelligently managing your lighting environment, Elgato Key Light ensures consistent quality outputs in your video productions or live streams.

Monitoring Elgato Key Light With Netdata

To monitor Elgato Key Light devices, Netdata utilizes an OpenMetrics (Prometheus) exporter. This allows users to gather valuable metrics by periodically sending HTTP requests to the Elgato Key Light exporter. Netdata excels in ingesting data from any Prometheus exporter, providing automated dashboards, alerts, and more, without requiring a Prometheus server or Grafana. This seamless integration enables you to keep track of lighting metrics in real-time, enhancing control and management of your environment.

Elitecore Technologies Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Eltek Energy AS SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Eltek Valere Inc Formerly Valere Power Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Eltex Enterprise Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Email

Mon, 01 Jan 0001 00:00:00 +0000

Emc Clariion Advanced Storage Solutions SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Emc Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Emc Data General Division SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Empire Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Endace Technology SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Endrun Technologies LLC SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Energi Core Wallet Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is Energi Core Wallet?

Energi is a cryptocurrency that is built on the Ethereum blockchain. It is designed to be a self-funding and self-governing cryptocurrency that uses a hybrid Proof-of-Stake (PoS) and Proof-of-Work (PoW) consensus model to improve scalability and security. The Energi Core Wallet is the official wallet software for Energi. It is a software application that allows users to securely store, send and receive Energi coins. The wallet also includes advanced features such as staking, Masternode setup, and coin control.

Energomera smart power meters

Mon, 01 Jan 0001 00:00:00 +0000

Energomera Smart Power Meters Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Energomera Smart Power Meters Monitoring

What Is Energomera?

Energomera smart power meters are innovative devices designed to efficiently measure and manage electricity usage. These meters offer high accuracy and reliability, providing essential data for effective energy management. Used widely in the industrial and consumer sectors, Energomera helps in monitoring power consumption patterns, optimizing energy usage, and reducing costs.

Monitoring Energomera With Netdata

To effectively monitor Energomera smart power meters, Netdata leverages an openmetrics (Prometheus) exporter. One of Netdata’s strengths lies in its ability to ingest data from any Prometheus exporter, presenting users with automated dashboards, alerts, and more—all without the need for a Prometheus server or Grafana. This ensures that monitoring solutions remain lightweight, easy to deploy, and comprehensive in their data collection.

Engenio Information Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Enlogic Systems LLC SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Enter our waitlist to trial Netdata's AI features

Mon, 01 Jan 0001 00:00:00 +0000

Enterasys Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Enterasys Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Enterprise 1004849 SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Entropy

Mon, 01 Jan 0001 00:00:00 +0000

Envoy

Mon, 01 Jan 0001 00:00:00 +0000

Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures

Mon, 01 Jan 0001 00:00:00 +0000

Envoy 502 and upstream resets: rx_reset, tx_reset, and mid-response failures

A client request made it past Envoy’s routing, established a TCP connection to an upstream host, and then the connection died before the response completed. The access log shows a UC or UPE response flag, upstream_rq_rx_reset is climbing, and the client received a 502 or 503.

This is a different failure class from “no healthy upstream” (503, NR flag) or “connection refused” (503, UF flag). Those happen before a connection is established. Resets happen after the handshake succeeds, which means the upstream was reachable and then failed during request processing. The diagnosis and fix are completely different.

Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend

Mon, 01 Jan 0001 00:00:00 +0000

Envoy 503 with response flag UO: a tripped circuit breaker, not a dead backend

You see 503 responses in access logs with the UO response flag. The flag is what pins the cause: UO means upstream overflow. A circuit breaker fast-failed the request locally before forwarding. The backend did not refuse the connection, did not time out, and may not be aware the request existed. Envoy decided the cluster was at capacity and returned an immediate 503.

Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag

Mon, 01 Jan 0001 00:00:00 +0000

Envoy 504 upstream timeout: upstream_rq_timeout, per-try timeouts, and the UT flag

A 504 from Envoy with response flag UT means the upstream request timeout fired: Envoy sent the request to a backend and did not receive a complete response within the configured budget. The backend may be genuinely slow, the timeout may be too tight for the workload, or a filter may be interfering with the timeout clock. Each case needs a different fix.

Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests

Mon, 01 Jan 0001 00:00:00 +0000

Envoy circuit breaker open: cx_open, rq_pending_open, and fast-failed requests

You see cx_open=1 or rq_pending_open=1 on a production cluster. Access logs show 503 responses tagged with the UO response flag. Clients receive fast-failed requests, sometimes with an x-envoy-overloaded header. The circuit breaker gauges are binary: 0 means the breaker has headroom and can admit more work, 1 means it is at capacity and rejecting. Each gauge is scoped per-cluster and per-priority (default or high), so you need the right cluster and priority combination to read the signal correctly.

Envoy clusters stuck warming: warming_clusters non-zero and routes returning 503

Mon, 01 Jan 0001 00:00:00 +0000

cluster_manager.warming_clusters reading non-zero during steady state is a config convergence problem. Envoy has accepted a new cluster (or a cluster update) but cannot activate it because a dependency has not resolved: DNS, an SDS secret, an EDS endpoint set, or an active health-check initialization. Until warming completes, the cluster is invisible to routing for new additions, or held in swap-for-update for modifications. Routes targeting a freshly added but still-warming cluster return 503 with response flag NC (no cluster), or 404 with NR (no route), depending on whether the route table entry has been pushed yet.

Envoy connection churn: a low reuse ratio and keepalive misconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Envoy connection churn: a low reuse ratio and keepalive misconfiguration

Envoy’s upstream connection pool amortizes TCP (and TLS) handshakes across many requests. When it stops doing that, you have connection churn: the pool establishes a fresh connection for nearly every request, and the reuse ratio collapses toward 1.0.

The headline signal is the ratio upstream_rq_total / upstream_cx_total. For HTTP/1.1 with keepalive working, expect a number well above 1, often in the tens. For HTTP/2 with stream multiplexing, expect far higher still. When the ratio sits near 1.0, every request pays the full handshake tax.

Envoy connection pool exhaustion: a slow upstream that fills the pool

Mon, 01 Jan 0001 00:00:00 +0000

Envoy connection pool exhaustion: a slow upstream that fills the pool

Clients are seeing 503 responses. Access logs show response flag UO on those requests. But upstream health looks fine: membership_healthy is stable and hosts are passing active health checks. The upstream is not down, yet Envoy is refusing to forward new requests to it.

This is connection pool exhaustion. An upstream that was previously fast has become slow. Each request now holds a connection longer, so the same request rate fills more connection slots. The per-worker connection pool saturates, new requests queue in the pending buffer, the buffer overflows, and Envoy fast-fails those requests with 503 UO rather than piling on more load.

Envoy control_plane.connected_state = 0: running on stale xDS config

Mon, 01 Jan 0001 00:00:00 +0000

control_plane.connected_state is a gauge that flips to 0 the moment Envoy’s gRPC stream to its xDS management server drops. The flip produces no user-visible symptom: existing traffic keeps flowing, listener sockets stay open, and Envoy keeps serving its last-known-good configuration. There is no default expiry on that cached config.

That silence is why this signal is under-monitored. By the time anyone notices, new endpoints have been invisible for hours, removed endpoints have been sending traffic to dead or reassigned IPs, routes never updated, and SDS-managed certificates stopped rotating. The disconnect happened long before the visible incident, which makes root-cause correlation non-obvious.

Envoy DNS resolution failures: STRICT_DNS clusters serving stale endpoints

Mon, 01 Jan 0001 00:00:00 +0000

Envoy DNS resolution failures: STRICT_DNS clusters serving stale endpoints

A STRICT_DNS cluster stops picking up new endpoints, and endpoints you removed from DNS hours ago are still receiving traffic. Envoy’s own dashboard shows the cluster as healthy, error rates look flat, and the upstream service reports traffic to IPs that no longer exist. The first concrete evidence is often a wave of 502s or connection failures when a stale IP gets reassigned to an unrelated workload.

Envoy downstream 4xx spike: 401s, 403s, and 404s from the client side

Mon, 01 Jan 0001 00:00:00 +0000

Envoy downstream 4xx spike: 401s, 403s, and 404s from the client side

A spike in http.<stat_prefix>.downstream_rq_4xx is usually a client-side story, not an Envoy story. The proxy is reporting that clients sent bad, unauthorized, or unroutable requests. The single counter lumps 400s, 401s, 403s, and 404s together, and Envoy does not expose per-status-code downstream counters. You will not find downstream_rq_401 or downstream_rq_403 in the stats dump.

Envoy downstream connection flood: slowloris, the cx-to-rq ratio, and oversized requests

Mon, 01 Jan 0001 00:00:00 +0000

Envoy downstream connection flood: slowloris, the cx-to-rq ratio, and oversized requests

The proxy is rejecting new connections, latency is climbing, and upstreams look healthy. The problem is at the front door. A downstream connection flood exhausts Envoy’s resources before a request reaches a filter chain. The classic signal: a high new-connection rate paired with a low request rate. Many TCP handshakes, few HTTP requests. That ratio is the cx-to-rq ratio, and when it inverts you are looking at a slowloris-style attack or a misconfigured client pool.

Envoy downstream request rate dropping to zero: the traffic blackhole

Mon, 01 Jan 0001 00:00:00 +0000

Envoy downstream request rate dropping to zero: the traffic blackhole

A sudden collapse of http.<stat_prefix>.downstream_rq_total to near zero is one of the loudest availability signals Envoy can emit, and one of the easiest to misread. Operators trained to chase 5xx spikes often treat a falling request rate as “the system is calm.” It is not. A drop below roughly 10% of baseline on a listener that should be receiving traffic is a traffic blackhole: clients have stopped reaching Envoy, Envoy has stopped accepting requests, or traffic is reaching Envoy but being answered by local replies before any upstream work happens.

Envoy downstream_cx_active growing: connection leaks and idle-timeout gaps

Mon, 01 Jan 0001 00:00:00 +0000

Envoy downstream_cx_active growing: connection leaks and idle-timeout gaps

listener.<address>.downstream_cx_active is a gauge counting the downstream (client-facing) connections currently held open by a listener. When it climbs without a matching rise in request rate, the usual first call is “connection leak”. That diagnosis is sometimes correct and sometimes wrong, and the fixes are completely different.

The gauge reflects three independent inputs: how fast new connections arrive (downstream_cx_total), how fast old connections close (downstream_cx_destroy), and how long each connection is allowed to live once idle (idle, stream, drain, and connection-duration timers). A leak is only one explanation. Others include clients that legitimately hold connections open (HTTP/2 and gRPC multiplexing), loopback sidecar traffic, an idle timeout set far longer than the workload needs, or a slow upstream that holds streams, and therefore connections, open past the point the idle timer could reclaim them.

Envoy downstream_cx_overflow and overload_reject: connections turned away at the door

Mon, 01 Jan 0001 00:00:00 +0000

Envoy downstream_cx_overflow and overload_reject: connections turned away at the door

When Envoy rejects a new TCP connection at the listener, the client gets no HTTP error or retry hint: the SYN is answered with a RST or dropped, and the client sees connection refused or a connect timeout. The only evidence is three listener-level counters that should always be zero: downstream_cx_overflow, downstream_cx_overload_reject, and downstream_global_cx_overflow.

Each counter corresponds to a distinct rejection mechanism with a different root cause and fix. Confusing them wastes time tuning the wrong limit while the real problem is memory pressure, file descriptor exhaustion, or a missing overload manager.

Envoy downstream_rq_time high: client-observed latency and proxy overhead

Mon, 01 Jan 0001 00:00:00 +0000

Envoy downstream_rq_time high: client-observed latency and proxy overhead

High downstream_rq_time means client-observed latency through the proxy is climbing. The histogram lives under http.<stat_prefix>.downstream_rq_time, measured in milliseconds, and it maps most directly to user-perceived slowness. When it breaches SLO, you are in an incident whether the upstreams are healthy or not.

The common reflex is to subtract upstream_rq_time from downstream_rq_time and label the remainder “Envoy overhead.” That delta is useful as a trend, but it is not a clean measurement of proxy processing time. It folds in filter execution, downstream upload and download behavior, retry time across multiple attempts, buffering delays, streaming duration, and how fast the client reads the response. A high delta can mean Envoy is working hard, a client is slow, a request was retried, or a long-lived SSE stream is open. The histogram alone will not tell you which.

Envoy ext_authz failure_mode_allowed: unauthenticated traffic when auth is down

Mon, 01 Jan 0001 00:00:00 +0000

Envoy ext_authz failure_mode_allowed: unauthenticated traffic when auth is down

The Envoy ext_authz filter calls an external authorization service on every request that matches the filter chain. That call is on the request critical path: nothing is forwarded upstream until the auth service returns a decision. When the auth service is unreachable, returns an HTTP 5xx, or exceeds its timeout, Envoy either lets the request through without an auth decision or rejects it. That choice is controlled by failure_mode_allow, and every time the fail-open branch fires, Envoy increments http.<stat_prefix>.ext_authz.failure_mode_allowed.

Envoy file descriptor exhaustion: the FD cliff that refuses every new connection

Mon, 01 Jan 0001 00:00:00 +0000

Envoy file descriptor exhaustion: the FD cliff that refuses every new connection

When Envoy runs out of file descriptors, it is not a slow degradation. At the RLIMIT_NOFILE ceiling every new downstream accept() and every new upstream connect() fails at once. Clients see resets, timeouts, or 503s, and the proxy looks dead even though established streams keep flowing. Unlike memory or CPU pressure, there is no graceful ramp and rarely an overload-manager warning before the cliff.

Envoy health checks vs outlier detection: two systems that eject hosts differently

Mon, 01 Jan 0001 00:00:00 +0000

Envoy health checks vs outlier detection: two systems that eject hosts differently

Envoy has two independent mechanisms for removing unhealthy upstream hosts from load balancing. Active health checks send synthetic probes on a schedule. Outlier detection watches real request traffic and ejects hosts based on observed errors. They are not redundant, and they do not always agree.

A host can pass every health check and still be ejected by outlier detection. The two systems run on different threads, observe different signals, and act on different timelines. Operators who watch only membership_healthy or only outlier_detection.ejections_active miss half the picture.

Envoy hot restart races: draining, epoch churn, and dropped connections

Mon, 01 Jan 0001 00:00:00 +0000

Envoy hot restart races: draining, epoch churn, and dropped connections

You deployed a new Envoy binary or triggered a reload that invoked a hot restart. Seconds later, clients report connection resets, your 5xx rate ticks up, and server.hot_restart_epoch is climbing on your dashboard. The new process was supposed to inherit listen sockets gracefully while the old one drained.

Hot restart is Envoy’s mechanism for zero-downtime binary upgrades and certain reloads. A new process launches, coordinates with the old one over a Unix domain socket, takes over the listen sockets, and the old process enters a drain sequence. Both processes run simultaneously during the handoff. That coexistence is where the races live.

Envoy listener_create_failure: a listener config Envoy could not apply

Mon, 01 Jan 0001 00:00:00 +0000

Envoy listener_create_failure: a listener config Envoy could not apply

The listener_manager.listener_create_failure counter increments when Envoy cannot add a listener object to its workers. In a healthy deployment this counter is zero. Any non-zero value means Envoy received a listener configuration it could not apply: a failed bind (port conflict, permission denied) or invalid configuration.

The signal is easy to miss because Envoy does not crash. It keeps serving the previous listener configuration, traffic on existing routes flows normally, and the listener the operator intended to add or modify never becomes active. Deployment pipelines report success. The control plane believes the config was accepted. Only the process log and this counter reveal the failure.

Envoy membership_healthy dropping: reading the single most important cluster signal

Mon, 01 Jan 0001 00:00:00 +0000

Envoy membership_healthy dropping: reading the single most important cluster signal

If you watch only one availability metric per Envoy cluster, watch the ratio of cluster.<name>.membership_healthy to cluster.<name>.membership_total. When the ratio collapses, the remaining healthy hosts take proportionally more load, and the cluster is one or two failures away from panic mode. Upstream 5xx rate, latency, circuit breaker state, and retries are all downstream of host availability.

The hard part is reading the gauge correctly, not collecting it. A drop can mean the upstream is genuinely broken, the control plane removed endpoints, or outlier detection ejected hosts that are still passing active health checks. Each case has a different response.

Envoy memory pressure spiral: server.memory_allocated climbing toward OOM

Mon, 01 Jan 0001 00:00:00 +0000

Envoy memory pressure spiral: server.memory_allocated climbing toward OOM

server.memory_allocated is climbing, traffic is flat, and you are hours or minutes from an OOM kill. In a sidecar deployment, the pod dies and the application gets the blame. In an edge or gateway deployment, Envoy vanishes and clients see connection resets across the board.

Envoy has a built-in protection mechanism (the overload manager), but many deployments never configure it. Without it, there is no graceful degradation: Envoy goes straight from “memory looks fine” to “OOM killed” with nothing in between.

Envoy memory_heap_size vs memory_allocated: tcmalloc fragmentation that fools dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Envoy memory_heap_size vs memory_allocated: tcmalloc fragmentation that fools dashboards

Envoy exposes three memory gauges that operators naturally try to map onto process RSS: server.memory_allocated, server.memory_heap_size, and server.memory_physical_size. On a busy proxy the first two diverge by 2x-4x routinely, and dashboards that graph heap_size look like a slow leak even when nothing is wrong.

The gap is not a bug. Envoy ships with tcmalloc as its default allocator, and tcmalloc is built for allocation latency, not for prompt return of freed memory to the OS. It parks freed pages in per-thread caches, central caches, and the page heap free lists so the next allocation is fast. Those pages stay mapped into the process, counted in memory_heap_size, and very often still resident in RSS.

Envoy Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Envoy Monitoring

What Is Envoy?

Envoy is an open-source edge and service proxy, designed for cloud-native applications and microservices architectures. It plays a pivotal role in managing ingress and egress traffic between microservices, thus ensuring seamless communications within distributed systems. As a Layer 7 proxy, it provides advanced load balancing, traffic management, and observability features, crucial for high-availability applications.

Monitoring Envoy With Netdata

When it comes to monitoring Envoy proxies, the Netdata monitoring tool provides comprehensive insights into various metrics that are key to maintaining optimal performance. Netdata’s real-time monitoring capabilities allow you to track the health of your Envoy instances, detect anomalies, and troubleshoot issues promptly. With its ease of configuration and powerful visualization features, Netdata is an ideal solution for monitoring Envoy.

Envoy monitoring checklist: the signals every production proxy needs

Mon, 01 Jan 0001 00:00:00 +0000

Envoy monitoring checklist: the signals every production proxy needs

This checklist organizes Envoy monitoring into four maturity levels: survival, operational, mature, and expert. Each level catches failure modes the previous level misses. The levels are cumulative. Level 2 assumes Level 1 is in place. A team alerting on outlier detection ejections without basic server liveness has gaps in the wrong direction.

Use this as an audit tool. Walk each level, confirm each signal is collected and alerted on (or deliberately omitted), and note the gaps. Most production Envoy deployments plateau around Level 2 with a few Level 3 additions.

Envoy monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Envoy monitoring maturity model: from survival to expert

Envoy exposes thousands of stats and dozens of admin endpoints. The trap is not lack of data; it is lack of the right data at the right depth for your operational maturity. A team monitoring /ready and membership_healthy will survive most outages but will be blind to the failure modes that cause multi-hour incidents: silent xDS NACKs, retry amplification, connection pool exhaustion, CFS throttling.

Envoy NC no cluster: the cluster vanished mid-request

Mon, 01 Jan 0001 00:00:00 +0000

Envoy NC no cluster: the cluster vanished mid-request

A request lands in Envoy, the route matches, the router filter looks up the cluster the route points at, and the cluster manager has no cluster by that name. Envoy fast-fails the request with the NC (No Cluster) response flag. This is not an upstream health problem or a transient network issue. Any non-zero rate of NC in production is a configuration or timing bug.

Envoy no healthy upstream: the 503 when a cluster has no host to route to

Mon, 01 Jan 0001 00:00:00 +0000

Envoy no healthy upstream: the 503 when a cluster has no host to route to

When Envoy returns a 503 with body “no healthy upstream” and access log flag UH, the cluster has zero hosts available for load balancing. The cluster exists, routes point to it, traffic is flowing, but the load balancer cannot select a host.

The response body is literal. The flag UH appears in the %RESPONSE_FLAGS% access log field and means “No healthy upstream hosts in upstream cluster in addition to 503 response code.” It is distinct from UO (circuit breaker overflow), UF (upstream connection failure), and NR (no route).

Envoy NR no route: 404s and 503s after a bad xDS route push

Mon, 01 Jan 0001 00:00:00 +0000

Envoy NR no route: 404s and 503s after a bad xDS route push

The NR response flag in Envoy access logs means no route matched the request. The HTTP connection manager walked its route table (delivered via RDS or static config) and found no match for the incoming Host header, path, or other match criteria. In production, sustained NR is a configuration error. A spike right after an xDS route push points to a bad or rejected config update.

Envoy outlier detection mass ejection: when passive health checks empty a cluster

Mon, 01 Jan 0001 00:00:00 +0000

Envoy outlier detection mass ejection: when passive health checks empty a cluster

membership_healthy is collapsing. outlier_detection.ejections_active is climbing toward membership_total, and ejections_overflow is ticking up because Envoy wanted to eject more hosts than max_ejection_percent allows. This is the outlier detection mass ejection pattern: one of the few Envoy failure modes that can amplify a partial upstream degradation into a cluster-wide outage.

The mechanism is subtle because outlier detection is doing exactly what it was configured to do. It is a passive health check: it ejects hosts based on the actual traffic they are serving, not on synthetic probes. When one host starts returning 5xx or its success rate drops, ejecting it is correct. The problem is what happens next. Load concentrates on the survivors, they get slower, their success rates drop, and they get ejected too. The cascade feeds itself.

Envoy overload manager actions: stop_accepting_connections and the last line before OOM

Mon, 01 Jan 0001 00:00:00 +0000

Envoy overload manager actions: stop_accepting_connections and the last line before OOM

When server.overload_manager.envoy.overload_actions.stop_accepting_connections.active flips to 1, Envoy has stopped accepting new TCP connections on its listeners. When stop_accepting_requests is active, Envoy returns 503 to new HTTP requests before they reach an upstream. These are not bugs. They are Envoy’s last intentional actions before the kernel OOM-kills the process.

The overload manager connects resource pressure (heap size, connection counts) to a cascade of protective actions. When it fires, you are looking at both a root cause (memory or connection pressure) and a symptom (traffic being refused). Many deployments never configure it, skipping every graceful degradation step and going straight from “fine” to OOM-killed with nothing in between.

Envoy panic threshold: why traffic routes to unhealthy hosts at 50%

Mon, 01 Jan 0001 00:00:00 +0000

Envoy panic threshold: why traffic routes to unhealthy hosts at 50%

You are looking at an Envoy cluster where membership_healthy has dropped below half of membership_total. The error rate has climbed. The first instinct is that something new has broken: the health checks are wrong, outlier detection is misfiring, or the upstream has a second fault. In most cases none of that is true. Envoy has entered panic mode, and the elevated error rate is a direct and expected consequence of the design.

Envoy rate limiting: over_limit, 429s, and fail-open vs fail-closed

Mon, 01 Jan 0001 00:00:00 +0000

Envoy rate limiting: over_limit, 429s, and fail-open vs fail-closed

Envoy has two HTTP filters for rate limiting with very different failure characteristics. The global rate limit filter delegates every applicable request to an external rate limit service (RLS) over gRPC. The local rate limit filter applies an in-process token bucket with no external dependency. Both can produce 429 responses, but the signals that tell you what happened live in different stat namespaces and mean different things.

Envoy RBAC: access denied: 403s from the RBAC filter and shadow-mode tuning

Mon, 01 Jan 0001 00:00:00 +0000

Envoy RBAC: access denied: 403s from the RBAC filter and shadow-mode tuning

A 403 with the body RBAC: access denied is the fingerprint of Envoy’s HTTP RBAC filter blocking a request. It is not the upstream service refusing the request, and it is not the network RBAC filter, which closes the TCP connection instead of returning an HTTP status. When this counter rises you have two questions to answer: which policy matched, and was the match correct.

Envoy response flags: decoding UO, NR, UF, UT, UC and the rest

Mon, 01 Jan 0001 00:00:00 +0000

Envoy response flags: decoding UO, NR, UF, UT, UC and the rest

A 503 from Envoy can mean a dozen different things. The HTTP status code tells you what the client saw; it does not tell you why Envoy generated that response. The %RESPONSE_FLAGS% access log field separates a circuit breaker trip from a missing route, a dead upstream, or a client that hung up.

Response flags are the most precise debugging signal Envoy emits, and the most operationally misunderstood. They are not aggregate stats; they appear only in access logs. Most teams discover this the first time they try to alert on UO and find no Prometheus counter for it.

Envoy retry policy tuning: retry_on, budgets, idempotency, and hedging

Mon, 01 Jan 0001 00:00:00 +0000

Envoy retry policy tuning: retry_on, budgets, idempotency, and hedging

Retries are one of the few Envoy features that can make an outage measurably worse. A correctly tuned retry policy absorbs transient failures cleanly. A poorly tuned one multiplies upstream load by 2x-3x during a partial failure and accelerates the collapse it was supposed to mask. This article covers the knobs that matter operationally: retry_on conditions, retry budgets, per_try_timeout, and hedging. It also covers the one thing Envoy cannot decide for you: whether a request is safe to repeat.

Envoy retry storm: when the upstream-to-downstream request ratio climbs

Mon, 01 Jan 0001 00:00:00 +0000

Envoy retry storm: when the upstream-to-downstream request ratio climbs

A retry storm is one of the most dangerous failure modes in an Envoy-based service mesh or edge proxy. An upstream service starts failing a fraction of requests. Envoy’s retry policy, configured to mask transient errors, fires retries on the failures. The extra load lands on an already-degraded upstream. More requests fail under the additional load. More retries fire. Within minutes, the upstream receives two to three times its normal traffic, mostly retries, and collapses under amplification it cannot escape.

Envoy server.state not LIVE: draining, initializing, and the /ready probe

Mon, 01 Jan 0001 00:00:00 +0000

Envoy server.state not LIVE: draining, initializing, and the /ready probe

A Kubernetes readiness probe starts failing. The pod shows NotReady, the service stops sending traffic, and a rolling deploy stalls. You exec in and hit Envoy’s /ready endpoint:

# Check readiness state on the admin port
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:9901/ready
503

The probe is doing its job: Envoy is reporting server.state != LIVE. The real question is whether this is expected (rolling deploy, hot restart, normal warm-up) or genuine capacity loss (stuck init, orphaned drain, crashed child). Non-LIVE is normal during deploys, so alerting on server.state != LIVE alone is noisy. You need to combine it with listener state and uptime before paging.

What this means

Envoy exposes its lifecycle as a single gauge, server.state:

Envoy ssl.fail_verify_error: certificate verification failures on the TLS path

Mon, 01 Jan 0001 00:00:00 +0000

Envoy ssl.fail_verify_error: certificate verification failures on the TLS path

When ssl.fail_verify_error starts climbing on an Envoy proxy, TLS handshakes are failing peer certificate verification. A small burst during a planned rotation is operational noise. A sustained climb past roughly 1% of handshakes means the trust relationship between Envoy and its peers has broken.

This counter is one of several SSL stats Envoy tracks, and the others help narrow the failure mode. ssl.connection_error covers protocol-level issues such as TLS version or cipher mismatch. ssl.no_certificate fires when a client presented no certificate where one was required. ssl.fail_verify_error means a certificate was presented and parsed, but Envoy could not validate it against its configured trust context, SAN matchers, or pin hashes.

Envoy stats cardinality explosion: the stats region fills and new metrics vanish

Mon, 01 Jan 0001 00:00:00 +0000

Envoy stats cardinality explosion: the stats region fills and new metrics vanish

Envoy builds each stat name from the resource names in your config: clusters, listeners, HTTP connection managers, routes. When those names are stable, cardinality is bounded. When they are dynamic, every distinct value mints a new metric. The budget that backs those stats fills, and new stats stop appearing with no error and no log line.

The failure is quiet by design. Envoy treats stat registration as best-effort relative to its memory budget. A missing metric raises no alert, increments no hot-path error counter, and never appears in access logs. You usually discover it weeks later, when an incident sends you hunting for a per-cluster or per-route metric that was never recorded.

Envoy stuck initializing: waiting on xDS while the pod never turns ready

Mon, 01 Jan 0001 00:00:00 +0000

Envoy stuck initializing: waiting on xDS while the pod never turns ready

The pod sits at 0/1 Running indefinitely. kubectl describe shows failing readiness probes, the Envoy sidecar never reports ready, and the application container may be healthy but unreachable because the sidecar gate is closed. Envoy’s /ready admin endpoint returns HTTP 503. /server_info reports PRE_INITIALIZING or INITIALIZING. The process is alive, but it never received its initial xDS configuration, so the listener sockets exist but no listeners are active.

Envoy symptoms from the kernel: conntrack exhaustion and CFS throttling

Mon, 01 Jan 0001 00:00:00 +0000

Envoy symptoms from the kernel: conntrack exhaustion and CFS throttling

Envoy looks healthy. The admin endpoint returns 200, cluster membership is stable, upstream_rq_time is in its normal band. But every few minutes a small fraction of requests fail with cluster.<name>.upstream_cx_connect_fail ticking up, and nothing inside Envoy explains it. Upstream hosts are up, the network path looks clean, the response flag is UF with no further detail.

Or: server.watchdog_miss starts incrementing during a traffic burst, tail latency spikes, and yet the cgroup CPU chart shows Envoy using well under its declared limit. No hot worker, no expensive filter, no lock contention. Envoy’s own telemetry says “I am not the problem.”

Envoy TLS certificate expiry: expired certs, broken SDS rotation, and total outage

Mon, 01 Jan 0001 00:00:00 +0000

Envoy TLS certificate expiry: expired certs, broken SDS rotation, and total outage

TLS certificate expiry in Envoy is one of the few failure modes that can take down an entire service mesh at once. A single expired server certificate breaks new handshakes on one listener. An expired CA root breaks every mTLS connection in the data plane, every Envoy-to-control-plane link, and every health check that uses TLS.

Expiry is silent until handshakes start failing. Envoy does not expose certificate expiration as a standard metric. The only authoritative source is the /certs admin endpoint, which must be polled externally. Teams running automated rotation via SDS or cert-manager often assume rotation is working and discover otherwise when TLS breaks across the fleet.

Envoy update_rejected (xDS NACK): the config the control plane pushed and Envoy quietly refused

Mon, 01 Jan 0001 00:00:00 +0000

Envoy update_rejected (xDS NACK): the config the control plane pushed and Envoy quietly refused

You deploy a config change through your control plane. The deployment reports success. Traffic flows, error rates are flat, control_plane.connected_state stays at 1. But the change never took effect: the new route is missing, the timeout override is gone, the certificate rotation did not happen.

Envoy received the new config, validated it, found it invalid, and rejected it. It kept the previous config and kept routing. The rejection is recorded in update_rejected and, for listeners, in listener_manager.listener_create_failure. The reason itself is only in the Envoy process log, not in stats. The control plane reports success because the gRPC stream is healthy, not because Envoy accepted the config. Nothing fails visibly. The only symptoms are the things that should have changed but did not.

Envoy upstream connect error or disconnect/reset before headers: reading the reset reason

Mon, 01 Jan 0001 00:00:00 +0000

The 503 body upstream connect error or disconnect/reset before headers. reset reason: <reason> means Envoy selected an upstream host, tried to open a request stream on a pooled connection, and the stream was reset before any response headers came back. The body looks generic, but the trailing reset reason: field is the diagnostic payload. It is one of the values in Envoy’s StreamResetReason enum , and each value points at a different failure mechanism.

Envoy upstream_cx_active near max_connections: the pool filling up

Mon, 01 Jan 0001 00:00:00 +0000

Envoy upstream_cx_active near max_connections: the pool filling up

The cluster.<name>.upstream_cx_active gauge reports active connections across all hosts in a cluster. When it trends toward the circuit-breaker max_connections limit, the cluster is approaching saturation. Past the limit, Envoy stops opening new connections, queues new requests in upstream_rq_pending_active, and once max_pending_requests is hit, returns 503s with response flag UO.

Envoy is fast-failing locally to protect an upstream that cannot absorb more concurrent work. Raising max_connections without addressing the upstream removes that protection. The operator’s job is to find what is shrinking effective pool capacity.

Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts

Mon, 01 Jan 0001 00:00:00 +0000

Envoy upstream_cx_connect_fail: failed TCP connections to upstream hosts

cluster.<name>.upstream_cx_connect_fail is a per-cluster counter that increments each time Envoy fails to establish a TCP connection to an upstream host. On a healthy cluster the rate is flat. The threshold for concern is connect_fail / connect_total > 0.05 , meaning more than 5% of connection attempts are failing. Sustained nonzero rates point at one of four root causes: the upstream process is down, the upstream is out of file descriptors, a firewall or ACL changed, or the upstream listen backlog is overflowing.

Envoy upstream_cx_connect_ms high: slow TCP connects to upstream hosts

Mon, 01 Jan 0001 00:00:00 +0000

Envoy upstream_cx_connect_ms high: slow TCP connects to upstream hosts

When cluster.<name>.upstream_cx_connect_ms starts climbing, understand what this histogram actually measures: the time Envoy spent establishing a TCP connection to an upstream host. When upstream TLS is configured, the TLS handshake is rolled into the same number. It is not request latency, not application processing time, and not pure network RTT once TLS is in the path.

Same-zone connections typically sit under 2ms at P99. The alert threshold is a sudden 5x rise over rolling baseline. Anything beyond that means the network path has changed, the upstream kernel cannot accept connections fast enough, or something is forcing Envoy to open far more new connections than usual.

Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin

Mon, 01 Jan 0001 00:00:00 +0000

Envoy upstream_rq_pending_overflow: the pending queue fills and 503s begin

You see cluster.<name>.upstream_rq_pending_overflow incrementing on one or more clusters. Access logs show 503 with response flag UO. Clients get fast-fail 503s, not timeouts. The curve is cliff-edge: requests were flowing fine, and now a slice of them are rejected with no upstream attempt at all.

This is Envoy’s pending request circuit breaker doing exactly what it was configured to do. Requests arrive, there is no available upstream connection to attach them to, so they wait in upstream_rq_pending_active. When that queue hits max_pending_requests, Envoy stops queueing and starts rejecting. The overflow counter is the rejected count.

Envoy upstream_rq_retry_overflow: the retry budget exhausted and retries dropped

Mon, 01 Jan 0001 00:00:00 +0000

Envoy upstream_rq_retry_overflow: the retry budget exhausted and retries dropped

upstream_rq_retry_overflow is climbing on a cluster. Error rates are elevated, and retries should absorb the failures. But upstream_rq_retry is not growing proportionally, and the system looks like it stopped retrying. It did: the retry circuit breaker or retry budget is full, and Envoy is dropping retries to protect the upstream.

The retry system is working as designed, but legitimate retries are being dropped at the moment they are needed most. A budget that is too small loses the resilience retries provide. A budget that is too large risks 2x-3x traffic amplification during partial failures.

Envoy upstream_rq_time high: upstream latency as the proxy sees it

Mon, 01 Jan 0001 00:00:00 +0000

Envoy upstream_rq_time high: upstream latency as the proxy sees it

cluster.<name>.upstream_rq_time is the histogram that answers “how long did the upstream interaction take, as Envoy observed it?” When it climbs, the instinct is to page the backend team. That instinct is often wrong, or at least incomplete. The metric is wall-clock time measured at the HTTP router filter. It bundles several distinct latencies: TCP connect, upstream TLS handshake (for new connections), request transmission, upstream processing, and response transfer. It is not pure backend service time.

Envoy URX upstream retry limit exceeded: all retries used, last error returned

Mon, 01 Jan 0001 00:00:00 +0000

Envoy URX upstream retry limit exceeded: all retries used, last error returned

When you see URX in Envoy access logs, every configured retry attempt has fired and failed, and the last upstream error is what the client receives. This is distinct from retry_overflow, where retries were never attempted because the retry budget or circuit breaker was full. The distinction matters because the two conditions have opposite root causes and opposite fixes.

Envoy watchdog_miss and the hot worker: one saturated event loop hiding in the average

Mon, 01 Jan 0001 00:00:00 +0000

server.watchdog_miss just incremented. Aggregate CPU shows Envoy at 40%, comfortably under capacity. But a slice of requests are spiking P99, P50 is fine, and latency variance is high.

This is the hot worker pattern. Envoy is single-threaded per worker: each worker owns its connections for their entire lifetime and runs its own event loop. When one worker’s event loop blocks past the watchdog timeout (200ms by default), the main thread increments watchdog_miss. A single saturated worker is invisible in process-level CPU averages because the other workers idle along.

EOS

Mon, 01 Jan 0001 00:00:00 +0000

EOS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

EOS Monitoring

What Is EOS?

EOS is a high-performance, scalable storage service developed by CERN. It is primarily used for managing massive volumes of scientific data, providing a reliable and efficient storage solution that ensures data integrity and accessibility.

Monitoring EOS With Netdata

To ensure optimal performance and reliability of EOS, monitoring it with Netdata can provide real-time insights into system metrics and help troubleshoot issues proactively. Netdata uses an openmetrics (Prometheus) exporter to collect EOS metrics efficiently. With Netdata, you can ingest data from any Prometheus exporter, offering you automated dashboards and alerts without the need for a Prometheus server or Grafana. This makes Netdata an ideal EOS monitoring tool that simplifies the monitoring process while delivering comprehensive metrics insights.

Epicenter Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Equallogic SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Equinox Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Equipe Communications Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Era A S SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ericsson AB Packet Core Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ericsson AB SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ericsson Inc Formerly Redback Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Essential Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

etcd

Mon, 01 Jan 0001 00:00:00 +0000

etcd Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

etcd Monitoring

What Is etcd?

etcd, a distributed key-value store, is a critical component for service discovery and storing all of a cluster’s data reliably. It ensures that the systems in production environments are available and consistent. etcd is often implemented in environments requiring fault tolerance, distributed networking, or consensus building. Learn more about etcd.

Monitoring etcd With Netdata

When it comes to monitoring etcd, Netdata stands out as a powerful tool. Netdata leverages an OpenMetrics (Prometheus) exporter to monitor etcd, facilitating effortless integration. This capability permits Netdata to ingest metrics from any Prometheus exporter, delivering dynamic dashboards, alerts, and comprehensive insights that require neither a dedicated Prometheus server nor Grafana setup.

Etherwan Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Eurologic Systems Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Exablaze SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Exabyte Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Exagrid

Mon, 01 Jan 0001 00:00:00 +0000

Exalt Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Exanet SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Exceliance SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Exim

Mon, 01 Jan 0001 00:00:00 +0000

Exim Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Exim Monitoring

What Is Exim?

Exim is a mail transfer agent (MTA) used on Unix-like operating systems. It is highly configurable and can handle email routing, spam control, and address verification. For more details, you can visit the Exim website.

Monitoring Exim With Netdata

Monitoring Exim’s performance is crucial to ensure the optimal operation of the mail server. With Netdata, you can monitor Exim effortlessly using the go.d.plugin, which tracks the Exim mail queue. This integration provides real-time insights, helping you troubleshoot and maintain reliable email service. Learn more in the Exim collector documentation.

Exinda Networks Pty Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Expand Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Extended Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Extrahop Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Extreme Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Extreme Switching

Mon, 01 Jan 0001 00:00:00 +0000

Extricomltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

F5 BGP

Mon, 01 Jan 0001 00:00:00 +0000

F5 BIG IP

Mon, 01 Jan 0001 00:00:00 +0000

F5 Labs Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

F5 Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fail2ban

Mon, 01 Jan 0001 00:00:00 +0000

Fail2ban Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Fail2ban Monitoring

What Is Fail2ban?

Fail2ban is an open-source intrusion prevention software framework that protects servers from brute-force attacks. It monitors log files and bans IPs that exhibit malicious behavior, such as too many failed login attempts, by modifying firewall rules.

Monitoring Fail2ban With Netdata

Netdata is a powerful monitoring solution that offers real-time insights into the performance and security of your systems. With Netdata, you can monitor Fail2ban, ensuring your servers remain secured from unwanted access. Check out the Live Demo to see Netdata’s capabilities in action.

Fair Usage Policy

Mon, 01 Jan 0001 00:00:00 +0000

Fastd

Mon, 01 Jan 0001 00:00:00 +0000

Fastd Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Fastd Monitoring

What Is Fastd?

Fastd, or Fast and Secure Tunneling Daemon, is a VPN solution renowned for its simplicity and flexibility. It’s widely used in various network environments, particularly in community wireless networks. Fastd enables encrypted internet connections and is known for its efficient resource usage and adaptability across different platforms.

Monitoring Fastd With Netdata

To monitor Fastd, Netdata employs an openmetrics (Prometheus) exporter. With Netdata, you can ingest data from any Prometheus exporter, offering automated dashboards, real-time alerts, and more, without the need for a Prometheus server or Grafana. This seamless integration makes it a powerful Fastd monitoring tool, providing comprehensive insights into your VPN’s performance and health.

FDB / MAC Forwarding Topology

Mon, 01 Jan 0001 00:00:00 +0000

Fedora

Mon, 01 Jan 0001 00:00:00 +0000

Fial Computer Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fiberhome Telecommunication Technologies Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fibernet International SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fibrolan SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fibronics SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Files and directories

Mon, 01 Jan 0001 00:00:00 +0000

Files and Directories Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Files and Directories Monitoring

What Is Files and Directories Monitoring?

Files and directories are integral components of any computing system. Monitoring these elements involves keeping track of their existence, modification times, sizes, and other changes. Effective monitoring ensures data integrity, improves security, and supports efficient system maintenance.

Monitoring Files and Directories With Netdata

Netdata provides an intuitive and powerful files monitoring tool leveraging its Filecheck module. This tool simplifies the process of gathering key metrics in real time. With Netdata, you can visualize data from multiple files and directories across different servers, offering a comprehensive view of your infrastructure.

Fireeye

Mon, 01 Jan 0001 00:00:00 +0000

Fireeye Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

First-observation baselining: don't page on lifetime counters at rollout

Mon, 01 Jan 0001 00:00:00 +0000

First-observation baselining: don’t page on lifetime counters at rollout

When you deploy SMART monitoring on an existing fleet for the first time, every in-service drive carries accumulated history in its lifetime counters. Offline Uncorrectable sectors, NVMe Media and Data Integrity Errors, Power-On Hours, Unsafe Shutdowns, Reallocated Sector Count, UDMA CRC Error Count. These counters started incrementing the moment the drive left the factory and never reset.

If your alerting fires on any non-zero absolute value, every drive with any history pages within minutes of enabling monitoring. A drive with 3 reallocated sectors from factory QA, a drive with 200 uncorrectable errors from a past thermal event three years ago, a drive with 5 unsafe shutdowns from a UPS failure. All produce identical alerts to a naive greater-than-zero rule.

Flarion Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Flock

Mon, 01 Jan 0001 00:00:00 +0000

Flow export-to-ingest latency: why your NetFlow data is minutes behind

Mon, 01 Jan 0001 00:00:00 +0000

Flow export-to-ingest latency: why your NetFlow data is minutes behind

Flow export-to-ingest latency accumulates across a pipeline: the exporter’s active timeout, the UDP transport path, the kernel socket buffer, the collector’s parser, and the storage write queue. Each stage can add seconds or minutes, and each has a different fix.

The most common cause is the active timeout default on most network devices: 30 minutes. Long-lived flows (VPN tunnels, database connections, bulk transfers) are not exported until the timer expires. The collector is not slow and the network is not congested. The device is behaving as configured. But if you need near-real-time visibility, a 30-minute export delay is indistinguishable from broken telemetry.

Fluentd

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd average flush time rising: the earliest sign of destination slowdown

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd average flush time rising: the earliest sign of destination slowdown

Your Fluentd process is alive. retry_count is zero. The buffer queue looks flat. And yet, if you are computing average flush time from flush_time_count and write_count, you can see the destination getting slower, sometimes hours before the first retry fires. This is the earliest signal of destination or network degradation in a Fluentd pipeline, and most teams never look at it because the monitor_agent API does not expose it as a ready-made field.

Fluentd broken pipe / connection reset: dropped output connections and LB timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd broken pipe / connection reset: dropped output connections and LB timeouts

Your Fluentd logs show broken pipe or Connection reset by peer during flush, retries start climbing, and chunks roll back into the buffer queue. The destination is not down. It answers health checks, other clients reach it fine, and the errors come and go in a pattern that looks almost random.

The usual explanation: Fluentd’s output plugin is holding a long-lived TCP connection to the destination, something in the middle (a load balancer, NAT gateway, firewall, or the destination itself) has an idle timeout, and it silently drops the connection after a period of inactivity. Fluentd does not find out until the next flush writes into the dead socket. The kernel returns EPIPE (“broken pipe”) or the peer returns RST (“connection reset by peer”), and the chunk goes back for retry.

Fluentd buffer available space low: computing time-to-overflow before it fires

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd buffer available space low: computing time-to-overflow before it fires

The monitor_agent metric buffer_available_buffer_space_ratios tells you what percentage of an output plugin’s configured buffer capacity is still free. When it drops, the buffer is filling. When it hits zero, the overflow_action fires: with the default throw_exception, new events are rejected at the input with a BufferOverflowError; with block, input threads stall; with drop_oldest_chunk, your oldest undelivered data is thrown away. None of those are outcomes you want to discover after the fact.

Fluentd buffer queue length growing: the output cannot keep pace with the input

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd buffer queue length growing: the output cannot keep pace with the input

You are looking at a graph of buffer_queue_length for one of your Fluentd outputs and it has been climbing for twenty minutes. No alerts have fired, nothing has crashed, but the trend only goes one direction. This is the earliest visible stage of the most common Fluentd failure mode: the output destination is falling behind the input, and the buffer is absorbing the difference.

Fluentd buffer stage vs queue: telling healthy batching from backpressure

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd buffer stage vs queue: telling healthy batching from backpressure

A common mistake in Fluentd operations is treating “the buffer” as a single number. Teams graph one buffer metric, see it climb, and either panic over normal batching or ignore a real backpressure signal because “the buffer number looks like it always does.” The buffer is not one thing. It is two distinct states with two distinct gauges, and they mean opposite things.

Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd buffer_oldest_timekey lag: how far behind the oldest buffered data is

buffer_oldest_timekey is a timestamp: the timekey of the oldest chunk still sitting in an output plugin’s buffer, waiting to be delivered. Subtract it from the current time and you get a number that answers the question operators care about during an incident: how old is the oldest data that has not reached its destination yet?

That number is more honest than the alternatives. Comparing input and output emit_records rates works for steady streams, but it misleads you for time-sliced outputs (which legitimately hold chunks until the slice expires) and for bursty workloads (where rates oscillate and any short window looks alarming). The oldest timekey does not care about rates. It tells you directly how stale the data is.

Fluentd BufferOverflowError: buffer space has too many data

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd BufferOverflowError: buffer space has too many data

Fluentd just started logging BufferOverflowError: buffer space has too many data and your destination stopped receiving events. The buffer has reached total_limit_size (512MB by default for memory buffers, 64GB for file buffers) and the default overflow_action, throw_exception, is rejecting every new event at the input. Those events are gone. They do not retry, they do not queue, and no error counter reliably tracks them.

Fluentd config integrity: leaked credentials and silent tampering

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd config integrity: leaked credentials and silent tampering

A Fluentd config file is not just routing logic. In most production deployments it is also a credential store: Elasticsearch passwords, S3 access keys, Kafka SASL secrets, TLS client certificates, and forward shared keys all live in the same file that tells Fluentd where your logs go. If that file is world-readable, every local user and every compromised process on the host can read those credentials. If it is writable outside the deploy pipeline, an attacker can redirect your log stream to their own endpoint, quietly drop collection for the services they are touching, or add an output you never approved.

Fluentd config reload failed: SIGHUP that partially applies

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd config reload failed: SIGHUP that partially applies

You edited the Fluentd config, sent SIGHUP, watched the log line saying the config reloaded, and moved on. Hours later you notice a new output never started receiving data, or an old filter is still dropping events you told it to keep. The process never crashed. No error fired. But the pipeline running in memory is not the pipeline in the config file.

Fluentd CPU bottleneck: the Ruby GVL caps a single worker at one core

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd CPU bottleneck: the Ruby GVL caps a single worker at one core

Fluentd throughput has flatlined. The host has idle cores, the destination is healthy, retry_count is zero, and yet the buffer is growing and in_tail is falling behind the files it watches. top shows the Fluentd process pinned at 100% CPU, but 100% of exactly one core.

This is the CRuby Global VM Lock (GVL) doing what it is designed to do. Within a single Fluentd worker process, only one Ruby thread can execute Ruby code at a time. The event router, parsers, filters, and flush threads all compete for that lock. The moment one thread does sustained CPU-bound work, such as regex parsing or JSON serialization, everything else in the process waits.

Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd CrashLoopBackOff: rapid restart cycling in Kubernetes

A Fluentd DaemonSet pod in CrashLoopBackOff is not “down.” It is oscillating: the container starts, runs for seconds or minutes, dies, and kubelet restarts it with an exponentially growing backoff. kubectl get pods shows the pod flapping between Running, Error, and CrashLoopBackOff while the restart count climbs.

Because kubelet keeps resurrecting the process, you rarely see one long outage. Instead you see repeated brief absences: buffers drain partially and refill, and downstream destinations see gaps and duplicate windows as file-backed buffers replay on each restart. The auto-restart that keeps the node “mostly covered” is also what masks the root cause.

Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd drop_oldest_chunk_count incrementing: confirmed buffer data loss

drop_oldest_chunk_count is not a warning signal. It is a receipt for data that no longer exists. Every increment means Fluentd discarded the oldest buffered chunk to make room for incoming events, and those events are permanently gone. No retry, no secondary output, no replay will bring them back.

This counter only moves when you have explicitly configured overflow_action drop_oldest_chunk in an output’s <buffer> section. That setting is a deliberate trade: keep the pipeline alive under sustained output failure by sacrificing the oldest data first. The failure mode is that nothing else in the system complains when it fires. Fluentd keeps running, input keeps flowing, output keeps writing. The only evidence of loss is this counter and a warning line in Fluentd’s own log, which most pipelines never scrape.

Fluentd duplicate events: why the same log shows up twice downstream

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd duplicate events: why the same log shows up twice downstream

You query Elasticsearch, S3, or your log backend and the same event appears twice. Sometimes the duplication is a burst that lines up exactly with a Fluentd restart or a pod reschedule. Sometimes it is a slow, persistent trickle that inflates dashboards and breaks counts. The two situations have different root causes and different fixes.

Fluentd does not guarantee exactly-once delivery. It guarantees at-least-once for file-backed buffers, and the brief duplication window after a restart is by design. Persistent or large-scale duplication almost always traces back to one of a small set of mechanisms: position tracking resets at the input, or full-chunk retries at the output after a partial write.

Fluentd emit_error_count: the number-one under-monitored data-loss signal

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd emit_error_count: the number-one under-monitored data-loss signal

Most Fluentd monitoring setups watch retry_count, buffer_queue_length, and process liveness. Almost none watch emit_error_count. The usual discovery is postmortem: hours of logs are missing, and the missing window is the exact window the incident needed.

emit_error_count counts emit transactions that failed inside the pipeline: events that could not be handed off and will never be delivered. Any nonzero rate means data loss is happening right now. This is not a degradation signal or a leading indicator. It is confirmation that events are gone.

Fluentd end-to-end pipeline latency: stale logs during an incident

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd end-to-end pipeline latency: stale logs during an incident

Your dashboards show errors spiking at 02:00, but when you query the log store for the same window, the newest events are 40 minutes old. Fluentd is running, the process is healthy, no alerts fired. The pipeline is alive but the data is stale, and you are debugging an incident on a delay you did not know existed.

Fluentd end-to-end pipeline latency is the time from event generation to arrival at the destination. Fluentd does not expose this as a native metric. There is no pipeline_latency_seconds field in the monitor_agent API. You have to derive it: compare event timestamps to arrival time at the destination, or inject synthetic events with known timestamps and measure round-trip time.

Fluentd failed to flush the buffer: the output cannot deliver and retries begin

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd failed to flush the buffer: the output cannot deliver and retries begin

Your Fluentd log starts emitting lines like this, over and over:

[warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2026-07-21 22:50:11 +0000 chunk="5e1a2b..." error_class=Net::OpenTimeout error="execution expired"

An output plugin tried to flush a buffer chunk to its destination and failed. Fluentd did not lose the chunk; it rolled the chunk back into the queue and scheduled a retry with exponential backoff. The warning repeats once per failed attempt, with retry_time climbing and next_retry_seconds drifting further into the future.

Fluentd file buffer filling the disk: when the buffer partition runs out

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd file buffer filling the disk: when the buffer partition runs out

A file-backed buffer is supposed to be the durable option: the output stalls, chunks pile up on disk, the destination recovers, the backlog drains. That story only holds while the partition underneath the buffer directory has free space. When it does not, the failure is abrupt. At 100% full, Fluentd cannot stage new chunks, incoming events are lost, and anything else writing to the same partition fails at the same time.

Fluentd in_tail not reading: the file is growing but no events are emitted

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd in_tail not reading: the file is growing but no events are emitted

The application is writing logs. ls -la shows the file growing. But Fluentd’s input emit_records for that in_tail source has been flat for minutes or hours, and nothing is arriving downstream. The daemon runs, the buffers stay empty, and the only evidence is missing data. This is an input-side stall, and it is one of the quieter Fluentd failures because nothing page-worthy breaks.

Fluentd in_tail with multi-worker: pinning to a single worker

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd in_tail with multi-worker: pinning to a single worker

You added workers 4 to <system> to get past the single-core ceiling the Ruby GVL imposes on one Fluentd process, and now Fluentd refuses to start. The log says: Plugin 'tail' does not support multi workers configuration (Fluent::Plugin::TailInput). Or worse: it starts, and you discover weeks later that some log files were read twice by different workers while others were never read at all.

Fluentd input emit_records dropped to zero: ingestion has stopped

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd input emit_records dropped to zero: ingestion has stopped

The emit_records counter for an input plugin is the number of records that plugin has handed to the Fluentd router since process start. It is cumulative and monotonically increasing. If its computed rate drops to zero and holds there while the upstream source is still writing logs, ingestion is broken: events are being generated somewhere and silently going nowhere.

This is one of the few Fluentd conditions that justifies paging immediately. A dead process is obvious. A live process that has stopped ingesting is a silent observability blackout, and it will not announce itself anywhere except this counter.

Fluentd input emit_records stuck at zero: enable_input_metrics on older versions

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd input emit_records stuck at zero: enable_input_metrics on older versions

You open the monitor_agent API (or a dashboard built on top of it) and every input plugin reports emit_records: 0. First read: ingestion has stopped. Then you notice output plugins are delivering records, buffers are cycling, and downstream log storage is receiving fresh data. The pipeline is fine. The metric is blind.

On Fluentd versions before v1.19.0, input plugin metrics are disabled by default. Unless you set enable_input_metrics true in the <system> block, every input plugin’s emit_records counter stays at 0 forever, no matter how much data flows through it. This is a common false alarm, and it also cuts the other way: teams running real ingestion stalls on older versions have no input-side visibility at all because they never turned the metrics on.

Fluentd input spike: a log storm that overwhelms buffers and outputs

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd input spike: a log storm that overwhelms buffers and outputs

Your dashboards show input emit_records suddenly at 5x or 50x baseline. Within minutes, buffer_queue_length starts climbing, average flush time rises, and you are watching the buffer fill in real time. This is a log storm: an input spike large enough that the output side of the pipeline cannot drain it.

The danger is not the spike itself; Fluentd is built to absorb bursts, that is what the buffer is for. The danger is the cascade: the buffer fills to total_limit_size, overflow_action fires, and depending on configuration you either drop new events (throw_exception, the default), stall all input threads (block), or discard your oldest buffered data (drop_oldest_chunk). None of these outcomes is obvious unless you know which counters to watch.

Fluentd log rotation loss: copytruncate, rotate_wait, and missed lines

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd log rotation loss: copytruncate, rotate_wait, and missed lines

Log rotation is the most common source of quiet data loss in a Fluentd deployment. Pipeline metrics look healthy, the destination keeps receiving data, and yet there is a gap in the log stream at exactly 00:00 every night, or a burst of duplicate records right after logrotate runs. The cause is almost always the interaction between the rotation method used by logrotate (or the container runtime) and the assumptions Fluentd’s in_tail plugin makes about how files change on disk.

Fluentd memory growing: leak versus the normal Ruby fragmentation plateau

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd memory growing: leak versus the normal Ruby fragmentation plateau

Fluentd RSS has been climbing for hours or days and you are trying to decide whether you have a memory leak or whether this is just what Ruby does. The answer matters because the two cases have completely different responses: one requires no action at all, the other ends in an OOM kill and, if your buffers are memory-backed, permanent loss of buffered log data.

Fluentd memory vs file buffer: why the default buffer loses data on restart

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd memory vs file buffer: why the default buffer loses data on restart

You restarted Fluentd for a config change, or the OOM killer restarted it for you, or Kubernetes rescheduled the pod. The process came back healthy. Every metric looks normal. But downstream there is a gap in the logs covering the minutes before the restart, and no error anywhere explains it.

The explanation is almost always the same: the output was using the memory buffer, and every chunk that had not been flushed at the moment the process died was deleted with it. This is not a bug. It is the documented behavior of the memory buffer. The Fluentd troubleshooting documentation itself lists “change buffer type from memory to file” as a standard remediation for exactly this symptom.

Fluentd monitor_agent not responding: a process that is up but hung

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd monitor_agent not responding: a process that is up but hung

Your liveness check says Fluentd is fine. The PID exists, systemctl status shows active, but curl http://localhost:24220/api/plugins.json hangs until it times out, or returns something other than 200, and no logs are moving.

This is the zombie state: a process that passes every cheap liveness check but is functionally dead. The monitor_agent endpoint is the cheapest honest probe you have for this. If it does not answer, the internal runtime is not making progress, regardless of what the process table says.

Fluentd Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd Monitoring

What Is Fluentd?

Fluentd is a powerful open-source data collector that allows you to unify the collection and consumption of data streams. It helps streamline log data management and enables real-time data processing.

Monitoring Fluentd With Netdata

Using Netdata as your Fluentd monitoring tool provides an efficient, real-time overview of the performance and health of your Fluentd instances. Netdata’s lightweight but comprehensive monitoring capabilities make it a preferred choice for DevOps and IT professionals.

Fluentd monitoring checklist: the signals every production log pipeline needs

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd monitoring checklist: the signals every production log pipeline needs

Most Fluentd monitoring setups answer one question: is the process running? That is necessary and nowhere near sufficient. A Fluentd process can be alive, responsive, and completely idle because the buffer is full and the default overflow_action (throw_exception) is dropping every new event at the input. No error counter reliably increments for that path. The process looks healthy while the pipeline loses data.

Fluentd monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd monitoring maturity model: from survival to expert

Most teams monitoring Fluentd stop at “is the process running?” and discover the gap during an incident, when the SIEM is missing the exact logs they need for a postmortem. Fluentd can be alive, responsive, and green on every dashboard while silently dropping events, accumulating a buffer that will overflow in forty minutes, or retrying into a backoff so deep the pipeline is effectively dead.

Fluentd OOM killed: memory bloat, the OOM killer, and lost buffers

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd OOM killed: memory bloat, the OOM killer, and lost buffers

Fluentd disappeared. The process is gone, systemd or Kubernetes restarted it, and there is a gap in your log storage covering the last few minutes or hours. A check of dmesg shows the OOM killer picked Fluentd as its victim. Memory grew for hours, GC fought harder and harder, and then the kernel ended it.

This failure mode is expensive because of what dies with the process. If your outputs use memory-backed buffers, every staged and queued chunk that had not yet been flushed is gone. The OOM kill is not a graceful shutdown. Ruby cleanup code does not run, so flush_at_shutdown never gets a chance to fire. In Kubernetes the same mechanism shows up as CrashLoopBackOff: the pod restarts, memory climbs back to the cgroup limit, and the kubelet kills it again.

Fluentd output authentication errors: 401/403 and rejected credentials

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd output authentication errors: 401/403 and rejected credentials

A 401, 403, unauthorized, forbidden, or TLS certificate error in the Fluentd log means the destination is actively rejecting the output plugin’s connection. The pipeline does not crash. Fluentd keeps accepting input, buffering events, and attempting flushes that keep failing, so from the outside the agent looks alive while no data reaches that destination.

The operational risk is the backlog that builds behind the failure. Every rejected flush pushes chunks back into the buffer, and if credentials stay broken long enough the buffer fills and the configured overflow_action decides what data you lose. With the default throw_exception, new events are silently discarded. With drop_oldest_chunk, the oldest buffered chunks are destroyed. Neither emits an obvious alert unless you are watching the right counters.

Fluentd output rate lower than input rate: the deficit that is quietly dropping logs

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd output rate lower than input rate: the deficit that is quietly dropping logs

Your Fluentd process is up. The monitor agent responds. retry_count is zero. And yet, when you query the destination, whole time windows of logs are missing. The cause is usually the same: output emit_records has been running below input emit_records for hours or days, and nobody was comparing them.

Most teams chart input rate and output rate independently and never compute the ratio. A sustained 5% deficit on a host doing 200 events per second is over 6 million events lost per week. The two rates should converge over any reasonable window. When they do not, data is either being dropped or piling up in a buffer that will eventually overflow and drop it anyway.

Fluentd overflow_action: throw_exception, block, and drop_oldest_chunk

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd overflow_action: throw_exception, block, and drop_oldest_chunk

When a Fluentd output buffer reaches total_limit_size, the pipeline does not degrade gradually. It hits a binary transition: the buffer was accepting events, and now it is not. What happens in that moment is decided by a single parameter in the <buffer> section: overflow_action. It accepts three values: throw_exception (the default), block, and drop_oldest_chunk.

Most teams assume “buffer full = pipeline blocks and waits.” That is only true if you explicitly configure block. The default, throw_exception, means new events are rejected when the buffer is full, and depending on the input plugin, those events are simply never ingested. There is no reliable metric that counts these losses.

Fluentd pattern not matched: the parser is silently dropping log lines

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd pattern not matched: the parser is silently dropping log lines

You found this in the Fluentd log:

[warn]: #0 pattern not match: "2026-07-21T22:58:11.123456789Z stdout F some application log line"

One warning line per dropped record. Or worse: someone turned the warnings off to silence the noise, and now log lines are vanishing with no signal at all. This is input-level data loss. The events never enter the filter chain, never reach the buffer, and never appear in your destination. Pipeline metrics can look healthy because most counters only count what the parser accepted.

Fluentd per-worker imbalance: one struggling worker hidden by aggregate metrics

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd per-worker imbalance: one struggling worker hidden by aggregate metrics

Your Fluentd dashboards look fine. Input rate is steady, summed output rate roughly matches, and average buffer usage across the instance is comfortably below the limit. Then someone notices a three-hour gap in the destination for a subset of logs, or the node starts flapping on disk pressure, and the aggregate metrics still insist nothing is wrong.

This is the defining trap of multi-worker Fluentd. When you set workers N in <system>, Fluentd spawns N independent Ruby processes. Each worker has its own event router, its own buffers, its own flush threads, its own retry state, and its own memory footprint. Workers do not share buffer state. If worker 1’s output is stuck in a retry storm while workers 0, 2, and 3 are draining normally, any metric you sum or average across workers will report “mostly healthy” right up until worker 1 starts losing data.

Fluentd plugin load error at startup: LoadError and missing gems

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd plugin load error at startup: LoadError and missing gems

Fluentd fails to start, or starts with part of the pipeline missing, and the log shows LoadError, cannot load such file, or a load_plugin failure. If the failed plugin is an output, that destination silently receives no data while everything else looks healthy. If the plugin is critical to the config, the process refuses to start entirely and you find out via your process-alive check, or via a gap in downstream logs.

Fluentd poison pill crash loop: one bad log line that kills the process on every restart

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd poison pill crash loop: one bad log line that kills the process on every restart

Fluentd is restarting every few seconds. The service comes up, CPU spikes to 100 percent almost immediately, the process dies, systemd (or Kubernetes) restarts it, and the cycle repeats. Input throughput is effectively zero, but the process is “running” most of the time, so a naive process-alive check flaps between green and red without telling you anything.

Fluentd pos_file corruption: duplicates and gaps after a restart

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd pos_file corruption: duplicates and gaps after a restart

You restarted Fluentd (a deploy, an OOM kill, a pod reschedule) and now one of two things is wrong downstream: the same log lines appear twice, or a window of logs never arrived. Both symptoms point at the same component: the in_tail position file.

The pos_file is how in_tail remembers where it stopped reading. It is a plain text file with one line per tailed file, recording the file path, a byte offset in hexadecimal, and an inode number in hexadecimal. There are no checksums and no integrity metadata. On startup, Fluentd reads it and trusts it completely.

Fluentd process not running: the log pipeline is dead and the host has gone dark

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd process not running: the log pipeline is dead and the host has gone dark

The Fluentd process is gone. No logs are being collected, parsed, buffered, or forwarded from this host. Your downstream systems (Elasticsearch, S3, a SIEM, an aggregator tier) are now receiving nothing from here, and most of them will not tell you that. Log pipelines fail silently at the consumer side: the absence of data looks identical to a quiet host.

Fluentd read_from_head replay: a memory and CPU spike that re-reads whole files

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd read_from_head replay: a memory and CPU spike that re-reads whole files

Fluentd has just started (or restarted) and the process is immediately pinned: CPU at or near 100% of one core, RSS climbing fast, and the pipeline emitting a burst of events with old timestamps. Alerts on process CPU, memory growth, or “events older than X arriving at the destination” are probably firing.

In most cases this is read_from_head true doing exactly what you configured. The in_tail plugin is reading each watched file from byte zero, parsing every historical line, and pushing those events through the filter chain, buffer, and outputs. The spike is real resource consumption, but it is one-time per file per position entry, and it stops when the read catches up.

Fluentd retry backoff: a pipeline that is 'retrying' but effectively dead

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd retry backoff: a pipeline that is ‘retrying’ but effectively dead

The Fluentd process is up. The monitor agent responds. retry_count is non-zero, which you already knew, because the destination had a bad night. What retry_count does not tell you is that the next retry attempt is scheduled 4 hours from now. No data is flowing, none will flow for hours, and every dashboard that only tracks the retry counter shows the same flat number it showed an hour ago. The pipeline is technically retrying and operationally dead.

Fluentd retry storm: thundering-herd resonance that keeps a destination down

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd retry storm: thundering-herd resonance that keeps a destination down

Your log destination had a hiccup. It recovered. Then it fell over again. And again. Each time it comes back, it survives for a minute or two, gets hit by a wall of buffered log traffic, and falls over. Fluentd’s own metrics look confusing: retries intermittently succeed, the buffer queue drains a little, then grows again. The destination team insists their service “works fine when we test it.”

Fluentd retry_count climbing: the destination is rejecting or unreachable

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd retry_count climbing: the destination is rejecting or unreachable

You pulled /api/plugins.json from the monitor_agent and one of your output plugins shows a nonzero retry_count, and it keeps going up. At the same time, write_count has stopped incrementing and buffer_queue_length is growing. Fluentd itself is alive and inputs are still collecting.

This is the destination-unavailable failure pattern: the output plugin cannot deliver chunks, the retry engine has taken over, and Fluentd is now in exponential backoff against a destination that is rejecting connections, rejecting data, or simply gone. Every event that arrives from now on accumulates in the buffer. The clock you are racing is buffer capacity, not the retry count itself.

Fluentd rollback_count: chunks recycling back into the queue after failed flushes

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd rollback_count: chunks recycling back into the queue after failed flushes

rollback_count is one of the output-plugin counters exposed by Fluentd’s monitor_agent. It answers a specific question: how many times has a buffer chunk been taken out of the queue for flushing, failed, and been put back? If buffer_queue_length sits stubbornly non-zero while write_count refuses to move, rollback_count tells you chunks are actively cycling through the failure path rather than sitting idle.

Fluentd Ruby GC pressure: garbage-collection pauses that stall event processing

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd Ruby GC pressure: garbage-collection pauses that stall event processing

Fluentd runs on CRuby. Every parsed record, buffered chunk, and serialized payload creates Ruby objects that the garbage collector eventually has to reclaim. Under memory pressure, GC stops being background work and starts blocking the pipeline: inputs stop reading, flush threads miss their windows, chunks roll back into the queue, and retries allocate even more objects.

The outside symptom is easy to misread. The process is alive, CPU is high, and throughput dips in pulses or sags steadily. Buffer metrics can make it look like a slow destination; CPU metrics can make it look like parser load. The distinguishing pattern is Ruby GC activity rising with memory pressure while destination health remains otherwise explainable.

Fluentd silent data loss: logs missing downstream with no error at all

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd silent data loss: logs missing downstream with no error at all

Someone queries the log store for an incident window and the logs are not there. Not delayed, not misindexed: missing. You check Fluentd. The process is running, retry_count is zero, no error logs, buffer metrics look unremarkable. Everything is green, and the data is gone.

This is the hardest Fluentd failure to detect because it is designed into the defaults. When the buffer fills and overflow_action is throw_exception (the default), new events are rejected at the input and lost. When overflow_action is drop_oldest_chunk, chunks are discarded with only a log warning and one counter that almost nobody alerts on. Parse failures can drop records with no visible trace if the error stream is not routed anywhere. In all three cases the pipeline looks healthy from every angle except the one that matters: the destination.

Fluentd slow flush: buffer flush took longer than slow_flush_log_threshold

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd slow flush: buffer flush took longer than slow_flush_log_threshold

You see this in the Fluentd log:

[warn]: buffer flush took longer time than slow_flush_log_threshold: elapsed_time=... slow_flush_log_threshold=20.0 plugin_id="..."

This is a performance warning, not a data-loss signal. Fluentd delivered (or is still retrying) a buffer chunk to the destination, and the write took longer than slow_flush_log_threshold, which defaults to 20 seconds. Each occurrence also increments the slow_flush_count counter on that output plugin.

Fluentd throttled_log_count: in_tail rate limiting is dropping lines at the source

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd throttled_log_count: in_tail rate limiting is dropping lines at the source

throttled_log_count is a cumulative counter exposed by the in_tail input plugin. Every increment means Fluentd deferred log lines at the source because a configured rate limit was exceeded. Unlike buffer-side loss, this happens before the event enters the pipeline: no filter sees it, no buffer holds it, and no output will ever deliver it.

The counter only moves when you have configured throttling in in_tail, either via the <group> section with rate rules or via byte-rate limiting on reads. If you never configured throttling, this counter stays at zero forever, and a zero value tells you nothing. If you did configure it, any non-zero increment rate is a decision point: either the throttle is doing what you designed it to do, or it is silently eating log volume you expected to keep.

Fluentd TLS certificate expiry: a cliff-edge that stops every output at once

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd TLS certificate expiry: a cliff-edge that stops every output at once

Every TLS output on a Fluentd node was working. Then, at one exact second, all of them started failing with the same handshake error. retry_count is climbing on every output that uses TLS, write_count has flatlined, and the buffer is filling. Nothing was deployed. Nothing changed on the network. The only thing that changed is the wall clock: the certificate crossed its notAfter timestamp.

Fluentd too many open files: file descriptor exhaustion stalls the pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd too many open files: file descriptor exhaustion stalls the pipeline

Fluentd logs Errno::EMFILE: Too many open files and the pipeline stops making progress. Log files keep growing on disk, the destination sees nothing new, and the Fluentd process is still alive. It is not crashed; it is wedged against its file descriptor limit, and almost every operation it needs to do next requires opening something.

What makes this failure nasty is the cliff edge. Everything works until the limit is reached, and then several things fail at once: in_tail cannot open newly created log files (and may stop watching them without a loud error), the buffer cannot create new chunk files, and outputs cannot establish new connections. A single resource limit takes out input, buffer, and output simultaneously.

Fluentd unauthorized in_forward connections: log injection on the aggregator tier

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd unauthorized in_forward connections: log injection on the aggregator tier

You ran ss -tn on your Fluentd aggregator and saw connections to port 24224 from IP addresses you do not recognize. Or a security review turned up the fact that your aggregator-tier Fluentd accepts forwarded events from anything that can reach it. Either way: your in_forward input is unauthenticated, and any network-reachable host can inject events directly into your log pipeline.

Fluentd worker died: a partial outage the supervisor hides

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd worker died: a partial outage the supervisor hides

systemctl status td-agent says active (running). The supervisor PID exists. Host-level checks are green. Meanwhile, one of your four Fluentd workers died twenty minutes ago, and a quarter of the log pipeline is gone or degraded.

This is the trap of Fluentd multi-worker mode. When you set workers N in <system>, Fluentd starts a supervisor process and N independent Ruby workers. Each worker has its own event loop, buffers, output threads, and, if configured, monitor_agent endpoint. Workers do not share in-memory state.

Fluentd write_secondary_count: the primary output has failed to its backup

Mon, 01 Jan 0001 00:00:00 +0000

Fluentd write_secondary_count: the primary output has failed to its backup

write_secondary_count is a per-output counter in Fluentd’s monitor_agent API. A nonzero value means the primary output plugin exhausted its retries for at least one chunk, and Fluentd wrote that chunk to the configured <secondary> backup destination instead. The primary pipeline for that output is broken.

This is a ticket-level signal. Data is not lost yet (that is the point of the secondary), but it is no longer flowing where downstream systems expect it. Dashboards, SIEM rules, and alerts that read from the primary destination are now working from a gap.

Force10 Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Forcepoint LLC SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fore Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fort Telecom SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Forte Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fortinet

Mon, 01 Jan 0001 00:00:00 +0000

Fortinet Appliance

Mon, 01 Jan 0001 00:00:00 +0000

Fortinet Fortigate

Mon, 01 Jan 0001 00:00:00 +0000

Fortinet Fortiswitch

Mon, 01 Jan 0001 00:00:00 +0000

Fortinet Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fortinet Licensing

Mon, 01 Jan 0001 00:00:00 +0000

Fraunhofer Fokus SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

FreeBSD

Mon, 01 Jan 0001 00:00:00 +0000

FreeBSD Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Why FreeBSD?

FreeBSD is a free and open source Unix-like operating system descended from the Berkeley Software Distribution (BSD). FreeBSD is the most widely used open source BSD distribution. It is used by companies such as Netflix, Baidu, the United States Department of Defense, and the European Organization for Nuclear Research (CERN).

FreeBSD is a high-quality, stable, and secure operating system used in a wide variety of applications. FreeBSD is developed by a large and passionate community of volunteers and is an excellent choice if you are looking for a high-quality, stable, and secure operating system.

FreeBSD NFS

Mon, 01 Jan 0001 00:00:00 +0000

FreeBSD NFS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

FreeBSD NFS Monitoring

What Is FreeBSD NFS?

FreeBSD Network File System (NFS) is a protocol that allows file and directory sharing across different systems over a network. Designed for efficient, streamlined connections, FreeBSD NFS is a reliable choice for environments requiring seamless file sharing capabilities.

Monitoring FreeBSD NFS With Netdata

Monitoring FreeBSD NFS efficiently is crucial to ensure optimal performance and reliability across networked systems. Netdata offers an effective FreeBSD NFS monitoring tool that leverages an OpenMetrics (Prometheus) exporter. Netdata can ingest data from any Prometheus exporter, meaning users can enjoy automated dashboards, alerts, and more without the need for running a full Prometheus server or Grafana setup. Simply connect to the FreeBSD NFS Exporter, and Netdata takes care of the rest, from data collection to visualization and alerting.

FreeBSD RCTL-RACCT

Mon, 01 Jan 0001 00:00:00 +0000

FreeBSD RCTL-RACCT Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

FreeBSD RCTL-RACCT Monitoring

What Is FreeBSD RCTL-RACCT?

FreeBSD RCTL-RACCT provides an intricate framework for resource consumption accounting and control on machines running the FreeBSD operating system. It enables precise monitoring and limiting of resource usage, which is vital for maintaining optimal system performance. Key tasks include controlling CPU, memory, and other resource utilizations across different processes and users.

Monitoring FreeBSD RCTL-RACCT With Netdata

To monitor FreeBSD RCTL-RACCT, Netdata employs an openmetrics (Prometheus) exporter. Users can rely on the FreeBSD RCTL Exporter to gather crucial metrics seamlessly. Netdata excels in ingesting data from any Prometheus exporter, generating automated dashboards, real-time alerts, and more without necessitating a separate Prometheus server or Grafana setup. This streamlined approach leverages robust open-source technology to ensure comprehensive observability of your FreeBSD resources.

FreeRADIUS

Mon, 01 Jan 0001 00:00:00 +0000

FreeRADIUS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

FreeRADIUS Monitoring

What Is FreeRADIUS?

FreeRADIUS is an open-source implementation of the RADIUS protocol aimed at providing centralized Authentication, Authorization, and Accounting (AAA) services for networked applications. Its broad adoption and flexible configuration options make it a preferred choice for ISPs and enterprises looking to manage network access efficiently.

Monitoring FreeRADIUS With Netdata

Monitoring FreeRADIUS is crucial for maintaining the reliability and performance of your network services. With Netdata’s FreeRADIUS monitoring tool, you can achieve real-time observability, receive timely alerts, and access detailed visualizations of your FreeRADIUS servers’ performance. Netdata’s collector gathers comprehensive metrics, allowing you to keep a keen eye on the health of your RADIUS server in a single dashboard.

Freifunk network

Mon, 01 Jan 0001 00:00:00 +0000

Freifunk Network Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Freifunk Network Monitoring

What Is Freifunk?

Freifunk is a grassroots initiative to create free and open wireless community networks. It enables various communities to contribute and connect using decentralized technology. The Freifunk network operates on the principle of crowd-sourced internet, where each node in the network can communicate with others, facilitating a global mesh of connections.

Monitoring Freifunk Networks With Netdata

Monitoring the Freifunk network is crucial for maintaining seamless connectivity and optimizing network performance. With Netdata, this task becomes efficient and manageable. By leveraging an openmetrics (Prometheus) exporter such as the Freifunk Exporter, Netdata can ingest valuable data to provide real-time insights. Notably, Netdata does not require a Prometheus server or Grafana to operate. The platform offers automated dashboards and alerts, significantly easing the monitoring process. Learn more about how Netdata can ingest data from any Prometheus exporter.

FRRouting

Mon, 01 Jan 0001 00:00:00 +0000

FRRouting Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

FRRouting Monitoring

What Is FRRouting?

FRRouting (FRR) is a robust, high-performance suite for software-based routing on Linux and Unix systems. It offers a comprehensive set of tools aimed at providing scalability and flexibility, allowing networks to handle both small and large network infrastructures effectively. With FRR, network administrators can ensure optimal routing protocols are in place for efficient network traffic management.

Monitoring FRRouting With Netdata

Monitoring FRRouting with Netdata is simplified by utilizing the openmetrics (Prometheus) exporter. Netdata has streamlined the process, allowing users to easily ingest data from any Prometheus exporter. This functionality ensures automated dashboards, alerts, and insights without necessitating a standalone Prometheus server or Grafana. You can get the community exporter here.

Fs Com Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fujitsu Access Limited SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fujitsu Germany GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fujitsu Limited SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fujitsu Network Communications Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Fusionio SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Future Software SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

G-Sense_Error_Rate rising: shock and vibration reaching the drive

Mon, 01 Jan 0001 00:00:00 +0000

G-Sense_Error_Rate rising: shock and vibration reaching the drive

G-Sense_Error_Rate (SMART attribute ID 221 on most drives, ID 191 on some) is a cumulative counter of shock and vibration events that exceeded the drive’s internal threshold, as detected by its built-in accelerometer. The attribute is HDD-only. SSDs and NVMe drives do not report it. Some enterprise HDD models do not expose the attribute at all, which does not mean vibration is absent, only that the drive does not instrument it.

Gadzoox Microsystems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Gamatronic Electronic Industries Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Gandalf SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Garderos GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Gcom Technologies Co Ltd Formerly Greennet Technology Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

GCP GCE

Mon, 01 Jan 0001 00:00:00 +0000

GCP GCE Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

GCP GCE Monitoring

What Is GCP GCE?

Google Cloud Platform’s Compute Engine (GCE) is a robust cloud service that provides scalable virtual machines on demand. GCE is designed to be flexible, offering many choices in terms of operating systems, resource configurations, and networking, making it an ideal choice for developers and enterprises looking to leverage Google’s infrastructure for building, testing, and deploying applications.

Monitoring GCP GCE With Netdata

To effectively monitor GCP GCE, Netdata leverages a Prometheus-based approach through an openmetrics exporter specifically designed for Google Cloud. The GCP GCE Exporter collects rich and detailed metrics, enabling seamless cloud resource management and enhanced performance monitoring. Netdata can ingest data from any Prometheus exporter, providing automated dashboards and alerts without the need for a standalone Prometheus server or Grafana. This capability ensures that monitoring your cloud infrastructure is as efficient and real-time as possible.

GCP IP Ranges

Mon, 01 Jan 0001 00:00:00 +0000

Gearman

Mon, 01 Jan 0001 00:00:00 +0000

Gearman Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Gearman Monitoring

What is Gearman?

Gearman is a distributed job processing system that allows applications to farm out work to other machines or processes that are better suited to do it. It consists of clients, workers, and a Gearman server to queue tasks. This decouples the workload from the tasks, helping in distributing various tasks in a wide set of machines or processes.

Monitoring Gearman With Netdata

Monitor Gearman effectively using the Netdata monitoring tool, designed to keep an eye on your Gearman instances in real-time. Netdata provides significant insight into jobs’ activity, priority, and available workers through its advanced monitoring solutions. This helps in maintaining an optimal work balance and swiftly addressing potential issues.

Gemtek Systems Holding B.V. SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

General Datacomm Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

General Instrument SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Generex Systems GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Generic BGP (BGP4-MIB)

Mon, 01 Jan 0001 00:00:00 +0000

Generic JSON-over-HTTP IPAM

Mon, 01 Jan 0001 00:00:00 +0000

Generic SNMP Device

Mon, 01 Jan 0001 00:00:00 +0000

Generic storage enclosure tool

Mon, 01 Jan 0001 00:00:00 +0000

Generic storage enclosure tool Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Generic storage enclosure tool Monitoring

What Is Generic storage enclosure tool?

The Generic storage enclosure tool, accessible from its GitHub repository, is a pivotal utility for managing storage enclosure metrics. It aids in effective management and performance evaluation of storage devices. Specifically created for IT infrastructures heavily relying on storage, it ensures your IT assets are running smoothly and efficiently by tracking critical metrics.

Monitoring Generic storage enclosure tool With Netdata

Monitoring your storage environment is crucial and Netdata offers a comprehensive solution for it. Using Netdata, you can monitor the Generic storage enclosure tool without needing complex setups like a full Prometheus server or Grafana. Netdata uses an openmetrics (Prometheus) exporter to pull metrics, automatically generating dashboards and alerts for a seamless monitoring experience. Netdata can ingest data effortlessly from any Prometheus exporter, offering profound insights and easy setup.

Generic UPS (UPS-MIB)

Mon, 01 Jan 0001 00:00:00 +0000

Genie Network Resource Management SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Get support!

Mon, 01 Jan 0001 00:00:00 +0000

getifaddrs

Mon, 01 Jan 0001 00:00:00 +0000

getmntinfo

Mon, 01 Jan 0001 00:00:00 +0000

Gigamon

Mon, 01 Jan 0001 00:00:00 +0000

Gigamon Systems LLC SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Giganet Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

GitHub API rate limit

Mon, 01 Jan 0001 00:00:00 +0000

GitHub API Rate Limit Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

GitHub API Rate Limit Monitoring

What Is GitHub API Rate Limit?

GitHub API rate limits are set thresholds that guide the number of requests a user can make within a certain time frame using the GitHub API. Monitoring these limits ensures that your applications and services make efficient use of API calls without exceeding the permitted usage, preventing disruptions and maintaining performance reliability.

Monitoring GitHub API Rate Limit With Netdata

Netdata provides a seamless approach to monitor GitHub API rate limits using an openmetrics (Prometheus) exporter. The GitHub API rate limit Exporter efficiently gathers metrics on rate limits, allowing you to gain insights into your API usage. Netdata can ingest data from any Prometheus exporter, delivering automated dashboards and alerts without the need for a Prometheus server or Grafana setup. This unique capability sets Netdata apart, simplifying the process of tracking API consumption in real-time and enhancing your operational insights.

GitHub repository

Mon, 01 Jan 0001 00:00:00 +0000

GitHub Repository Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

GitHub Repository Monitoring

What Is GitHub Repository Monitoring?

GitHub repository monitoring involves tracking a plethora of metrics essential for maintaining robust and high-performing code repositories. As various developers contribute code, issues may arise that affect performance and reliability. Monitoring these aspects is crucial for ensuring smooth DevOps processes. A well-rounded approach to monitoring examines pull requests, commits, issues, and broader analytics related to repository activity.

Monitoring GitHub Repository With Netdata

Netdata provides a seamless path to monitor GitHub repositories. To monitor GitHub repositories, Netdata uses an openmetrics (Prometheus) exporter, the GitHub Exporter. Netdata is designed to ingest data from any Prometheus exporter, allowing you to benefit from automated dashboards, alerts, and real-time insights without requiring standalone Prometheus servers or Grafana dashboards. This streamlined approach facilitates effective monitoring while saving overhead on setting up intricate monitoring systems.

GitLab Runner

Mon, 01 Jan 0001 00:00:00 +0000

GitLab Runner Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

GitLab Runner Monitoring

What Is GitLab Runner?

GitLab Runner is an open-source project that is used to run your jobs and send the results back to GitLab. It is a key component in GitLab’s CI/CD pipeline, responsible for executing code and providing continuous feedback. GitLab Runner supports multiple platforms, cloud-native applications, and many advanced configuration options, making it versatile and essential for developers looking to automate their CI/CD processes.

Monitoring GitLab Runner With Netdata

Netdata is a powerful tool to monitor GitLab Runner. Using a Prometheus exporter, Netdata collects detailed metrics from GitLab Runner without the need for a Prometheus server or Grafana installation. This setup enables users to access automated dashboards and real-time alerts which significantly ease the process of monitoring your CI/CD environment. Moreover, Netdata is capable of ingesting data from any Prometheus exporter, providing flexibility and scalability for growing operations.

Gnocchi

Mon, 01 Jan 0001 00:00:00 +0000

Go applications (EXPVAR)

Mon, 01 Jan 0001 00:00:00 +0000

Go Beyond 'Good Enough' Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Go Beyond 'Good Enough' Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Go-ethereum

Mon, 01 Jan 0001 00:00:00 +0000

Go-ethereum Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Go-ethereum Monitoring

What Is Go-ethereum?

Go-ethereum, often abbreviated as Geth, is one of the most popular implementations of the Ethereum protocol. Written in Go, it enables developers and enthusiasts to run a full Ethereum node, providing functionalities such as mining, creating contracts, and deploying DApps on the Ethereum blockchain. Go-ethereum is a fundamental tool within the blockchain ecosystem and is highly valued for its robustness and performance.

Monitoring Go-ethereum With Netdata

For anyone invested in the blockchain landscape, ensuring optimal performance and reliability of Go-ethereum nodes is crucial. With Netdata, a highly efficient monitoring solution, you can keep an eye on the performance and health of your Go-ethereum instances in real-time. As a powerful Go-ethereum monitoring tool, Netdata offers deep insights into blockchain operations, facilitating rapid troubleshooting and ensuring seamless operations.

Gobetween

Mon, 01 Jan 0001 00:00:00 +0000

Gobetween Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Gobetween Monitoring

What Is Gobetween?

Gobetween is a high-performance, protocol-agnostic load balancer designed for distributed network traffic management. As an open-source solution, it allows for the efficient routing of requests between multiple servers, offering improved performance, scalability, and reliability for modern web applications.

Monitoring Gobetween With Netdata

Monitor Gobetween effectively using Netdata’s robust monitoring capabilities. Netdata enhances the $name monitoring tool by utilizing an openmetrics (prometheus) exporter, which allows for seamless data collection without the need for a Prometheus server or Grafana setup. With Netdata, users can ingest data from any Prometheus exporter and gain access to automated dashboards, real-time alerts, and a wealth of metrics that facilitate optimized network traffic management and performance analysis.

Google BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

Google Cloud Platform

Mon, 01 Jan 0001 00:00:00 +0000

Google Cloud Platform Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Google Cloud Platform Monitoring

What Is Google Cloud Platform?

Google Cloud Platform (GCP) is a suite of cloud computing services provided by Google. It offers a wide range of services including computing power, storage, and machine learning capabilities, all of which are scalable to meet business needs. Whether you are deploying applications in Kubernetes, analyzing big data, or developing AI/ML models, GCP provides the infrastructure and tools to do so efficiently.

Google Cloud Pub Sub

Mon, 01 Jan 0001 00:00:00 +0000

Google Colab Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Google Colab offers an exceptional platform for running Notebooks, developing machine learning models, and conducting various data science and analytics tasks. To optimize your experience, it is crucial to understand the performance of your Colab instance.

Why monitor Colab?

Monitoring your Google Colab instance provides vital insights into its performance, enabling you to optimize your work and make the most of available resources. The benefits of monitoring your Colab instance include:

Google Pagespeed

Mon, 01 Jan 0001 00:00:00 +0000

Google Pagespeed Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Google Pagespeed Monitoring

What Is Google Pagespeed?

Google Pagespeed is a widely-utilized performance tool that offers valuable insights into the speed and optimization of web pages. It’s an essential service in the realm of cloud computing, providing metrics that help ensure efficient website performance and user experience.

Monitoring Google Pagespeed With Netdata

Monitor Google Pagespeed effectively with Netdata, using the robust capabilities of Netdata’s openmetrics (Prometheus) exporter. Netdata allows you to seamlessly ingest data from any Prometheus exporter, including Google Pagespeed, providing automated dashboards, alerts, and more. This toolkit eliminates the need for a Prometheus server or Grafana, streamlining your monitoring setup and enabling real-time insights.

Google Secret Manager

Mon, 01 Jan 0001 00:00:00 +0000

Google Stackdriver

Mon, 01 Jan 0001 00:00:00 +0000

Google Stackdriver Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Google Stackdriver Monitoring

What Is Google Stackdriver?

Google Stackdriver, now part of Google Cloud’s operations suite, provides powerful monitoring, logging, and diagnostics services for applications hosted on Google Cloud Platform (GCP) and Amazon Web Services (AWS). With Stackdriver, you can proactively identify and troubleshoot issues across cloud services to ensure optimal performance and reliability.

Monitoring Google Stackdriver With Netdata

To monitor Google Stackdriver, Netdata employs an openmetrics (prometheus) exporter, allowing users to gather diverse metrics without needing a Prometheus server or Grafana. The integration seamlessly ingests data, providing automated dashboards, alerts, and insights for real-time monitoring. You can explore the community-supported Google Stackdriver exporter to start monitoring with accuracy and efficiency.

Gotify

Mon, 01 Jan 0001 00:00:00 +0000

gpsd

Mon, 01 Jan 0001 00:00:00 +0000

GPSD Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

GPSD Monitoring

What Is GPSD?

GPSD (GPS Daemon) is a service daemon that monitors one or more GPS or AIS receivers, which share data with various applications. It manages the communication between GPS devices and client applications, ensuring that all connected applications receive accurate and timely positional and navigational data.

Monitoring GPSD With Netdata

Netdata offers robust solutions for monitoring GPSD by leveraging the gpsd-exporter. This tool uses openmetrics (Prometheus) to collect data seamlessly. With Netdata, you can ingest this data without the need for a dedicated Prometheus server or custom Grafana dashboards. Netdata automates dashboards, alerts, and allows you to keep a finger on the pulse of your GPS data systems effortlessly.

Grafana

Mon, 01 Jan 0001 00:00:00 +0000

Grafana Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Grafana Monitoring

What Is Grafana?

Grafana is a powerful open-source platform for monitoring and observability that allows you to query, visualize, and alert on your metrics. By enabling diverse data visualization options, Grafana helps DevOps, SREs, and IT admins understand complex datasets in real-time, which makes it an invaluable tool in modern IT environments.

Monitoring Grafana With Netdata

Monitoring Grafana effectively is crucial for leveraging its full capabilities in visualizing data. Netdata offers a comprehensive solution to monitor Grafana using its OpenMetrics (Prometheus) exporter. Unlike traditional setups where a Prometheus server and Grafana dashboard need to be maintained separately, Netdata simplifies the process by allowing direct ingestion of Prometheus metrics, offering automated dashboards, alerts, and more without the need for additional server infrastructure.

Grand Junction Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Grandstream Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Graphite

Mon, 01 Jan 0001 00:00:00 +0000

Graylog Server

Mon, 01 Jan 0001 00:00:00 +0000

Graylog Server Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Graylog Server Monitoring

What Is Graylog Server?

Graylog Server is a powerful open-source log management platform designed to aggregate and analyze log data effectively. It centralizes log data from across an infrastructure, providing DevOps, SREs, developers, IT admins, and engineers with valuable insights for operational intelligence and security.

Monitoring Graylog Server With Netdata

To monitor Graylog Server, Netdata uses an openmetrics exporter, commonly known as a Prometheus exporter. Netdata seamlessly integrates with any Prometheus exporter, allowing you to get instantaneous access to automated dashboards and alerts, without the need for a complete Prometheus setup or Grafana. This ease of use makes Netdata an excellent choice for tools for monitoring Graylog Server, ensuring efficient log management and analysis.

GreptimeDB

Mon, 01 Jan 0001 00:00:00 +0000

Gude Analog Und Digitalsysteme GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Gw Technologies Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

H3C SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Hadoop Distributed File System (HDFS)

Mon, 01 Jan 0001 00:00:00 +0000

Hadoop Distributed File System (HDFS) Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Hadoop Distributed File System (HDFS) Monitoring

What Is HDFS?

The Hadoop Distributed File System (HDFS) is the primary data storage system used by Hadoop applications. It’s designed to store very large data sets reliably and to stream those data sets at high bandwidth to user applications.

Monitoring HDFS With Netdata

Monitoring HDFS is pivotal for ensuring data reliability and system performance. Netdata’s HDFS monitoring tool provides deep insights by efficiently gathering metrics via JMX from HDFS daemons. This ensures that any anomaly within HDFS is quickly detected and addressed.

Halcyon Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Halon

Mon, 01 Jan 0001 00:00:00 +0000

Halon Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Halon Monitoring

What Is Halon?

Halon is a high-performance email security and delivery platform designed to streamline email management and enhance protection against spam and malicious content. Halon offers a modular environment whereby organizations can tailor configurations to address specific email handling and security needs effectively.

Monitoring Halon With Netdata

Monitoring Halon is crucial to ensure seamless email operations and security. With Netdata, you can monitor Halon using the powerful openmetrics (Prometheus) exporter. Netdata can ingest data from any Prometheus exporter, enabling automated dashboards, alerts, and more without the need for setting up a separate Prometheus server or Grafana. This integration streamlines the monitoring process and delivers real-time insights on Halon’s performance and health.

HANA

Mon, 01 Jan 0001 00:00:00 +0000

HANA Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

HANA Monitoring

What Is HANA?

SAP HANA is an in-memory, column-oriented, relational database management system designed to handle both high transaction rates and complex query processing. As a powerful database server, it’s essential to monitor HANA to ensure efficient data storage and quick query performance.

Monitoring HANA With Netdata

To monitor HANA, Netdata leverages an openmetrics (Prometheus) exporter. By integrating with HANA Exporter, Netdata collects and visualizes key metrics from your HANA database. One of the standout benefits of using Netdata is that it can ingest data from any Prometheus exporter, providing automated dashboards, alerts, and insights into your system’s performance—without the need for a Prometheus server or Grafana setup.

HAProxy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 400 and 408 spikes: bad requests, request timeouts, and Slowloris

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 400 and 408 spikes: bad requests, request timeouts, and Slowloris

Your frontend hrsp_4xx counter just doubled, and it is not 404s. The spike is 400s and 408s, which means HAProxy itself is rejecting or timing out requests before they reach a backend. Unlike backend-generated 4xx, these errors say something about the bytes arriving at your frontend: malformed requests, protocol mismatches, oversized headers, or clients that open connections and never finish the request.

HAProxy 502 Bad Gateway: backend connection and response failures

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 502 Bad Gateway: backend connection and response failures

Clients are getting 502 Bad Gateway from HAProxy. The backend may look healthy, health checks may be green, and yet some fraction of requests comes back with a 502 the application never logged. That is the signature: a 502 from HAProxy is usually generated by HAProxy itself, not passed through from the backend.

HAProxy emits a 502 in two situations. First, it could not establish or use a connection to the backend server at all. Second, it connected fine, but the server sent an invalid, truncated, or incomplete response, or closed the connection mid-response. These are different failure modes with different fixes, and the fastest path to resolution is separating them before touching anything.

HAProxy 503 Service Unavailable: no server is available to handle this request

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 503 Service Unavailable: no server is available to handle this request

Clients are getting 503 Service Unavailable with the message no server is available to handle this request. HAProxy generated this error. Unlike a 500 from your application or a 502 from a broken backend response, a 503 with this message never touched a backend server. HAProxy looked at its own state, concluded it had nowhere to send the request, and answered on its own.

HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 504 Gateway Timeout: timeout server and the backend timeout cascade

Your frontend error rate jumps. Clients report “504 Gateway Timeout.” HAProxy looks alive: the process is running, servers are UP, health checks are green. Yet requests are dying on a timer.

This is the backend timeout cascade, and the 504 is its final symptom. The servers are reachable (TCP accepts succeed), but the application behind them has slowed past the timeout server budget. HAProxy holds the client connection, waits, gives up, and generates a 504 itself.

HAProxy 5xx delta: telling HAProxy-generated errors from backend errors

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy 5xx delta: telling HAProxy-generated errors from backend errors

Your frontend hrsp_5xx counter is climbing and users are seeing errors. The first question that decides the next hour of your incident response: is HAProxy generating these errors, or is it passing through errors your backend servers produced?

HAProxy counts 5xx responses in both places. Frontend rows count every 5xx the client saw, including responses HAProxy fabricated itself. Backend and server rows count only 5xx responses that actually came back from a server. The difference between the two answers “is the backend failing or is HAProxy failing?” without walking every backend one by one.

HAProxy backend losing servers: active server count and cascade risk

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy backend losing servers: active server count and cascade risk

A backend can lose half its servers and HAProxy will still report the BACKEND aggregate row as UP. As long as one server is available, the rollup status says traffic is being served, and status-based dashboards stay green. Meanwhile the surviving servers are absorbing redistributed load, their session counts are climbing, and you are one health check failure away from a full 503 outage for that backend.

HAProxy backend queue building (qcur): requests waiting for a free server slot

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy backend queue building (qcur): requests waiting for a free server slot

You are looking at HAProxy stats and qcur is nonzero on a backend, or worse, climbing. Clients have not seen errors yet, but latency is drifting up. That instinct that something is wrong is correct: qcur is the earliest signal that backend capacity is exhausted, and it shows up before any error counter moves.

qcur is the instantaneous count of connections sitting in a queue because every server that could take them is already at its maxconn limit. Those requests are not being served. They are parked, consuming their client’s timeout budget, waiting for a slot to free up. If the queue grows faster than slots free up, the outcome is queue timeouts and 503s.

HAProxy bandwidth (bin/bout): bytes in, bytes out, and capacity planning

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy bandwidth (bin/bout): bytes in, bytes out, and capacity planning

Every row in HAProxy’s stats output carries two cumulative byte counters: bin (bytes in) and bout (bytes out). They are the ground truth for how much data actually moves through the proxy, per frontend, per backend, and per server. Most teams only look at them when a network link saturates. Used properly, they are a baseline signal, an anomaly detector, and the raw material for bandwidth capacity planning.

HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy compression overhead: comp_byp, CPU cost, and when compression backfires

HTTP compression in HAProxy is a trade: CPU cycles on the proxy in exchange for fewer bytes on the wire. When it works, text responses shrink by 50 to 80 percent and clients on slow links see faster page loads. When it backfires, the proxy burns event-loop CPU compressing content that does not compress, or it sheds compression work under load and you lose the bandwidth savings exactly when traffic is highest.

HAProxy connect time (ctime) high: network latency and backend accept-queue overflow

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy connect time (ctime) high: network latency and backend accept-queue overflow

Your HAProxy backend shows ctime climbing from a steady 1-2ms to tens or hundreds of milliseconds. Total request latency (ttime) rises with it, and if the trend continues, connect timeouts (econ increments, 502/504s on the frontend) are next. The connect phase of the HAProxy-to-backend path is degrading, and it almost always means one of three things: the network path is congested or losing packets, the backend’s kernel accept queue is overflowing, or the backend is too CPU-starved to answer the handshake promptly.

HAProxy connection errors (econ): backend connections refused, timed out, or reset

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy connection errors (econ): backend connections refused, timed out, or reset

Your HAProxy backend shows a rising econ counter. Clients may not see errors yet, because HAProxy retries failed connections and redispatches to other servers, but the counter means HAProxy is trying to open TCP connections to a backend server and failing. The connection is being refused, timing out, or getting reset.

econ is one of the highest-signal error counters HAProxy exposes, and one of the most misread. It is cumulative, it only increments on new connection attempts (so connection reuse can mask a real problem for hours), and it lumps three very different failure modes into one number. Telling “the server process is dead” apart from “a firewall is silently dropping SYNs” requires correlating econ with ctime and a few other signals.

HAProxy connection reuse dropping: http-reuse, pooling, and handshake overhead

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy connection reuse dropping: http-reuse, pooling, and handshake overhead

Backend latency crept up, CPU on the HAProxy host is climbing, and ctime went from zero to measurable milliseconds, but the backends are healthy and the network looks fine. Before you blame the application or the network, check the one ratio almost nobody graphs: connection reuse, reuse / (connect + reuse) on the backend. If it fell off a cliff, HAProxy stopped reusing pooled backend connections and is now paying a full TCP (and possibly TLS) handshake for every request.

HAProxy counter resets on reload: phantom drops and spikes in your metrics

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy counter resets on reload: phantom drops and spikes in your metrics

Your HAProxy 5xx dashboard just showed a cliff-edge drop to zero, then a vertical spike an hour later. No user complaints. No backend incidents. No deploys. If this pattern repeats on a schedule, or right after config changes, you are almost certainly looking at counter resets from HAProxy reloads, not real traffic events.

Every time HAProxy reloads its configuration, a new worker process starts. That new process has its own in-memory statistics, and every cumulative counter starts at zero: hrsp_5xx, econ, eresp, bin, bout, wretr, wredis, chkfail, the SSL counters, all of them. A reload is a new process for stats purposes, not a continuation of the old one.

HAProxy DNS resolver failure: stale IPs and silently misrouted traffic

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy DNS resolver failure: stale IPs and silently misrouted traffic

Your backend servers report UP. Health checks pass. The stats page is green. And traffic is going to IP addresses that belong to pods deleted an hour ago, or to a node that now runs something else entirely.

This is the HAProxy DNS resolver failure mode, and it only bites deployments that resolve backends dynamically: server-template with Consul DNS, Kubernetes headless services, SRV records, or any setup where server addresses come from DNS instead of static config lines. When the resolver times out or a TTL expires without a successful re-resolution, HAProxy keeps routing to the last IPs it knew about. There is no stats CSV counter for this. show resolvers is the only direct view into it, and most teams never look.

HAProxy DroppedLogs: losing log lines exactly when you need them

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy DroppedLogs: losing log lines exactly when you need them

You are mid-incident. Traffic is spiking, 5xx rates are climbing, and you go to the HAProxy logs for per-request timing and termination codes. The logs are not there. Whole seconds of traffic are missing, right at the peak. Then you notice DroppedLogs in show info has been incrementing for hours and nobody was watching it.

DroppedLogs counts log messages HAProxy tried to emit but could not deliver. The cause is almost never HAProxy itself: it is the log pipeline downstream. A syslog daemon that cannot keep up, a socket buffer that overflowed, a log target that stalled. There is no error in the HAProxy log, because the line that would have told you is one of the ones that got dropped.

HAProxy ephemeral port exhaustion: TIME_WAIT and backend connection churn

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy ephemeral port exhaustion: TIME_WAIT and backend connection churn

Your backends are healthy. Every server is UP, health checks pass, the application team sees nothing wrong on their side. Yet HAProxy is logging failed backend connections, econ counters are climbing, and clients see intermittent 502s or stalls. On the HAProxy host, tens of thousands of sockets sit in TIME_WAIT.

This is ephemeral port exhaustion on the backend-facing side of HAProxy. Every new TCP connection from HAProxy to a backend consumes one local ephemeral port. When the connection closes, that port is held in TIME_WAIT for 60 seconds before it can be reused. If the rate of new backend connections is high enough, the port range empties and connect() starts failing with EADDRNOTAVAIL. HAProxy reports these as connection errors even though nothing is wrong with the backend.

HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy health check L4TOUT and L4CON: server marked DOWN and unreachable

You look at the HAProxy stats page (or your monitoring) and a server row is red. The last_chk field says L4TOUT or L4CON, status says DOWN, and HAProxy has stopped routing traffic to that server. Both codes mean the failure happened before any HTTP, before any application code, at the plain TCP layer.

L4TOUT means HAProxy opened a socket and the TCP connect did not complete within the check timeout. L4CON means the connect attempt failed immediately with an error, typically “Connection refused” (TCP RST) or “No route to host” (ICMP). The server is unreachable at the TCP layer, and after fall consecutive failures (default 3), HAProxy marks it DOWN and redistributes its traffic.

HAProxy health check L7STS: server DOWN on the wrong HTTP status

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy health check L7STS: server DOWN on the wrong HTTP status

A backend server flips to DOWN in the stats page, last_chk shows L7STS, and the log line says something like Layer7 wrong status, code: 503, info: "Service Unavailable". The server process is running. The port is open. You can curl it by hand and get a response. But HAProxy has pulled it out of rotation and traffic is concentrating on the survivors.

HAProxy health checks green but the application is broken: when UP does not mean healthy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy health checks green but the application is broken: when UP does not mean healthy

Every server in the backend shows UP. Health checks are passing. And users are getting 500s. This is one of the most common HAProxy postmortem findings: “health checks said everything was fine.”

The failure is conceptual, not a bug. HAProxy’s health check engine runs independently of traffic and only proves that the specific probe you configured succeeds. A TCP check proves the port accepts connections. An HTTP check to /health proves that one URL returns the expected status. Neither proves that real requests, which exercise the database, auth, and downstream dependencies, actually work.

HAProxy hot-thread skew: when average Idle_pct hides a saturated thread

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy hot-thread skew: when average Idle_pct hides a saturated thread

Latency is climbing, throughput has plateaued below what the hardware should deliver, and show info reports Idle_pct at a comfortable 60 or 70 percent. System CPU shows one core pegged while the others coast. The bottleneck is not HAProxy as a whole. It is one thread.

With nbthread > 1 (the default since HAProxy 2.0, where nbthread is set to the number of available CPUs), work is spread across per-thread event loops. Idle_pct is a process-level average across all of them. One thread pinned at 100% while the rest sit at 80% idle averages out to “looks fine,” but that hot thread caps overall throughput: every connection it owns gets processed late, and no other thread can take them over.

HAProxy Idle_pct dropping: event-loop CPU saturation and the busy-polling trap

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy Idle_pct dropping: event-loop CPU saturation and the busy-polling trap

Your HAProxy latency is climbing, nothing is queuing on the backends, and the one metric that looks wrong is Idle_pct sliding toward zero. Or worse: Idle_pct has been pinned at zero for weeks and you only just noticed because someone finally asked what it meant. Both situations are common, and they have very different explanations.

Idle_pct is HAProxy’s self-reported measure of event-loop headroom: the share of time the loop spends waiting in poll() versus processing events. It measures HAProxy CPU headroom, not system CPU. A host at 50% system CPU with Idle_pct at 80% is fine. A host at 50% system CPU with Idle_pct at 10% means HAProxy itself is saturated. That distinction is the entire point of the metric, and it is also where the traps live.

HAProxy in TCP mode: the HTTP signals you lose and what to watch instead

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy in TCP mode: the HTTP signals you lose and what to watch instead

When a frontend and its backend run with mode tcp, HAProxy stops parsing the protocol inside the connection. It forwards bytes and nothing more. That is often correct: databases, gRPC, Redis, TLS passthrough, or protocols HAProxy does not understand. But it amputates a large part of the monitoring surface most HAProxy alerting assumes exists.

The failure mode this creates is not an outage; it is a blind spot. Dashboards that worked in HTTP mode start reporting zeros that look like “no traffic” or “no errors.” Health checks stay green while the application behind them is broken. Latency alerts keyed on rtime never fire again because rtime is permanently zero. The proxy is fine; your visibility into what flows through it is not.

HAProxy kernel accept-queue overflow: silent SYN drops and somaxconn

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy kernel accept-queue overflow: silent SYN drops and somaxconn

Clients report connection timeouts. You open the HAProxy stats page and everything looks fine: session rate is normal, no 5xx spike, servers are UP, Idle_pct is healthy. The network team sees nothing. The clients insist the load balancer is dropping them.

Both sides are right. The kernel is silently dropping SYN packets in the accept queue before they ever reach HAProxy’s event loop. From HAProxy’s perspective, those connections never existed. There is no log line, no error counter, no stat field that increments. The only evidence lives in two kernel counters, ListenOverflows and ListenDrops, that almost nobody graphs.

HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy maxconn hierarchy: global, frontend, backend, and per-server limits

HAProxy does not have one connection limit. It has four, enforced independently at the global, frontend, backend, and per-server levels. Any one of them can throttle traffic while the other three show headroom. This is why “global maxconn is at 30%” tells you almost nothing during an incident: the request path can be blocked at a per-server limit you never set, or queuing behind a backend constraint you did not know existed.

HAProxy maxconn reached: new connections queued and rejected

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy maxconn reached: new connections queued and rejected

HAProxy is refusing work. Clients see connection timeouts or resets, load tests fail at a suspiciously round concurrency number, and yet the HAProxy process looks healthy: CPU is fine, memory is fine, the process is up, health checks pass. The stats tell the real story: CurrConns is flat at Maxconn, or scur is pinned at slim on a frontend, and it does not move.

HAProxy memory growth and OOM: connection buffers, pools, and memory ceilings

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy memory growth and OOM: connection buffers, pools, and memory ceilings

The symptom usually arrives as a dead process, not a slow one. HAProxy was proxying fine, then the PID changed, connections dropped for a few seconds, and dmesg shows the OOM killer picked haproxy as the victim. Or, before it gets that far, RSS climbs day over day and nobody knows whether it is a leak or just load.

HAProxy Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy Monitoring

What Is HAProxy?

HAProxy, short for High Availability Proxy, is a popular open-source software widely used for load balancing and proxying for TCP and HTTP-based applications. It is known for its reliability, performance, and feature-rich nature, making it a staple in server infrastructure for distributing workloads efficiently.

Monitoring HAProxy With Netdata

Monitoring HAProxy is crucial for ensuring your application infrastructure performs optimally and remains highly reliable. Netdata offers comprehensive and real-time monitoring of HAProxy, delivering rich visualization of health and performance metrics. By leveraging such a tool, you can observe metrics like response times, session rates, and network throughput in a live, interactive interface. You can get started by checking out the Live Demo or signing up for a free trial.

HAProxy monitoring checklist: the signals every production proxy needs

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy monitoring checklist: the signals every production proxy needs

Most HAProxy outages are visible in its own stats long before users notice. The problem is that the stats socket exposes dozens of fields, and teams usually wire up the five metrics their dashboard template shipped with and stop there. Then a reload storm, a retry storm, or a silent stick-table overflow teaches them what they were missing.

This checklist organizes the signals that catch production failures into four maturity levels: survival, operational, mature, and expert. Each level builds on the previous one. You do not need to reach expert on day one, but you should know which level you are at, because that determines which failure modes are invisible to you.

HAProxy monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy monitoring maturity model: from survival to expert

Most HAProxy monitoring setups fall into one of two states: a process check and a prayer, or a wall of charts nobody reads. Neither answers the question that matters during an incident: is HAProxy the problem, the messenger, or the victim.

This article defines four maturity levels for HAProxy monitoring: Survival, Operational, Mature, and Expert. Each level adds signals that catch a class of failures the previous level cannot see. The goal is not to collect everything. The goal is to know, at any moment, which questions your monitoring can answer and which it cannot.

HAProxy peers sync broken: divergent stick tables across instances

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy peers sync broken: divergent stick tables across instances

You have two or more HAProxy instances fronting the same service, and their behavior no longer matches. Rate limiting triggers on one node but lets the same client through on another. A failover happens and every sticky session evaporates, even though you configured stick tables and a peers section specifically to survive that event. Or you compare show table output across instances and the entry counts are wildly different when they should be near-identical.

HAProxy per-source-IP rate limiting: stick tables, conn_rate, and http_req_rate

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy per-source-IP rate limiting: stick tables, conn_rate, and http_req_rate

Per-source-IP rate limiting in HAProxy is built on stick tables: in-memory key-value stores that track counters per client key (usually the source IP) and expose those counters to ACLs. When it breaks, it breaks silently. A full table, a mismatched expire, or an IPv6 client population can disable your limiting without a single error in the logs, and most teams find out only after an abuse incident.

HAProxy PoolFailed: buffer starvation and memory-allocator failures

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy PoolFailed: buffer starvation and memory-allocator failures

PoolFailed in show info counts failed internal memory-pool allocations since process start. It is normally zero. Any nonzero value means HAProxy tried to allocate a buffer, connection, or session object and could not get one. Work was dropped or stalled because of it.

The tricky part is how this surfaces. Buffer starvation rarely looks like a memory problem from the outside. It looks like mysterious latency: rtime and backend health are fine, CPU has headroom, the network is clean, and yet clients see stalls and sporadic failures. PoolFailed is one of the few direct signals that HAProxy itself is refusing work because it ran out of an internal resource.

HAProxy queue time (qtime) rising: the earliest backend-saturation signal

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy queue time (qtime) rising: the earliest backend-saturation signal

You are looking at a backend whose qtime has crept from zero to a few hundred milliseconds, or a few seconds, and nothing is “down” yet. Health checks are green, error rates are flat, and no alert has fired. That is the point of qtime: it moves before the failure signals.

Nonzero qtime means requests spent time in HAProxy’s queue because every usable server connection slot was busy when they arrived. It is usually the first measurable symptom of backend saturation: ahead of 503s, ahead of timeout server expiries, ahead of user complaints.

HAProxy reload storm: lingering processes leaking file descriptors and memory

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy reload storm: lingering processes leaking file descriptors and memory

You log into the HAProxy host because memory and file descriptor usage have been climbing for days, and you find a dozen haproxy processes running. Only one is serving traffic. The rest are old workers stuck in soft-stop, each holding open connections, connection buffers, and file descriptors it will never release. Traffic looks fine, per-process metrics look fine, and the host is still slowly running out of resources.

HAProxy response errors (eresp): truncated and invalid backend responses

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy response errors (eresp): truncated and invalid backend responses

The eresp counter is climbing on a backend, and clients are seeing broken pages, truncated downloads, or intermittent 502s. The backend servers report healthy. Health checks are green. The 5xx rate, if it moved at all, does not explain the volume of user complaints.

This is the failure mode eresp exists to capture: HAProxy connected to the backend successfully, sent the request, and then received a response it could not treat as valid. Either the bytes were not parseable HTTP, or the server closed the connection before the response finished. That is a protocol-level failure, not an application returning an error status code, and it needs a different investigation path than a 5xx spike.

HAProxy response time (rtime) climbing: slow backends behind the proxy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy response time (rtime) climbing: slow backends behind the proxy

Your HAProxy dashboards show rtime creeping up on one or more backends. Clients have not started timing out yet, or maybe they just have. HAProxy itself looks fine: CPU is reasonable, no servers are DOWN, error rates are flat or only slightly elevated. But the average time between HAProxy forwarding a request and the backend’s first response byte is growing, and it is not coming back down.

HAProxy retries and redispatches (wretr/wredis): the backend instability nobody watches

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy retries and redispatches (wretr/wredis): the backend instability nobody watches

Your dashboards are clean. No 5xx spike, latency averages look normal, every backend server is UP. Then the backend falls over in what feels like seconds, and the postmortem shows the application was flapping for six hours before the outage. The warning was there the whole time, sitting in two counters almost nobody charts: wretr and wredis.

These counters record HAProxy’s built-in compensation mechanism at work. When a connection to a backend server fails, HAProxy does not immediately return an error to the client. It retries. If option redispatch is set, it can also give up on the original server and dispatch the request to a different one. From the client’s perspective the request succeeds, possibly a bit slower. From the proxy’s perspective, something in your backend just failed and got papered over.

HAProxy run queue rising: task backlog and scheduler pressure

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy run queue rising: task backlog and scheduler pressure

HAProxy’s event loop processes every connection, request, health check, and timer as tasks scheduled across its threads. When tasks arrive faster than threads can drain them, they pile up in the run queue. That backlog is the earliest internal signal that HAProxy is CPU-saturated: it appears before latency spikes, before backend queuing, and before client timeouts.

It is also the signal to use when Idle_pct is meaningless. With busy-polling enabled, the event loop never sleeps, so Idle_pct sits near zero by design. The Run_queue field from show info and the per-thread counters from show activity are the replacement.

HAProxy scur approaching slim: the concurrent-session saturation signal

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy scur approaching slim: the concurrent-session saturation signal

Your alert says scur is at 85% of slim on a frontend, or CurrConns is closing on Maxconn in show info. HAProxy is still up. Traffic is still flowing. But you are running out of concurrent-session headroom, and what happens next depends entirely on which of the four independent limits you are about to hit.

The danger is that the system still appears responsive: HAProxy is alive and healthy, but clients are about to see queuing delays or 503 rejections. The scur/slim ratio is the warning, if you read it correctly.

HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy server flapping: rise, fall, and health checks oscillating UP and DOWN

A backend server that alternates between UP and DOWN is worse than one that is cleanly down. Every transition redistributes traffic: when the server goes DOWN its connections move to the survivors, when it comes back UP it absorbs a fresh share of load before it has warmed up, and every cycle costs retries, redispatches, and connection churn. If the flap rate is high enough, the oscillation itself becomes the incident.

HAProxy server-template and dynamic backends: monitoring service-discovery routing

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy server-template and dynamic backends: monitoring service-discovery routing

With a static HAProxy config, the server list lives in the config file. Capacity review is config review: count the server lines and you know what the backend can absorb. With server-template, the server list lives in DNS. HAProxy pre-provisions a pool of empty server slots, and its internal resolver fills and empties them as A or SRV records change. This is the mechanism behind HAProxy fronting Kubernetes Ingress, Consul, and most autoscaled service-discovery setups.

HAProxy show pools: finding which internal pool is consuming memory

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy show pools: finding which internal pool is consuming memory

HAProxy’s memory footprint is not one number. The process holds dozens of internal object pools: buffers, connections, sessions, tasks, stick-table entries, SSL state. When RSS climbs, when PoolFailed goes nonzero, or when you are sizing a memory budget for a new instance, the question is never “how much memory is HAProxy using” but “which pool is using it”. show pools is the runtime API command that answers that.

HAProxy Slowloris and idle-connection pile-up: high scur, low request rate

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy Slowloris and idle-connection pile-up: high scur, low request rate

Your HAProxy frontend shows scur pinned at or near slim, but req_rate is a fraction of what those connections should be producing. Throughput (bin/bout) is near zero, backend queue depth (qcur) is zero because backends are doing nothing, and ereq with 408 responses is climbing. HAProxy is not down. It is not slow. It is full: thousands of connections are holding session slots open and sending data so slowly they may as well be dead.

HAProxy SSL certificate expired: total TLS failure and how to catch it first

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy SSL certificate expired: total TLS failure and how to catch it first

Every client connecting to the affected frontend gets a TLS handshake failure. Not a percentage of clients, not a slow degradation: all of them, immediately, from the moment the certificate’s notAfter timestamp passes. Browsers show NET::ERR_CERT_DATE_INVALID, API clients throw certificate validation errors, and HAProxy itself is perfectly healthy, running, and passing every process-liveness check you have.

The mechanism is trivial, the blast radius is total, and HAProxy gives you no built-in metric, counter, or log warning that a certificate is about to expire. If you are reading this during an incident, skip to Quick checks and Fixes. If you are reading it afterward, the Prevention section is the part that matters.

HAProxy SSL session cache misses: full handshakes and lost resumption

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy SSL session cache misses: full handshakes and lost resumption

Your HAProxy CPU is climbing but connection counts look normal. Idle_pct is dropping, latency on new connections is rising, and existing sessions seem fine. When you pull show info, SslFrontendKeyRate is far above baseline and the SSL session cache numbers tell the story: almost every lookup is a miss.

A high cache miss rate means most TLS clients are paying for a full handshake instead of resuming a previous session. Full handshakes are the most CPU-intensive operation HAProxy performs; resumed handshakes skip most of that cost. When resumption breaks, the load balancer loses a large fraction of its capacity without any change in traffic volume.

HAProxy SslFrontendKeyRate: the handshake rate that drives CPU cost

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy SslFrontendKeyRate: the handshake rate that drives CPU cost

SslFrontendKeyRate is the number of new (non-resumed) TLS handshakes per second across your HAProxy frontends. Because full TLS handshakes are the most CPU-intensive operation HAProxy performs, this metric is effectively the CPU cost of your TLS traffic expressed as a rate.

Operators usually look for this metric in two situations: HAProxy CPU is climbing and they need to know whether TLS is the cause, or they are doing capacity planning and need a defensible number for “how many handshakes per second can this box absorb.” Both depend on understanding what the metric counts, where it comes from, and what breaks it.

HAProxy stick-table overflow: rate limiting that silently stops working

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy stick-table overflow: rate limiting that silently stops working

Your HAProxy config has rate limiting. It has session persistence. Both are enforced by stick tables, and both can stop working without an error counter, a log line, or a page.

Stick tables are fixed-size, in-memory key-value stores. When a table reaches its configured size, HAProxy has two possible behaviors and neither one alerts you. With the default configuration, HAProxy flushes expired entries to make room for new ones. If the table is full of entries that have not yet expired, new entries are refused. With the nopurge option, eviction is disabled entirely and new entries are silently rejected. In both paths, whatever the table was doing (rate limiting, sticky sessions, abuse tracking) degrades or stops for affected clients.

HAProxy stuck in soft-stop: draining connections and hard-stop-after

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy stuck in soft-stop: draining connections and hard-stop-after

You reloaded HAProxy minutes ago, but pgrep haproxy still shows two, three, maybe a dozen PIDs. The old process is not serving new traffic, but it refuses to die. Memory and file descriptor usage on the host keep climbing with every reload, and your monitoring loses counter history each time it happens.

The old process is in soft-stop: it has unbound from its listeners and is waiting for its existing connections to close before exiting. A drain of a few seconds to a couple of minutes is normal. A process draining for more than 10 minutes is stuck, and the cause is almost always long-lived connections (WebSocket, Server-Sent Events, gRPC streaming, or plain TCP keepalive sessions) that have no reason to close on their own.

HAProxy TLS handshake failures: broken clients after a cert or cipher change

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy TLS handshake failures: broken clients after a cert or cipher change

A certificate rotation or TLS policy change goes out, HAProxy reloads cleanly, health checks pass, traffic graphs look normal, and then the tickets start: a subset of clients can no longer connect. The load balancer itself is fine. The handshake rate even looks normal. What changed is the failure rate within handshakes, and HAProxy has no dedicated stats counter that shows it to you directly.

HAProxy TLS handshake storm: new-connection floods that saturate the event loop

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy TLS handshake storm: new-connection floods that saturate the event loop

Traffic looks normal at the TCP layer. Connections are arriving, the frontend is accepting them, but latency is climbing, some clients time out before ever sending a request, and backend servers look healthy. When you check show info, Idle_pct is near zero and SslFrontendKeyRate is far above anything you have tested. You are in a TLS handshake storm: HAProxy is burning every CPU cycle on cryptographic handshake work and the event loop has nothing left for the rest of the pipeline.

HAProxy Too many open files: file descriptor exhaustion and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy Too many open files: file descriptor exhaustion and refused connections

HAProxy is refusing new connections. Clients see connection timeouts or resets, health checks start failing for no application reason, and the logs show Too many open files. The process is alive and CPU looks fine, which makes this confusing the first time you hit it: nothing is overloaded in the usual sense. The proxy has run out of file descriptors.

HAProxy tune.bufsize and 400 errors: buffers too small for large headers

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy tune.bufsize and 400 errors: buffers too small for large headers

Your frontend 400 rate just spiked. The backends insist they never saw the requests, and they are telling the truth: HAProxy generated these 400s itself, during header parsing, before any backend was selected. The usual trigger is a request header block that no longer fits in the per-connection buffer, and the usual suspects are large cookies, JWT bearer tokens, or a newly added header that pushed total header size past the limit.

Hardware information collected from kernel ring.

Mon, 01 Jan 0001 00:00:00 +0000

Harmonic Lightwaves SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

HDD temperature

Mon, 01 Jan 0001 00:00:00 +0000

HDD Temperature Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

HDD Temperature Monitoring

What Is HDD Temperature?

HDD temperature refers to the heat emitted by your hard disk drive during its operation. Monitoring the HDD temperature can prevent overheating, which could compromise data integrity and hardware stability. Using an efficient HDD temperature monitoring tool is crucial for maintaining your system’s overall health.

Monitoring HDD Temperature With Netdata

Netdata provides a comprehensive HDD temperature monitoring solution. This open-source platform leverages the hddtemp daemon to collect and display real-time temperature data across all your disks, ensuring you stay informed about any potential overheating issues. With Netdata’s live demo, you can see Netdata’s monitoring capabilities in action.

Hewlett Packard Enterprise SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Hewlett Packard SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Hitachi Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Hitron CODA Cable Modem

Mon, 01 Jan 0001 00:00:00 +0000

Hitron CODA Cable Modem Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Hitron CODA Cable Modem Monitoring

What Is Hitron CODA Cable Modem?

The Hitron CODA Cable Modem is a device that facilitates high-speed internet connectivity, often used in homes and businesses. This modem is designed to enhance internet performance through efficient data management and superior networking capabilities. Monitoring is crucial for maintaining optimal performance and troubleshooting any connectivity issues that may arise.

Monitoring Hitron CODA Cable Modem With Netdata

To monitor the Hitron CODA Cable Modem, Netdata utilizes an openmetrics (Prometheus) exporter developed by the community. This integration allows Netdata to ingest data from any Prometheus exporter, offering users automated dashboards, alerts, and more—without the need for setting up a complex Prometheus server or Grafana. The seamless integration with Netdata means you get actionable insights with minimal setup.

Homebridge

Mon, 01 Jan 0001 00:00:00 +0000

Homebridge Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Homebridge Monitoring

What Is Homebridge?

Homebridge is a lightweight Node.js server that emulates the iOS HomeKit API. It allows you to connect all your smart home devices, which may not natively support HomeKit, to the Apple ecosystem. By integrating a wide array of home automation devices from various brands into a single cohesive environment, Homebridge plays a pivotal role in efficient home automation management.

Monitoring Homebridge With Netdata

To monitor Homebridge effectively, utilizing the Homebridge Prometheus Exporter, Netdata provides a comprehensive solution. Netdata uses an openmetrics (Prometheus) exporter approach, capable of ingesting data from any Prometheus exporter, offering you automated dashboards, alerts, and more—all without the need for a separate Prometheus server or Grafana. This streamlined, powerful integration makes Homebridge monitoring seamless and highly efficient.

Homey

Mon, 01 Jan 0001 00:00:00 +0000

Homey Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Homey Monitoring

What Is Homey?

Homey is a smart home controller that allows users to manage and automate their home appliances and devices, offering a seamless and integrated smart living experience. Its user-friendly interface and compatibility with a wide range of smart devices make it a popular choice for home automation enthusiasts.

Monitoring Homey With Netdata

Netdata provides a comprehensive and efficient way to monitor Homey using the Homey Prometheus Exporter. By leveraging an openmetrics (Prometheus) exporter, Netdata enables the ingestion of data from any Prometheus exporter. This means you can monitor Homey efficiently without the need for a Prometheus server or Grafana. Netdata offers automated dashboards, alerts, and detailed real-time monitoring, making it an essential tool for maintaining the optimal performance of your smart home setup.

Honeypot

Mon, 01 Jan 0001 00:00:00 +0000

Honeypot Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Honeypot Monitoring

What Is Honeypot?

A honeypot is a security mechanism that creates a virtual trap to entice cyber attackers. It essentially offers an isolated and controlled environment where malicious activities can be detected and studied without compromising real systems. Used prominently in cybersecurity, a honeypot mimics a legitimate system to lure hackers, allowing organizations to analyze attack techniques and enhance system defenses.

Monitoring Honeypot With Netdata

Monitoring Honeypot with Netdata leverages the Intrinsec honeypot_exporter using an openmetrics (Prometheus) exporter. Netdata excels by ingesting data from any Prometheus exporter, providing users automated dashboards, alerts, and insights without the need for a Prometheus server or Grafana. This seamless integration enables real-time visibility into Honeypot metrics, facilitating efficient threat detection and management.

How ActiveMQ Classic actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How ActiveMQ Classic actually works in production: a mental model for operators

Most ActiveMQ incidents are misdiagnosed in the first hour because the operator is looking at the wrong subsystem. A “hung producer” is flow control doing its job. A “slow broker” is fsync latency on the KahaDB device. A “healthy” broker that is losing messages is a full temp store with flow control disabled. Each of these looks like a different problem until you know which subsystem is actually involved.

How an NVIDIA GPU actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How an NVIDIA GPU actually works in production: a mental model for operators

GPU runbooks fail when they treat the GPU as a black box that emits a utilization percentage. “100% utilization but slow” makes sense only when you know what utilization measures. “nvidia-smi hangs” makes sense only when you know where nvidia-smi gets its data.

This guide covers the two pieces underneath GPU troubleshooting: the subsystems that produce production failures, and the telemetry plane through which you observe them. It applies primarily to datacenter GPUs such as V100, A100, and H100, with differences noted for consumer cards.

How Apache HTTPD actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Apache HTTPD actually works in production: a mental model for operators

Most Apache incidents are misdiagnosed for the same reason: the operator is reading signals without knowing which execution model produced them. A scoreboard full of K states is a crisis on prefork and a rounding error on event. “Apache is down” and “the backend is down” look identical from outside a reverse proxy. MaxRequestWorkers means something different depending on whether your concurrency unit is a process or a thread.

How Apache Pulsar actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Apache Pulsar actually works in production: a mental model for operators

Most Pulsar incidents are misdiagnosed in the first thirty minutes because the operator is looking at the wrong layer. A publish latency spike gets chased on the broker when the real bottleneck is a bookie journal disk. A “consumer problem” turns out to be a cursor that stopped advancing. A broker that looks healthy on its HTTP endpoint is fenced off from ZooKeeper and losing topic ownership in a loop.

How BIND actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How BIND actually works in production: a mental model for operators

During an incident, named can feel like a black box. A single process handles packet reception, DNS parsing, ACL evaluation, cache lookup, zone lookup, recursive fetch, DNSSEC validation, RPZ policy enforcement, and response serialization. There is no separate resolver process, no separate authoritative process, no separate cache daemon. Every query flows through the same pipeline, and every subsystem competes for the same pool of CPU, memory, file descriptors, and kernel network buffers.

How Cassandra actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Cassandra actually works in production: a mental model for operators

If you operate Cassandra in production, you are managing a distributed, partitioned, replicated log-structured merge tree where every node is a peer. There is no master to restart, no central query planner to tune, and no automatic load balancing that will save you from a hot partition. Every write is a sequential append to a commitlog and an in-memory update on multiple replicas. Every read is a merge of memtables and immutable SSTables, filtered by probabilistic bloom filters and reconciled by timestamp.

How Ceph actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Ceph actually works in production: a mental model for operators

Most Ceph incidents become legible the moment you stop reasoning about RBD, CephFS, and RGW as separate products and start reasoning about one system: RADOS placing objects on OSDs. The three client interfaces are thin translations. Underneath them, every write is an object, every object lives in a placement group, and every placement group is mapped to a set of OSDs by a deterministic algorithm.

How ClickHouse actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How ClickHouse actually works in production: a mental model for operators

ClickHouse is a column-oriented OLAP database built around the MergeTree family of table engines. It does not fail like a transactional database. Its failure modes are distinctive and almost always originate in the storage layer rather than the query layer. In production, most incidents begin as storage-structure debt: immutable data parts accumulate faster than background processes can consolidate them. By the time queries slow down or inserts fail, the underlying merge crisis has been developing for hours.

How CockroachDB actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How CockroachDB actually works in production: a mental model for operators

CockroachDB is a distributed, strongly-consistent SQL database built on a replicated key-value store. It layers SQL execution on top of a transactional KV engine that uses Raft consensus for replication and MVCC for concurrency control. To reason about its failures, you must hold several interacting subsystems in your head simultaneously.

This is not a tutorial. It is the mental model that experienced operators use when diagnosing slow queries, unavailable ranges, clock skew, and compaction death spirals. Each subsystem has its own failure modes, but the most damaging incidents happen when failures cascade across layers.

How Consul actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Consul actually works in production: a mental model for operators

Most Consul incidents are debugged with the wrong mental model. Operators check consul members, see all nodes alive, conclude the cluster is healthy, and miss that Raft has no leader or that the catalog is stale by minutes. Consul runs three concurrent subsystems with different failure modes, signals, and consistency guarantees. Most monitoring collapses them into one “is Consul up?” check.

How CoreDNS actually works in production: the plugin chain mental model

Mon, 01 Jan 0001 00:00:00 +0000

How CoreDNS actually works in production: the plugin chain mental model

Most CoreDNS incidents are pipeline incidents, not DNS incidents. An upstream resolver gets slow, goroutines pile up, memory climbs, and the pod gets OOMKilled. Or the Kubernetes API watch drops silently and CoreDNS keeps answering every query correctly except the answers are three hours stale. Each failure mode is a direct consequence of how CoreDNS is built internally, and most CoreDNS runbooks assume you already hold that internal model in your head.

How Elasticsearch actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Elasticsearch actually works in production: a mental model for operators

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. Every node runs interacting subsystems that compete for the same resources. If you treat it as a black box that stores JSON, you will miss the failure modes that kill clusters at 3 a.m. The interactions between subsystems determine whether a cluster survives a traffic spike or collapses into a heap-pressure death spiral.

How Envoy actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Envoy actually works in production: a mental model for operators

Envoy is a multi-threaded, event-driven L4/L7 proxy written in C++. Its architecture directly shapes what you see in metrics, access logs, and user-visible behavior during incidents. If you do not know that worker threads share nothing in the hot path, aggregate CPU utilization will mislead you. If you do not know that circuit breaker 503s are indistinguishable from upstream-generated 503s at the counter level, you will blame the wrong component. If you do not know that Envoy keeps serving traffic on stale configuration after an xDS disconnect, you will miss the slow-burn failure that surfaces hours later.

How Fluentd actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Fluentd actually works in production: a mental model for operators

Most Fluentd incidents are not random. They are the predictable output of a small set of internal mechanisms: a single-threaded event router, a chunked buffer with a hard limit, Ruby threads fighting over one GVL, and a retry engine with exponential backoff. If you understand those four, you can predict almost every Fluentd failure mode before you open a dashboard.

How HAProxy actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How HAProxy actually works in production: a mental model for operators

Most HAProxy incidents come from an operator applying the wrong mental model: treating HAProxy like a web server, like nginx, or like a black box that either works or does not. HAProxy is an event-driven state machine that multiplexes hundreds of thousands of connections across a small number of threads, and almost every confusing symptom traces back to one of a handful of internal mechanisms: the accept pipeline, the maxconn hierarchy, the buffer and connection pools, the health check engine, or a subsystem you forgot was running.

How Kafka actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Kafka actually works in production: a mental model for operators

What it is and why it matters

Each broker is a concurrent system of subsystems sharing disk, network, page cache, and threads. Tail latency spikes and offline partitions usually stem from interactions between these subsystems, not single component failures.

Underneath the distributed commit log abstraction, each broker runs a reactor-pattern server with three saturation layers: network threads, a bounded request queue, and I/O handler threads. Between I/O threads and the log sits a delayed-operation timer wheel called the purgatory. A separate controller maintains cluster metadata in a single-threaded event queue. Consumer groups are managed by a Group Coordinator that hashes group IDs to the __consumer_offsets topic. Log data lives in segment files that rely on OS page cache and zero-copy sendfile. These subsystems compete for the same disk I/O, network bandwidth, file descriptors, and cache. Treating them as independent pipelines leads to unnecessary restarts and missed bottlenecks.

How Logstash actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Logstash actually works in production: a mental model for operators

Most Logstash incidents are diagnosed badly because the operator is reasoning about the wrong system. They see a “log shipper” and check whether the process is up. The process is up. The API returns 200. Nothing is being delivered. Or they see high CPU and assume the host is undersized, when the real problem is one grok pattern with catastrophic backtracking. Or they watch queue depth stay flat for hours and conclude all is well, while the persistent queue quietly absorbs a downstream outage that will page someone at 4 a.m. when it fills.

How LVM actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How LVM actually works in production: a mental model for operators

Most LVM incidents are caused by operators holding the wrong mental model: thinking of LVM as “partitioning with extra steps” rather than what it actually is, a userspace management layer that programs the kernel’s device-mapper subsystem. Once you internalize that, the confusing behaviors stop being confusing. A full thin pool freezing every process in D-state, a snapshot silently invalidating, an lvs command hanging during the exact incident you need it for: all of these follow directly from the architecture.

How Memcached actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Memcached actually works in production: a mental model for operators

Memcached is a multi-threaded, in-memory key-value cache daemon built on libevent. No persistence, no replication, no clustering. Clients handle sharding. A restart means total data loss. These are the design, not limitations to work around.

Before you can debug a memcached incident, you need three abstractions: how memory is partitioned (the slab allocator), how items age within each partition (the segmented LRU), and how connections and threads interact under load. Without these, the stats output is noise. With them, the same numbers tell you exactly which resource is saturated.

How Microsoft SQL Server actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Microsoft SQL Server actually works in production: a mental model for operators

Most SQL Server incidents look confusing because operators bring a Linux-process mental model to a system that is not one. sqlservr (sqlservr.exe on Windows) is a single multi-threaded process, but inside it runs SQLOS, a user-mode operating system with its own scheduler, memory manager, and I/O completion handling. The host OS does not schedule your queries, does not cache your data pages, and does not decide which transaction gets a lock. SQL Server does all of that itself.

How MongoDB actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How MongoDB actually works in production: a mental model for operators

A write latency spike in MongoDB is rarely a query problem. It is more often a storage journal stall, cache eviction cascade, ticket exhaustion, or replication flow-control throttle. Operators who treat MongoDB as a monolithic document database often tune indexes while the real bottleneck is a WiredTiger checkpoint falling behind.

This guide is a runtime mental model for the subsystems that matter in production: the WiredTiger storage engine, replication state machine, thread-per-connection execution model, and sharding coordination layer. Understanding how these compete for memory, disk, tickets, and connections lets you read signals correctly instead of guessing.

How MySQL actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How MySQL actually works in production: a mental model for operators

MySQL is a connection-oriented request processor with a pluggable storage engine. In production, that engine is almost always InnoDB. Operating MySQL at scale means managing memory pressure, write-ahead log capacity, version-chain traversal, and fsync latency. The optimizer matters, but the stalls that wake you up start inside InnoDB.

This guide maps the subsystems that fail in production. It assumes you know how to run SHOW PROCESSLIST and need to move from symptoms to root cause: why the server stalls, why replication drifts, or why a fast query suddenly saturates disk I/O.

How NATS actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How NATS actually works in production: a mental model for operators

Most NATS incidents come from a mismatch between how operators think NATS works and how it actually works. Teams coming from Kafka or RabbitMQ carry a queue-centric mental model: messages go somewhere, wait there, and can be retrieved later. Core NATS does not work like that. The mismatch shows up at 3 a.m. as silent message loss, slow consumer cascades, or a Raft election storm nobody saw coming.

How NGINX actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How NGINX actually works in production: a mental model for operators

NGINX is not a multi-threaded server that spawns a thread per connection. It is an event-driven, non-blocking, single-threaded-per-worker process architecture. If you are debugging a production incident where connections are timing out, memory is climbing, or CPU is pinned, this architecture is the lens through which every symptom must be interpreted.

Most production issues involving NGINX are not NGINX bugs. They are resource accounting problems: file descriptors, connection slots, buffer boundaries, or event loop latency. An operator who understands the internal mechanics can read the signals correctly instead of chasing phantom upstream problems or adding hardware that hits the same limit.

How NVMe actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How NVMe actually works in production: a mental model for operators

Most storage incidents on NVMe are misdiagnosed for the same reason: operators debug them as if NVMe were a faster SATA disk. It is not. NVMe is a host-to-controller communication protocol that exposes flash storage over PCIe, and the failure modes live in places generic disk monitoring never looks: the PCIe link, the controller firmware, and the flash translation layer sitting between your filesystem and the NAND.

How Oracle Database actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Oracle Database actually works in production: a mental model for operators

Oracle is dense, but four abstractions make most production incidents legible: the shared memory region (SGA), per-process private memory (PGA), the background processes that move data to disk, and the wait-event model that tells you where time is going. When you see log file sync climbing, or free buffer waits appearing, or the database hanging while basic health checks still pass, you should know immediately which subsystem is involved and what it competes for.

How PgBouncer actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How PgBouncer actually works in production: a mental model for operators

Most PgBouncer incidents are the predictable consequence of a handful of internal mechanisms interacting under load: a single-threaded event loop, per-pool FIFO wait queues, fixed-size socket buffers, a small set of server connection states, and a pool mode that decides when server connections change hands. Hold these in your head and the alert thresholds stop being arbitrary numbers.

This article is the model, not the triage. It explains what PgBouncer is doing internally so that when cl_waiting spikes or sv_login won’t drain, you already know which part of the machine is hurting and why.

How PHP-FPM actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How PHP-FPM actually works in production: a mental model for operators

PHP-FPM looks like a single service from the outside: a socket, a process, a stream of responses. Inside, it is a process-based concurrency system with a small number of moving parts, each of which fails in a specific, predictable way. Most production incidents trace back to a misunderstanding of one of those parts.

The single most important fact about PHP-FPM: each worker process handles exactly one request at a time. There is no in-process concurrency. Your maximum concurrent request capacity equals your number of active worker processes. Everything else in the system exists to feed, queue, recycle, or protect those workers.

How Postfix actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Postfix actually works in production: a mental model for operators

Postfix is not a monolith. It is a collection of small, specialized programs that a supervisor process spawns on demand, coordinated by a single-threaded queue manager, with all durable state living as files on disk. Every operational failure in Postfix, from queue gridlock to inode exhaustion to silent DNS degradation, traces back to how these pieces interact.

The specific symptom matters less than the architecture underneath it. The same model that explains why one slow destination stalls all outbound mail also explains why a burst of inbound traffic degrades delivery performance, and why a restart on a large queue looks like recovery before it looks like failure again.

How PostgreSQL actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How PostgreSQL actually works in production: a mental model for operators

PostgreSQL is a reliable relational database, but that reliability is not magic. It is the emergent behavior of five concrete subsystems that interact in specific, observable ways: the Write-Ahead Log, Multi-Version Concurrency Control, checkpoints, autovacuum, and the process-per-connection model. If you do not understand how these interact, you will misdiagnose table bloat as a missing index, interpret checkpoint I/O spikes as disk failures, and respond to connection exhaustion by raising max_connections until the OOM killer arrives.

How ProxySQL actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How ProxySQL actually works in production: a mental model for operators

When ProxySQL fails in production, the layer where it fails determines what you see: connection exhaustion at the frontend pool, mis-routing in the query processor, pool starvation from multiplexing collapse, or false health decisions in the monitor module. Understanding these layers is prerequisite to debugging any of them.

This article covers the request path from client connection to backend response, the components that make routing and pooling decisions, and the failure modes characteristic of each. The ProxySQL runbooks in this section build on the terminology and component relationships described here.

How Redis actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Redis actually works in production: a mental model for operators

Production incidents do not come from the Redis API. They come from invisible internals: a fork that doubles memory usage, a replication backlog that wraps around, a single slow command that freezes every client. You cannot debug a cascading failure at 3 a.m. without knowing which abstractions compete for which resources.

What it is and why it matters

Redis is a single-threaded event loop around an in-memory dataset. Every incident traces back to resource competition inside one process: memory consumed by the dataset, client buffers, replication backlogs, and allocator fragmentation; CPU consumed by command execution, active expiry, and defragmentation; disk I/O consumed by AOF fsync and RDB snapshots; and network bandwidth consumed by replication and Pub/Sub fan-out.

How S.M.A.R.T. actually works: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How S.M.A.R.T. actually works: a mental model for operators

S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) is not a tool, an agent, or a daemon. It is firmware-level self-instrumentation embedded in every modern HDD, SSD, and NVMe drive. The drive itself continuously monitors its internal health and reports what it finds. The smartctl utility from smartmontools is simply a reader: it queries the drive and prints what the firmware already knows. It has no independent intelligence about drive health.

How the Kubernetes control plane works: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How the Kubernetes control plane works: a mental model for operators

The Kubernetes control plane is often drawn as a single box labeled “Master.” In production, that abstraction fails the moment kubectl hangs, a node disappears, or a Deployment stays pending. The control plane is not a monolith. It is a pipeline of specialized components that hand off state through a single API server, and understanding those handoffs is what lets you triage a cluster outage without guessing.

How Tomcat actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Tomcat actually works in production: a mental model for operators

Tomcat looks like a single process serving HTTP, but inside it is a nested container hierarchy with a three-tier buffering model in front of your servlet code. Most production incidents are not bugs in Tomcat; they are mismatches between what the operator thought was happening and what the connector and thread pool were doing. The thread pool exhausts while CPU sits at 5%. The JVM is alive but every request hangs. Sessions fill the heap while request rate looks normal. Clients see connection timeouts while Tomcat logs nothing.

How Traefik actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Traefik actually works in production: a mental model for operators

Most Traefik incidents are confusing because operators reason about it like nginx: a static proxy with a config file that gets reloaded on change. Traefik is not that. It is two systems sharing one process: a data-plane reverse proxy that terminates and forwards connections, and a control-plane reconciler that continuously watches configuration providers and rebuilds the routing table without a restart.

How uWSGI actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How uWSGI actually works in production: a mental model for operators

uWSGI is a pre-fork application server. A single master process spawns a pool of worker processes, each running a full copy of your application. The master never serves requests. It manages worker lifecycle, enforces timeouts, and coordinates graceful reloads. Every request flows through the same path: a client connects to a socket, the kernel queues that connection in a listen backlog, and a worker calls accept() to pull it out and process it synchronously.

How Varnish actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How Varnish actually works in production: a mental model for operators

Varnish Cache is a reverse HTTP proxy that serves cached content from memory. Behind that description sits a set of interacting subsystems, each with its own failure modes and saturation points. Before you can interpret a counter or diagnose an incident, you need to understand how a request moves through the process, where worker threads are consumed, how the cache store is managed, and what runs in the background.

How vSphere actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How vSphere actually works in production: a mental model for operators

vSphere is a layered virtualization stack with three interdependent planes, and most production incidents cross plane boundaries. A guest that “feels slow” may be starved at the hypervisor scheduler, throttled by a forgotten CPU limit, fighting for IOPS at the storage layer, or sitting behind a vCenter whose database has bloated to the point that DRS stopped rebalancing the cluster.

How ZFS actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How ZFS actually works in production: a mental model for operators

Every ZFS incident traces back to a small set of internal mechanisms. A write freeze at 3 a.m. is the transaction group pipeline stalling. A server that “runs out of memory” with gigabytes free is the ARC doing exactly what it was designed to do. A pool that was fine at 78% full and unusable at 88% is the metaslab allocator crossing a threshold, not a disk dying.

How ZooKeeper actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How ZooKeeper actually works in production: a mental model for operators

ZooKeeper is a distributed coordination service built on a replicated state machine. It maintains a hierarchical namespace of data nodes (znodes) entirely in memory and replicates every mutation across an ensemble of servers using the ZAB (ZooKeeper Atomic Broadcast) protocol.

This is the mental model the rest of the ZooKeeper runbooks assume. It is the set of abstractions an on-call engineer needs to reason about why writes stall, sessions expire, a “healthy” node can return stale reads, and why the leader matters more than any other node in the cluster.

HP H3C Switch

Mon, 01 Jan 0001 00:00:00 +0000

HP ICF Switch

Mon, 01 Jan 0001 00:00:00 +0000

HP ILO

Mon, 01 Jan 0001 00:00:00 +0000

HP Ilo4

Mon, 01 Jan 0001 00:00:00 +0000

HPE Bladesystem Enclosure

Mon, 01 Jan 0001 00:00:00 +0000

HPE MSA

Mon, 01 Jan 0001 00:00:00 +0000

HPE Nimble

Mon, 01 Jan 0001 00:00:00 +0000

HPE Proliant

Mon, 01 Jan 0001 00:00:00 +0000

HPE Smart Arrays

Mon, 01 Jan 0001 00:00:00 +0000

HPE Smart Arrays Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

HPE Smart Arrays Monitoring

What Is HPE Smart Arrays?

HPE Smart Arrays are hardware RAID controllers from Hewlett Packard Enterprise designed to manage data storage that is highly reliable and enhances system performance. These controllers are integral for businesses that require robust and scalable storage systems, often used in server environments to support critical applications.

Monitoring HPE Smart Arrays With Netdata

Netdata provides a comprehensive monitoring tool specifically designed for HPE Smart Arrays, which allows real-time insights into the health and performance of your storage systems. By leveraging the HPE Smart Arrays monitoring tool, you can stay informed about any potential issues or performance bottlenecks before they impact your operations.

HTTP endpoint

Mon, 01 Jan 0001 00:00:00 +0000

HTTP Endpoints

Mon, 01 Jan 0001 00:00:00 +0000

HTTP Endpoints Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

HTTP Endpoints Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

HTTP Endpoints Monitoring

What Is HTTP Endpoints?

HTTP Endpoints are specific URLs or web services that applications or users interact with. Monitoring these endpoints involves tracking their availability, response times, and reliability to ensure smooth and efficient operation of web services.

Monitoring HTTP Endpoints With Netdata

Netdata offers a comprehensive HTTP Endpoints monitoring tool, designed to provide real-time insights into your web services. With Netdata, you can monitor HTTP Endpoints to identify issues like increased response times or server connectivity problems before they impact user experience. You can sign up for a free trial or check out a live demo to see how it works.

HTTPD

Mon, 01 Jan 0001 00:00:00 +0000

Huawei

Mon, 01 Jan 0001 00:00:00 +0000

Huawei Access Controllers

Mon, 01 Jan 0001 00:00:00 +0000

Huawei BGP

Mon, 01 Jan 0001 00:00:00 +0000

Huawei Routers

Mon, 01 Jan 0001 00:00:00 +0000

Huawei Switches

Mon, 01 Jan 0001 00:00:00 +0000

Huawei Symantec Technologies Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Huawei Technology Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Hubble

Mon, 01 Jan 0001 00:00:00 +0000

Hubble Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Hubble Monitoring

What Is Hubble?

Hubble is an advanced, open-source network observability tool that provides detailed telemetry and security observability for cloud-native environments. By leveraging Hubble, DevOps engineers and IT professionals can gain deep insights into the network traffic and infrastructure within their Kubernetes clusters, powered by the Cilium network plugin.

Monitoring Hubble With Netdata

Monitoring Hubble with Netdata allows you to leverage its openmetrics (Prometheus) exporter capabilities. Netdata seamlessly ingests data from any Prometheus exporter, allowing you to monitor Hubble efficiently. With Netdata, users benefit from automated dashboards and smart alerting systems without the need for setting up a Prometheus server or Grafana. This streamlined approach facilitates a faster and more efficient workflow, enabling SREs and IT admins to focus on responding to the alerts and insights rather than setting up and maintaining complex systems.

Huber Suhner Bktel GmbH Hfc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Hw Group S R O SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

hw.intrcnt

Mon, 01 Jan 0001 00:00:00 +0000

Hyper-V

Mon, 01 Jan 0001 00:00:00 +0000

Hypercom Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

I/O errors in dmesg with clean SMART: the failure the drive can't see

Mon, 01 Jan 0001 00:00:00 +0000

I/O errors in dmesg with clean SMART: the failure the drive can’t see

I/O error, dev sda scrolling through dmesg. The application is timing out on disk reads. You run smartctl -H /dev/sda and it returns PASSED. Reallocated sectors, pending sectors, offline uncorrectable: all zero. The ATA error log is empty.

SMART reports what the drive firmware can observe about itself: media integrity, mechanical health, thermal state, NAND endurance. It cannot see failures in the transport layer between the host and the drive. When the failure lives in the SATA cable, the backplane connector, the HBA firmware, a SCSI error recovery loop, or a PCIe link, the drive firmware has no way to observe it. The host kernel sees every failed transaction. The drive does not.

IBM

Mon, 01 Jan 0001 00:00:00 +0000

IBM AIX systems Njmon

Mon, 01 Jan 0001 00:00:00 +0000

IBM AIX Systems Njmon Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

IBM AIX Systems Njmon Monitoring

What Is IBM AIX Systems Njmon?

IBM AIX systems Njmon is a state-of-the-art tool designed for performance monitoring and management of IBM AIX operating environments. It effectively provides insights into system health and performance metrics, ensuring smooth IT infrastructure operation. With Njmon, you can seamlessly monitor data collection in real time without impacting system performance.

Monitoring IBM AIX Systems Njmon With Netdata

To monitor IBM AIX Systems Njmon efficiently, Netdata provides a robust solution utilizing an openmetrics (Prometheus) exporter. This integration allows users to ingest data directly from any Prometheus endpoint without needing a dedicated Prometheus server or Grafana setup. With Netdata, you gain access to automated dashboards, alerts, and insights, drastically simplifying the process of monitoring complex systems like IBM AIX.

IBM CryptoExpress (CEX) cards

Mon, 01 Jan 0001 00:00:00 +0000

IBM CryptoExpress (CEX) Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

IBM CryptoExpress (CEX) Monitoring

What Is IBM CryptoExpress (CEX)?

IBM CryptoExpress (CEX) cards are specialized hardware devices used in IBM systems for cryptographic operations, providing secure encryption and decryption services. These cards are crucial for businesses that require high-security standards for their data transactions and management.

Monitoring IBM CryptoExpress (CEX) With Netdata

To efficiently monitor IBM CryptoExpress (CEX), Netdata utilizes the openmetrics (Prometheus) exporter available for these cards. This allows Netdata to seamlessly integrate with any existing Prometheus exporter to collect vital metrics that give insights into cryptographic performance and management. With Netdata, automated dashboards and alerts are readily available without the need for setting up a separate Prometheus server or Grafana, simplifying the monitoring process.

IBM Datapower Gateway

Mon, 01 Jan 0001 00:00:00 +0000

IBM DB2

Mon, 01 Jan 0001 00:00:00 +0000

Ibm Eserver X SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ibm Https W3 Ibm Com Standards SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

IBM i (AS/400)

Mon, 01 Jan 0001 00:00:00 +0000

IBM Lenovo Server

Mon, 01 Jan 0001 00:00:00 +0000

IBM MQ

Mon, 01 Jan 0001 00:00:00 +0000

IBM MQ

Mon, 01 Jan 0001 00:00:00 +0000

IBM MQ Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

IBM MQ Monitoring

What Is IBM MQ?

IBM MQ is a powerful messaging middleware designed to simplify, accelerate, and facilitate the transport of messages between different platforms and applications. It allows for secure and reliable message transfer, ensuring that businesses can integrate multiple complex systems seamlessly.

Monitoring IBM MQ With Netdata

To monitor IBM MQ effectively, Netdata provides a comprehensive solution using the MQ Exporter, which is an openmetrics (prometheus) exporter. With Netdata, you can effortlessly ingest data from any Prometheus exporter without the need for a Prometheus server or Grafana for visualizations. This means you can access automated dashboards, receive intelligent alerts, and enjoy rich visual insights into your IBM MQ metrics—all integrated seamlessly within the Netdata platform.

IBM Spectrum

Mon, 01 Jan 0001 00:00:00 +0000

IBM Spectrum Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

IBM Spectrum Monitoring

What Is IBM Spectrum?

IBM Spectrum is a suite of data management software from IBM, designed to manage storage resources as efficiently and cost-effectively as possible. With solutions that optimize analytics, data protection, and storage, IBM Spectrum is essential for organizations that rely on large-scale data environments.

Monitoring IBM Spectrum With Netdata

To monitor IBM Spectrum, Netdata leverages an openmetrics, specifically a Prometheus exporter, known as the IBM Spectrum Exporter. Netdata’s integration provides dynamic monitoring capabilities, turning data collected from IBM Spectrum into insightful analytics. Netdata is unique in its ability to ingest data from any Prometheus exporter, and users benefit from automated dashboards, real-time alerts, and more—all without needing a separate Prometheus server or Grafana.

IBM Spectrum Virtualize

Mon, 01 Jan 0001 00:00:00 +0000

IBM Spectrum Virtualize Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

IBM Spectrum Virtualize Monitoring

What Is IBM Spectrum Virtualize?

IBM Spectrum Virtualize is a sophisticated software that enables efficient storage virtualization, streamlining storage management and enhancing performance. It is widely utilized in various IT environments to manage storage resources, boosting performance and scalability across different platforms.

Monitoring IBM Spectrum Virtualize With Netdata

Monitoring IBM Spectrum Virtualize becomes seamless and highly efficient with Netdata. By using an openmetrics (Prometheus) exporter, specifically the spectrum_virtualize_exporter, Netdata can effortlessly ingest data from any Prometheus exporter. This functionality allows users to access automated dashboards, receive real-time alerts, and gain deep insights without requiring a Prometheus server or Grafana. The integration facilitates comprehensive visibility into your storage virtualization metrics, ensuring optimal performance and resource utilization.

IBM WebSphere JMX

Mon, 01 Jan 0001 00:00:00 +0000

IBM WebSphere MicroProfile

Mon, 01 Jan 0001 00:00:00 +0000

IBM WebSphere PMI

Mon, 01 Jan 0001 00:00:00 +0000

IBM Z Hardware Management Console

Mon, 01 Jan 0001 00:00:00 +0000

IBM Z Hardware Management Console Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

IBM Z Hardware Management Console Monitoring

What Is IBM Z Hardware Management Console?

The IBM Z Hardware Management Console (HMC) is a critical component for managing enterprise-class mainframe systems. It provides an interface for controlling hardware, managing partitions, and configuring features of IBM Z servers, enabling efficient mainframe management.

Monitoring IBM Z Hardware Management Console With Netdata

To optimize performance and ensure reliable operation, monitoring the IBM Z HMC is crucial. Netdata provides a robust solution for this task by using an openmetrics (Prometheus) exporter, specifically the IBM Z HMC Exporter. This enables seamless integration with Netdata’s monitoring platform, allowing users to ingest metrics without needing a standalone Prometheus server or Grafana setup.

Ibrix Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Icecast

Mon, 01 Jan 0001 00:00:00 +0000

Icecast Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Icecast Monitoring

What Is Icecast?

Icecast is a free and open-source streaming media server which supports various streaming formats, including MP3. It’s widely used for setting up online radio stations and creating or distributing online audio content. For more information, visit the Icecast official website.

Monitoring Icecast With Netdata

Netdata offers an unparalleled solution for monitoring Icecast, providing real-time insights and detailed metrics to optimize performance. As a comprehensive Icecast monitoring tool, Netdata allows you to track listener counts, server performance, and more, ensuring your streaming server operates at its best.

Idirect SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Idle OS Jitter

Mon, 01 Jan 0001 00:00:00 +0000

IDRAC

Mon, 01 Jan 0001 00:00:00 +0000

Ieee 802 SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ieee Lldp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

IIS

Mon, 01 Jan 0001 00:00:00 +0000

ilert

Mon, 01 Jan 0001 00:00:00 +0000

ilert

Mon, 01 Jan 0001 00:00:00 +0000

Image Processing Techniques Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Image Project Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Imv Victron B.V. SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Inca Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Incognito Software Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Independence Technologies Inc Iti SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Infinera Coriant Groove

Mon, 01 Jan 0001 00:00:00 +0000

Infinet LLC Formerly Aqua Project Group SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

InfiniBand

Mon, 01 Jan 0001 00:00:00 +0000

Inflection Systems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

InfluxDB

Mon, 01 Jan 0001 00:00:00 +0000

InfluxDB Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

InfluxDB Monitoring

What Is InfluxDB?

InfluxDB is a high-performance time series database designed for storing and analyzing time-stamped data. Used extensively for monitoring, it offers powerful data retention, querying, and analytics features that make it a popular choice for real-time applications. If you’re working with metrics, logs, or any form of timestamped data, understanding how to effectively monitor InfluxDB can significantly enhance your infrastructure’s performance.

Monitoring InfluxDB With Netdata

To monitor InfluxDB seamlessly, Netdata utilizes an openmetrics (Prometheus) exporter. The flexibility of Netdata allows it to ingest data from any Prometheus exporter. This means you can access automated dashboards, receive alerts, and more, without the need for a dedicated Prometheus server or Grafana. By leveraging the InfluxDB exporter, Netdata provides comprehensive visibility into your InfluxDB instances, enabling you to optimize performance efficiently.

Infoblox Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Infoblox Ipam

Mon, 01 Jan 0001 00:00:00 +0000

Infortrend Technology Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Infosim SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Infratec Plus GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ingrasys SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Inktomi Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Innominate Security Technologies AG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Innovaphone GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Innovative Circuit Technology Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Inoc Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Inova Dc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Inovonics Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Integrations

Mon, 01 Jan 0001 00:00:00 +0000

Intel Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Intel GPU

Mon, 01 Jan 0001 00:00:00 +0000

Intel GPU Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Intel GPU Monitoring

What Is Intel GPU?

Intel Graphics Processing Units (GPUs) are integrated into Intel CPUs, offering powerful graphics capabilities for a variety of applications. These GPUs are particularly important for managing graphics-intensive workloads and improving the overall performance of systems using Intel-based hardware.

Monitoring Intel GPU With Netdata

Netdata’s Intel GPU monitoring tool provides real-time, insightful data that allows you to track the performance and health of your Intel GPU seamlessly. With Netdata, you can monitor Intel GPU metrics like frequency, power consumption, and engine utilization without compromising system security or performance.

Intelligent Platform Management Interface (IPMI)

Mon, 01 Jan 0001 00:00:00 +0000

Inter Process Communication

Mon, 01 Jan 0001 00:00:00 +0000

Interface discards with low utilization: diagnosing ifInDiscards/ifOutDiscards

Mon, 01 Jan 0001 00:00:00 +0000

Interface discards with low utilization: diagnosing ifInDiscards/ifOutDiscards

ifOutDiscards is climbing on a critical uplink. Utilization sits at 35%. No CRC errors, no input errors, no physical-layer alarms. The link is up and passing traffic, but something is silently dropping packets, and your averaged utilization metrics are not telling you why.

The gap between what the counters show and what the silicon is doing comes down to two things: the averaging window on utilization, and the fact that discards happen at buffer-queue granularity, not at link-rate granularity.

Interface flapping: link up/down storms and their blast radius

Mon, 01 Jan 0001 00:00:00 +0000

Interface flapping: link up/down storms and their blast radius

Interface flapping is when a network interface oscillates rapidly between up and down states. Each transition generates a linkDown/linkUp trap pair, syslog entries, and an STP topology change notification. At low rates this is operational noise. At high rates it becomes a multi-layer failure: the trap receiver overflows, the syslog pipeline saturates, STP reconvergence flushes MAC tables across the VLAN, and the monitoring platform reports misleading availability because the poll interval is slower than the flap cadence.

Interface input/output errors: finding the bad link with ifInErrors/ifOutErrors

Mon, 01 Jan 0001 00:00:00 +0000

Interface input/output errors: finding the bad link with ifInErrors/ifOutErrors

A switch port reports 47,000 input errors and climbing. The interface is up and traffic is flowing. Your monitoring fired an alert on ifInErrors crossing threshold. Now you need to determine whether this is a dirty fiber, a dying SFP, a duplex mismatch, buffer exhaustion, or a counter artifact from an interface flap.

ifInErrors (.1.3.6.1.2.1.2.2.1.14) and ifOutErrors (.1.3.6.1.2.1.2.2.1.20) are aggregate counters. They tell you something is wrong, but not what. The counter is a sum of multiple error types: CRC, alignment, runts, giants, overruns, frame errors, and on some platforms, input drops. A frame that arrives with both a CRC error and a runt condition increments ifInErrors by exactly 1, not 2. You cannot reconcile the sub-counters against the aggregate by simple addition.

Interface saturation: measuring utilization against ifHighSpeed correctly

Mon, 01 Jan 0001 00:00:00 +0000

Interface saturation: measuring utilization against ifHighSpeed correctly

Interface utilization is one of the most frequently miscomputed network metrics. The formula is simple: divide throughput by capacity, multiply by 100. But the SNMP objects for throughput and capacity have different precision, units, and failure modes. An interface that is genuinely saturated can report 0% utilization. A healthy link can report 300%.

The root cause is almost always the denominator. ifSpeed (IF-MIB .1.3.6.1.2.1.2.2.1.5) is a 32-bit Gauge that caps at 4,294,967,295 bps, approximately 4.29 Gbps. For any link faster than that, ifSpeed saturates at its maximum value and utilization computed against it is wrong. The correct denominator is ifHighSpeed (.1.3.6.1.2.1.31.1.1.1.15), which reports speed in units of 1,000,000 bps (Mbps) and has no practical upper bound. A value of 10000 means 10 Gbps; 100000 means 100 Gbps.

Internet Information Services (IIS) Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

IIS (Internet Information Services)

IIS stands for Internet Information Services, which is a web server software package designed for Windows Server. IIS can host static or dynamic websites, and serve content such as HTML, JavaScript, or media files to a user’s browser. IIS is versatile and stable, and it has been widely used in production for many years.

IIS is composed of the following basic components:

Web Server
- The most common use-case IIS is used for is to host websites and ASP.NET web applications.
Security
- IIS provides robust security through Windows authentication, and is capable of filtering requests, and management of TLS certificates, so that you can enable HTTPS or SFTP on your web server.
Management
- IIS can be managed locally or remotely through the Console, CLI, PowerShell and other tools

IIS Monitoring

To holistically monitor IIS, you need to track various aspects of its performance, such as:

Interrupts

Mon, 01 Jan 0001 00:00:00 +0000

Intersystems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Invidi Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

IOPing

Mon, 01 Jan 0001 00:00:00 +0000

Ip Infusion Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

IP Virtual Server

Mon, 01 Jan 0001 00:00:00 +0000

IP2Location LITE IP-Country

Mon, 01 Jan 0001 00:00:00 +0000

IPDeny Country Zones

Mon, 01 Jan 0001 00:00:00 +0000

Ipf Technology Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

IPFIX

Mon, 01 Jan 0001 00:00:00 +0000

IPFS

Mon, 01 Jan 0001 00:00:00 +0000

IPFS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

IPFS Monitoring

What Is IPFS?

IPFS (InterPlanetary File System) is a protocol and peer-to-peer network for storing and sharing data in a distributed file system. IPFS allows users to host and receive content in a decentralized manner, making it well-suited for distributed and resilient applications.

Monitoring IPFS With Netdata

To effectively monitor IPFS, you need a robust tool that provides real-time insights into network activity, file storage, and other critical metrics. Netdata’s IPFS monitoring tool offers comprehensive visibility, enabling you to ensure the stability and performance of your IPFS nodes.

ipfw

Mon, 01 Jan 0001 00:00:00 +0000

IPIP Country Database

Mon, 01 Jan 0001 00:00:00 +0000

IPtoASN

Mon, 01 Jan 0001 00:00:00 +0000

IPv6 Socket Statistics

Mon, 01 Jan 0001 00:00:00 +0000

IRC

Mon, 01 Jan 0001 00:00:00 +0000

IRONdb

Mon, 01 Jan 0001 00:00:00 +0000

Ironport Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

ISC DHCP

Mon, 01 Jan 0001 00:00:00 +0000

ISC DHCP Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

ISC DHCP Monitoring

What Is ISC DHCP?

ISC DHCP is a widely used open-source DHCP server software that automates the assignment and management of IP addresses, enabling efficient network management. It is capable of serving both IPv4 and IPv6 clients and supports various network topologies.

Monitoring ISC DHCP With Netdata

Monitoring ISC DHCP efficiently is crucial for ensuring network reliability and performance. With Netdata, a leading ISC DHCP monitoring tool, you gain real-time insights into DHCP lease usage and pool utilization. Netdata provides comprehensive data visualization to help you track key metrics and quickly diagnose any issues in your DHCP environment.

ISC DHCPd Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is ISC DHCPd?

ISC DHCP is a DHCP server that supports both IPv4 and IPv6, and is suitable for use in high-volume and high-reliability applications.

Monitoring ISC DHCPd with Netdata

The prerequisites for monitoring ISC DHCPd with Netdata are to have ISC DHCPd and Netdata installed on your system.

Netdata auto discovers hundreds of services, and for those it doesn’t turning on manual discovery is a one line configuration. For more information on configuring Netdata for ISC DHCPd monitoring please read the collector documentation.

ISCBind (RNDC) Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is ISCBind (RNDC)?

Monitoring ISCBind (RNDC) with Netdata

The prerequisites for monitoring ISCBind with Netdata are to have ISCBind and Netdata installed on your system.

Netdata auto discovers hundreds of services, and for those it doesn’t turning on manual discovery is a one line configuration. For more information on configuring Netdata for ISCBind monitoring please read the collector documentation.

You should now see the ISCBind section on the Overview tab in Netdata Cloud already populated with charts about all the metrics you care about.

Isilon

Mon, 01 Jan 0001 00:00:00 +0000

Isilon Ststems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

It Watchdogs Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ixsystems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Ixsystems Truenas

Mon, 01 Jan 0001 00:00:00 +0000

Jacarta Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Jacobs University Bremen SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Jacques Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Janitza Electronics GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Jarvis Standing Desk

Mon, 01 Jan 0001 00:00:00 +0000

Jarvis Standing Desk Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Jarvis Standing Desk Monitoring

What Is Jarvis Standing Desk?

The Jarvis Standing Desk is an ergonomic workspace solution designed to promote better posture and flexibility in the office or home environment. Its ability to adjust height with ease helps users to maintain productive and healthy working conditions. With the increasing need for workspace ergonomics, monitoring the usage of standing desks like Jarvis becomes crucial to ensure optimal usage and benefit from the ergonomic advantages they offer.

Java Springboot Applications Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is Java Springboot2?

Java Springboot2 is an open source Java-based framework used to create web applications. It is based on the Spring framework and provides a range of features such as enhanced dependency injection, abstract data access layers, and a powerful configuration system. Springboot2 also enables developers to quickly create and deploy applications and services. It can be used to rapidly develop applications and services that are cloud-native and highly secure.

Jds Uniphase Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Jenkins

Mon, 01 Jan 0001 00:00:00 +0000

Jenkins Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Jenkins Monitoring

What Is Jenkins?

Jenkins is an open-source automation server that is used to automate the process of building, testing, and deploying software. It is a continuous integration tool that enables developers to continually test and integrate their codebase. It supports a range of plugins for building and deploying applications.

Monitoring Jenkins With Netdata

To effectively monitor Jenkins, Netdata employs a robust methodology using an openmetrics (Prometheus) exporter. This integration is seamless as Netdata can ingest data from any Prometheus exporter. It saves users from having to maintain separate Prometheus server and Grafana dashboard setups. With Netdata, automated dashboards and alerts become instantly accessible, providing real-time insights and alerting on critical metrics.

JMX

Mon, 01 Jan 0001 00:00:00 +0000

JMX Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

JMX Monitoring

What Is JMX?

Java Management Extensions (JMX) is a technology that facilitates the management and monitoring of Java applications. It provides crucial insights into application performance through MBeans (Managed Beans), which represent resources such as applications or any device that needs to be managed.

Monitoring JMX With Netdata

Leveraging Netdata for JMX monitoring adds a powerful dimension to your DevOps toolkit. With Netdata, you can monitor JMX seamlessly by using an OpenMetrics (Prometheus) exporter. Netdata can ingest data from any Prometheus exporter, offering automated dashboards, alerts, and more, without requiring a Prometheus server or Grafana. This means you can have real-time visibility into JMX metrics with minimal setup.

Johnson Controls Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

journald

Mon, 01 Jan 0001 00:00:00 +0000

Journald Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Journald Monitoring

What Is Journald?

Journald is a component of the systemd suite that manages and facilitates logging within Linux systems. It collects and stores logs in a structured manner, offering a unified logging system to better manage system events and logs. Understanding the intricacies of journald is essential for system administrators and developers who rely on consistent and efficient log management.

Monitoring Journald With Netdata

To monitor journald, Netdata employs an openmetrics (Prometheus) exporter, specifically the journald-exporter. This approach allows Netdata to ingest data from any Prometheus exporter, providing users with automated dashboards and alerts without the need for a Prometheus server or Grafana. This capability makes Netdata an excellent journald monitoring tool due to its simplicity and efficiency in onboarding and dashboard automation.

JSON

Mon, 01 Jan 0001 00:00:00 +0000

Juniper

Mon, 01 Jan 0001 00:00:00 +0000

Juniper BGP

Mon, 01 Jan 0001 00:00:00 +0000

Juniper EX

Mon, 01 Jan 0001 00:00:00 +0000

Juniper MX

Mon, 01 Jan 0001 00:00:00 +0000

Juniper Networks Funk Software SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Juniper Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Juniper Networks Unisphere SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Juniper Pulse Secure

Mon, 01 Jan 0001 00:00:00 +0000

Juniper QFX

Mon, 01 Jan 0001 00:00:00 +0000

Juniper SRX

Mon, 01 Jan 0001 00:00:00 +0000

K2Net SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

K8s Kubelet Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is K8s Kubelet?

Kubelet is an agent that runs on each node in the cluster. It makes sure that containers are running in a pod.

Monitoring K8s Kubelet with Netdata

The prerequisite for monitoring K8s Kubelet with Netdata is to Netdata installed on your system.

Netdata auto discovers hundreds of services, and for those it doesn’t turning on manual discovery is a one line configuration. For more information on configuring Netdata for K8s Kubelet monitoring please read the collector documentation.

Kafka

Mon, 01 Jan 0001 00:00:00 +0000

Kafka

Mon, 01 Jan 0001 00:00:00 +0000

Kafka __consumer_offsets growing huge: compaction failure on the offsets topic

Mon, 01 Jan 0001 00:00:00 +0000

Kafka __consumer_offsets growing huge: compaction failure on the offsets topic

One broker’s disk is climbing faster than its peers, or __consumer_offsets has grown to multiple gigabytes while producer traffic is flat. This internal topic is compacted by default; its size should stay roughly proportional to active consumer groups and partitions. Unbounded growth means the log cleaner has stalled or crashed. The failure is silent: producers and consumers keep working, but every offset commit appends a record that compaction will never remove. Growth is not evenly distributed: __consumer_offsets is partitioned by group.id hash, so a stalled cleaner on one broker affects only the partitions in that broker’s log.dirs. Eventually the log directory fills, and the broker may mark it offline.

Kafka 'Broker may not be available': clients that can't connect or stay connected

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ‘Broker may not be available’: clients that can’t connect or stay connected

Client logs show a warning like this:

WARN [Producer clientId=...] Connection to node -1 could not be established. Broker may not be available.

Python clients may raise kafka.errors.NoBrokersAvailable, while librdkafka-based clients report Connection refused against the broker address. The bootstrap server is often reachable; ping and telnet succeed, yet the client still fails. This happens because Kafka clients use the bootstrap connection only to fetch metadata. After that, they disconnect and try to open fresh TCP connections to the host:port pairs advertised in the metadata response. If those endpoints are unreachable, misconfigured, or secured differently than the client expects, the connection fails even though the bootstrap succeeded.

Kafka ActiveControllerCount not equal to 1: no controller or split brain

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ActiveControllerCount not equal to 1: no controller or split brain

When the cluster-wide sum of kafka.controller:type=KafkaController,name=ActiveControllerCount is not 1, the cluster has either no active controller or multiple active controllers. A sum of 0 means no broker is steering metadata. A sum greater than 1 means split brain. Either way, this is a control-plane failure. Existing partition leaders usually keep serving produce and fetch requests, so the data plane may look healthy at first. Any broker failure, partition reassignment, or topic operation will stall because there is no controller to process it, or multiple controllers are processing conflicting operations.

Kafka authentication failures: SASL/mTLS errors, credential rotation, and brute force

Mon, 01 Jan 0001 00:00:00 +0000

Kafka authentication failures: SASL/mTLS errors, credential rotation, and brute force

AuthenticationException lines in broker logs are spiking and failed-authentication-total is climbing on one or more listeners. Producers or consumers are failing to connect, and every failed attempt consumes a network thread and a file descriptor before the broker closes the connection. The same metric pattern can mean routine credential rotation, a client misconfiguration, or an active brute-force attempt. Telling the difference determines whether you need a config fix, a secret rotation, or a security incident response.

Kafka authorization failures: ACL denials, wrong-topic clients, and audit trails

Mon, 01 Jan 0001 00:00:00 +0000

Kafka authorization failures: ACL denials, wrong-topic clients, and audit trails

Your consumers or producers throw TOPIC_AUTHORIZATION_FAILED or CLUSTER_AUTHORIZATION_FAILED. Your security team spots repeated log lines in the broker logs. You need to know within minutes whether this is a deployment misconfiguration, a client pointing at the wrong topic, or a security incident.

In clusters where authorizer.class.name is configured, Kafka writes authorization decisions to kafka-authorizer.log. Denials log at INFO level; allowed operations are silent unless you enable DEBUG logging for the authorizer.

Kafka broker out of disk: log.dirs full, the cliff-edge shutdown, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kafka broker out of disk: log.dirs full, the cliff-edge shutdown, and recovery

When a Kafka broker exhausts disk space on a log directory, it does not throttle. It marks that directory offline or crashes entirely. Partitions on the failed directory become unavailable immediately. If all log.dirs fail, the broker exits. You may see under-replicated partitions or producer timeouts seconds before failure, but often the first sign is a hard process exit with No space left on device.

Kafka CommitFailedException: rebalanced-out consumers and poll loop timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Kafka CommitFailedException: rebalanced-out consumers and poll loop timeouts

CommitFailedException with the message that the group has already rebalanced and assigned the partitions to another member means the time between poll() calls exceeded max.poll.interval.ms. The coordinator evicted the consumer and rejected the in-flight offset commit.

When one consumer is evicted, the group rebalances. If other consumers are also slow, or if the rebalance itself takes long enough that healthy consumers miss the same deadline, the group enters a rebalance storm: it oscillates between JoinGroup and SyncGroup without stabilizing, and lag grows without bound.

Kafka connection storms: connection-count spikes, FD pressure, and network threads

Mon, 01 Jan 0001 00:00:00 +0000

Kafka connection storms: connection-count spikes, FD pressure, and network threads

Connection-count jumps on one or more brokers. An alert fires for open file descriptors, or consumers report timeouts while broker logs show nothing dramatic. In Kafka, a connection storm is insidious: every TCP connection costs one file descriptor and network-thread time, and the reactor architecture means network-thread saturation blocks everything, including metadata requests.

Incremental fetch sessions keep consumer connections open even when idle. This is normal, but it means brokers run with a high steady-state connection count. The danger is the rate of change, not the absolute number. When connections spike above twice the established baseline, network threads drop below 30% idle, and FD usage approaches the process limit, the broker is in a connection storm.

Kafka consumer group lag growing: detection, lag-as-time, and root causes

Mon, 01 Jan 0001 00:00:00 +0000

Kafka consumer group lag growing: detection, lag-as-time, and root causes

When a consumer group falls behind, lag climbs monotonically. If the committed offset crosses the retention boundary, consumers hit OffsetOutOfRangeException and must reset to earliest or latest, reprocessing or skipping data. Restarting the consumer is a common first reaction, but broker-side fetch latency, page cache eviction, and network saturation are equally common culprits. Detect lag accurately, convert it to time, and trace the root cause without guessing.

Kafka consumer group rebalancing too often: heartbeats, session timeout, and assignors

Mon, 01 Jan 0001 00:00:00 +0000

Kafka consumer group rebalancing too often: heartbeats, session timeout, and assignors

Consumer group lag is growing, application logs are full of JoinGroup and SyncGroup messages, and kafka-consumer-groups.sh shows the group flipping between Stable and PreparingRebalance. Healthy groups rebalance only during membership changes and planned deployments. More than two or three rebalances per hour for a stable group signals instability. The usual cause is a mismatch between processing latency and one of three timeouts: session.timeout.ms, heartbeat.interval.ms, or max.poll.interval.ms. The assignor strategy and static membership configuration determine how painful each rebalance is.

Kafka consumer group stuck Empty or Dead: no members consuming

Mon, 01 Jan 0001 00:00:00 +0000

Kafka consumer group stuck Empty or Dead: no members consuming

A consumer group that should be active shows Empty or Dead state with no members. For a real-time pipeline, every new message piles up unread. For a batch job, Empty between runs is expected. The difference between an intentional idle state and a silent outage is simple: is lag growing, and was the group supposed to be active?

An Empty group still exists in the coordinator. Committed offsets are retained until they expire, so consumers can resume on restart. A Dead group has been removed, usually because offsets expired while no members were active. If the group is Dead and data is still being produced, new consumers join a fresh group state and default to auto.offset.reset, which can skip or reprocess data.

Kafka Consumer Lag

Mon, 01 Jan 0001 00:00:00 +0000

Kafka Consumer Lag Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kafka Consumer Lag Monitoring

What Is Kafka Consumer Lag?

Kafka Consumer Lag represents the difference between the latest offset of a partition and the offset of the consumer group. It is a crucial metric within Kafka’s architecture as it indicates the latency for messages consumed by consumers in a Kafka topic. Properly monitoring Kafka Consumer Lag ensures efficient message queue management and prevents bottlenecks in your streaming applications.

Monitoring Kafka Consumer Lag With Netdata

To monitor Kafka Consumer Lag, Netdata uses an openmetrics (Prometheus) exporter. Netdata’s $name monitoring tool allows you to ingest data from any Prometheus exporter seamlessly, offering automatic dashboards, alerts, and comprehensive insights without requiring a Prometheus server or Grafana. By leveraging Netdata’s dashboards, DevOps and IT teams can effectively track Kafka performance, ensuring real-time data processing stays smooth and efficient.

Kafka consumer rebalance storm: stuck in PreparingRebalance and max.poll.interval.ms

Mon, 01 Jan 0001 00:00:00 +0000

Kafka consumer rebalance storm: stuck in PreparingRebalance and max.poll.interval.ms

Consumer group lag climbs while brokers report zero under-replicated partitions, normal produce and fetch latency, and normal request handler idle percent. The consumer group oscillates between Stable, PreparingRebalance, and CompletingRebalance without settling long enough to make progress. Every rebalance cycle pauses consumption; lag grows because time spent rebalancing dwarfs time spent fetching. This is a consumer rebalance storm. It is almost always a client-side timeout or processing issue. Look for CommitFailedException or max.poll.interval.ms exceeded in consumer logs.

Kafka controller event queue backing up: overwhelmed controller and stalled metadata

Mon, 01 Jan 0001 00:00:00 +0000

Kafka controller event queue backing up: overwhelmed controller and stalled metadata

You see NOT_LEADER_FOR_PARTITION errors spike in client logs. Leader elections stop completing. The active controller’s event queue grows and does not drain. Partitions that need new leaders stay offline. The single thread processing partition state changes cannot keep up, and metadata operations stall.

In ZooKeeper mode, each event writes to ZooKeeper. In KRaft mode, each event appends to the Raft metadata log. The active controller processes them sequentially. When the queue backs up, the control plane slows. The data plane may continue serving existing leaders, but any failure requiring a state change gets stuck.

Kafka disk I/O latency high: await, LocalTimeMs, and the slow-disk broker

Mon, 01 Jan 0001 00:00:00 +0000

Kafka disk I/O latency high: await, LocalTimeMs, and the slow-disk broker

iostat shows await climbing, maybe a disk alert fired. But await alone is not a pageable event. On SSDs and RAID arrays, %util hits 100% under modest load because it measures device busy time, not saturation. What matters is await, the average time for I/O requests to be served. At the Kafka layer, the mirror image is LocalTimeMs in the request latency breakdown. Both spike during normal operations – broker restart with cold page cache, log compaction, and partition reassignment all drive up disk latency without indicating hardware fault. This guide shows how to distinguish a transient spike from a slow disk that will shrink your ISR and block produce requests.

Kafka disk space planning: retention, replication, and runway estimation

Mon, 01 Jan 0001 00:00:00 +0000

Kafka disk space planning: retention, replication, and runway estimation

When a Kafka broker exhausts a log.dirs volume, the directory goes offline. If it is the only volume, the broker halts. Partitions on that volume become unavailable and the broker cannot rejoin until space is freed. Disk planning is a continuous operational calculation, not a one-time purchase. You need a working estimate of steady-state usage, a runway model against an operational full threshold, and an explicit map of the retention rules that actually delete data.

Kafka enable.auto.commit data loss: committed offsets that outrun processing

Mon, 01 Jan 0001 00:00:00 +0000

Kafka enable.auto.commit data loss: committed offsets that outrun processing

Downstream systems are missing events, but your Kafka consumer group reports lag near zero and the group state is Stable. There are no broker errors, no rebalances, and no visible backpressure. The application crashed or deployed minutes ago, then caught up instantly on recovery. Messages are lost silently. This is the signature of enable.auto.commit=true: offsets are committed to __consumer_offsets on a fixed schedule, not after processing completes.

Kafka FailedProduceRequestsPerSec rising: the single best 'producers are hurting' signal

Mon, 01 Jan 0001 00:00:00 +0000

Kafka FailedProduceRequestsPerSec rising: the single best ‘producers are hurting’ signal

FailedProduceRequestsPerSec rising is broker-side confirmation that producers are actively being rejected. Unlike client-side timeouts from network blips or producer memory pressure, it only increments when the broker receives a produce request and fails to complete it. The MBean kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec rolls up server-side produce failures into a single rate: NotEnoughReplicasException, NotLeaderOrFollowerException, CorruptRecordException, and others. A sustained nonzero rate means data is not being accepted, and the window for data loss or unavailability is open.

Kafka fetch request latency high: FetchConsumer vs FetchFollower and page cache misses

Mon, 01 Jan 0001 00:00:00 +0000

Kafka fetch request latency high: FetchConsumer vs FetchFollower and page cache misses

Your tail consumers are lagging, or your under-replicated partition count is climbing. On the broker, kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer or FetchFollower is elevated. Raw fetch latency is a poor signal: consumer long-polling means TotalTimeMs routinely includes the full fetch.max.wait.ms wait even on an idle topic, and follower fetches are paced by the leader’s ability to serve segments. The actionable metric is LocalTimeMs, the time the leader spends reading the log. When FetchConsumer LocalTimeMs spikes, the data was not in the OS page cache. When FetchFollower LocalTimeMs spikes, the leader is slow to serve replication reads and ISR shrinks will follow. The sections below show how to tell the two apart, confirm page cache misses, and fix the root cause.

Kafka ISR shrinking: IsrShrinksPerSec, flapping, and the cascade to offline

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ISR shrinking: IsrShrinksPerSec, flapping, and the cascade to offline

IsrShrinksPerSec is climbing on your leaders and UnderReplicatedPartitions is no longer zero. If it is flapping – shrinks followed by expands every few minutes – the path ends with OfflinePartitionsCount rising and acks=all producers throwing NotEnoughReplicasException. This guide covers that path: how a lagging follower becomes a cluster-wide problem, how to separate flapping from one-way degradation, and how to stop the cascade before partitions go offline.

Kafka JVM heap and Full GC pauses: ISR drops, session timeouts, and right-sizing the heap

Mon, 01 Jan 0001 00:00:00 +0000

Kafka JVM heap and Full GC pauses: ISR drops, session timeouts, and right-sizing the heap

Sporadic UnderReplicatedPartitions and ISR shrinks that do not correlate with disk I/O or network faults, combined with consumer rebalances and NotEnoughReplicasException from producers using acks=all, point to broker JVM heap pressure. Check broker logs for GC pauses in the Old Generation lasting several seconds.

Brokers use the JVM heap for metadata, request buffers, and message format conversion. They do not store messages on the heap; the OS page cache handles that. When the heap is misconfigured or under pressure, garbage collection pauses can freeze a broker long enough to trigger ZooKeeper session expirations, follower lag, and cascading availability issues. Full GC pauses exceeding five seconds are the common threshold where these symptoms begin.

Kafka KRaft metadata log lag: standby controllers and brokers falling behind

Mon, 01 Jan 0001 00:00:00 +0000

Kafka KRaft metadata log lag: standby controllers and brokers falling behind

When client errors like NOT_LEADER_FOR_PARTITION or LeaderNotAvailableException hit a stable-looking cluster, check whether standby controllers and broker metadata lag are climbing. In KRaft mode, the active controller is the Raft quorum leader. It appends metadata changes to __cluster_metadata. Standby controllers and brokers apply that log asynchronously. When they fall behind, they act on stale partition leadership, outdated ISR memberships, and old broker registrations. This guide shows how to distinguish a slow follower, a sick leader, and a network partition.

Kafka KRaft quorum has no leader: current-leader = -1 and frozen metadata

Mon, 01 Jan 0001 00:00:00 +0000

Kafka KRaft quorum has no leader: current-leader = -1 and frozen metadata

Topic creation hangs. Partition reassignments stall. Broker logs show metadata operations timing out. On controller nodes, JMX reports kafka.server:type=raft-metrics,attribute=current-leader with value -1, and the quorum state is frozen. Existing producers and consumers continue to read and write, but the control plane is stuck.

What this means

The quorum leader acts as the active controller. When current-leader is -1, the local controller has not discovered a leader, which means no node in the quorum can commit new entries to the metadata log. Topic creation, deletion, configuration updates, reassignments, and ISR changes are blocked. Leader elections for partitions that lose their broker also cannot proceed.

Kafka LEADER_NOT_AVAILABLE: causes during elections, restarts, and topic creation

Mon, 01 Jan 0001 00:00:00 +0000

Kafka LEADER_NOT_AVAILABLE: causes during elections, restarts, and topic creation

LEADER_NOT_AVAILABLE means a client asked a broker to produce or fetch from a partition that has no assigned leader. In healthy clusters this is brief during rolling restarts, controller elections, or topic creation. Persistent errors correlate with OfflinePartitionsCount > 0 and indicate the data plane is broken for those partitions. Distinguish this from NOT_LEADER_FOR_PARTITION, which means a leader exists but the client contacted the wrong broker and needs a metadata refresh.

Kafka LeaderElectionRateAndTimeMs spiking: election storms and slow elections

Mon, 01 Jan 0001 00:00:00 +0000

Kafka LeaderElectionRateAndTimeMs spiking: election storms and slow elections

LeaderElectionRateAndTimeMs climbing outside maintenance windows means the controller is struggling to keep the partition map consistent. Producers and consumers may see NOT_LEADER_FOR_PARTITION or request timeouts. Partitions can hang under-replicated or offline while the controller works through a backlog.

This metric has two dimensions. The rate is how often leadership changes. The time is how long the metadata store takes to commit each change. High rate with low time means broker flapping or repeated administrative actions. Normal rate with high time means the controller event queue is backed up, ZooKeeper writes are slow, or the KRaft quorum is lagging. Determine which mode you are in before the cluster degrades further.

Kafka leadership imbalance: LeaderCount skew and preferred replica election

Mon, 01 Jan 0001 00:00:00 +0000

Kafka leadership imbalance: LeaderCount skew and preferred replica election

One broker in your cluster is handling 40 percent of all produce requests. Its RequestHandlerAvgIdlePercent is dropping toward 0.3 while the rest of the cluster is nearly idle. Cluster-level throughput dashboards look healthy because aggregate ingress is within capacity, but tail latency on the hot broker is climbing and consumers connected to it are falling behind. The root cause is leadership skew: a disproportionate number of partition leaders have landed on one broker, and every leader carries the full request load for its partitions. This happens silently after rolling restarts, controlled shutdowns, or broker decommissions. Even a perfectly even PartitionCount across brokers can hide severe leadership imbalance. You will not see it without monitoring LeaderCount per broker.

Kafka log compaction falling behind: the dead cleaner thread and unbounded disk growth

Mon, 01 Jan 0001 00:00:00 +0000

Kafka log compaction falling behind: the dead cleaner thread and unbounded disk growth

Disk utilization climbs steadily on one or more brokers while producer traffic stays flat. __consumer_offsets balloons from a few gigabytes to hundreds. Under-replicated partitions are at zero, producers show no errors, and standard Kafka alerts are silent. The usual culprit is a dead log cleaner thread: it hit a corrupt record or an OOM during compaction, crashed, and never restarted. Every compacted topic now accumulates records without bound.

Kafka Log directory failed / OfflineLogDirectoryCount > 0: disk errors and JBOD recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kafka Log directory failed / OfflineLogDirectoryCount > 0: disk errors and JBOD recovery

What this means

When Kafka catches an IOException on a log.dirs path, it marks that log directory offline. The broker increments kafka.log:type=LogManager,name=OfflineLogDirectoryCount and logs the failure. Partitions with replicas on the failed directory lose those replicas. If the partition leader was on that directory and unclean.leader.election.enable=false, the partition becomes unavailable until the controller elects a new leader from the remaining ISR. Producers with acks=all see NotEnoughReplicasException when the surviving ISR drops below min.insync.replicas.

Kafka LogFlushRateAndTimeMs high: fsync latency and a failing disk

Mon, 01 Jan 0001 00:00:00 +0000

Kafka LogFlushRateAndTimeMs high: fsync latency and a failing disk

When kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs p99 stays above 200 ms, the broker’s fsync path is slow. On SSD-backed clusters, p99 should stay below 50 ms; sustained values above 2 s point to disk degradation. Because most deployments leave log.flush.interval.messages and log.flush.interval.ms unset and rely on replication plus OS lazy flush, this metric reflects kernel-driven flushes or explicit fsyncs. A slow flush raises produce LocalTimeMs, blocks request handler threads, and can push followers out of ISR. This guide shows how to tell whether the cause is a failing disk, a bad flush policy, or transient I/O contention.

Kafka min.insync.replicas and acks: configuring durability you actually have

Mon, 01 Jan 0001 00:00:00 +0000

Kafka min.insync.replicas and acks: configuring durability you actually have

Most operators set producers to acks=all and assume the cluster acks only when every replica has the message. It does not. With acks=all, the broker waits only for the current in-sync replica set (ISR). Because the ISR shrinks dynamically when followers lag, a partition with replication factor three can have an ISR of one – the leader itself. Without raising min.insync.replicas from its default, the leader acks with zero followers caught up. Your durability guarantee collapses to leader-only persistence, and you only find out when the leader dies and data is missing.

Kafka Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kafka Monitoring

What Is Kafka?

Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. It is used by thousands of companies for streaming analytics, data integration, and data pipelines. As a message broker, Kafka is crucial for managing big data and ensuring real-time data streaming operations. Monitoring Kafka is vital to ensure that it operates efficiently and to prevent costly downtime or data loss.

Kafka monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Kafka monitoring checklist: the signals every production cluster needs

Kafka failures follow predictable paths: an ISR shrinks, a controller queue backs up, a disk fills while the cleaner thread hangs, or a consumer rebalance storm hides behind healthy broker metrics. You need to know which signals matter and when they justify a 3 AM page.

This checklist organizes broker-side signals into four levels. Each builds on the last: Level 1 prevents data loss. Level 2 prevents surprises. Level 3 exposes leading indicators. Level 4 catches silent killers. Use it to audit dashboards, tune alert severity, and justify instrumentation.

Kafka monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Kafka monitoring maturity model: from survival to expert

Broker logs and a single health check are not enough for production Kafka. Monitoring everything at once creates noise and hides the signals that prevent outages. This model gives you a prioritized path: start with telemetry that prevents total data loss, then add layers that catch degradation before it becomes an incident.

Use this as an onboarding checklist for new clusters, an incident reference, and a roadmap when you have instrumentation budget. Signals come from Kafka JMX, OS metrics, and admin APIs. In KRaft mode (mandatory in Kafka 4.0+), substitute ZooKeeper-specific signals with the corresponding Raft quorum metrics.

Kafka network egress saturation: BytesOutPerSec, replication amplification, and fan-out

Mon, 01 Jan 0001 00:00:00 +0000

Kafka network egress saturation: BytesOutPerSec, replication amplification, and fan-out

Kafka is usually sized for producer ingress because that is what the business reports. In practice, the first resource to saturate is almost always network egress. Every byte written to a partition leader is read at least once by each follower replica and once by every active consumer group keeping up with the log. A broker can hit its NIC ceiling while producer throughput looks comfortable and disk I/O is barely warmed up.

Kafka NetworkProcessorAvgIdlePercent low: network thread saturation and TLS overhead

Mon, 01 Jan 0001 00:00:00 +0000

Kafka NetworkProcessorAvgIdlePercent low: network thread saturation and TLS overhead

NetworkProcessorAvgIdlePercent drops on one or more brokers. A ticket may fire when the value falls below 0.3, or producers and consumers may timeout while broker CPU looks fine and I/O threads are not obviously saturated. Network thread saturation blocks all socket activity, including metadata requests, so every client suffers. With the default num.network.threads set to 3, brokers running TLS and dense client pools are especially vulnerable. Sustained values below 0.1 make the broker effectively unreachable even though the process is still running.

Kafka NOT_LEADER_FOR_PARTITION: stale metadata, controller lag, and client retries

Mon, 01 Jan 0001 00:00:00 +0000

Kafka NOT_LEADER_FOR_PARTITION: stale metadata, controller lag, and client retries

Producers and consumers log NOT_LEADER_FOR_PARTITION. Broker response metrics show spikes in failed produce or fetch requests. The cluster usually self-heals within seconds as clients refresh metadata. When the error persists for minutes, or flaps across many partitions, the root cause is typically a controller that cannot keep up with leadership changes. Distinguishing a routine leader election from a controller queue backup that blocks metadata propagation is the first step.

Kafka NotEnoughReplicasException: acks=all writes rejected below min.insync.replicas

Mon, 01 Jan 0001 00:00:00 +0000

Kafka NotEnoughReplicasException: acks=all writes rejected below min.insync.replicas

Producers are throwing org.apache.kafka.common.errors.NotEnoughReplicasException or NotEnoughReplicasAfterAppendException, and acks=all writes are failing while acks=1 or acks=0 writes may still succeed. The affected partitions no longer have enough in-sync replicas to satisfy min.insync.replicas. The immediate operational question is whether the ISR shrink is a transient recovery blip or a sustained degradation that will block writes until you fix the follower.

What this means

The leader tracks followers caught up within replica.lag.time.max.ms in the In-Sync Replica set (ISR). For acks=all, the leader waits for all current ISR members before acknowledging the producer. If the ISR size drops below min.insync.replicas, the leader rejects the produce request. The broker-level default for min.insync.replicas is 1, so a lone leader can acknowledge alone. In practice, with replication.factor=3 and acks=all, set min.insync.replicas=2 so a single follower loss blocks writes instead of silently weakening durability.

Kafka OfflinePartitionsCount > 0: partitions with no leader and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

Kafka OfflinePartitionsCount > 0: partitions with no leader and how to recover

When kafka.controller:type=KafkaController,name=OfflinePartitionsCount is nonzero, at least one partition has no active leader. Those partitions are completely unavailable: producers receive errors, consumers stall, and no data is written or read until a leader is elected. This is a data-plane outage.

This metric is only meaningful on the active controller. Non-controller brokers always report zero. If the controller itself is down, the metric may be stale or unreachable at the exact moment you need it. Brief spikes can occur during controller re-election or ungraceful broker shutdown, but any sustained nonzero value past 60 seconds is an active incident that requires immediate intervention.

Kafka OffsetOutOfRangeException: when retention deletes data before the consumer reads it

Mon, 01 Jan 0001 00:00:00 +0000

Kafka OffsetOutOfRangeException: when retention deletes data before the consumer reads it

OffsetOutOfRangeException means the consumer requested an offset the broker no longer holds. The log segment containing the consumer’s committed position was deleted by retention before the consumer caught up. This is not a transient fetch error; it is data loss, and the outcome depends entirely on auto.offset.reset. Many clients default to latest, which turns this exception into silent skipping.

Kafka page cache thrashing: the backfill consumer that 100x's tail latency

Mon, 01 Jan 0001 00:00:00 +0000

Kafka page cache thrashing: the backfill consumer that 100x’s tail latency

Tail latency jumps from milliseconds to seconds while producer throughput and replication stay flat. CPU is normal. Every fetch request from tail consumers starts hitting disk. The culprit is usually a single backfill consumer reading historical data, evicting the hot working set from the OS page cache and turning a memory-speed system into a disk-bound one.

The write path stays green. Under-replicated partitions do not increase. No broker has crashed. The only visible signs are elevated disk read latency and slow consumers. If the cluster slowed suddenly with no broker fault, suspect page cache thrashing from a backfill consumer.

Kafka produce request latency high: reading the TotalTimeMs breakdown

Mon, 01 Jan 0001 00:00:00 +0000

Kafka produce request latency high: reading the TotalTimeMs breakdown

Producers are timing out or retrying. Client-side request-latency-avg is elevated and broker TotalTimeMs p99 is spiking. The total is unactionable by itself. Kafka breaks it into five sub-components, each implicating a different subsystem. You need all five via JMX or an equivalent metrics collector to know whether to fix disk, scale a thread pool, or replace a follower.

What this means

TotalTimeMs is the wall-clock time from when the broker’s network thread receives a produce request until the response is fully sent. It is the arithmetic sum of five phases:

Kafka producer timeout cascade: when retries pile load onto a slow broker

Mon, 01 Jan 0001 00:00:00 +0000

Kafka producer timeout cascade: when retries pile load onto a slow broker

Your producers are timing out. P99 produce latency is climbing. You see more requests hitting the brokers, yet actual throughput of new messages is flat or falling. This is not a traffic spike. It is a producer timeout cascade: one slow broker causes clients to retry, and those retries add load to the same overloaded broker, closing the loop until the cluster is pinned.

Kafka purgatory size growing: delayed produce and fetch operations

Mon, 01 Jan 0001 00:00:00 +0000

Kafka purgatory size growing: delayed produce and fetch operations

JMX shows kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize climbing on one or more brokers. Produce purgatory holds acks=all requests waiting for ISR completion. Fetch purgatory holds consumer and follower requests waiting for fetch.min.bytes. A growing queue means requests spend more time inside the broker than clients expected. Producers time out and retry. Consumers sit idle. The cluster is not dead, but it is backing up at a precise choke point. This guide distinguishes normal long-polling from replication crisis.

Kafka quota throttling: throttle-time, runaway clients, and protecting the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Kafka quota throttling: throttle-time, runaway clients, and protecting the cluster

Your Kafka producers are timing out. Consumer lag is growing. Client dashboards show elevated request latency, but the brokers are healthy: UnderReplicatedPartitions is zero and disk I/O looks fine. The culprit is often quota throttling: the cluster is enforcing per-client or per-user byte-rate limits, and one or more clients have hit the wall. The broker exposes this as throttle-time in JMX. That backpressure slows the client, but if the client is a runaway service or a backfill consumer, the throttling can cascade into widespread latency.

Kafka RecordTooLargeException / MESSAGE_TOO_LARGE: message size limits across the path

Mon, 01 Jan 0001 00:00:00 +0000

Kafka RecordTooLargeException / MESSAGE_TOO_LARGE: message size limits across the path

Two errors surface: the producer client throws RecordTooLargeException before the request hits the wire, or the broker returns MESSAGE_TOO_LARGE in the produce response. Both mean a record batch exceeds a limit somewhere in the path, but the fix depends on exactly which limit and where the size is measured. These settings are not a single knob: max.request.size, message.max.bytes, max.message.bytes, replica.fetch.max.bytes, fetch.max.bytes, max.partition.fetch.bytes, and socket.request.max.bytes must all align. Raise one limit without raising the others and you will wedge replication, strand consumers, or silently lose data on the next large message.

Kafka replica MaxLag growing: slow followers and replica fetcher health

Mon, 01 Jan 0001 00:00:00 +0000

Kafka replica MaxLag growing: slow followers and replica fetcher health

When kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica climbs on a broker, the worst follower is failing to replicate fast enough. This metric is the maximum offset distance between a leader and its most lagging follower. In a healthy cluster it stays near zero. If the gap persists longer than replica.lag.time.max.ms, the leader removes the follower from the ISR. Once enough replicas drop, partitions can fall below min.insync.replicas, and producers using acks=all hit NotEnoughReplicasException.

Kafka request queue filling up: RequestQueueSize, queued.max.requests, and backpressure

Mon, 01 Jan 0001 00:00:00 +0000

Kafka request queue filling up: RequestQueueSize, queued.max.requests, and backpressure

Your Kafka producers are timing out. Broker logs show no errors, but client-side metrics reveal growing latency and retries. On the broker, RequestQueueSize climbs toward queued.max.requests (default 500). Once the queue fills, network threads block on enqueue and stop reading from their sockets. Clients see TCP backpressure, time out, and retry. Retries add load, deepening the queue. This feedback loop correlates with falling RequestHandlerAvgIdlePercent.

Kafka REQUEST_TIMED_OUT: produce requests that expire before replication completes

Mon, 01 Jan 0001 00:00:00 +0000

Kafka REQUEST_TIMED_OUT: produce requests that expire before replication completes

Producers using acks=all throw TimeoutException (error code REQUEST_TIMED_OUT) when the broker accepts a produce request but cannot complete replication before request.timeout.ms expires. The leader appends the record to its local log and waits in purgatory for acknowledgments from all in-sync replicas. If the ISR ack does not arrive before the client deadline, the producer disconnects and surfaces the error. The broker may still finish the write, but the producer has already moved on, creating a hidden replication backlog and a potential retry storm.

Kafka RequestHandlerAvgIdlePercent low: I/O thread saturation and overload

Mon, 01 Jan 0001 00:00:00 +0000

Kafka RequestHandlerAvgIdlePercent low: I/O thread saturation and overload

You get paged because RequestHandlerAvgIdlePercent is below 0.3 and falling. This is an exponentially weighted moving average, not a spike metric. A low value means the broker’s I/O handler threads have been saturated long enough that the request queue is backing up and clients are timing out.

Treat above 0.5 as healthy, below 0.3 as critical, and below 0.1 as active overload. Even a stable 0.45 is dangerous: one broker failure can shift enough load to collapse the survivors.

Kafka retention not deleting old segments: retention.ms, retention.bytes, and the active segment

Mon, 01 Jan 0001 00:00:00 +0000

Kafka retention not deleting old segments: retention.ms, retention.bytes, and the active segment

You set retention.ms to 24 hours, but broker disk keeps climbing. A partition shows segment files older than the threshold, or the active segment has grown so large it consumes most of the volume. A topic with both retention.ms and retention.bytes may still appear to ignore them.

Kafka retention is not a continuous sweep. retention.ms and retention.bytes apply independently to closed segments, and the check interval adds latency. The active segment is never deleted by retention alone. On compacted topics, retention.bytes is ignored. Per-topic overrides shadow broker defaults.

Kafka Too many open files: file descriptor exhaustion from segments and connections

Mon, 01 Jan 0001 00:00:00 +0000

Kafka Too many open files: file descriptor exhaustion from segments and connections

Producers time out, consumers disconnect, and the broker log shows java.io.IOException: Too many open files. The broker may still run, but it cannot open new log segments or accept additional TCP connections. File descriptor exhaustion is a cliff-edge failure: the broker operates normally until it hits the hard limit, then the data path stops.

Kafka brokers hold a file descriptor for every log segment and every network connection. Each partition maintains an active segment and older retained segments. A broker with thousands of partitions and tens of segments each, plus hundreds of client connections, can hold tens of thousands of open file descriptors. The default Linux per-process limit of 1024 is far below production requirements.

Kafka too many partitions per broker: controller load, recovery time, and the 4000 guideline

Mon, 01 Jan 0001 00:00:00 +0000

Kafka too many partitions per broker: controller load, recovery time, and the 4000 guideline

Every replica on a Kafka broker carries fixed overhead: file descriptors, memory-mapped index files, replication fetcher load, and controller metadata. The guideline of roughly 4000 partitions per broker is not a hard architectural limit. It is an operational warning based on a serial bottleneck: the controller processes partition state changes one at a time. When a broker with 5000 partitions fails, the active controller queues thousands of events. Until the queue drains, leader elections stall, metadata propagation slows, and clients see NOT_LEADER_FOR_PARTITION. Restarting the broker forces every replica to recover its logs and catch up from leaders before rejoining the ISR. Recovery time is a step function; a routine restart can become a multi-hour incident. Partition count is easy to ignore in steady state and catastrophic to discover during an outage.

Kafka UncleanLeaderElectionsPerSec > 0: confirmed silent data loss

Mon, 01 Jan 0001 00:00:00 +0000

Kafka UncleanLeaderElectionsPerSec > 0: confirmed silent data loss

Your UncleanLeaderElectionsPerSec alert fired. The JMX metric kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec shows OneMinuteRate > 0, or Count has incremented since your last check. This is confirmed data loss: a partition leader was elected from outside the ISR, and acknowledged records that the new leader does not possess are silently truncated.

Producers that received acks for those records were given a durability guarantee the cluster just broke. Stop additional loss, understand scope, and fix the conditions that allowed it.

Kafka UnderMinIsrPartitionCount: confirming the write path is blocked

Mon, 01 Jan 0001 00:00:00 +0000

Kafka UnderMinIsrPartitionCount: confirming the write path is blocked

UnderMinIsrPartitionCount is non-zero and acks=all producers are failing. Unlike UnderReplicatedPartitions, which only opens a durability window, this metric confirms the broker is actively rejecting writes. It counts leader partitions where the in-sync replica set has shrunk below min.insync.replicas. In steady state it must be zero.

This article explains the metric, when to PAGE, and the exact checks to run when it fires.

Kafka UnderReplicatedPartitions > 0: the most important metric and how to clear it

Mon, 01 Jan 0001 00:00:00 +0000

Kafka UnderReplicatedPartitions > 0: the most important metric and how to clear it

UnderReplicatedPartitions climbing above zero means a follower has fallen behind, the ISR has shrunk, and the cluster’s durability guarantee is degraded. One more failure on the wrong broker could make partitions unavailable or cause acks=all writes to be rejected. Determine whether this is a transient blip from maintenance or the start of a cascading replication failure.

What this means

The JMX MBean kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions reports the number of partitions where this broker is the leader and the ISR count is below the configured replication factor. It is a per-broker gauge. A broker without leadership always reports zero, even when the cluster is degraded, so aggregate across all brokers to see the full picture.

Kafka ZooKeeper

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ZooKeeper Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ZooKeeper Monitoring

What Is Kafka ZooKeeper?

Kafka ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. It is a critical component in the architecture of distributed systems, especially for managing and coordinating services like Apache Kafka.

Monitoring Kafka ZooKeeper With Netdata

Monitoring Kafka ZooKeeper with Netdata offers a comprehensive view of the system’s performance and health. To monitor Kafka ZooKeeper, Netdata uses an openmetrics (Prometheus) exporter called the Kafka ZooKeeper Exporter. This allows Netdata to ingest data from any Prometheus exporter efficiently. With Netdata, you get automated dashboards, alerts, and more without the need for a Prometheus server or Grafana. This streamlined approach with the Netdata $name monitoring tool ensures ease of use and rapid insights into your Kafka ZooKeeper instances.

Kafka ZooKeeper request latency high: metadata slowdowns in ZK mode

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ZooKeeper request latency high: metadata slowdowns in ZK mode

If you run Kafka in ZooKeeper mode and kafka.server:type=ZooKeeperClientMetrics,name=ZooKeeperRequestLatencyMs is climbing, a sustained p99 above 100 ms means the controller event queue is already backing up. Above 1 s, you approach the zookeeper.session.timeout.ms boundary and brokers risk session expiry. In ZK mode, every leader election, ISR change, and topic update flows through the ensemble. When ZK stalls, the controller stalls, metadata propagation freezes, and the cluster degrades in a way that looks like a broker problem but originates in the metadata plane.

Kafka ZooKeeper session expired: GC pauses, ISR drops, and controller loss

Mon, 01 Jan 0001 00:00:00 +0000

Kafka ZooKeeper session expired: GC pauses, ISR drops, and controller loss

In ZooKeeper mode, a session expiry means ZK declared a broker dead. The Java process may still be running, but the cluster treats it as gone. If the broker was the controller, the metadata plane re-elects. If it was a leader, every partition it led starts a new leader election. If it was a follower, leaders remove it from the ISR. The default zookeeper.session.timeout.ms is 18000 ms, and the most common trigger is a Full GC pause longer than that window. This guide applies to Kafka clusters running in ZooKeeper mode; KRaft mode does not use ZK sessions.

KairosDB

Mon, 01 Jan 0001 00:00:00 +0000

Kannel

Mon, 01 Jan 0001 00:00:00 +0000

Kannel Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kannel Monitoring

What Is Kannel?

Kannel is an open-source WAP and SMS gateway used widely for mobile communication gateways. It enables applications to send and receive SMS messages and connect to mobile network carriers. Given its crucial role in mobile communication, oversight of its operational metrics is paramount.

Monitoring Kannel With Netdata

To effectively monitor Kannel, Netdata offers an intuitive solution that employs an openmetrics (Prometheus) exporter. This integration does not require a standalone Prometheus server or Grafana setup. Netdata strives as a comprehensive Kannel monitoring tool by allowing users to visualize data in real time with automated dashboards and alerts across various metrics provided by any compatible Prometheus exporter. To get started, you can use the Kannel Exporter.

Kashya SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Kaspersky Lab Zao SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Katron Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Kavenegar

Mon, 01 Jan 0001 00:00:00 +0000

Kcp Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Keepalived

Mon, 01 Jan 0001 00:00:00 +0000

Keepalived Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Keepalived Monitoring

What Is Keepalived?

Keepalived is a robust software used primarily for high-availability and load-balancing on Linux systems. It leverages VRRP (Virtual Router Redundancy Protocol) to increase network uptime by offering failover protocols. By monitoring network interfaces, Keepalived helps in ensuring that servers are capable of handling traffic efficiently.

Monitoring Keepalived With Netdata

Monitoring Keepalived with Netdata is a seamless process that ensures you have real-time insights into the performance and availability of your network infrastructure. Netdata uses an openmetrics (Prometheus) exporter to retrieve Keepalived metrics. This means that, to monitor Keepalived, Netdata can accept data from any Prometheus exporter, allowing you to benefit from automated dashboards and alerts without needing a dedicated Prometheus server or Grafana. This integration allows for comprehensive monitoring of Keepalived metrics, ensuring your high-availability setup runs smoothly.

Kentix GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Kentrox SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

kern.cp_time

Mon, 01 Jan 0001 00:00:00 +0000

kern.ipc.msq

Mon, 01 Jan 0001 00:00:00 +0000

kern.ipc.sem

Mon, 01 Jan 0001 00:00:00 +0000

kern.ipc.shm

Mon, 01 Jan 0001 00:00:00 +0000

Kernel Same-Page Merging

Mon, 01 Jan 0001 00:00:00 +0000

Kernel Same-page Merging (KSM) Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kernel Same-page Merging (KSM)?

KSM can greatly reduce the amount of memory used by VMs. When it finds two or more identical pages, it replaces them with a single page that is shared by all VMs that are using it.

Kevin Ether Boulain SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Knuerr AG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Kube-proxy Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is Kube-proxy?

Kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service.

Monitoring Kube-proxy with Netdata

The prerequisites for monitoring Kube-proxy with Netdata are to have Kube-proxy and Netdata installed on your system.

Netdata auto discovers hundreds of services, and for those it doesn’t turning on manual discovery is a one line configuration. For more information on configuring Netdata for Kube-proxy monitoring please read the collector documentation.

Kubelet

Mon, 01 Jan 0001 00:00:00 +0000

Kubelet Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kubelet Monitoring

In today’s fast-paced DevOps environments, monitoring the Kubelet effectively is crucial for maintaining healthy Kubernetes clusters. Understanding its metrics allows teams to ensure high availability and performance efficiency for their applications.

What Is Kubelet?

Kubelet is a core component of Kubernetes that runs on each node in the cluster. It ensures that containers are running as expected, watching for changes in Pod specifications and reporting back to the Kubernetes API server.

Kubeproxy

Mon, 01 Jan 0001 00:00:00 +0000

Kubeproxy Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kubeproxy Monitoring

What Is Kubeproxy?

Kubeproxy is a critical component of the Kubernetes ecosystem, acting as a network proxy that runs on each node in your cluster. It manages the IP table rules and forwards traffic to correct Pod IP addresses, playing a vital role in facilitating seamless communication within your Kubernetes deployment. Its efficient operation is crucial to maintaining healthy network traffic in Kubernetes environments.

Monitoring Kubeproxy With Netdata

Netdata provides a comprehensive Kubernetes monitoring tool that allows you to monitor Kubeproxy in real-time. By harnessing the Kubeproxy collector, you can gather pivotal metrics that give insights into your cluster’s network performance. These metrics help in diagnosing issues swiftly and ensuring optimal Kubeproxy operations. You can even check out the live demo here to see these monitoring capabilities in action.

Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes (Helm)

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes admission webhook death spiral: detection and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes admission webhook death spiral: detection and recovery

You deploy a mutating webhook to enforce policy. Later, a node drains and the webhook pod evicts. Now no pods can start, including the webhook’s own replacement. The cluster does not crash, but it stops moving. Every kubectl apply hangs. Horizontal autoscalers freeze. Rolling updates stall. This is the admission webhook death spiral: a circular dependency where the webhook must admit a pod it itself requires.

Kubernetes anonymous API access: detection, audit, and lockdown

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes anonymous API access: detection, audit, and lockdown

Anonymous requests to the Kubernetes API server are authenticated as system:anonymous and evaluated as part of the system:unauthenticated group. On many self-managed clusters this behavior is enabled by default, which means requests that arrive without a valid client certificate, bearer token, or other credential are passed to the authorization layer instead of being rejected immediately. Public discovery endpoints and health checks are common legitimate uses, but anonymous access to namespaced resources, secrets, or RBAC objects is a direct security exposure.

Kubernetes API Server

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server audit logging: policy, backends, and forensics

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server audit logging: policy, backends, and forensics

Kubernetes API server audit logging is the authoritative record of every request that reaches the control plane. It captures the identity of the caller, the resource and verb, the timestamp, the stage, and the outcome. Without it, a security investigation into unauthorized access, a compliance audit, or a postmortem into a failed certificate rotation is built on inference rather than evidence.

Kubernetes API server certificate rotation: detection and grace handling

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server certificate rotation: detection and grace handling

Kubernetes control plane certificates created by kubeadm expire after one year. The API server does not auto-rotate its serving certificate. When it expires, etcd rejects control plane connections, kubelets cannot authenticate, and the cluster becomes unreachable. The failure is sudden and total.

This guide covers kubeadm-managed clusters where you own the control plane. Distinguish between the API server serving certificate and the broader control plane bundle, detect expiration before the outage, and renew with minimal disruption.

Kubernetes API server etcd latency: detection and cascading failures

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server etcd latency: detection and cascading failures

When etcd slows down, the entire control plane slows with it. A few extra milliseconds on disk fsync turns into hung kubectl commands, backed-up controller queues, and eventually a cluster that cannot schedule pods or update endpoints. Detect the etcd latency cascade, confirm whether storage is the root cause, and break the feedback loop before the cluster becomes effectively read-only.

What this means

etcd serializes every Kubernetes mutation. Every API server write becomes a Raft proposal that must fsync to the WAL before etcd acknowledges it. When the disk under etcd is slow, every fsync waits longer. The API server holds mutating requests open until etcd responds. Requests pile up in the inflight queue. Once the queue hits the limit, the API server returns 429 Too Many Requests. Controllers that depend on writes (scheduler, replica set controller, and others) fall behind and retry. Retries generate more write load. The result is a feedback loop: slow disk -> slow etcd -> slow API server -> retry storm -> amplified etcd load.

Kubernetes API server FlowSchemas and PriorityLevels: design and tuning

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server FlowSchemas and PriorityLevels: design and tuning

Before Kubernetes 1.20, the API server protected itself with two global hard limits: --max-requests-inflight and --max-mutating-requests-inflight. Every request, whether a kubelet heartbeat or a runaway controller LIST, competed for the same pool. API Priority and Fairness (APF), enabled by default since 1.20, replaces that coarse model with a two-stage classification and fair-queuing system. It separates requests into priority levels, isolates flows within each level, and rejects or queues traffic before it can starve critical control plane operations.

Kubernetes API server memory pressure: OOM cycle and tuning

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server memory pressure: OOM cycle and tuning

Your control plane is crash-looping. The kube-apiserver process climbs toward its container memory limit, the Go garbage collector stalls trying to reclaim space, and the kernel OOM killer terminates it. The replacement pod starts with cold caches; every connected client immediately re-lists watched resources, and memory spikes again before caches warm. The root cause is usually a mismatch between how the API server uses memory and how much headroom you have given it.

Kubernetes API server rate limiting: APF priority levels and starvation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server rate limiting: APF priority levels and starvation

Your API server is running. /healthz returns 200. /readyz passes. Yet nodes drop to NotReady, the scheduler stops placing pods, and controller logs fill with context deadline exceeded. The cluster is not down, but it is frozen. This pattern often points to API Priority and Fairness (APF) starvation: low-priority traffic consumes the API server’s concurrency budget, and critical control plane requests queue or get rejected.

Kubernetes API server slow or unresponsive: causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server slow or unresponsive: causes and fixes

When kubectl hangs, controllers log context deadline exceeded, and deployments stall, the Kubernetes API server is usually the bottleneck. It is the single funnel for every read and write to cluster state. Slowness propagates to scheduling, pod lifecycle, service discovery, and external automation.

This article covers operational causes and gives a step-by-step diagnostic flow to run during an incident. Use it to distinguish etcd latency, admission webhook stalls, request saturation, and memory pressure.

Kubernetes API server watch storm: re-list cascades and connection floods

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes API server watch storm: re-list cascades and connection floods

A sudden wall of LIST requests pins API server CPU, climbs memory, spikes etcd read latency, and lags controllers. The culprit is usually a watch storm: hundreds or thousands of clients simultaneously re-listing because their watch connections failed or fell behind. Each re-list triggers an expensive etcd range scan and serializes all matching objects. Under load, this saturates CPU, fills network bandwidth, and can trigger APF throttling or memory pressure. If the API server restarts before the storm subsides, the cycle repeats.

Kubernetes bound service account tokens: rotation, audience, and expiry

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes bound service account tokens: rotation, audience, and expiry

Pods fail with 401 Unauthorized, CSI volume mounts hang with token errors, or security audits surface long-lived credentials that never rotate. Bound tokens are projected, audience-scoped, and short-lived. Legacy tokens are static Secrets that persist forever. After Kubernetes 1.24, both coexist in most upgraded clusters. This guide covers the TokenRequest API lifecycle, kubelet rotation behavior, audience binding, and version-specific changes from 1.24 through 1.33 to help you diagnose auth failures and remove stale credentials safely.

Kubernetes Cluster State

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Cluster State Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Cluster State Monitoring

What Is Kubernetes Cluster State?

Kubernetes Cluster State refers to the real-time condition or the status of your Kubernetes cluster, including nodes, pods, and containers. Monitoring these elements ensures that your clusters are operating at optimal capacity and helps in rapidly identifying any discrepancies or issues.

Monitoring Kubernetes Cluster State With Netdata

Netdata provides a comprehensive Kubernetes Cluster State monitoring tool that allows you to effortlessly oversee node and pod metrics. The integration with the go.d.plugin module makes it simple for you to garner insights into the dynamic environment of Kubernetes.

Kubernetes conntrack exhaustion: dropped connections under load

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes conntrack exhaustion: dropped connections under load

Intermittent connection timeouts under load in Kubernetes often trace to a full nf_conntrack table on the node. Existing TCP sessions stay open, but new connections fail silently. DNS resolution becomes unreliable. Application logs show timeouts to healthy dependencies. The root cause is usually not the application, network policy, or CNI, but kernel connection tracking exhaustion.

Every connection that traverses kube-proxy NAT rules creates an entry in the node’s nf_conntrack table. This finite, node-level table is shared by all workloads and invisible to most application monitoring. When it fills, the kernel drops new connection attempts without sending a TCP reset or ICMP error. The application sees a timeout.

Kubernetes container runtime shim failures: containerd, CRI-O troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes container runtime shim failures: containerd, CRI-O troubleshooting

Pods stuck in ContainerCreating, nodes flapping NotReady, and PLEG timeouts that clear only after a node reboot usually point to the container runtime shim layer, not the kubelet or network. The shim sits between the kubelet and the low-level runtime. When it hangs, crashes, or leaks, the kubelet cannot enumerate containers, start sandboxes, or reap terminated pods. Existing containers may keep running, but the node stops accepting new work.

Kubernetes Containers

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes controller-manager leader election failures

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes controller-manager leader election failures

Your Deployment has stopped scaling. Nodes cordoned hours ago are still draining. Garbage collection is paused, and orphaned volumes are not being cleaned up. The kube-controller-manager runs these reconciliation loops, and in an HA cluster only the leader performs work. When leader election fails, the controller-manager exits, and the control plane stops acting on desired state. Existing workloads keep running, but nothing new is managed.

Kubernetes CSI driver failures: detection, recovery, and version skew

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes CSI driver failures: detection, recovery, and version skew

When a workload pod hangs in ContainerCreating with FailedMount or FailedAttach events, the root cause is often a CSI driver pod that has crashed, a node plugin that is missing on the target node, or a version mismatch between the driver and its sidecars. Unlike application pods, CSI drivers sit on the critical path for every volume operation. Their failure is not isolated; it blocks scheduling, provisioning, and recovery.

Kubernetes DaemonSet pods Pending: scheduling and tolerations

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes DaemonSet pods Pending: scheduling and tolerations

A DaemonSet pod in Pending on a node where it should run is a scheduling failure, not a workload crash. Since Kubernetes 1.18, DaemonSet pods pass through the default scheduler like any other pod. The DaemonSet controller creates the pod and pins it to a target node via nodeAffinity, but the scheduler still evaluates predicates: taints, tolerations, resource requests, and node state. If the scheduler rejects the pod, it stays Pending. The controller will not create a replacement; it waits for the scheduler to succeed.

Kubernetes Deployment rollout stuck: stalled rollouts and ready replicas

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Deployment rollout stuck: stalled rollouts and ready replicas

A Deployment rollout that stalls is a silent capacity leak. The old ReplicaSet scales down, the new ReplicaSet stops halfway, and kubectl rollout status blocks indefinitely. Kubernetes does not automatically recover. The controller sets ProgressDeadlineExceeded only after progressDeadlineSeconds elapses, and takes no corrective action. You need to distinguish between a Pod lifecycle blockage, a readiness probe or gate failure, and a rare controller bug that freezes the rollout entirely.

Kubernetes DNS resolution failures inside pods

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes DNS resolution failures inside pods

DNS failures inside pods break service discovery. A single overloaded CoreDNS replica or saturated conntrack table on one node can look like a multi-service outage. Before fixing, determine whether the failure is cluster-wide, node-specific, or workload-specific.

What this means

Kubernetes injects an /etc/resolv.conf into every pod that points to the cluster DNS service, typically CoreDNS. CoreDNS resolves cluster-internal names via the kubernetes plugin and forwards external queries to an upstream resolver. A failure at any point produces the same symptom: the name cannot be resolved.

Kubernetes etcd defragmentation: when, how, and what breaks

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes etcd defragmentation: when, how, and what breaks

etcd’s database file grows over time even when you are not adding objects. Compaction removes old revisions logically, but the bbolt backend does not shrink the file on disk. Without defragmentation, the gap between physical file size and actual data footprint widens until the cluster hits a NOSPACE alarm and rejects all writes. This guide covers measuring that gap, sequencing defragmentation across an HA cluster without triggering leader elections, and the monitoring signals that predict when you need to act.

Kubernetes etcd disk fsync latency: detection and tuning

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes etcd disk fsync latency: detection and tuning

etcd serializes every Kubernetes write. When disk fsync latency rises, Raft heartbeats stall, leaders step down, and the API server returns 500s and 429s while controllers retry into a death spiral. Unlike CPU or memory pressure, etcd disk latency is invisible until it is catastrophic. This guide shows how to detect it early, isolate the root cause between storage hardware, database size, and configuration, and tune the cluster to survive production load.

Kubernetes etcd snapshot failures: backup, restore, and verification

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes etcd snapshot failures: backup, restore, and verification

An etcd snapshot failure usually surfaces during an incident, not during backup. A snapshot from an unhealthy member, a corrupted transfer to object storage, or a restore that writes no data renders disaster recovery useless. This guide gives the checks, commands, and decision logic to verify snapshot integrity, fix backup failures, and perform clean restores.

What this means

An etcd snapshot captures the entire key-value store at a point in time. Because committed Raft log entries exist on a majority of members, a snapshot from any healthy member contains the full cluster state. The snapshot file includes a SHA-256 hash computed at save time. If that hash does not match after transfer, or if the snapshot is taken while etcd is under NOSPACE alarm or leader instability, the file may be inconsistent.

Kubernetes eviction cascade: when one node failure takes down the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes eviction cascade: when one node failure takes down the cluster

You see pods entering Evicted status across multiple nodes. Nodes flap between Ready and MemoryPressure or DiskPressure. The scheduler keeps placing replacements, but the new pods are evicted again before they become ready. Workloads never stabilize, and every remediation attempt seems to make the cluster more volatile.

This is a node-pressure eviction cascade. It happens when the scheduler’s view of capacity diverges from the kubelet’s view. One node under pressure evicts pods; those pods land on other nodes that are also overcommitted; those nodes tip into pressure and evict more pods. The result is a cluster-wide feedback loop that looks like a resource shortage but is often a scheduling and configuration problem.

Kubernetes headless service resolution: SRV records and pod discovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes headless service resolution: SRV records and pod discovery

You deployed a StatefulSet with a headless Service so peers can discover each other, but nslookup returns NXDOMAIN or only a single IP when several pods are running. Your application might rely on SRV records for port discovery and the lookup returns nothing. Or a pod rescheduled onto a new node and clients kept trying the old IP for minutes because the TTL behavior surprised you.

Kubernetes imagePullSecrets: configuration, propagation, and rotation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes imagePullSecrets: configuration, propagation, and rotation

Pods stuck in ImagePullBackOff are rarely caused by a missing or incorrect image tag. More often, the kubelet lacks valid registry credentials. Kubernetes uses imagePullSecrets, namespaced Secrets of type kubernetes.io/dockerconfigjson, to inject registry auth into a pod. These can be attached directly to the pod spec or propagated through a ServiceAccount.

This guide covers propagation from registry to kubelet, verification at each link, and rotation without forcing a rolling restart of every workload.

Kubernetes init container fails: blocking main container start

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes init container fails: blocking main container start

An init container failure blocks the entire pod. Until every init container exits zero, no main container starts. In automated clusters, one failing init container can stall a deployment while pods sit in PodInitializing. This guide covers init container retry mechanics, how to read pod status to find the failing step, and how to distinguish application bugs, resource limits, and kubelet issues that leave pods permanently stuck.

Kubernetes Job and CronJob troubleshooting: history, backoff, and missed runs

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Job and CronJob troubleshooting: history, backoff, and missed runs

You deployed a CronJob to run every minute, but the last successful run was three hours ago. Or a critical data-processing Job failed with BackoffLimitExceeded after six silent retries, leaving a trail of failed Pods and no clear signal about what broke. Batch workloads fail differently from long-running services: they are time-bound, retry-sensitive, and leave debris in etcd if you do not clean them up. Read the failure signals, distinguish retry storms from control plane delays, and fix the root cause without guessing.

Kubernetes kube-proxy and CNI rule conflicts: detection and fix

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kube-proxy and CNI rule conflicts: detection and fix

Pods stuck in ContainerCreating while the node reports Ready. Services time out despite existing endpoints. Intermittent connection resets during rolling updates. These symptoms usually indicate a conflict between kube-proxy and the container network interface (CNI) plugin over netfilter rules. Both components program the same kernel tables, compete for the same locks, and can corrupt each other’s chains. The data plane degrades while the control plane stays healthy.

Kubernetes kube-proxy iptables sync stall: causes and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kube-proxy iptables sync stall: causes and recovery

Pods fail to start. Services intermittently route traffic to dead endpoints. The kube-proxy health endpoint still returns HTTP 200, so the DaemonSet looks healthy, yet rules drift further behind with every sync cycle.

An iptables sync stall is not a crash. It is a slowdown or blockage in the control loop that translates Service and EndpointSlice state into kernel NAT rules. When kube-proxy cannot acquire the global xtables lock, when iptables-restore hangs, or when the rule set grows too large to reconcile within the sync period, the node forwards packets using stale rules. New endpoints are invisible. Terminated pods still receive connections. CNI plugins that also need the xtables lock time out, and pod sandbox creation fails.

Kubernetes kube-proxy IPVS: stale rules and session affinity issues

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kube-proxy IPVS: stale rules and session affinity issues

DNS queries start timing out from one node after a CoreDNS rolling update. A UDP Service returns timeouts for some clients but not others. New Services are unreachable from a specific node while older Services continue to work.

In IPVS mode, kube-proxy programs the kernel’s IPVS table with virtual servers and real servers. The IPVS connection table lives outside kube-proxy’s direct control and outside nf_conntrack. That separation creates two IPVS-specific failure modes: stale rules that diverge from EndpointSlice state, and UDP session affinity that sticks to dead backends long after a pod terminates.

Kubernetes kubelet certificate expired: detection, rotation, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet certificate expired: detection, rotation, and recovery

A healthy node suddenly shows NotReady. Pods keep running, but the kubelet stops reporting status. kubectl logs and kubectl exec fail with TLS errors. The cluster event stream is quiet. This is usually an expired kubelet client certificate that failed to rotate.

Every kubelet maintains two independent TLS credentials: a client certificate that authenticates it to the kube-apiserver, and a serving certificate that secures the kubelet’s own HTTPS endpoints. Both typically have a one-year validity. When the client certificate expires, the kubelet cannot authenticate to the API server. The node goes NotReady. Workloads may continue running, but they are unmanaged: no evictions, no probe execution, no status updates, and no new pod scheduling. When the serving certificate expires, metrics-server, kubectl exec, and kubectl logs break even if the node is otherwise Ready.

Kubernetes kubelet goroutine leaks: detection and bisection

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet goroutine leaks: detection and bisection

A kubelet goroutine leak is a slow-burn failure. The process stays up, the node often remains Ready for hours, and then PLEG timeouts start, sync loops lag, and the kubelet is eventually OOM-killed or unresponsive. By the time the node flips to NotReady, the original leak signature is usually obscured by secondary symptoms.

What this means

The kubelet spawns goroutines for pod workers, probes, API server watches, volume operations, PLEG relists, and CRI calls. In a healthy node, goroutine count correlates with pod density and returns to a stable floor when churn stops. A leak means goroutines are created but never exit, causing three cascading effects:

Kubernetes kubelet memory leak: detection and OOM cycle

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet memory leak: detection and OOM cycle

Kubelet memory growth ends one of two ways: the process hits its cgroup limit or the node runs out of memory. The kernel OOM killer sends SIGKILL. Systemd restarts kubelet, but the new process has cold caches and immediately runs a full reconciliation pass: relisting all containers, re-syncing every pod status, and re-attaching every volume. On a busy node, that burst spikes CPU and memory, which can push the fresh kubelet back over the edge and create a Ready/NotReady flap cycle.

Kubernetes kubelet not responding: PLEG, runtime, and certificate issues

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Kubelet Not Responding: PLEG, Runtime & Certificate Issues

A Kubernetes node flipping to NotReady while containers keep running is one of the most confusing production failure modes. The kubelet is the node agent that reconciles API server intent with running containers. When it stops responding or reports unhealthy subsystems, the control plane marks the node NotReady and reschedules workloads, even though the data plane may still serve traffic.

This guide covers three failure domains: Pod Lifecycle Event Generator (PLEG) stalls, container runtime disconnections, and kubelet certificate expiration or rotation failures. Distinguish these symptoms, run safe targeted diagnostics, and apply fixes without blind node reboots.

Kubernetes kubelet pod CIDR changes: detection and rolling fix

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet pod CIDR changes: detection and rolling fix

Pod sandbox creation errors, nodes registering without an IP range, and cross-node traffic failures are symptoms of pod CIDR drift. The kubelet does not assign its own pod CIDR; the kube-controller-manager node IPAM controller writes the range into node.spec.podCIDR and node.spec.podCIDRs. When that assignment fails or diverges, the result is usually a slow fracture: some nodes host pods while others cannot, or pods on different nodes lose connectivity.

Kubernetes kubelet pprof troubleshooting: capturing heap and goroutine profiles

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet pprof troubleshooting: capturing heap and goroutine profiles

When a node starts flapping between Ready and NotReady, or kubelet memory climbs steadily while pod count stays flat, node-level metrics like kubelet_goroutines and process_resident_memory_bytes will tell you that the kubelet process is struggling. They will not tell you whether the leak is in the PLEG relist path, the volume manager, or a probe goroutine pool. For that, you need a profile.

Kubernetes kubelet volume deadlock: detection and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes kubelet volume deadlock: detection and recovery

When pods hang in ContainerCreating or Terminating while the node stays Ready=True, the kubelet volume manager is often the culprit. A blocked mount, unmount, attach, or detach operation consumes a goroutine in the volume manager’s finite pool. Once enough operations hang, the queue saturates. Subsequent pods needing volumes stall indefinitely, while pods without volumes start normally. The scheduler, seeing a healthy node, may keep placing volume-bound pods onto the saturated node, deepening the backlog. This guide covers how to confirm a volume deadlock, distinguish it from slow storage, and recover safely.

Kubernetes monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Monitoring Checklist: The Signals Every Production Cluster Needs

This article is a reference checklist for senior engineers who are wiring up, auditing, or hardening monitoring for a production Kubernetes cluster. It assumes you already understand the control plane architecture and focuses on what to collect, where to find it, and which symptoms matter. Use it during greenfield instrumentation, post-incident gap analysis, or routine health audits.

The signals are grouped by domain. Each entry leads with a short noun phrase, followed by one sentence explaining why it matters, and a concrete warning sign to alert on. Thresholds are drawn from upstream SLOs, kubelet defaults, and etcd operational limits documented in the Kubernetes source and production playbooks. If you run a managed service such as EKS, GKE, or AKS, treat control-plane metrics as provider-mediated; many etcd and API server internals are opaque in those environments.

Kubernetes monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes monitoring maturity model: from survival to expert

Kubernetes failures rarely announce themselves. A slow etcd disk cascades into API latency, which backs up controller workqueues, which delays pod scheduling, which triggers autoscaling, which amplifies the load that caused the original latency spike. Without the right signals, you will debug the autoscaling event while the real problem is a WAL fsync that crossed 100ms ten minutes earlier.

A monitoring maturity model is not a tooling checklist. It is a coverage framework that tells you which signals you are missing and what those gaps cost you during an incident. The four levels below map the progression from “Is the cluster on fire?” to “Why did that single pod take three milliseconds longer to start on node twelve?” Each level assumes the previous and adds signals that change your mean time to detection and your mean time to root cause.

Kubernetes NetworkPolicy debugging: when traffic is denied silently

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes NetworkPolicy debugging: when traffic is denied silently

A pod that could reach its dependency yesterday now times out today. There is no TCP RST, no ICMP unreachable, and often no application log. The packet is dropped in the CNI data plane. If a policy change, namespace reorganization, or cluster upgrade preceded the outage, you are likely dealing with silent NetworkPolicy denial. This guide shows how to confirm it, find the rule or semantic gap responsible, and restore connectivity without opening the cluster.

Kubernetes node CPU saturation: load, throttling, and runqueue depth

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes node CPU saturation: load, throttling, and runqueue depth

Application latency climbs and pods slow down. kubectl top nodes reports 70 percent CPU, so you assume headroom exists. It does not. CPU percent is a time-average that masks micro-bursts, runqueue backlog, and CFS throttling. A container can throttle to a crawl while node utilization looks comfortable, and a node can show 50 percent utilization with every runnable thread queued behind a noisy neighbor. Distinguish node-level CPU contention from limit-induced throttling using runqueue depth, CFS bandwidth metrics, and Pressure Stall Information (PSI).

Kubernetes node DiskPressure: detection, eviction, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Node DiskPressure: Detection, Eviction & Recovery

A node reporting DiskPressure is actively shedding workloads. The kubelet has detected that nodefs or imagefs has crossed an eviction threshold. It is garbage collecting images, terminating pods, and applying the node.kubernetes.io/disk-pressure taint to block new scheduling. Existing pods may continue running, but any pod requiring disk for logs, emptyDir volumes, or image pulls is at risk.

Disk pressure builds predictably, unlike memory pressure. This guide covers how the kubelet evaluates disk pressure, how to distinguish nodefs from imagefs exhaustion, how to find the specific consumer, and how to recover without causing a cascading eviction loop.

Kubernetes node MemoryPressure: detection, eviction order, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes node MemoryPressure: detection, eviction order, and prevention

Before adding RAM, determine whether kubelet is evicting because workloads are genuinely starving or because memory requests are misaligned with reality.

What this means

Kubelet evaluates memory.available against an eviction threshold. On Linux the default hard threshold is memory.available < 100Mi. Kubelet derives this from cgroup stats, not free -m. It measures working-set memory (RSS plus active file-backed pages) and subtracts that from total capacity. When the threshold is crossed, kubelet sets the node condition MemoryPressure=True and adds the taint node.kubernetes.io/memory-pressure:NoSchedule. New pods are blocked from scheduling until the condition clears.

Kubernetes node NotReady: kubelet, runtime, and network diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Node NotReady: Kubelet, Runtime & Network Diagnosis

When a Kubernetes node becomes NotReady, existing containers usually keep running, but the cluster stops scheduling new pods, removes endpoints from Services, and eventually evicts workloads after the pod eviction timeout. Root causes fall into three domains: kubelet health, container runtime responsiveness, and CNI or control plane connectivity.

What This Means

Kubernetes marks a node NotReady when the kubelet Ready condition is False, or when the node controller has not received a heartbeat within --node-monitor-grace-period (default 40 seconds). The node receives the node.kubernetes.io/not-ready:NoSchedule taint. If the condition persists longer than the pod eviction timeout (default 5 minutes), the controller manager marks pods on the node for rescheduling.

Kubernetes node PIDPressure: detection and remediation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes node PIDPressure: detection and remediation

PID exhaustion is a cliff-edge failure: once the kernel cannot fork, containers fail to start, health checks fail, and ssh to the node may hang. Kubernetes surfaces this through the PIDPressure node condition, but many clusters ship without PID-based eviction thresholds. Without them, the first symptom is usually EAGAIN or ENOMEM from fork failures, not a kubelet eviction.

This guide shows how to detect PIDPressure before it triggers an outage, distinguish between application leaks, runtime shim accumulation, and kernel limits, and remediate the root cause. You will correlate node-level PID utilization with specific pods, validate kubelet cgroup enforcement, and configure thresholds that provide lead time.

Kubernetes PLEG is not healthy: runtime stalls and node degradation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes PLEG is not healthy: runtime stalls and node degradation

A node suddenly flips to NotReady with the message “PLEG is not healthy.” Containers on the node keep running, but the control plane evicts workloads and reschedules them elsewhere. New pods cannot start, and existing pods run without health checks or status updates. This is one of the most common kubelet failure modes in production.

The Pod Lifecycle Event Generator (PLEG) is the kubelet subsystem that polls the container runtime every second and emits events when containers start, stop, or change state. When the runtime becomes slow or unresponsive, the PLEG relist loop stalls. If the elapsed time since the last successful relist exceeds three minutes, kubelet declares PLEG unhealthy, marks the node NotReady, and skips pod synchronization.

Kubernetes pod CrashLoopBackOff: causes, diagnosis, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Pod CrashLoopBackOff: Causes, Diagnosis & Fixes

CrashLoopBackOff means a container in a Pod has terminated after starting, and the kubelet is delaying the next restart with exponential backoff. The status describes behavior, not root cause. Underlying failures include application panics, OOM kills, misconfigured liveness probes, missing secrets, or node-level resource pressure.

Use pod status, previous container logs, node conditions, and kubelet events to narrow the cause. Monitor restart rate, node pressure, and probe failures to catch loops before they degrade capacity.

Kubernetes pod creation fails: admission, quota, and CRI errors

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod creation fails: admission, quota, and CRI errors

Pre-scheduling failures happen when the API server or container runtime rejects a Pod before the scheduler assigns it. You apply a Deployment, but kubectl get pods returns nothing. Or a Pod hangs in ImagePullBackOff before ContainerCreating. These cases surface as missing Pods, FailedCreate events on ReplicaSets or Jobs, or explicit API rejections. This guide covers admission control, quota and policy limits, and CRI-level image pull and sandbox failures.

Kubernetes pod Evicted: detection, root cause, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod Evicted: detection, root cause, and prevention

Pods with status Evicted are not application crashes. They are the kubelet’s emergency response to node-level resource pressure. When memory, disk, inodes, or PIDs approach exhaustion, the kubelet terminates pods to reclaim resources and protect node availability. The pod phase changes to Failed with reason Evicted, and the node reports conditions such as MemoryPressure or DiskPressure.

This guide covers node-pressure eviction triggered by the kubelet, not voluntary disruption from kubectl drain or PodDisruptionBudget enforcement.

Kubernetes pod exits immediately: how to diagnose it

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod exits immediately: how to diagnose it

When a pod shows Completed or Error with zero restarts, the container exited on its first run. The diagnostic evidence lives in termination metadata, not in a growing restart count. This is distinct from CrashLoopBackOff, where the kubelet has already applied exponential backoff after multiple restarts.

This guide covers how to distinguish a clean exit, an OOM kill, an application crash, and a configuration error using only the kubelet’s reported state and the previous container logs, plus which node-level and control-plane signals to check when the container produced no logs.

Kubernetes pod ImagePullBackOff: registry, auth, and network diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod ImagePullBackOff: registry, auth, and network diagnosis

ImagePullBackOff means the kubelet cannot pull a required image. After each ErrImagePull failure, the kubelet retries with exponential backoff capped at five minutes. When serializeImagePulls is true, a single slow pull blocks every subsequent pull on that node. Read the exact error from the CRI in pod events, test the registry directly from the node, and fix the root cause without blindly recreating pods.

Kubernetes pod liveness probe killing healthy containers

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod liveness probe killing healthy containers

A container that is processing requests, not OOMKilled, and not crashed can still be restarted repeatedly by the kubelet because a liveness probe failed. The application is alive, but the probe says it is not. This usually shows up as a pod stuck in CrashLoopBackOff with Liveness probe failed events, even though application logs show no fatal error. The restarts waste resources, break active connections, and can trigger cascading load on the cluster as other pods absorb the shifted traffic.

Kubernetes pod network isolation: when one node loses pod connectivity

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod network isolation: when one node loses pod connectivity

One node in your cluster drops off the pod network. Pods still show Running, but they cannot reach Services, other pods, or external endpoints. The rest of the cluster is unaffected. This is single-node pod network isolation, and it is almost always a node-local CNI or data path failure.

Symptoms are subtle compared to a full cluster outage. Applications log connection timeouts. Health checks fail. New pods stick in ContainerCreating. Because the node often stays Ready, operators may blame the application rather than the network layer.

Kubernetes pod OOMKilled: cgroup limits, evictions, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod OOMKilled: cgroup limits, evictions, and fixes

A pod status of OOMKilled means the container restarted after the kernel sent SIGKILL because it could not satisfy a memory allocation. There is no graceful shutdown.

Distinguish whether the kill happened at the container cgroup level (a limit you set) or at the node level (a system-wide shortage). Then separate kernel OOM kills from kubelet evictions, identify the correct fix, and prevent recurrence without guessing at memory limits.

Kubernetes pod readiness probe failures: traffic exclusion and debug

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod readiness probe failures: traffic exclusion and debug

You check a Deployment and see pods in Running phase but not Ready. Traffic to the Service drops. The containers have not restarted. This is a readiness probe failure. Unlike liveness, readiness does not restart the container. It removes the pod from Service endpoints. The container may be starting slowly, temporarily overloaded, or waiting on a dependency that should not be in the probe path. This guide explains how Kubernetes excludes traffic, how to distinguish readiness from liveness failures, and how to debug the root cause.

Kubernetes Pod Security Standards violations: detection and remediation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Pod Security Standards violations: detection and remediation

You apply a Deployment and the ReplicaSet creates zero pods. You roll out a DaemonSet update and new pods vanish without events. You add a namespace label to improve security posture, and CI pipelines fail with opaque admission errors. Pod Security Admission (PSA) blocks non-compliant workloads silently when enforce mode is active. To fix these violations, correlate namespace labels with enforcement outcomes, parse audit log annotations into exact spec changes, distinguish admission-time rejections from runtime failures, and prevent enforcement from silently blocking future rollouts.

Kubernetes pod stuck ContainerCreating: volume, network, and image issues

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod stuck ContainerCreating: volume, network, and image issues

A pod stuck in ContainerCreating never produces logs or readiness events. The kubelet accepted the spec but blocked during initialization after scheduling and before the container runtime starts the user process. The dominant failure domains are volume mount deadlocks, CNI sandbox creation failures, and image pull problems. They all surface the same status but need different fixes. This guide shows how to identify the stuck subsystem and resolve it.

Kubernetes pod stuck on volume mount: CSI, permissions, and timeouts

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod stuck on volume mount: CSI, permissions, and timeouts

You scale a StatefulSet and the new pods sit in ContainerCreating for ten minutes. The node is Ready. The CSI driver pods are running. kubectl describe shows no FailedMount events, yet the containers never start. The absence of volume events often misleads operators into checking image registries or resource quotas instead of the storage path. The kubelet volume manager is blocked somewhere between attach and mount, and Kubernetes will not retry fast enough to hide the problem.

Kubernetes pod stuck Pending: scheduling failures explained

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes pod stuck Pending: scheduling failures explained

A Deployment scales up and the new replicas stay Pending. No containers start, no logs appear, and kubectl logs returns nothing because no node is assigned yet. When a pod is stuck in Pending, the scheduler has either not yet evaluated it or has rejected every candidate node. Containers cannot start until the pod is assigned, so this blocks rollouts, autoscaling, and recovery.

Kubernetes pod stuck Terminating: finalizers, grace periods, and force delete

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Pod Stuck Terminating: Finalizers, Grace Periods & Force Delete

A pod stuck in Terminating stays visible in the API server after kubectl delete, sometimes for minutes or hours. Usually a finalizer blocks removal, a CSI volume is still attached, or the container is ignoring SIGTERM. Force deleting without diagnosis orphans containers and can violate StatefulSet guarantees. Check the signals first, then decide whether to wait, patch a finalizer, or force delete.

Kubernetes priority class evictions: how preemption actually works

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes priority class evictions: how preemption actually works

Scheduler preemption and kubelet node-pressure eviction both use Pod Priority, but they follow different rules, interact differently with PodDisruptionBudgets, and produce different failure signatures. Misunderstanding the two mechanisms leads to critical workloads being terminated unexpectedly, high-priority pods staying pending while lower-priority pods run, or system pods being preempted after a PriorityClass change.

The scheduler preempts lower-priority pods to make room for a higher-priority pending pod. The kubelet evicts pods when a node is under resource pressure. A PodDisruptionBudget protects against voluntary disruptions, but the scheduler treats PDB violations during preemption as best-effort, not guaranteed.

Kubernetes PV reclaim policies: Retain, Delete, Recycle in practice

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes PV reclaim policies: Retain, Delete, Recycle in practice

Deleting a PVC does not always delete the underlying storage. If the PV enters Released state and the cloud bill keeps growing, or the application fails after redeployment because the PV is still bound to a deleted claim, the reclaim policy is misaligned with operator intent. This guide explains what happens when a bound PVC is deleted under each reclaim policy, how to recover a Retained volume, why Delete can orphan cloud resources, and why Recycle should not be used in modern clusters. The focus is on practical checks, CSI-specific gotchas, and preventing data loss.

Kubernetes PV volumeBindingMode WaitForFirstConsumer: when and why

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes PV volumeBindingMode WaitForFirstConsumer: when and why

A PersistentVolumeClaim that stays Pending after creation often triggers an incident response. When the StorageClass uses volumeBindingMode: WaitForFirstConsumer, that Pending state is usually intentional. The cluster defers provisioning and binding until a Pod that references the PVC is created and the scheduler tentatively selects a node. This article explains what Immediate and WaitForFirstConsumer mean, why Kubernetes defers binding, where the mechanism appears in production, and the operational tradeoffs you accept when you choose one mode over the other.

Kubernetes PVC stuck Pending: storage class, provisioner, and quota

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes PVC stuck Pending: storage class, provisioner, and quota

A PersistentVolumeClaim stuck in Pending is a storage-layer failure. Unlike a pod that is Pending due to CPU or memory, a PVC in Pending means the cluster cannot provision the volume. The cause usually sits in one of three layers: the StorageClass and its provisioner, namespace-level ResourceQuota limits, or topology and cloud constraints that prevent volume creation and attachment.

When a PVC stays unbound, dependent pods cannot start. StatefulSets stall, rolling updates hang, and storage-dependent applications remain offline.

Kubernetes RBAC permission denied: detection and minimum-permission fix

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes RBAC permission denied: detection and minimum-permission fix

A 403 Forbidden from the Kubernetes API server means the caller was authenticated but RBAC refused the action. In production, this appears as a Deployment stuck creating pods, a CI pipeline failing to patch a ConfigMap, a controller logging repeated forbidden errors, or an operator unable to finalize a custom resource. One missing verb on one resource in one namespace blocks the entire workflow.

Kubernetes ResourceQuota exceeded: detection and remediation

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes ResourceQuota exceeded: detection and remediation

A Deployment looks healthy in kubectl get deployment but the new ReplicaSet has zero pods. A Job is accepted but never creates a pod. A CI/CD pipeline fails with opaque 403 errors from the API server. These symptoms point to ResourceQuota exhaustion. Confirm the quota is the blocker, identify the exhausted resource, and fix it.

What this means

ResourceQuota is a namespace-scoped admission controller that enforces hard aggregate limits on resource consumption. When a tracked resource hits its hard limit, the API server rejects subsequent create requests with HTTP 403 Forbidden and a message containing exceeded quota: <quota-name>. Existing pods keep running; quota is enforced at admission time, not by terminating workloads.

Kubernetes scheduler not scheduling pods: queue depth and failure reasons

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes scheduler not scheduling pods: queue depth and failure reasons

Pods stay Pending for many reasons, but the scheduler process being down is rarely one. More often, pods accumulate in internal queues because the cluster is out of capacity, a control plane dependency stalls the binding cycle, or a filter plugin rejects every candidate node. Distinguishing “unschedulable” (no node fits) from “not scheduling” (the scheduler cannot keep up or the binding cycle is failing) prevents wasted node scaling when the real problem is an etcd latency spike or a volume affinity conflict.

Kubernetes secrets mount failures: detection and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes secrets mount failures: detection and recovery

Pods that reference a Secret volume can hang in ContainerCreating for minutes, crash on startup with missing files, or silently run with stale credentials. The kubelet mounts Secret data as a tmpfs-backed volume and validates the Secret object and its requested keys before any container starts. If the Secret does not exist, if a specific key is absent, or if the kubelet cannot reach the API server, the mount fails. Non-optional mounts block every container from starting. Optional mounts allow startup but leave the mount point empty, which can cause failures later.

Kubernetes Service Not Reachable: Kube-Proxy, Endpoints & DNS

Mon, 01 Jan 0001 00:00:00 +0000

A Service fails when the chain between the client and backend breaks. That chain depends on EndpointSlices to list healthy pods, kube-proxy to program kernel rules, and cluster DNS to resolve names to ClusterIPs. This guide covers the gap between healthy backend pods and an unreachable Service, focusing on kube-proxy data-plane programming, endpoint state, and DNS dependencies. It does not cover application-level bugs inside the pod.

What This Means

Reachability is a control-loop problem. kube-proxy watches Services and EndpointSlices, then programs iptables, IPVS, or nftables rules to DNAT traffic to healthy endpoints. These rules persist in the kernel if kube-proxy crashes, but updates stop until it resumes. DNS resolution targets the CoreDNS ClusterIP, so a kube-proxy failure often appears as a DNS outage before a direct Service timeout.

Kubernetes stale conntrack during rolling updates: intermittent connection resets

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes stale conntrack during rolling updates: intermittent connection resets

You deploy a new version of a service. The rollout reports healthy. Error budgets look fine. Then you notice sporadic connection resets in application logs, a handful of timeout errors, or brief latency spikes that correlate exactly with pod terminations. The failures are intermittent, last only seconds, and never trigger a full outage. This is the stale conntrack race during rolling updates.

Kubernetes StatefulSet pod not ready: ordering, PVCs, and recovery

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes StatefulSet pod not ready: ordering, PVCs, and recovery

A StatefulSet pod stuck at 0/1 Ready, Pending, or Unknown blocks the entire chain when the default OrderedReady policy is in effect. The controller creates pods sequentially from ordinal 0 to N-1, so one failure halts all higher ordinals. A volumeClaimTemplate adds dependencies on storage binding, node affinity, and CSI attachment that can outlive the pod. Force-deleting a pod risks violating at-most-one semantics and causing split-brain in quorum-sensitive workloads.

Kubernetes volume snapshot failures: VolumeSnapshot CRDs and CSI snapshotter

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes volume snapshot failures: VolumeSnapshot CRDs and CSI snapshotter

A VolumeSnapshot that never becomes readyToUse, a restore yielding an empty PVC, or a CSI snapshotter sidecar logging GRPC errors with no cluster progress all indicate the same problem: the snapshot pipeline broke between the API server and the storage backend.

This guide covers failure modes between applying a VolumeSnapshot manifest and the storage backend completing the operation. It covers verifying CRDs, the snapshot-controller, the CSI snapshotter sidecar, driver names, capabilities, and secrets, and reading the error signatures each misalignment produces.

Kyocera Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Kyocera Printer

Mon, 01 Jan 0001 00:00:00 +0000

Lanart Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lancast Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lancity Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lancom Systems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lanex Sp Z O O SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lannair Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lannet Company SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lantronix SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Last Mile Gear SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Latitude Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Laurel Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lefthand Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lenovo Enterprise Business Group SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lenovoemc Ltd Formerly Iomega SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lexmark International SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Librenms SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Libreswan

Mon, 01 Jan 0001 00:00:00 +0000

Libvirt VMs and Containers

Mon, 01 Jan 0001 00:00:00 +0000

License expiry silently disabling features: monitor days-to-expiry

Mon, 01 Jan 0001 00:00:00 +0000

License expiry silently disabling features: monitor days-to-expiry

Your firewall dashboard shows green. Interfaces are up, CPU and memory are normal, traffic is flowing. But at 09:00, someone reports VPN connections failing, IPS no longer blocking threats, or URL filtering not enforcing policy. A feature license expired at midnight, and the device silently stopped performing the licensed function without raising a visible alarm.

The device stays up, counters keep incrementing, throughput looks normal. The license-expiry message in syslog is low severity and gets buried under routine noise. By the time someone notices, the feature has been disabled for hours.

Lighttpd

Mon, 01 Jan 0001 00:00:00 +0000

Lighttpd Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Lighttpd Monitoring

Welcome to the comprehensive guide on monitoring Lighttpd, a flexible and lightweight web server. This guide will take you through everything you need to know about monitoring Lighttpd using the Netdata platform. From understanding essential metrics to employ advanced performance monitoring techniques, Netdata provides a powerful Lighttpd monitoring tool for all your DevOps, SRE, and IT admin needs.

What Is Lighttpd?

Lighttpd is an open-source web server optimized for speed-critical environments while maintaining a low memory footprint. Its feature set and performance make it a popular choice for many developers and engineers looking to implement efficient web solutions.

Ligowave SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Linksys

Mon, 01 Jan 0001 00:00:00 +0000

Linksys SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Linode

Mon, 01 Jan 0001 00:00:00 +0000

Linode Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Linode Monitoring

What Is Linode?

Linode is a cloud hosting service that provides virtual servers to deploy applications, manage web hosting, and scale infrastructure as demands grow. Known for its simplicity and cost-effectiveness, Linode empowers developers, DevOps teams, and IT Administrators to manage cloud-based services with flexibility and efficiency.

Monitoring Linode With Netdata

Monitoring Linode is crucial to ensure optimal performance, cost management, and resource allocation. Netdata serves as an effective Linode monitoring tool, leveraging an openmetrics (Prometheus) exporter to collect real-time data without the need for standalone Prometheus or Grafana servers. This integration provides you with automated dashboards, alerts, and comprehensive insights into your Linode instances.

Linux

Mon, 01 Jan 0001 00:00:00 +0000

Linux Audit Subsystem

Mon, 01 Jan 0001 00:00:00 +0000

Linux Hardware Sensors (libsensors)

Mon, 01 Jan 0001 00:00:00 +0000

Linux kernel SLAB allocator statistics

Mon, 01 Jan 0001 00:00:00 +0000

Linux Sensors

Mon, 01 Jan 0001 00:00:00 +0000

Linux Sensors Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Linux Sensors Monitoring

What Is Linux Sensors?

Linux Sensors is a comprehensive suite for monitoring hardware sensors in Linux systems, gathering real-time data about temperature, voltage, fan speed, energy consumption, and more. Leveraging the sysfs interface, it provides a robust way for systems and applications to access sensor data, crucial for maintaining optimal server performance and health.

Monitoring Linux Sensors With Netdata

Netdata is a powerful, real-time Linux sensors monitoring tool that allows you to effortlessly monitor Linux Sensors. By utilizing the sysfs interface, Netdata automatically detects all available sensors on your system, providing you with instant visibility into sensor metrics such as temperature, voltage, current, and power usage.

Linux ZSwap

Mon, 01 Jan 0001 00:00:00 +0000

Litespeed

Mon, 01 Jan 0001 00:00:00 +0000

Litespeed Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Litespeed Monitoring

What Is Litespeed?

Litespeed is a powerful web server technology that boosts website speed and security. It is renowned for its high-performance capabilities, delivering superior HTTP/HTTPS content and optimizing traffic handling. Discover more about Litespeed.

Monitoring Litespeed With Netdata

Utilizing Netdata to monitor Litespeed provides comprehensive insights into your web server’s performance. As a real-time, distributed monitoring tool, Netdata captures extensive metrics, ensuring your server operates efficiently and responds proactively to issues. You can begin monitoring Litespeed effortlessly with Netdata’s Litespeed Monitoring Tool.

Live Network Connections

Mon, 01 Jan 0001 00:00:00 +0000

Livingston Enterprises Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

LLDP Topology

Mon, 01 Jan 0001 00:00:00 +0000

Local listening processes

Mon, 01 Jan 0001 00:00:00 +0000

Locating endpoints behind NAT and wireless: the positioning problem

Mon, 01 Jan 0001 00:00:00 +0000

Locating endpoints behind NAT and wireless: the positioning problem

Endpoint positioning maps a MAC address or IP to a specific switch port, access point, or VLAN. It underpins security investigations, access control enforcement, and day-to-day troubleshooting. When the endpoint sits behind a NAT boundary, the Layer 2 and Layer 3 signals that topology engines rely on (FDB entries, ARP tables, flow records) all report the NAT device’s identity, not the endpoint behind it. The endpoint becomes operationally invisible upstream.

Logstash

Mon, 01 Jan 0001 00:00:00 +0000

Logstash _dateparsefailure: timestamp formats that stop parsing

Mon, 01 Jan 0001 00:00:00 +0000

Logstash _dateparsefailure: timestamp formats that stop parsing

The _dateparsefailure tag appears when Logstash’s date filter cannot parse a timestamp field against any of its configured match patterns. The event still flows through the pipeline and reaches its output. @timestamp stays at ingestion time instead of being set to the event’s actual time.

The result: time-based dashboards show gaps or misplaced data points, index lifecycle management operates on ingestion time so events land in the wrong time bucket or expire at the wrong time, and the issue can persist for days because throughput metrics stay green. The date filter tries each format string in the match array sequentially. The first successful parse wins. If none succeed, the filter appends its tag_on_failure tag (default _dateparsefailure) and leaves @timestamp unchanged.

Logstash _grokparsefailure: why grok stops matching and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

Logstash _grokparsefailure: why grok stops matching and how to fix it

Events tagged with _grokparsefailure pass through your pipeline unstructured. Grok could not match any of its configured patterns against the event’s input field, so it appended the failure tag and moved on. The event is not dropped. It is not counted as filtered. Unless you conditionally route it, it reaches your output destination carrying raw, unparsed data alongside the failure tag.

Logstash _jsonparsefailure: malformed JSON and codec mismatches

Mon, 01 Jan 0001 00:00:00 +0000

Logstash _jsonparsefailure: malformed JSON and codec mismatches

When events arrive at your Elasticsearch indices carrying the _jsonparsefailure tag, they contain raw, unparsed data instead of the structured fields your downstream consumers expect. Throughput metrics look healthy, events-out counts keep climbing, and no alerts fire. But the data is wrong.

The _jsonparsefailure tag is added by either the json codec or the json filter when it receives input it cannot parse as valid JSON. The event is not dropped. It flows through to the output carrying whatever raw data was received, plus the failure tag. This is a silent correctness failure, not an availability failure.

Logstash address already in use: input port conflicts on Beats, TCP, and HTTP

Mon, 01 Jan 0001 00:00:00 +0000

Logstash address already in use: input port conflicts on Beats, TCP, and HTTP

Logstash logs Address already in use (a Java BindException) when a listening input plugin, typically the Beats input on 5044, a tcp, http, or syslog input, tries to bind a port that is already taken. The bind fails, the input cannot start, and the pipeline that owns it stops ingesting.

The blast radius is the problem. In a multi-pipeline deployment the JVM usually stays alive and the other pipelines keep running. Process-level checks stay green, the monitoring API answers, and one pipeline silently stops ingesting. Upstream Filebeat agents queue, syslog senders drop, and nobody notices until someone asks where a stream of logs went.

Logstash API unreachable on port 9600: crash, GC pause, or startup

Mon, 01 Jan 0001 00:00:00 +0000

Logstash API unreachable on port 9600: crash, GC pause, or startup

Your liveness check against http://127.0.0.1:9600/ just started timing out or returning connection refused. That single fact tells you almost nothing by itself. The Logstash monitoring API shares the JVM with the pipeline, so an unreachable API means one of four things: the process is dead, the JVM is frozen in a garbage collection pause, Logstash is still starting up, or something is wrong with how the API is bound to the network.

Logstash Beats input: Filebeat backpressure and connection health

Mon, 01 Jan 0001 00:00:00 +0000

Logstash Beats input: Filebeat backpressure and connection health

Filebeat has stopped delivering logs. Its harvesters are running, files are being read, but the registry on the Filebeat host has not advanced in twenty minutes and downstream dashboards are going stale. Or the opposite variant: Filebeat’s log is a rotating wall of “Failed to publish events” and “connection reset by peer” while Logstash looks perfectly healthy on process checks.

The Beats input is the front door to Logstash. Filebeat speaks the Lumberjack protocol over TCP, conventionally on port 5044, and that connection is where Logstash-side trouble shows up first. When Logstash cannot drain its queue fast enough, the backpressure does not stay inside Logstash. It propagates out the Beats input, across the TCP connection, and into Filebeat, where it appears as a stalled registry, growing disk spool, or publish errors.

Logstash certificate expiry: the silent, total outage no built-in metric shows

Mon, 01 Jan 0001 00:00:00 +0000

Logstash certificate expiry: the silent, total outage no built-in metric shows

Every TLS connection into or out of Logstash depends on a certificate with a hard expiry date. When that date passes, the failure is not gradual: every new TLS handshake is rejected, every client disconnects, and every output stalls at the exact second the certificate lapses. Filebeat agents stop shipping. Elasticsearch outputs throw handshake errors. Nothing is delivered.

The dangerous part is what Logstash does not tell you. The Node Stats API exposes JVM, pipeline, event, queue, and plugin metrics, but there is no certificate expiry date, no days-until-expiry gauge, and no TLS handshake failure counter anywhere in it. Your dashboards can be completely green at T-minus one hour and show a total ingestion outage at T-plus zero. Certificate expiry is a property of files on disk, so it needs an external check, not a Logstash metric.

Logstash config reload failed: reloads.failures and invisible configuration drift

Mon, 01 Jan 0001 00:00:00 +0000

Logstash config reload failed: reloads.failures and invisible configuration drift

You pushed a config change. The reload counter incremented, the process never restarted, and nothing paged. Two days later someone notices the new filter logic is not in the data. The deploy “worked” in every system that tracks deploys, but Logstash is still running the old pipeline.

With config.reload.automatic enabled, Logstash polls config files, validates changes, and hot-swaps the pipeline. When validation fails, it keeps the old pipeline running. That is the safe choice, but it means the config on disk and the config in memory diverge with no process-level symptom. The only evidence lives in three counters almost nobody graphs: reloads.failures, reloads.last_error, and reloads.successes.

Logstash configuration drift: when the running config no longer matches the deployed one

Mon, 01 Jan 0001 00:00:00 +0000

Logstash configuration drift: when the running config no longer matches the deployed one

You pushed a config fix to /etc/logstash/conf.d/ an hour ago. The pipeline is running, throughput is normal, no alert fired. But the fix never took effect: the reload failed validation, Logstash kept the old pipeline running, and nothing in your standard monitoring noticed. The running config and the deployed config are now two different things.

This is Logstash configuration drift, and it is dangerous because it is invisible by default. Logstash has no built-in drift detection: there is no API endpoint that returns the currently active config text or a hash of it. The reload machinery is deliberately safe (a failed reload keeps the old pipeline alive rather than dropping it), but that safety creates a silent gap between what you think is deployed and what is actually processing your events.

Logstash configuration integrity: detecting unexpected changes to pipeline files

Mon, 01 Jan 0001 00:00:00 +0000

Logstash configuration integrity: detecting unexpected changes to pipeline files

Logstash has no built-in file integrity monitoring. It will load, reload, and run whatever it finds in /etc/logstash/conf.d/, pipelines.yml, and logstash.yml, and it will not tell you that those files changed outside your deployment pipeline. A hand edit at 02:00, a config-management agent fighting your last deploy, or an unauthorized change all look identical to the process: new config, reload attempt, keep running.

Logstash could not be started: another instance is using the configured data.dir

Mon, 01 Jan 0001 00:00:00 +0000

Logstash could not be started: another instance is using the configured data.dir

Logstash exits immediately at startup with an error like:

Logstash could not be started because there is already another instance
using the configured data.dir. Please change the value of path.data or
configure a different instance.

The wording varies slightly by version, but the meaning is always the same: Logstash tried to take an exclusive lock on its data directory and failed. This is a hard startup block. No pipeline loads, no events flow, the process exits.

Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue

Mon, 01 Jan 0001 00:00:00 +0000

Logstash CPU-bound filters (grok hell): high CPU, saturated workers, growing queue

Worker utilization is pinned above 90%. Host CPU is near its ceiling on allocated cores. The queue is growing, per-event processing duration is climbing, and output errors are absent. The pipeline is not blocked downstream. It is starved for compute because one or more filter plugins cannot keep up with the event rate.

The usual culprit is complex grok regex evaluation, though Ruby filters, heavy JSON manipulation, and enrichment plugins can produce the same signature. The critical diagnostic fork: if CPU is high, you have a compute problem. If CPU is low or moderate with the same queue growth, you have backpressure or I/O blocking, and the fixes are entirely different. This article covers the compute-bound path.

Logstash dead letter queue growing: DLQ diversion, replay, and disabled-by-default risk

Mon, 01 Jan 0001 00:00:00 +0000

Logstash dead letter queue growing: DLQ diversion, replay, and disabled-by-default risk

A growing dead_letter_queue.queue_size_in_bytes means an output is permanently rejecting events and diverting them to the DLQ instead of delivering them downstream. This is a data-integrity failure, not just a metric ticking up. Every event in the DLQ is data your destination never received.

Two design properties compound the risk. The DLQ is disabled by default in all Logstash versions; many production deployments run without it, meaning permanently failed events are logged and silently lost with no on-disk record. And the DLQ only captures specific output failure classes. Filter and parse failures (grok, json, date) do not go to the DLQ. They pass through the pipeline with _grokparsefailure or similar tags and reach the output as malformed events.

Logstash disk full: PQ, DLQ, and log volumes competing for space

Mon, 01 Jan 0001 00:00:00 +0000

Logstash disk full: PQ, DLQ, and log volumes competing for space

Logstash is down, the logs show disk write failures, and the persistent queue metrics look innocent: queue_size_in_bytes is well under max_queue_size_in_bytes. The queue never filled. The disk did.

This is a distinct outage path from queue fullness. queue.max_bytes limits how much the persistent queue itself allocates. It says nothing about total disk consumption on the partition. When the PQ shares a filesystem with the dead letter queue, Logstash’s own log files, or the OS, any of those consumers can fill the partition while the PQ stays comfortably within its configured limit. Once the filesystem returns no-space errors, page writes fail, checkpoint writes fail, and the process dies.

Logstash downstream backpressure cascade: when a slow output stalls the whole pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Logstash downstream backpressure cascade: when a slow output stalls the whole pipeline

Logstash is up, the API returns 200, and your dashboards are going dark. Beats agents are buffering, Kafka consumer lag is climbing, and throughput collapsed while CPU sits at 15%. The process looks healthy from the outside. It is not.

A slow or failing output (Elasticsearch, Kafka, HTTP endpoint, syslog receiver) raises output duration. Worker threads block waiting for acknowledgment. The queue fills. The persistent queue absorbs the gap for a while, sometimes hours. When it hits max_bytes, inputs are blocked. Upstream systems back up or drop events.

Logstash Elasticsearch 429: retrying failed action with response code 429

Mon, 01 Jan 0001 00:00:00 +0000

The log line retrying failed action with response code: 429 from logstash.outputs.elasticsearch means Elasticsearch refused a bulk request because its bulk thread pool queue was full (es_rejected_execution_exception). ES is saturated and cannot keep up with the incoming bulk request rate.

This is the most common trigger of the backpressure wedge in Logstash-to-ES pipelines. The ES output plugin retries the rejected batch with exponential backoff. Worker threads block on those retries. The internal queue fills. Inputs are backpressured, and upstream systems like Filebeat buffer locally or begin dropping events.

Logstash Elasticsearch mapping conflict: field type mismatches and rejected documents

Mon, 01 Jan 0001 00:00:00 +0000

Logstash Elasticsearch mapping conflict: field type mismatches and rejected documents

A mapping conflict occurs when Logstash sends a document whose field type does not match the index mapping. A field mapped as long arrives as a string, or an object appears where a scalar was expected. Elasticsearch rejects the document at the bulk API level.

The rejection is often invisible. Elasticsearch returns HTTP 200 for bulk requests that contain per-document failures. Logstash counts the batch as events.out and moves on. If the Dead Letter Queue (DLQ) is enabled, rejected documents divert there. If it is not (the default), documents are logged at WARN and silently lost. Pipeline throughput stays green while data disappears.

Logstash Elasticsearch partial bulk failure: HTTP 200 with per-document errors and silent data loss

Mon, 01 Jan 0001 00:00:00 +0000

Logstash Elasticsearch partial bulk failure: HTTP 200 with per-document errors and silent data loss

Your Logstash pipeline looks healthy. The process is up, the API responds on port 9600, events.out is climbing steadily, and the queue is empty. But someone asks why last Tuesday’s application logs are missing from Elasticsearch. You run a count query against the index and the numbers do not add up. Logstash says it delivered 50 million events. Elasticsearch shows 48 million documents. Two million vanished.

Logstash event duplication: in/out ratio drift, clones, and re-read sources

Mon, 01 Jan 0001 00:00:00 +0000

Logstash event duplication: in/out ratio drift, clones, and re-read sources

Logstash exposes three cumulative event counters at the pipeline level: events.in, events.out, and events.filtered. Their relationship reflects the pipeline’s intended transformation. A passthrough pipeline should see out approximately equal to in over any sustained window. A pipeline with drop {} filters should see out plus filtered approximately equal to in. Clone and split filters legitimately produce more output events than input events.

Logstash file descriptor pressure: leaks, tailed files, and reconnection churn

Mon, 01 Jan 0001 00:00:00 +0000

Logstash file descriptor pressure: leaks, tailed files, and reconnection churn

Logstash is up, throughput looks normal, and then new connections start failing, file opens error out, and the process dies with “too many open files.” The failure looks sudden. It usually is not. File descriptor pressure builds over days or weeks as process.open_file_descriptors climbs toward process.max_file_descriptors, and exhaustion is a cliff edge: everything works until the limit is hit, then new connections, new file opens, and sometimes logging itself fail at once.

Logstash file input and sincedb: re-read loops, duplicates, and FD pressure

Mon, 01 Jan 0001 00:00:00 +0000

Logstash file input and sincedb: re-read loops, duplicates, and FD pressure

Your Logstash process looks healthy. The API answers, the pipeline is running, workers are busy. Then someone downstream asks why the same log lines appear three times in Elasticsearch, or why events.in just spiked to ten times its baseline, or why Logstash is logging “too many open files” while traffic looks normal.

All three symptoms usually trace back to the same component: the file input plugin and its position-tracking file, the sincedb. The file input holds one file descriptor per tailed file and records how far it has read in a sincedb file on disk. When that tracking breaks (corruption, inode reuse after rotation, a remounted filesystem changing device numbers, a wildcard that suddenly matches thousands of files) the failure shows up as duplicate events, unexpected events.in spikes, or climbing process.open_file_descriptors.

Logstash flow.queue_backpressure: the input-throttling metric explained

Mon, 01 Jan 0001 00:00:00 +0000

Logstash flow.queue_backpressure: the input-throttling metric explained

You opened the Logstash node stats API during an incident, found flow.queue_backpressure sitting at 0.9, and now you need to know what that number is actually telling you. The short answer: it is the fraction of time your input threads spend blocked trying to push events into the pipeline queue. It measures ingestion throttling, not worker-side slowness, and that distinction drives the entire diagnosis.

Logstash GC death spiral: high garbage-collection overhead and collapsing throughput

Mon, 01 Jan 0001 00:00:00 +0000

Logstash GC death spiral: high garbage-collection overhead and collapsing throughput

The Logstash process is up. systemd reports it as active. But event throughput has collapsed to near zero, and CPU is pinned at full utilization. This is a garbage-collection death spiral: the JVM spends most of its wall-clock time in GC, reclaiming almost nothing, while the pipeline starves for compute.

The failure is misleading because the process appears alive. Process-liveness checks pass. The monitoring API on port 9600 may still respond, albeit slowly. But no useful work is happening. The JVM has entered a feedback loop: each GC cycle reclaims less than the last, so GC runs more frequently, steals CPU from event processing, causes events to accumulate, fills the heap faster, and triggers even more GC.

Logstash grok slow: catastrophic backtracking and per-event duration spikes

Mon, 01 Jan 0001 00:00:00 +0000

Logstash grok slow: catastrophic backtracking and per-event duration spikes

A Logstash pipeline that ran fine for months suddenly pegs CPU. Workers saturate, the persistent queue grows, and event throughput collapses. The input rate is unchanged, the output destination is healthy, and there are no error storms in the logs. The only visible anomaly is that one grok filter’s per-event processing duration has exploded.

This is the signature of catastrophic regex backtracking, also known as ReDoS. A single malformed or near-miss log line hits a pattern with nested unbounded quantifiers or overlapping alternation, and the Oniguruma regex engine that grok uses takes exponential time to determine that the match fails. One bad event can occupy a pipeline worker thread for seconds or longer.

Logstash heap usage high: why the post-GC floor matters more than the peak

Mon, 01 Jan 0001 00:00:00 +0000

Logstash heap usage high: why the post-GC floor matters more than the peak

Logstash operators hit a recurring trap with JVM heap monitoring. They set an alert for heap_used_percent above 80%, following standard JVM guidance. The alert fires constantly because the JVM heap naturally cycles through allocation and collection. They silence it or raise the threshold. Then a real memory leak or GC death spiral develops, heap stays elevated for hours, and nobody notices because the alert was muted.

Logstash input rate dropped to zero: upstream failure vs blocked inputs

Mon, 01 Jan 0001 00:00:00 +0000

Logstash input rate dropped to zero: upstream failure vs blocked inputs

The Logstash input rate metric (flow.input_throughput or the events.in counter delta) has dropped to zero. Before restarting anything, answer one question: is the queue empty or full?

An input rate of zero with an empty queue means events are not arriving from upstream. An input rate of zero with a queue at capacity means Logstash has applied backpressure to its own inputs because it cannot drain events fast enough. These conditions require opposite responses, and treating one as the other wastes time during an incident.

Logstash JVM heap sizing: -Xms/-Xmx, the 1GB default, and why they should match

Mon, 01 Jan 0001 00:00:00 +0000

Logstash JVM heap sizing: -Xms/-Xmx, the 1GB default, and why they should match

Logstash ships with a 1GB JVM heap (-Xms1g -Xmx1g in config/jvm.options), unchanged through 7.x, 8.x, and 9.x. That default works for trivial pipelines. It is catastrophically small for production and is the single most common cause of GC death spirals, throughput collapse, and OOM kills.

The fix: increase the heap, set minimum and maximum to the same value, and account for off-heap memory. Set the heap too large and you starve the OS and risk longer full-GC pauses. Set it without understanding off-heap allocation and you get OOM kills with a heap that looks comfortably under capacity. Ignore container cgroup limits and the JVM sizes itself against host RAM, not the container limit.

Logstash Kafka input: consumer group lag and rebalances

Mon, 01 Jan 0001 00:00:00 +0000

Logstash Kafka input: consumer group lag and rebalances

The consumer group lag for your logstash group is climbing and not draining, or the Kafka input periodically drops to zero for seconds or minutes at a time while the brokers look fine. In both cases the evidence lives outside Logstash: consumer-group lag is a Kafka-side signal, and you will not find it in the Logstash monitoring API on port 9600.

Rising lag means one thing: Logstash is consuming slower than producers are writing. Frequent rebalances mean the group cannot hold a stable assignment, and every rebalance is a window where consumption stops entirely. The two often appear together, because the most common rebalance trigger (slow event processing) is also the most common lag trigger.

Logstash memory queue vs persistent queue: durability, visibility, and failure modes

Mon, 01 Jan 0001 00:00:00 +0000

Logstash memory queue vs persistent queue: durability, visibility, and failure modes

Every Logstash pipeline has a queue between its inputs and its worker threads. The default is the memory queue: a small, bounded, in-memory buffer with no durability. The alternative is the persistent queue (PQ): a page-based, checkpointed, on-disk buffer that survives restarts.

This choice changes three things that matter operationally: what you lose when the process dies, what you can see in the metrics API, and how the pipeline fails when the queue fills. Teams enable PQ for durability and then discover it added disk I/O saturation, page corruption, and a false sense of safety to their failure catalogue. Teams running the memory queue often have no written answer for what a crash costs them.

Logstash Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Logstash Monitoring

What Is Logstash?

Logstash is a powerful, open-source tool for managing events and logs. It plays a pivotal role in data collection within the Elastic Stack, allowing for seamless data transportation from a multitude of sources to your target destinations for further analysis or storage.

Monitoring Logstash With Netdata

Netdata provides an effective Logstash monitoring tool that offers real-time insights into Logstash performance. Through the integration with Netdata Cloud, DevOps teams can oversee Logstash’s health and quickly identify any anomalies or inefficiencies that could affect their data pipelines. Check out our Live Demo to see Netdata in action.

Logstash monitoring API exposed: port 9600 on a routable interface without auth

Mon, 01 Jan 0001 00:00:00 +0000

Logstash monitoring API exposed: port 9600 on a routable interface without auth

Every Logstash node runs an HTTP monitoring API, by default on port 9600. It answers questions like “what pipelines are running”, “what plugins and versions are installed”, and “show me the hottest threads with stack traces”. By default it binds to 127.0.0.1, which is safe. Exposure happens when someone sets api.http.host to 0.0.0.0 or a routable IP to make remote monitoring easier, and leaves it there.

Logstash monitoring checklist: the signals every production pipeline needs

Mon, 01 Jan 0001 00:00:00 +0000

Logstash monitoring checklist: the signals every production pipeline needs

Most Logstash monitoring setups answer the wrong question. They answer “is the process running?” when the question that matters is “are events leaving the pipeline?” A Logstash JVM can be alive, healthy by systemd’s standards, and returning 200 from its monitoring API while the queue is full, workers are blocked on a dead Elasticsearch, and zero events have been delivered for an hour. Process liveness is necessary. It is nowhere near sufficient.

Logstash monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Logstash monitoring maturity model: from survival to expert

Most Logstash deployments are monitored at the wrong level. Teams alert on process liveness and heap percentage, then get paged by users asking where Wednesday’s logs went. The process was up, the API returned 200, and throughput looked fine the whole time. The gap is not tooling. It is which signals the team decided to watch.

This article defines a four-level maturity model for Logstash monitoring: Survival, Operational, Mature, and Expert. Each level names the specific signals to collect, why they matter, and what class of failure becomes visible that was invisible at the level below. Use it as an audit checklist against your current setup, and as a roadmap for what to add next. The levels are cumulative: every level assumes everything below it is already in place.

Logstash multi-pipeline monitoring: why aggregate stats hide a failed pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Logstash multi-pipeline monitoring: why aggregate stats hide a failed pipeline

You migrated to pipelines.yml to isolate workloads: one pipeline per team, per source, or per destination. Each pipeline got its own queue, its own workers, its own failure domain. Right call for fault isolation. But if your monitoring still polls the node once and looks at global event counts, you have given back the visibility the isolation bought you.

The concrete scenario: five pipelines, roughly equal traffic, one of them wedges. An output stalls, a config reload fails, a Kafka consumer group gets stuck rebalancing. That pipeline’s throughput goes to zero; the other four keep flowing. Aggregate node throughput drops by about 20 percent. If your alert is “throughput drops more than 50 percent” or “events per second below N”, nothing fires. The failed pipeline can sit dead for hours while node-level graphs show a normal dip.

Logstash one input stopped: per-input failures masked by aggregate metrics

Mon, 01 Jan 0001 00:00:00 +0000

Logstash one input stopped: per-input failures masked by aggregate metrics

One of three inputs in your Logstash pipeline stops receiving events. The pipeline-level events.in counter drops by a third. But because the other two inputs keep flowing, the aggregate rate never hits zero, and your threshold-based alert stays silent. The failed input’s upstream source starts accumulating: Kafka consumer lag grows, file tails fall behind, or Beats agents buffer locally. By the time someone notices, hours or days of data from that source are delayed or lost.

Logstash OutOfMemoryError: Java heap space and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

Logstash OutOfMemoryError: Java heap space and how to recover

java.lang.OutOfMemoryError: Java heap space in the Logstash log means the JVM could not satisfy an allocation request and GC could not reclaim enough heap to proceed. The process exits or gets kernel OOM-killed, and every pipeline it was running stops with it.

Before the hard crash, there is usually a warning period: the GC death spiral. Heap fills, garbage collection runs longer and more frequently, throughput collapses. This can last minutes or hours before the JVM finally fails to allocate. If you catch the spiral, you can intervene before data is lost. If you only catch the OOM, you are in recovery mode.

Logstash output errors and retries: reading downstream failure before the queue grows

Mon, 01 Jan 0001 00:00:00 +0000

Logstash output errors and retries: reading downstream failure before the queue grows

Output-side failures are the most common cause of Logstash queue growth and eventual outage. When the downstream destination (Elasticsearch, Kafka, an HTTP endpoint) starts rejecting, timing out, or slowing down, Logstash output plugins retry. Those retries preserve data temporarily but hide mounting delay. The pipeline appears alive while events accumulate.

The critical window for diagnosis is between the first output errors and the moment the queue fills. Once the queue reaches capacity, inputs are blocked, upstream systems buffer or drop events, and the incident has cascaded beyond Logstash.

Logstash persistent queue full: max_bytes reached and inputs blocked

Mon, 01 Jan 0001 00:00:00 +0000

Logstash persistent queue full: max_bytes reached and inputs blocked

The persistent queue has hit queue.max_bytes, input threads are blocked trying to push events in, and the pipeline has stopped accepting new data. A full PQ is almost never the root cause. It is the terminal stage of a downstream outage or capacity failure that the queue was absorbing. Fix the cause, drain the queue, prevent recurrence.

This guide covers confirming the state, finding the downstream cause, calculating how much time you have, and recovering without making things worse. For the underlying throttling metric, see Logstash flow.queue_backpressure: the input-throttling metric explained.

Logstash persistent queue not draining: page-release lag after downstream recovery

Mon, 01 Jan 0001 00:00:00 +0000

Logstash persistent queue not draining: page-release lag after downstream recovery

The downstream outage is over. Elasticsearch is green, output errors have stopped, and Logstash is delivering events again. But the persistent queue is shrinking far slower than it filled, disk I/O is pinned, and df shows the same used space it did an hour ago even though queue_size_in_bytes has clearly dropped.

Most of the time, nothing is stuck. A persistent queue (PQ) draining after recovery looks unhealthy from almost every angle: high disk I/O, reduced effective throughput, and disk usage that refuses to go down. This is expected behavior driven by how PQ pages are released. The failure modes you actually need to rule out are narrower: page or checkpoint corruption, poison events pinning pages, and disk I/O saturation becoming the new bottleneck.

Logstash persistent queue runway: how long until the PQ fills

Mon, 01 Jan 0001 00:00:00 +0000

Logstash persistent queue runway: how long until the PQ fills

A persistent queue changes the failure shape of a Logstash pipeline. When the output destination goes down, nothing looks broken for a while: inputs keep accepting events, throughput counters keep moving, the process stays green, and the queue absorbs the gap between arrival and delivery. The outage exists the whole time; the PQ just delays the moment anyone feels it.

Logstash pipeline not started: a healthy JVM with a missing pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Logstash pipeline not started: a healthy JVM with a missing pipeline

Systemd says Logstash is active. The monitoring API on port 9600 returns 200. Heap looks fine, GC is quiet, and yet one of your data sources has gone silent at the destination. When you query /_node/stats/pipelines, the pipeline ID you expected is not there, or its event counters have not moved in hours.

This is a partial outage that process-level checks cannot see. Since Logstash 7.11, a pipeline that crashes no longer takes the JVM down with it: the process stays alive, serving the API, with fewer pipelines running than you configured. A failed initial config load, a failed reload that stopped the old pipeline without starting the new one, or a worker error that terminated a single pipeline all produce the same shape: healthy JVM, missing pipeline.

Logstash pipeline stalled: output rate at zero while the process looks alive

Mon, 01 Jan 0001 00:00:00 +0000

Logstash pipeline stalled: output rate at zero while the process looks alive

The Logstash JVM is running. systemctl status logstash says active. The monitoring API on port 9600 returns 200. Every liveness check you have is green. But events.out has not moved in twenty minutes, and data stopped arriving downstream at about the same time.

This is the “living dead” state: the process exists, but the pipeline is functionally dead. It is one of the most common ways Logstash fails in production, and it is invisible to any monitoring that only checks process existence. Output throughput, not liveness, is the health signal.

Logstash pipeline.workers and batch_size: tuning the worker pool

Mon, 01 Jan 0001 00:00:00 +0000

Logstash pipeline.workers and batch_size: tuning the worker pool

The worker pool is the most important throughput lever in Logstash. pipeline.workers sets how many threads process events through the filter and output stages. pipeline.batch.size sets how many events each worker handles per trip through the queue. Together, these define the maximum in-flight event count, the pipeline’s memory footprint, and how much CPU and downstream capacity it can consume.

Tuning is not about finding a universal optimum. It is about matching the worker pool to three variables: CPU cost per event (dominated by filters), average event size, and the downstream system’s ability to absorb batches efficiently. Getting this wrong produces one of three outcomes: CPU saturation with queue growth, heap exhaustion from too many in-flight events, or output-blocking cascades that look like high worker utilization but produce no useful throughput.

Logstash queue events count growing: reading the in-flight backlog

Mon, 01 Jan 0001 00:00:00 +0000

Logstash queue events count growing: reading the in-flight backlog

queue.events_count trending upward on the pipeline stats API means one thing: events are arriving faster than workers can process and deliver them. It is the primary backpressure indicator, and the earliest warning you get before inputs block and upstream systems start backing up or dropping data.

What the number alone does not tell you is how much trouble you are in. The same rising curve means different things for the memory queue versus a persistent queue, and the absolute value is workload-dependent enough that fixed thresholds are nearly useless. A memory queue climbing by a few hundred events can be more urgent than a persistent queue holding a million.

Logstash queue full: inputs blocked and the backpressure wedge

Mon, 01 Jan 0001 00:00:00 +0000

Logstash queue full: inputs blocked and the backpressure wedge

The symptom usually arrives from upstream first: Filebeat stops shipping, Kafka consumer lag climbs, or an HTTP input starts refusing connections. Logstash itself looks alive. The API answers on port 9600, the process is running, CPU is often unremarkable. But events are not moving, because the internal queue between inputs and workers is full, and every input thread is blocked trying to push into it.

Logstash silent data loss: events dropped with no DLQ and green dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Logstash silent data loss: events dropped with no DLQ and green dashboards

The first sign is usually a question from downstream: “Where are last Wednesday’s logs?” Your Logstash dashboards show green. Process liveness is fine. Events-in and events-out curves track each other. The queue is empty. GC is quiet. Nothing paged.

Silent data loss happens when events leave the Logstash accounting system as “delivered” but never reach their destination in usable form. The events.out counter increments, throughput looks normal, and no alert fires. By the time anyone notices, the loss window may be hours or days old and the evidence is gone.

Logstash SSL/TLS handshake failures: expired certs and broken trust chains

Mon, 01 Jan 0001 00:00:00 +0000

Logstash SSL/TLS handshake failures: expired certs and broken trust chains

Your Logstash logs are filling with PKIX path building failed or handshake_failure errors. Depending on which side is failing, you either have a delivery outage in progress (an output that can no longer connect to its destination) or a security signal (clients failing to authenticate against an input). The triage path starts the same either way: find the failing plugin, read the actual certificate error, and determine whether this is an availability problem, a security problem, or a planned rotation you forgot about.

Logstash thread starvation: workers blocked, low CPU, and stalled throughput

Mon, 01 Jan 0001 00:00:00 +0000

Logstash thread starvation: workers blocked, low CPU, and stalled throughput

Logstash throughput has collapsed. The queue is growing. Your pipeline workers are all occupied. But CPU is sitting at 20% with no obvious explanation. Restarting the process clears it temporarily, then the pattern repeats. This is thread starvation. The symptoms look contradictory, which makes it easy to misdiagnose.

The mechanism: pipeline worker threads are stuck waiting on a shared resource rather than doing computation. Each worker pulls a batch from the queue and does not release it until every filter and output in the chain completes. When the thing that completes that chain is a slow network call, an exhausted connection pool, a mutex held by another worker, or a DNS resolver that hangs, the worker blocks. With enough workers blocked, no one drains the queue, backpressure propagates to inputs, and the pipeline stalls. CPU stays low because threads are parked in wait states, not burning cycles.

Logstash too many open files: file-descriptor exhaustion and refused connections

Mon, 01 Jan 0001 00:00:00 +0000

Logstash too many open files: file-descriptor exhaustion and refused connections

Logstash is logging java.io.IOException: Too many open files (EMFILE), Beats shippers report refused or reset connections, and new files are not being tailed. The JVM process is still alive, systemd still says active (running), and the monitoring API on port 9600 may still answer. Nothing looks crashed, yet the pipeline has stopped accepting new work.

That is the defining shape of file-descriptor exhaustion: a partial failure that is routinely misread as a network or disk problem. The process cannot open new sockets, new persistent queue page files, or new files for the file input, because every open of those consumes one file descriptor and the per-process limit has been reached. Existing connections and already-open files keep working, which is why some traffic continues to flow while new connections are refused.

Logstash won't start after a crash: persistent queue corruption and checkpoint errors

Mon, 01 Jan 0001 00:00:00 +0000

Logstash won’t start after a crash: persistent queue corruption and checkpoint errors

Logstash was killed uncleanly (OOM kill, kill -9, SIGKILL after a pod termination grace period, power loss) and now refuses to start. The log at /var/log/logstash/logstash-plain.log shows java.io.IOException and checkpoint-related errors during pipeline initialization, and the process exits before any events flow.

The persistent queue (PQ) is a page-based on-disk queue: events live in page files (default 250MB each, up to queue.max_bytes which defaults to 1GB), and checkpoint files record which events have been acknowledged as delivered. Both are updated continuously while the pipeline runs. An unclean shutdown can leave the checkpoint files and page files inconsistent with each other, and Logstash’s startup code refuses to open a queue it cannot reconcile.

Logstash worker utilization high: reading flow.worker_utilization and per-plugin skew

Mon, 01 Jan 0001 00:00:00 +0000

Logstash worker utilization high: reading flow.worker_utilization and per-plugin skew

High flow.worker_utilization by itself means workers are busy. It becomes actionable only when the queue is growing or there is no headroom for traffic spikes. The first check is always queue depth, not utilization.

flow.worker_utilization is a percentage from 0 to 100, available in Logstash 8.x at the pipeline level. Plugin-level breakdowns live under plugins.filters[].flow.worker_utilization and plugins.outputs[].flow.worker_utilization in the Node Stats API. Pipeline-level tells you whether workers are saturated. Plugin-level tells you which filter or output is responsible.

loki

Mon, 01 Jan 0001 00:00:00 +0000

Loki Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Loki Monitoring

What Is Loki?

Loki is an open-source log aggregation system developed by Grafana Labs. It is designed to manage, aggregate, and analyze log data efficiently. Inspired by Prometheus, it focuses on performance and scalability for search across different log streams. Unlike traditional logging tools, Loki uses dynamic tagging to allow logs to be efficiently stored and accessed, promoting cost-effective log management.

Monitoring Loki With Netdata

Netdata offers an advanced monitoring solution specifically tailored for Loki. Utilizing an openmetrics Prometheus exporter, Netdata effectively gathers metrics from your Loki instances without the need for a Prometheus server or Grafana setup. This means users can benefit from automated dashboards, real-time alerts, and more. Netdata’s integration simplifies the process, allowing you to intensely monitor Loki’s performance and health with ease.

Loop Telecommunication International Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Loral Wdl SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lotus Development Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lsi Logic SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lucent Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Luminous Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Lustre metadata

Mon, 01 Jan 0001 00:00:00 +0000

Lustre Metadata Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Lustre Metadata Monitoring

What Is Lustre Metadata?

Lustre is a type of parallel distributed file system, widely used for large-scale cluster computing. Originally developed for research and enterprise sectors due to its capacity and high performance, Lustre metadata refers to the internal management data that keeps track of file location, size, and storage attributes. Monitoring Lustre metadata is crucial for maintaining optimal file system operations and ensuring efficient management.

Monitoring Lustre Metadata With Netdata

Monitoring Lustre with Netdata provides seamless tracking of all critical metrics using an openmetrics (prometheus) exporter. To monitor Lustre metadata, Netdata taps into the Cluster Exporter which is capable of gathering essential data. With Netdata, you can ingest data from any Prometheus exporter; unlocking automated dashboards and alerts without needing a Prometheus server or Grafana. This integration supports efficient Lustre metadata monitoring and ensures constant observability over your cluster’s performance.

Luxn Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

LVM boot activation failure: emergency shell and missing mount points

Mon, 01 Jan 0001 00:00:00 +0000

LVM boot activation failure: emergency shell and missing mount points

You boot a server and land in a dracut emergency shell or initramfs rescue prompt. Or the system boots but /mnt/data, /var/lib/postgresql, or other non-root filesystems are not mounted. The common thread: LVM volumes that should have activated during early boot did not.

The initramfs activates LVM volumes before mounting root and before systemd mounts the rest of fstab. If the initramfs lacks the right LVM tools or configuration, or if a required PV is slow to appear (iSCSI not connected, multipath not configured, SAN LUN not presented), activation fails before you have a running system to diagnose from.

LVM cannot extend a logical volume: adding a PV when the VG is full

Mon, 01 Jan 0001 00:00:00 +0000

LVM cannot extend a logical volume: adding a PV when the VG is full

You ran lvextend and it refused with some variation of “insufficient free space” or “insufficient free extents,” while the filesystem that prompted all this sits at 99% and application writes fail. The fix is mechanical once you know which of three independent constraints is actually blocking you.

The trap is that “no space” means three different things in an LVM stack. The filesystem can be full while the LV has room. The VG can be out of free extents while every filesystem looks fine. A thin pool can be 100% full while the VG reports free space. Each constraint has a different remediation, and running the wrong one either does nothing or makes the incident worse.

LVM commands hang: when lvs, vgs, and pvs block on locks or dead devices

Mon, 01 Jan 0001 00:00:00 +0000

LVM commands hang: when lvs, vgs, and pvs block on locks or dead devices

You type lvs on a host during an incident and the prompt never comes back. Ctrl-C does nothing. kill -9 from another shell does nothing. The process is in D state, uninterruptible sleep, waiting on storage I/O or a metadata lock. Now vgs and pvs hang too, and your monitoring agent, which polls lvs every 60 seconds, has gone silent on exactly the host that is on fire.

LVM Couldn't find device with uuid: a physical volume has gone missing

Mon, 01 Jan 0001 00:00:00 +0000

LVM Couldn’t find device with uuid: a physical volume has gone missing

The error “Couldn’t find device with uuid” appears when LVM’s volume group metadata references a physical volume UUID that no block device on the system currently claims. Every LVM command that touches the affected VG prints the warning. The PV shows as [unknown] in pvs output, and the VG enters a partial state.

The recovery path depends entirely on why the device disappeared and what LV layout sits on top of it. The wrong recovery command, applied too quickly, causes permanent data loss. If system uptime is more than a few minutes and a PV is gone, something has failed at the hardware, fabric, cloud, or operator layer.

LVM device node permissions: raw block access that bypasses the filesystem

Mon, 01 Jan 0001 00:00:00 +0000

LVM device node permissions: raw block access that bypasses the filesystem

Every active logical volume on a Linux host exposes device nodes at /dev/mapper/<VG>-<LV> and /dev/<VG>/<LV>. The default permissions on these nodes are brw-rw---- root:disk (mode 0660). Anything more permissive than that, a mode like 0666, or a group assignment that includes unprivileged users, lets any local user read and write raw block data on the volume. Filesystem permissions, ACLs, and mount options do not apply at this layer. A user who can open the device node can read /etc/shadow straight off the disk or overwrite filesystem metadata directly.

LVM dm-N device numbers change after reboot: use /dev/mapper, not /dev/dm-N

Mon, 01 Jan 0001 00:00:00 +0000

LVM dm-N device numbers change after reboot: use /dev/mapper, not /dev/dm-N

After a reboot, you run lsblk or df -h and notice your logical volumes have different /dev/dm-N numbers. What was dm-0 before the reboot is now dm-2, and what was dm-1 is now dm-0. If anything on the system references /dev/dm-0 directly, it may now point at the wrong volume.

This is not a bug and not a sign of corruption. Device-mapper minor numbers are assigned dynamically at activation time based on the order devices are discovered and activated. The numbers are not stored persistently anywhere. The device-mapper tables that back logical volumes exist only in kernel memory and are torn down on every shutdown, then recreated on every boot.

LVM dmeventd not running: the auto-extend safety net is offline

Mon, 01 Jan 0001 00:00:00 +0000

LVM dmeventd not running: the auto-extend safety net is offline

You found this page because systemctl is-active lvm2-monitor.service returned something unexpected, or pgrep dmeventd came back empty, or lvs -o+seg_monitor showed “not monitored” on a thin pool you assumed was protected.

Here is the uncomfortable part: nothing is broken right now. No errors in lvs. No failed services your alerting noticed. No I/O hangs. The system looks healthy, and it is, right up until the moment a thin pool hits 100%, a mirror leg dies, or a snapshot overflows. Then you discover, during the incident, that the daemon which was supposed to fire the safety net has been dead for weeks.

LVM filesystem full while the volume group has space: the resize step everyone forgets

Mon, 01 Jan 0001 00:00:00 +0000

LVM filesystem full while the volume group has space: the resize step everyone forgets

df -h says the filesystem is 100% full. Applications are getting ENOSPC. vgs shows plenty of free space. Maybe you already ran lvextend and df still shows the old size. Nothing here is broken. You are looking at three layers that measure three different things, and one of them was never told to grow.

lvextend grows the logical volume. The filesystem on top keeps its old size until you explicitly resize it. The LV is a bigger container; the filesystem inside has not expanded into the new space.

LVM Found duplicate PV: multipath devices and the lvm.conf filter

Mon, 01 Jan 0001 00:00:00 +0000

LVM Found duplicate PV: multipath devices and the lvm.conf filter

When LVM prints “Found duplicate PV” during boot or routine commands, it has found the same PV UUID on two or more block devices. On multipath-attached storage, this almost always means LVM is scanning both the raw SCSI paths (/dev/sdb, /dev/sdc) and the aggregated multipath device (/dev/mapper/mpatha). Each path carries the same PV metadata header.

The warning is advisory at first: LVM picks one device and proceeds. The danger is which one. If LVM activates a logical volume through a raw path instead of the multipath device, you lose path redundancy and failover. I/O runs through a single HBA link until that link fails, then the volume goes dark. The system can boot fine for weeks until a udev timing race changes which device LVM selects, and the next path failure takes down production.

LVM has free space but striped or mirrored allocation still fails

Mon, 01 Jan 0001 00:00:00 +0000

LVM has free space but striped or mirrored allocation still fails

vgs shows the volume group 30% free. lvcreate for a striped or mirrored logical volume, or lvextend on an existing one, fails anyway:

Insufficient suitable allocatable extents for logical volume

The VG has headroom and the operation needs less than that headroom. Both facts are true at once, and that is the trap: vg_free is an aggregate across every physical volume in the group, but striped and mirrored allocations are not aggregate operations. They are per-PV placement problems. LVM must find extents on multiple distinct PVs at the same time, and if your free space is piled onto one PV, the allocation has nowhere legal to go.

LVM I/O hang: a suspended dm device and processes stuck in D state

Mon, 01 Jan 0001 00:00:00 +0000

LVM I/O hang: a suspended dm device and processes stuck in D state

Processes are stuck in D state. kill -9 does nothing. lvs hangs. Load average is climbing. The root cause is almost always a device-mapper device stuck in suspended state.

A suspended dm device blocks all I/O to the underlying logical volume. Every process performing I/O to that volume enters uninterruptible sleep (D state) and cannot be killed, even with SIGKILL. The only way to release them is to resolve the underlying I/O blockage and let queued I/O drain.

LVM Insufficient free extents: the volume group is out of space

Mon, 01 Jan 0001 00:00:00 +0000

LVM Insufficient free extents: the volume group is out of space

You ran lvcreate, lvextend, or lvresize and got one of these:

Insufficient free extents
Insufficient free space: 12800 extents needed, but only 0 available

Both messages mean the same thing: the volume group has no unallocated physical extents left for the allocation you requested. Every physical extent in the VG is already assigned to some logical volume, and the operation fails outright.

LVM lock contention: stale lock files and blocked commands

Mon, 01 Jan 0001 00:00:00 +0000

LVM lock contention: stale lock files and blocked commands

You run lvs and it sits there. You try vgs in another terminal and it hangs too. Storage is fine, VMs are running, application I/O is flowing, but every LVM command blocks. Your monitoring, which polls lvs every 10 seconds, goes stale at the exact moment you need it.

This is LVM metadata lock contention. Every LVM command (pvs, vgs, lvs, lvcreate, lvextend, pvmove) acquires locks before reading or modifying VG metadata, using lock files in /run/lock/lvm/. When one command holds a lock and stalls, everything else queues behind it. Because the monitoring commands are LVM commands too, the management plane goes blind under the same condition it is supposed to be observing.

LVM logical volume not active: lvchange -ay and why activation failed

Mon, 01 Jan 0001 00:00:00 +0000

LVM logical volume not active: lvchange -ay and why activation failed

A logical volume that should be serving data shows as inactive. In lvs output, lv_attr position 5 is - where you expect a. No device-mapper table is loaded, no /dev/dm-N device exists, and nothing can mount, open, or write to that LV. Applications see a missing block device, and if this LV backs root or a critical service, the system may not have booted properly.

LVM logical volume partial (p) flag: which LVs the missing disk took down

Mon, 01 Jan 0001 00:00:00 +0000

LVM logical volume partial (p) flag: which LVs the missing disk took down

You run lvs and see a lowercase p in the ninth character of lv_attr. That means a physical volume (PV) backing extents in this logical volume has disappeared from the system. The VG is now in partial mode, and some LVs may be silently degraded or completely inaccessible.

The p flag is direct evidence of device loss: a disk failed, a SAN LUN was unpresented, a multipath device lost all paths, or a cloud volume was detached. The question is how bad the damage is and which volumes took the hit.

LVM logical volumes

Mon, 01 Jan 0001 00:00:00 +0000

LVM logical volumes Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

LVM logical volumes Monitoring

What Is LVM?

LVM, or Logical Volume Manager, is a system for managing logical volumes, or filesystems, in Linux environments. It provides a high level of flexibility, allowing administrators to easily resize, extend, or shrink volumes according to their needs. This flexibility is particularly useful in dynamic and modern data environments.

Monitoring LVM With Netdata

Monitoring LVM logical volumes efficiently is where the Netdata monitoring tool excels. With Netdata, you can monitor LVM to ensure each logical volume’s health and performance. Netdata leverages the lvs CLI tool securely by using ndsudo to facilitate robust and secure data collection without the need for sudo.

LVM metadata backup stale: /etc/lvm/backup out of sync with the VG

Mon, 01 Jan 0001 00:00:00 +0000

LVM metadata backup stale: /etc/lvm/backup out of sync with the VG

Every LVM command that changes metadata (lvcreate, lvextend, lvremove, vgextend, snapshot creation) writes two things to the local filesystem: an archived copy of the new metadata in /etc/lvm/archive/ and a current-state backup in /etc/lvm/backup/<vg>. That backup file is what vgcfgrestore reads when you recover a VG after metadata corruption.

The failure mode here is quiet: something blocks those writes, the backup file stops updating, and nothing in day-to-day operation tells you. LVM keeps working. Months later you hit metadata corruption, reach for vgcfgrestore, and restore a VG layout from six months ago. LVs created since then are missing. LVs removed since then come back pointing at extents that have been reallocated. You have traded a recovery exercise for a data integrity problem.

LVM metadata corruption recovery: vgcfgrestore from /etc/lvm/archive

Mon, 01 Jan 0001 00:00:00 +0000

LVM keeps the complete map of your logical volumes in a small metadata area at the start of each physical volume. If that metadata is corrupted, by a power loss during a metadata write or a bad sector in the first few megabytes of a PV, the entire volume group becomes unreadable. Every LV in the VG becomes inaccessible at once, regardless of how healthy the actual data extents are.

LVM mirror resync storm: multiple rebuilds saturating disk I/O

Mon, 01 Jan 0001 00:00:00 +0000

LVM mirror resync storm: multiple rebuilds saturating disk I/O

Production latency is climbing. Multiple applications are slow. Storage shows high I/O across all physical volumes, but there is no single failed disk and no obvious error in the logs. When you check LVM, several mirrored logical volumes show copy_percent below 100%. They are all resyncing at the same time.

This is a mirror resync storm. After a system crash, a disk replacement, or bulk mirror creation, multiple mirrored LVs may need to resynchronize simultaneously. Each resync generates substantial sequential and random I/O as the mirror copies dirty regions or entire leg contents to the recovering side. When several run in parallel, the combined I/O overwhelms storage bandwidth. Production workloads suffer elevated latency, and each resync takes longer than it would alone because they compete for the same physical devices.

LVM monitoring checklist: the signals every production volume manager needs

Mon, 01 Jan 0001 00:00:00 +0000

LVM monitoring checklist: the signals every production volume manager needs

Most LVM incidents are not exotic. They are a thin pool that hit 100% because nobody watched metadata_percent, a mirror that ran degraded for three months because nobody alerted on copy_percent, or a PV that went missing on a Friday and paged nobody because the check only ran df. The failure modes are well known. What is usually missing is a deliberate, minimal set of signals with thresholds and severities attached.

LVM monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

LVM monitoring maturity model: from survival to expert

Most teams monitor LVM by accident. They alert on filesystem fullness, maybe on disk failure, and discover thin pool exhaustion or a silently degraded mirror only when applications start returning I/O errors. LVM has its own failure domains that filesystem metrics cannot see: a thin pool can be at 100% data usage while df shows 50% free, and a mirror can run on one leg for months with no visible symptom.

LVM pvmove stuck or interrupted: temporary mirror state and how to unwind it

Mon, 01 Jan 0001 00:00:00 +0000

LVM pvmove stuck or interrupted: temporary mirror state and how to unwind it

You ran pvmove to drain a disk, it has been running for hours, and now something is wrong. Maybe the SSH session died, maybe the host rebooted, maybe someone killed the process. Now lvs shows a mirror LV you never created, and every LVM command against the volume group feels slow or blocked.

This is the pvmove temporary mirror state. It is recoverable in almost every case, but only if you unwind it the way LVM expects: resume it with a bare pvmove, or abort it with pvmove --abort. The one move that turns a tedious situation into a bad one is kill -9 on the pvmove process, or worse, deleting the mirror LV by hand.

LVM RAID mismatch count: data integrity after a scrub

Mon, 01 Jan 0001 00:00:00 +0000

LVM RAID mismatch count: data integrity after a scrub

A non-zero raid_mismatch_count means the scrub found blocks where data on one leg disagrees with data on the other leg(s). This surfaces as the m flag in position 9 of lv_attr and in the raid_mismatch_count field of lvs. It demands investigation, though some mismatches turn out to be benign.

The operational question is not “do we have mismatches” but “is this real corruption or expected noise.” RAID1 and RAID10 arrays can report non-zero mismatch counts from documented edge cases in the kernel write path that produce alignment differences in transient data areas. Mismatches appearing after an unclean shutdown are expected until the scrub completes and should not trigger paging on initial boot.

LVM RAID or mirror degraded: a leg is dead and you are one failure from data loss

Mon, 01 Jan 0001 00:00:00 +0000

LVM RAID or mirror degraded: a leg is dead and you are one failure from data loss

Your LVM RAID1 or mirrored logical volume is still serving reads and writes. Applications see no errors. Filesystems are mounted. But one leg is dead and you are running on a single copy with zero redundancy. The next disk failure, cable disconnect, or SAN path loss is total data loss.

The default raid_fault_policy is warn. When a leg fails, dmeventd logs a warning but takes no repair action. No resync is triggered, no alert fires unless your monitoring explicitly checks for degraded state. The system runs indefinitely on the surviving leg until the second failure removes all copies.

LVM RAID resync stuck: copy_percent not progressing

Mon, 01 Jan 0001 00:00:00 +0000

LVM RAID resync stuck: copy_percent not progressing

You are watching lvs output on a RAID logical volume, and copy_percent has not moved for over an hour. The array is degraded, a resync or rebuild is supposed to be underway, and the percentage is frozen. Meanwhile, the volume is running on fewer healthy legs than intended, and every minute without full redundancy is a minute where a second disk failure means data loss.

LVM reached low water mark for data device: the thin pool warning before the freeze

Mon, 01 Jan 0001 00:00:00 +0000

LVM reached low water mark for data device: the thin pool warning before the freeze

You found this line in dmesg or the journal:

device-mapper: thin: 253:4: reached low water mark for data device: sending event

This is not an error. It is the dm-thin kernel target reporting that a thin pool’s data device has crossed its low water mark and that a device-mapper event has been sent to userspace. dmeventd listens for that event and can auto-extend the pool. On a correctly configured system, you may see this message once, dmeventd extends the pool, and usage drops back below the threshold.

LVM recovering a volume group after permanent PV loss: vgreduce --removemissing

Mon, 01 Jan 0001 00:00:00 +0000

LVM recovering a volume group after permanent PV loss: vgreduce –removemissing

When a physical volume (PV) disappears from the system, the volume group (VG) enters a partial state. LVM commands start printing “Couldn’t find device with uuid” warnings, and any logical volume (LV) with extents on the missing PV becomes inaccessible (linear/striped) or degraded (mirror/RAID). Once you have confirmed the PV is permanently lost, vgreduce --removemissing cleans the missing PV out of the VG metadata.

LVM snapshot COW usage climbing: extend or remove before it overflows

Mon, 01 Jan 0001 00:00:00 +0000

LVM snapshot COW usage climbing: extend or remove before it overflows

lvs shows a traditional snapshot with snap_percent at 78%. An hour ago it was 61%. You have a fixed-size COW exception store filling up from writes to the origin volume, and when it reaches 100% the snapshot is invalidated instantly and permanently. There is no warning state, no graceful degradation, and no recovery. If a backup is reading from that snapshot, the backup dies with it.

LVM snapshot invalid: the COW exception store filled and the snapshot is gone

Mon, 01 Jan 0001 00:00:00 +0000

LVM snapshot invalid: the COW exception store filled and the snapshot is gone

You run lvs and see it: a snapshot with snap_percent at 100.00 and an I in the fifth position of lv_attr. In the kernel log there is a line like Invalidating snapshot: Unable to allocate exception. The snapshot is dead. It did not degrade, it did not warn you at the application layer, and it cannot be brought back.

LVM snapshot slowing the origin: copy-on-write write amplification

Mon, 01 Jan 0001 00:00:00 +0000

LVM snapshot slowing the origin: copy-on-write write amplification

Write latency on a logical volume doubles or worse with no obvious cause. The disks are healthy, dmesg shows no I/O errors, no RAID resync is running, and the workload has not changed. The one thing that did change: someone created a snapshot, often hours or days ago, usually for a backup that has long since finished.

This is a traditional (thick) LVM snapshot working as designed. While the snapshot exists, the first write to every chunk of the origin triggers a copy-on-write (COW) cycle: read the original chunk, write it to the snapshot’s exception store, then write the new data. One application write becomes a minimum of three physical I/Os. Under write-heavy load, origin latency doubles or worse, and it degrades further as the exception store fills.

LVM thin pool auto-extend not working: threshold 100 means disabled

Mon, 01 Jan 0001 00:00:00 +0000

LVM thin pool auto-extend not working: threshold 100 means disabled

Your thin pool hit 100% data usage, writes to every thin LV in the pool started failing, and the auto-extend you thought was protecting you never fired. Or you are reading this before that happens, auditing a pool that has been “protected” for months without anyone ever confirming the mechanism works.

The most common root cause is a one-line surprise in /etc/lvm/lvm.conf: the default value of thin_pool_autoextend_threshold is 100, and a threshold of 100 does not mean “extend when the pool is full”. It means auto-extend is disabled. The commented-out line # thin_pool_autoextend_threshold = 100 in the shipped config looks like a sensible default. It is the off switch.

LVM thin pool check needed: thin_check and lvconvert --repair

Mon, 01 Jan 0001 00:00:00 +0000

LVM thin pool check needed: thin_check and lvconvert –repair

A thin pool that reports “check needed” is telling you the kernel no longer trusts the pool’s metadata B-tree. The pool may still be serving I/O, or it may already be erroring writes. Either way, the wrong move (resizing metadata, forcing activation, running the wrong repair path on an oversized metadata LV) can turn a recoverable state into permanent data loss.

LVM thin pool metadata full: the exhaustion that can corrupt the pool

Mon, 01 Jan 0001 00:00:00 +0000

LVM thin pool metadata full: the exhaustion that can corrupt the pool

Your thin pool’s metadata_percent has hit 100. Writes to every thin LV in the pool are failing or hanging, and yet lvs shows data_percent at 40%. That is not a contradiction. It is the defining feature of thin pool metadata exhaustion, and it is the LVM failure mode most likely to cost you data.

Metadata exhaustion is more dangerous than data exhaustion for one reason: a full data area stops new allocations, but a full metadata area can leave the pool structurally corrupted. The metadata LV holds the block mapping tables for every thin LV and snapshot in the pool. When the kernel can no longer allocate a metadata block mid-transaction, it aborts the transaction and switches the pool to read-only mode. From that point, recovery via lvconvert --repair is possible but not guaranteed, and the lvmthin(7) man page warns plainly that data from thin LVs may ultimately be unrecoverable.

LVM thin pool out of data space: every thin volume freezes at once

Mon, 01 Jan 0001 00:00:00 +0000

LVM thin pool out of data space: every thin volume freezes at once

The symptom usually arrives as “the server hung” rather than “the storage filled up.” Processes writing to any thin volume in the pool go into uninterruptible sleep (D state). Applications stop mid-write without returning errors at first. Databases stall, VMs pause, containers wedge. If root or swap sits on a thin LV in the pool, the whole system becomes unresponsive and SSH sessions freeze.

LVM thin pool overprovisioning: 60% full can still be dangerous

Mon, 01 Jan 0001 00:00:00 +0000

LVM thin pool overprovisioning: 60% full can still be dangerous

Your thin pool shows 60% data usage. Every dashboard is green, no alert has fired, and the pool has sat around that number for weeks. Then a single process on one thin volume starts writing aggressively, and eleven minutes later every thin LV in the pool is frozen: databases crash, VMs pause, and writes across the entire pool queue for 60 seconds and then fail with I/O errors.

LVM thin pool queue_if_no_space: why a full pool hangs instead of erroring

Mon, 01 Jan 0001 00:00:00 +0000

LVM thin pool queue_if_no_space: why a full pool hangs instead of erroring

The server looks dead. Processes pile up in uninterruptible sleep, shells that touch certain filesystems never return, and even lvs hangs. There are no I/O errors in the application logs, no kernel oops, nothing in dmesg that screams hardware. Teams burn the first hour of this incident investigating a kernel or storage fault, when the actual cause is an LVM thin pool that ran out of data space.

LVM thin pool space not reclaimed: discard, TRIM, and fstrim

Mon, 01 Jan 0001 00:00:00 +0000

LVM thin pool space not reclaimed: discard, TRIM, and fstrim

You deleted 200 GB of files from a filesystem on a thin LV. df shows the space as free, but lvs still reports the thin pool data_percent exactly where it was. Nothing is broken in the reporting: this is how thin provisioning works. Deleting a file only updates filesystem metadata. The pool blocks that held the data stay allocated until a discard (TRIM) operation travels from the filesystem, through the thin LV, into the pool.

LVM thin snapshots vs old-style COW snapshots: which failure mode you inherit

Mon, 01 Jan 0001 00:00:00 +0000

LVM thin snapshots vs old-style COW snapshots: which failure mode you inherit

LVM offers two snapshot mechanisms that look similar from the command line but fail in fundamentally different ways. The choice between them is about which failure mode you inherit when something goes wrong.

Old-style COW (copy-on-write) snapshots allocate a fixed exception store at creation time. When that store fills to 100%, the snapshot is invalidated permanently and silently. The origin volume continues running. You lose the snapshot and whatever backup or rollback point it represented, but production keeps going.

LVM unexpected metadata changes: auditing pvcreate, vgcreate, and lvcreate

Mon, 01 Jan 0001 00:00:00 +0000

LVM unexpected metadata changes: auditing pvcreate, vgcreate, and lvcreate

You found a logical volume nobody remembers creating. Or a volume group grew a new physical volume overnight. Or lvdisplay output does not match your configuration management records. None of your change tickets explain it, and now you need to answer two questions fast: what changed, and who changed it.

LVM metadata changes are a high-signal security event because every pvcreate, vgcreate, lvcreate, lvremove, and vgextend requires root. An unexpected LVM change outside a change window means root ran that command, which narrows the possibilities to an authorized root process you forgot about (automation, a package script, a provisioning tool) or an unauthorized one (a compromised service that escalated, a rogue script, an attacker with a shell).

LVM vgck metadata inconsistency: PVs disagree about the volume group

Mon, 01 Jan 0001 00:00:00 +0000

LVM vgck metadata inconsistency: PVs disagree about the volume group

vgck returned non-zero on a production VG, or you are seeing warnings like “Inconsistent metadata found for VG” or “ignoring metadata seqno N on /dev/sdX for seqno M on /dev/sdY.” The physical volumes in the volume group hold different versions of the VG metadata. One or more PVs are stale: they missed a metadata update because they were offline, unreachable, or had bad blocks when the write happened.

LVM volume group is partial: operating a VG with a missing PV

Mon, 01 Jan 0001 00:00:00 +0000

LVM volume group is partial: operating a VG with a missing PV

You ran vgs or lvs and saw something wrong. The VG attributes show a p in the fourth position (wz--pn- instead of wz--n-). One or more PVs show as [unknown] in pvs output. Commands that touch the VG now print warnings about a missing device and may refuse to run.

The VG is in partial mode. LVM metadata still references a PV UUID that no block device on the system currently claims. The metadata itself is intact on surviving PVs, which is why you can still see the VG at all. But any LV that had physical extents on the missing PV now has a hole in its mapping table.

LVM volume group running low on free space: vg_free and runway estimation

Mon, 01 Jan 0001 00:00:00 +0000

LVM volume group running low on free space: vg_free and runway estimation

Your volume group still works: LVs are active, filesystems are mounted, applications are writing. But vgs shows vg_free shrinking week over week, and the trend says you hit zero sometime next month. VG exhaustion is a cliff edge: there is no graceful degradation between 99% and 100% full. Everything works until an allocation request cannot be satisfied, and then it fails immediately. Watching vg_free is the warning the cliff never gives you.

LXC Containers

Mon, 01 Jan 0001 00:00:00 +0000

Lynis audit reports

Mon, 01 Jan 0001 00:00:00 +0000

Lynis Audit Reports Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Lynis Audit Reports Monitoring

What Is Lynis?

Lynis is a renowned security auditing tool designed to perform comprehensive scans and audits on Unix-based systems. Its diverse range of checks helps ensure robust system security and compliance, making it a crucial element in any security-conscious IT infrastructure.

Monitoring Lynis With Netdata

To effectively monitor Lynis audit reports, Netdata utilizes an openmetrics (prometheus) exporter. This setup allows Netdata to gather rich metrics by periodically sending HTTP requests to lynis_exporter. Netdata is adept at ingesting data from any Prometheus exporter, empowering users to visualize information with automated dashboards and receive instant alerts without needing a separate Prometheus server or Grafana installation.

M3DB

Mon, 01 Jan 0001 00:00:00 +0000

macOS

Mon, 01 Jan 0001 00:00:00 +0000

macOS

Mon, 01 Jan 0001 00:00:00 +0000

macOS Unified Logs

Mon, 01 Jan 0001 00:00:00 +0000

Madge Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Maipu Electric Industrial Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Manjaro Linux

Mon, 01 Jan 0001 00:00:00 +0000

Marc Hirsch SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

MariaDB

Mon, 01 Jan 0001 00:00:00 +0000

Matrix

Mon, 01 Jan 0001 00:00:00 +0000

Mattermost

Mon, 01 Jan 0001 00:00:00 +0000

MaxMind GeoIP / GeoLite2

Mon, 01 Jan 0001 00:00:00 +0000

MaxScale

Mon, 01 Jan 0001 00:00:00 +0000

MaxScale Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

MaxScale Monitoring

What Is MaxScale?

MaxScale is a database proxy that manages the traffic between client applications and a set of database servers. It is an integral part of the MariaDB ecosystem, coordinating database operations and optimizing performance and scalability for distributed database environments. Learn more about MaxScale.

Monitoring MaxScale With Netdata

Netdata provides a comprehensive MaxScale monitoring tool that helps you keep an eye on the performance and health of your MaxScale instances. The Netdata Agent collects real-time metrics from MaxScale, enabling you to monitor MaxScale servers efficiently. With Netdata’s extensive visualizations and intuitive interface, you can diagnose root causes of performance issues swiftly and effectively. For hands-on experience, check out our Live Demo or sign up for a free trial.

Mcafee Associates Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mcafee Formerly Secure Computing Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mcafee Inc Formerly Network Associates Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mcafee WEB Gateway

Mon, 01 Jan 0001 00:00:00 +0000

MD RAID

Mon, 01 Jan 0001 00:00:00 +0000

Media5 Corporation M5 Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Meeting Netdata

Mon, 01 Jan 0001 00:00:00 +0000

Meeting with Constantine

Mon, 01 Jan 0001 00:00:00 +0000

Meeting with Netdata

Mon, 01 Jan 0001 00:00:00 +0000

Meeting with Onboarding team

Mon, 01 Jan 0001 00:00:00 +0000

Meeting With Onboarding Team

Mon, 01 Jan 0001 00:00:00 +0000

Meeting with Satya

Mon, 01 Jan 0001 00:00:00 +0000

Meeting with Shyam

Mon, 01 Jan 0001 00:00:00 +0000

Meeting with Stuart

Mon, 01 Jan 0001 00:00:00 +0000

Mega System Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

MegaCLI MegaRAID

Mon, 01 Jan 0001 00:00:00 +0000

MegaCLI MegaRAID Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

MegaCLI MegaRAID Monitoring

What Is MegaCLI MegaRAID?

MegaCLI is a powerful command-line utility from Broadcom, primarily used to manage and monitor MegaRAID storage controllers. Learn more.

Monitoring MegaCLI MegaRAID With Netdata

Using Netdata as your MegaCLI monitoring tool allows you to monitor the health and performance of your RAID adapters, physical drives, and backup batteries seamlessly. Netdata’s real-time monitoring capabilities make it one of the most effective tools for monitoring MegaCLI.

Megapac SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Meilisearch

Mon, 01 Jan 0001 00:00:00 +0000

Meilisearch Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Meilisearch Monitoring

What Is Meilisearch?

Meilisearch is a powerful, open-source, and real-time search engine designed for swift search interactions across large datasets. Its lightweight design and focus on performance and developer-friendliness make it an ideal choice for modern applications demanding fast and accurate search capabilities.

Monitoring Meilisearch With Netdata

When it comes to tools for monitoring Meilisearch, Netdata stands out as a performance-efficient solution. Netdata uses an openmetrics (Prometheus) exporter to monitor Meilisearch, such as the Meilisearch Exporter, which efficiently gathers search engine metrics. With Netdata, you can ingest data from any Prometheus-compatible exporter, automatically generating on-demand dashboards and alerts without requiring a dedicated Prometheus server or Grafana setup. This streamlined monitoring process ensures you receive real-time insights without the need for extensive infrastructure.

Meinberg SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Melco Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mellanox Technologies Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Memcached

Mon, 01 Jan 0001 00:00:00 +0000

Memcached alive but not responding: the silent process hang

Mon, 01 Jan 0001 00:00:00 +0000

Memcached alive but not responding: the silent process hang

The port check passes. The process shows up in ps. The kernel still accepts TCP on 11211. But every stats and version probe times out, and dashboards show cmd_get and cmd_set falling off a cliff. This is the silent process hang: memcached is effectively down while appearing alive to every shallow health check.

Most liveness checks stop at “is something listening on the port?” That question is answered by the kernel’s TCP stack, not by memcached’s worker threads. A frozen main thread, a stuck slab rebalancer, a process suspended by SIGSTOP, or an OOM kill in progress can all leave the listen backlog accepting connections while no command ever executes. Clients connect, send a command, and wait forever.

Memcached auth_errors: SASL failures, brute force, and the no-auth default

Mon, 01 Jan 0001 00:00:00 +0000

Memcached auth_errors: SASL failures, brute force, and the no-auth default

Two counters, present since memcached 1.4.4, are the only native signal the daemon gives you about authentication activity: auth_cmds and auth_errors. They count attempts and failures. They do not tell you who attempted, from where, or whether authentication is even enforced.

The critical interpretation rule: if authentication is not enabled (no -S for SASL, no -Y for ASCII token auth

Memcached bound to 0.0.0.0: unauthenticated access on an open port

Mon, 01 Jan 0001 00:00:00 +0000

Memcached bound to 0.0.0.0: unauthenticated access on an open port

Memcached was designed for trusted internal networks. It has no authentication by default, no per-key access control, and no per-connection logging. Anyone who can open a TCP connection to the daemon can read every cached value, overwrite any key, enumerate key names and sizes, and issue flush_all to wipe the entire cache in one command. Session tokens, PII, and application secrets cached in plaintext are all readable by that caller.

Memcached cache stampede: a hot key expires and the backend takes the hit

Mon, 01 Jan 0001 00:00:00 +0000

Memcached cache stampede: a hot key expires and the backend takes the hit

A cache stampede happens when a popular cached key expires or is evicted and many concurrent requests miss at the same instant. Each miss falls through to the backend independently. The backend, usually sized to handle only the small fraction of traffic that misses the cache, receives the full load for that key at once. Memcached itself is not malfunctioning. It is correctly reporting misses and returning them as fast as it can. The victim is the backend, and the fix lives in the application layer.

Memcached cas_badval climbing: check-and-set contention and lost updates

Mon, 01 Jan 0001 00:00:00 +0000

Memcached cas_badval climbing: check-and-set contention and lost updates

cas_badval tracks check-and-set operations that failed because the CAS unique token changed between your gets read and your cas write. Each increment is a rejected write: another writer modified the key first, and memcached correctly refused the stale update to prevent a lost update.

A sustained cas_badval rate above 10% of total CAS attempts, computed as cas_badval / (cas_hits + cas_misses + cas_badval), indicates meaningful write contention. The application is spending CPU on writes that never land, and unbounded retry logic can amplify the problem: every failed CAS triggers an immediate re-read and re-write, increasing load on both the client and memcached without making progress.

Memcached command rate anomalies: cmd_get and cmd_set spikes and sudden drops

Mon, 01 Jan 0001 00:00:00 +0000

cmd_get, cmd_set, and cmd_touch are cumulative counters in memcached. They increase monotonically from process start and are meaningless as raw numbers. The signal is the derived rate: the delta between two samples divided by the interval. A sudden change in that rate is almost never the root cause. It is a proxy for something that changed upstream (clients stopped or started sending traffic), inside the cache (a hot key expired, the cache was flushed), or in the application logic (a write loop, a deploy with new key patterns).

Memcached conn_yields rising: one client's pipeline starving the others

Mon, 01 Jan 0001 00:00:00 +0000

conn_yields is climbing on a memcached instance. The name suggests thread contention or a locking bug. It is neither.

conn_yields is memcached’s per-connection fairness throttle. Each worker thread processes requests from its assigned connections in a libevent loop. The -R flag (default 20, since memcached 1.4.0) caps how many sequential requests a worker pulls from a single connection in one event loop pass. When a connection hits that cap, the worker yields it: moves it to the back of the processing queue, serves other connections, then comes back. The conn_yields stat increments on each yield.

Memcached connection churn: total_connections racing and TIME_WAIT buildup

Mon, 01 Jan 0001 00:00:00 +0000

Memcached connection churn: total_connections racing and TIME_WAIT buildup

total_connections is a cumulative counter that only moves up. When it climbs fast while curr_connections stays flat, clients are opening a fresh TCP connection per request and closing it immediately instead of pooling. Each cycle costs a syscall pair on both ends, a socket on both ends, and on Linux the closed socket sits in TIME_WAIT on the client host for roughly 60 seconds.

Memcached connection limit reached: accepting_conns=0 and clients being refused

Mon, 01 Jan 0001 00:00:00 +0000

Memcached connection limit reached: accepting_conns=0 and clients being refused

Memcached enforces a hard ceiling on concurrent client connections via the -c flag (default 1024). When curr_connections reaches that limit, the daemon flips accepting_conns to 0, disables the listen socket, and turns away every new TCP connection. Clients see connection-refused errors or timeouts, fall through to the backend, and the cache stops working for anyone not already connected.

This is purely a connection-slot problem. Memory can be at 40% of limit_maxbytes, CPU idle, evictions zero, and the process fully serving existing connections. The only thing wrong is that the connection table is full.

Memcached connection refused: telling a dead process from a hung or full one

Mon, 01 Jan 0001 00:00:00 +0000

Memcached connection refused: telling a dead process from a hung or full one

“Connection refused” from a memcached client is commonly misdiagnosed. Operators run a TCP port check, see that the port is open, and act on that single data point. A port test tells you whether the kernel accepts a TCP SYN on 11211, not whether memcached is processing commands.

Three distinct failure modes all surface to clients as “cannot reach memcached”:

Memcached CPU saturation: the per-thread ceiling that aggregate graphs hide

Mon, 01 Jan 0001 00:00:00 +0000

Memcached CPU saturation: the per-thread ceiling that aggregate graphs hide

Memcached is rarely CPU-bound. At normal request rates, worker threads spend most of their time in epoll waits. When CPU climbs on a memcached host, the reflex is to check aggregate rusage and conclude the daemon is busy. That reflex misses the actual failure mode.

Memcached dispatches each accepted connection round-robin to one of -t worker threads (default 4). Once assigned, a connection lives on that worker for its lifetime. The worker runs its own libevent loop and serves its assigned connections exclusively. If one worker saturates at 100% of a core, every connection pinned to that worker sees elevated latency. The other workers may be at 20%. The aggregate graph reads roughly 40%. Three quarters of your clients are fine.

Memcached curr_connections climbing: connection leaks and missing pooling

Mon, 01 Jan 0001 00:00:00 +0000

Memcached curr_connections climbing: connection leaks and missing pooling

curr_connections trends upward with no corresponding traffic increase, edging toward the -c ceiling (default 1024). When it hits the limit, the listen socket disables: accepting_conns flips to 0 and new client TCP connects are refused. To the application it looks like memcached is down. To memcached it is idle and healthy on the connections it already holds.

Most of the time this is not a memcached bug. It is a client-side problem: connections opened and never returned, a broken or absent pool, or a fleet of application instances each holding more sockets than you accounted for. The server is the victim, not the cause.

Memcached evicted_time low: distinguishing healthy turnover from cache thrash

Mon, 01 Jan 0001 00:00:00 +0000

Memcached evicted_time low: distinguishing healthy turnover from cache thrash

A high global eviction counter is one of the most over-alerted memcached signals. Operators see evictions climbing and page someone at 3 a.m. or reflexively add memory. Both reactions are usually wrong. Raw eviction count says nothing about whether the evicted items mattered. The discriminator that turns “evictions are high” into “evictions are harmful” is evicted_time, reported per slab class in the output of stats items.

Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads

Mon, 01 Jan 0001 00:00:00 +0000

Memcached evicted_unfetched and expired_unfetched: caching data nobody ever reads

Memcached exposes two counters that most operators never look at, and both describe the same failure: the application is storing data that nobody ever reads. evicted_unfetched counts items evicted from the LRU before they were ever touched by a read. expired_unfetched counts items that lived their full TTL and expired without ever being read. Both are efficiency signals, not reliability signals. They will not tell you the cache is down. They tell you the cache is being used as a write-only buffer, and that write-only data is displacing items that might actually be read.

Memcached evicting with memory to spare: the slab-class imbalance trap

Mon, 01 Jan 0001 00:00:00 +0000

Memcached evicting with memory to spare: the slab-class imbalance trap

Evictions are climbing on your memcached instance. Global memory is at 55% of limit_maxbytes. The natural response is to add memory, restart the daemon, or hunt for an eviction storm. None of those help here, because the cache is not out of memory in aggregate. One slab class is at 100% capacity and discarding recently-active items, while several other classes sit mostly empty.

Memcached eviction cascade: when a full cache overloads the backend

Mon, 01 Jan 0001 00:00:00 +0000

Memcached eviction cascade: when a full cache overloads the backend

The most common memcached-related production incident is not a memcached crash. The process stays up, serves hits at sub-millisecond latency, CPU is flat, and memory looks fine in aggregate. What breaks is the backend.

An eviction cascade starts when the working set outgrows the cache. Memory fills, evictions accelerate, and every evicted item becomes a future miss. Those misses fall through to the database, which was sized for cache-assisted load, not raw traffic. The backend saturates, latency climbs stack-wide, and slow responses drive retries that add still more load. It is a positive feedback loop, and memcached is not where it closes.

Memcached evictions climbing: the cache is full and discarding live data

Mon, 01 Jan 0001 00:00:00 +0000

Memcached evictions climbing: the cache is full and discarding live data

The evictions counter is climbing. At least one slab class is saturated, and the daemon is removing valid, non-expired items to make room for new sets. The severity question is whether the cache is discarding cold data the application no longer needs, or live data that will be requested again within seconds.

Adding memory is the right response only when the cache is globally undersized. If the problem is slab calcification (memory concentrated in the wrong size classes), adding RAM does nothing useful and may mask the real issue. First determine whether evictions are healthy LRU turnover or harmful thrash, and whether the pressure is global or per-slab.

Memcached flush_all: the accidental cache wipe and its cold-start blast radius

Mon, 01 Jan 0001 00:00:00 +0000

Memcached flush_all: the accidental cache wipe and its cold-start blast radius

You are paged because hit ratio collapsed and the backend database is saturating. The memcached process is up, uptime is stable, memory looks allocated, and there are no evictions. The signature is an instant hit ratio cliff with no gradual decline. The cause is almost certainly a flush_all command, and the cmd_flush counter will confirm it.

A single flush_all invalidates every item in the cache. It does not lock the server, does not free memory immediately, and does not require authentication by default. Anyone with TCP access to port 11211 can issue it. The result is a cold-cache thundering herd: every subsequent GET misses and hits the backend simultaneously. If the backend is sized for cache-assisted load, it just received a multiple of its designed capacity.

Memcached hash table expansion: hash_is_expanding, memory spikes, and item churn

Mon, 01 Jan 0001 00:00:00 +0000

Memcached hash table expansion: hash_is_expanding, memory spikes, and item churn

When you first see hash_is_expanding=1 in memcached stats, it looks like a warning. It is not. The daemon is doubling its internal key hash table to accommodate more items, a routine maintenance operation that runs on a dedicated background thread (since 1.4.0) and does not block request processing. It does temporarily spike memory and add CPU load on the maintenance thread, and in specific cases it can expose underlying problems: item-count oscillation, repeated expansion, or an overstretched memory budget.

Memcached high miss rate: separating cold start, new key patterns, and memory pressure

Mon, 01 Jan 0001 00:00:00 +0000

Memcached high miss rate: separating cold start, new key patterns, and memory pressure

A high miss rate on memcached is frequently misdiagnosed. The instinct is to assume the cache is too small and add memory. That instinct is usually wrong. Random or unique key access produces a near-0% hit rate no matter how healthy the cache is. A freshly restarted cache legitimately runs near 0% until it warms. A deploy that changed key naming produces instant misses on the new keys without any memory pressure.

Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness

Mon, 01 Jan 0001 00:00:00 +0000

Memcached hit ratio dropping: reading get_hits, get_misses, and cache effectiveness

The cache hit ratio tells you whether memcached is earning its keep. When it drops, the backend database inherits the miss traffic, and a slow decline can cascade into a failure before anyone pages on “cache.” The ratio is also frequently read wrong: computing it from lifetime counters hides the exact degradation you are trying to catch.

What the hit ratio actually measures

The cache hit ratio is the fraction of GET lookups that found the key in cache:

Memcached hot key: one key, one thread, and lopsided load

Mon, 01 Jan 0001 00:00:00 +0000

Memcached hot key: one key, one thread, and lopsided load

Aggregate cmd_get and cmd_set rates look normal. Hit ratio is healthy. Evictions are zero. CPU across the memcached fleet is moderate. Yet a specific subset of clients reports p99 latency an order of magnitude above baseline, and in a consistent-hashing cluster one node runs noticeably hotter than the rest. Memory pressure and connection exhaustion are absent.

The signature of a memcached hot key is concentration without saturation. A tiny number of keys, sometimes a single key, take a disproportionate share of traffic: a feature flag read on every request, a viral item page, a single bot’s session record, a shared rate-limit counter. All funnel traffic to one slab class on one node.

Memcached incr/decr misses: evicted counters that silently break rate limiters and locks

Mon, 01 Jan 0001 00:00:00 +0000

Memcached incr/decr misses: evicted counters that silently break rate limiters and locks

Most memcached operators never look at incr_misses and decr_misses. They sit in the per-operation hit/miss breakdown and stay near zero. When they start climbing, the cache itself usually looks healthy: hit ratio is fine, evictions are modest, memory is not full. The damage is happening somewhere else.

The pattern is specific. The application uses memcached as an atomic counter store. Rate limiters increment a per-client counter and reject when it crosses a threshold. Distributed locks hold a lease token with a TTL and decrement on release. Quota counters track usage per tenant. These counter keys are small, hot, and short-lived. When the slab class they live in comes under memory pressure, the LRU evicts them between operations.

Memcached latency high: diagnosing slow gets when the server exposes no latency metric

Mon, 01 Jan 0001 00:00:00 +0000

Memcached latency high: diagnosing slow gets when the server exposes no latency metric

Memcached does not expose a latency histogram, p50, or p99. The standard stats output is all counters and gauges at the moment of query: no time-series, no distribution, no per-operation duration. If your application is reporting slow cache reads, the daemon itself cannot confirm or deny the problem. You must measure latency at the client, then use server-side and OS signals to localize the cause.

Memcached LRU crawler: crawler_reclaimed, disabled crawlers, and lazy expiry

Mon, 01 Jan 0001 00:00:00 +0000

Memcached LRU crawler: crawler_reclaimed, disabled crawlers, and lazy expiry

Memcached does not remove items when their TTL expires. An expired item holds its slab slot until something touches it: a client request, an eviction scan, or the LRU crawler. The crawler is a background thread that walks LRU chains, finds expired items, and frees their memory before eviction pressure forces the issue.

A working crawler keeps slab slots available for new writes without evicting live data. Every expired item it reclaims is one fewer item evicted under pressure. Without it, expired items that are never accessed and never reach the eviction tail waste memory indefinitely.

Memcached maxconns and ulimit: why the default 1024 is too low for production

Mon, 01 Jan 0001 00:00:00 +0000

Memcached maxconns and ulimit: why the default 1024 is too low for production

Memcached is up, the port is open, existing connections work fine. But new clients get connection refused. Their requests fall through to the backend, backend load climbs, and the cache looks healthy from the outside: CPU is low, memory is nowhere near the limit, and the slab allocator is not evicting anything.

The most common cause is the default -c 1024 (maxconns) limit. It looks generous for a single application instance, but becomes a hard cliff-edge once you scale from 10 to 50 instances, each with its own connection pool. Monitoring agents, health checks, and load balancer probes also consume connections against this budget.

Memcached memory utilization: bytes vs limit_maxbytes and why the global number lies

Mon, 01 Jan 0001 00:00:00 +0000

Memcached memory utilization: bytes vs limit_maxbytes and why the global number lies

The bytes and limit_maxbytes fields from stats look like a clean fill gauge. Divide one by the other and you know how full the cache is. Many dashboards and alerts are built on exactly that ratio.

This article covers why the global ratio lies, what it actually measures, and which per-slab signals confirm real pressure. bytes includes per-item overhead and never quite reaches the ceiling, allocation happens in 1MB page jumps rather than smoothly, and the pool is partitioned into fixed-size slab classes that saturate independently. A cache at 50% global utilization can be evicting live, actively-requested data.

Memcached monitoring checklist: the signals every production cache needs

Mon, 01 Jan 0001 00:00:00 +0000

Memcached monitoring checklist: the signals every production cache needs

A production reference for memcached signals, organized by monitoring maturity. Use it to audit what you collect, identify gaps, and prioritize additions.

Levels are cumulative: Level 2 includes Level 1. Level 1 detects crashes and restarts but misses slab calcification and eviction quality. Level 3 catches those before they hit the backend. Level 4 adds signals teams adopt after being burned by subtle failures.

Memcached monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Memcached monitoring maturity model: from survival to expert

Memcached exposes most of its internal state through plain text counters. Most teams stop at hit ratio, global memory, and eviction rate, then discover during an incident that the signal they needed was buried in stats items or /proc/<pid>/status. The four-level model below stages the path from “is the process alive” to “is the segmented LRU moving items between HOT and COLD the way the workload expects”.

Memcached network saturation: large values, multiget, and a maxed-out NIC

Mon, 01 Jan 0001 00:00:00 +0000

Memcached network saturation: large values, multiget, and a maxed-out NIC

Memcached latency is climbing for every client, but the host looks healthy. CPU is well below saturation, memory utilization is far from the limit, evictions are zero, and curr_connections is well below -c. The daemon answers version and stats instantly. Yet every application reports slow gets, and the slowdown is uniform across keys and clients.

When latency degrades uniformly and the usual suspects are clear, look at the wire. Memcached pulls values from RAM and writes them to a socket. If the response stream exceeds what the NIC can push, the kernel queues, TCP congestion control engages, and every client waits. The cache is fine; the link is the bottleneck.

Memcached reclaimed vs evictions: reading memory pressure through reclamation

Mon, 01 Jan 0001 00:00:00 +0000

Memcached reclaimed vs evictions: reading memory pressure through reclamation

Most operators learn one signal first: evictions. It goes up, the cache is full, hit ratio drops, someone pages. The counter next to it, reclaimed, tells you how often memcached made room for a new item by reusing an expired item’s slot instead of throwing out live data. Read together, they describe the same process from two angles: how memcached finds space for new writes.

Memcached response_obj_oom: connections killed for lack of buffer memory

Mon, 01 Jan 0001 00:00:00 +0000

Memcached response_obj_oom: connections killed for lack of buffer memory

If response_obj_oom is climbing in your stats output, the daemon is closing client connections because it cannot allocate internal response buffers. This is not an item-storage problem and not an eviction problem. A cache can be otherwise healthy (zero evictions, free slab memory, low CPU) and still kill clients over this.

The stat has existed since 1.6.0, when memcached reworked its connection buffer management to allocate buffers on demand instead of reserving them per connection. That cut idle connection overhead but introduced a new failure surface: when the pool of memory for response objects runs out, the daemon closes the connection rather than blocking or queueing. The close is immediate when the allocation fails. There is no retry, no backpressure, no graceful degradation. Applications see timeouts or resets against a daemon that still answers version and stats probes.

Memcached RSS above the -m limit: connection buffers, hash table, and fragmentation

Mon, 01 Jan 0001 00:00:00 +0000

Memcached RSS above the -m limit: connection buffers, hash table, and fragmentation

Operators set -m to cap cache memory, then notice RSS is 40 percent or more above that number. This is not a leak. The -m flag bounds only the slab allocator, the pool that stores cached items. RSS also includes the hash table, per-connection buffers, worker thread stacks, and allocator overhead.

The bytes / limit_maxbytes gauge in memcached stats describes cache memory: the keys and values your application stores. RSS, reported as VmRSS in /proc/<pid>/status, describes the entire process. They measure different scopes and will almost never match.

Memcached segmented LRU: HOT/WARM/COLD tiers and what the move counters tell you

Mon, 01 Jan 0001 00:00:00 +0000

Memcached segmented LRU: HOT/WARM/COLD tiers and what the move counters tell you

Since memcached 1.5.0, every slab class runs a segmented LRU instead of a single flat queue. Items land in HOT, WARM, COLD, and optionally TEMP tiers, and a background thread moves them between tiers based on access patterns. The goal is to protect frequently-accessed items from being evicted by one-off scans that touch cold data.

The move counters under stats items describe how well that sorting is working. moves_to_cold counts items aging out of active use, moves_to_warm counts items rescued from COLD by a re-access, and moves_within_lru counts re-ranking within WARM. The ratios between these counters explain why a slab class evicts the way it does, even when global memory looks fine.

Memcached SERVER_ERROR object too large for cache: items over the max item size

Mon, 01 Jan 0001 00:00:00 +0000

Memcached SERVER_ERROR object too large for cache: items over the max item size

SERVER_ERROR object too large for cache is the protocol-level error memcached returns when a store operation targets a value larger than the daemon’s configured maximum item size. The default ceiling is 1 MB (1048576 bytes), set by the -I flag. The store is rejected cleanly: nothing is written, and any pre-existing item under that key is left untouched. From the application’s side this is usually the first sign that something upstream is serializing more data than intended.

Memcached SERVER_ERROR out of memory storing object: stores rejected instead of evicting

Mon, 01 Jan 0001 00:00:00 +0000

Memcached SERVER_ERROR out of memory storing object: stores rejected instead of evicting

When a memcached client receives SERVER_ERROR out of memory storing object, the daemon refused a SET because it could not allocate a chunk for the item. The store_no_memory counter increments. Under default operation, memcached evicts LRU items from the relevant slab class to make room. This error means eviction was either disabled or could not produce a free chunk for that allocation.

Memcached slab calcification: pages locked to the wrong item sizes after a workload shift

Mon, 01 Jan 0001 00:00:00 +0000

Memcached slab calcification: pages locked to the wrong item sizes after a workload shift

Hit ratio is dropping. Evictions are climbing. But global memory utilization sits at 50 to 70 percent, and adding memory does nothing. This is the signature of slab calcification: whole megabyte pages are locked into slab classes that serve item sizes the workload no longer produces, while the classes for the new sizes are starved and evicting actively used data.

Memcached slab_automove and slab_reassign: rebalancing pages between classes

Mon, 01 Jan 0001 00:00:00 +0000

Memcached slab_automove and slab_reassign: rebalancing pages between classes

Memcached’s slab allocator divides its memory budget into 1MB pages and permanently assigns each page to a slab class based on item size. Once assigned, a page traditionally stayed locked to that class forever. If the workload’s item-size distribution shifted after deployment, you got slab calcification: one class full and evicting while others sat idle with free chunks. Global memory utilization looked healthy while cache effectiveness collapsed for the saturated size range.

Memcached swapping: why any VmSwap on an in-memory cache is an incident

Mon, 01 Jan 0001 00:00:00 +0000

Memcached swapping: why any VmSwap on an in-memory cache is an incident

A nonzero VmSwap value in /proc/<pid>/status for a memcached process is a production incident, not a tuning concern. Every access to a swapped page costs milliseconds instead of nanoseconds, so a fraction of your cache lookups runs at disk speed. The point of an in-memory cache is lost for those items, and the resulting latency outliers are random and intermittent, almost impossible to trace from the application layer without checking swap first.

Memcached UDP amplification: udpport, CVE-2018-0115, and internet exposure

Mon, 01 Jan 0001 00:00:00 +0000

Memcached UDP amplification: udpport, CVE-2018-1000115, and internet exposure

You are here because a third party reported your server in a DDoS attack, a security scan flagged UDP 11211, or you are auditing an older deployment. The condition is the same in every case: memcached has UDP enabled (udpport is non-zero) and is reachable from a network you do not control.

This is CVE-2018-1000115. In early 2018 it powered reflection attacks peaking at 1.3 to 1.7 Tbps. A single small UDP GET request with a spoofed source IP can trigger a response of hundreds of kilobytes from an unauthenticated memcached instance, with observed amplification factors around 51,000x. The upstream fix in 1.5.6 (Feb 2018) made udpport default to 0, but that does not retroactively rewrite init scripts, container images, or distro configs.

Memcached unexpected restart: uptime reset, wiped cache, and the cold-start backend spike

Mon, 01 Jan 0001 00:00:00 +0000

Memcached unexpected restart: uptime reset, wiped cache, and the cold-start backend spike

The uptime stat in memcached’s stats output is a seconds-since-process-start counter. When it drops from hours or days back to near-zero, the process restarted. Because memcached has no persistence, a restart is a full cache wipe: every item is gone, curr_items falls to zero, and hit ratio collapses to 0%.

The restart is rarely the incident. The incident is the cold-start backend spike that follows. Every cache miss now hits the origin database or service at full production volume. If the backend was sized assuming 90%+ cache offload, it now sees several times its normal read load.

Memory modules (DIMMs)

Mon, 01 Jan 0001 00:00:00 +0000

Memory Statistics

Mon, 01 Jan 0001 00:00:00 +0000

Memory Statistics (Win)

Mon, 01 Jan 0001 00:00:00 +0000

Memory Usage

Mon, 01 Jan 0001 00:00:00 +0000

Memotec Communications SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Memotec Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Meraki

Mon, 01 Jan 0001 00:00:00 +0000

Meraki Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Merlin Gerin SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Meru Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mesos

Mon, 01 Jan 0001 00:00:00 +0000

Mesos Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Mesos Monitoring

What Is Mesos?

Apache Mesos is a powerful cluster manager that simplifies resource management and task scheduling across distributed systems. It’s widely used to organize and optimize processing tasks in cloud computing environments, taking the complexity out of managing large scale data centers.

Monitoring Mesos With Netdata

To monitor Mesos, Netdata leverages an openmetrics (prometheus) exporter, specifically the Mesos exporter. Netdata stands out by ingesting data from any Prometheus exporter, providing automated dashboards and alerts without the need for a separate Prometheus server or Grafana setup. This integration with Netdata enables users to effortlessly track performance metrics and ensure efficient resource allocation across all running instances. Discover the potential by trying out our Live Demo.

MessageBird

Mon, 01 Jan 0001 00:00:00 +0000

Metaswitch Networks Ltd Formerly Data Connection Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

MetricFire

Mon, 01 Jan 0001 00:00:00 +0000

Metro Ethernet Forum SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Micom Communication Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Microbursts: catching sub-second congestion that minute averages hide

Mon, 01 Jan 0001 00:00:00 +0000

Microbursts: catching sub-second congestion that minute averages hide

Your switches are dropping packets. The utilization charts say everything is fine. Interface counters show moderate load, error rates are clean, and no congestion alert has fired. But applications report retransmissions, latency spikes, and intermittent connectivity. The problem resolved between polls.

A microburst is a short, intense spike of traffic that fills a switch egress queue faster than the queue can drain. The burst may last 50 milliseconds or less, but during that window the queue overflows and packets are tail-dropped. By the time your SNMP poller arrives 60 or 300 seconds later, the burst is over, the queue has drained, and interface utilization has been averaged down to an unremarkable number.

Microchip Technology Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Micromuse Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Microsens GmbH Co KG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Microsoft Exchange Server Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Microsoft Exchange Server

Microsoft Exchange Server is a mail server and calendaring server that runs on Windows Server operating systems. It enables users to send and receive email messages, schedule appointments, store contacts, and access shared mailboxes and calendars within an organization.

Exchange Server Monitoring

To monitor Microsoft Exchange Server, you need to collect and analyze various performance metrics that reflect the health and performance of your server. Some of these metrics include:

Microsoft SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Microsoft SQL Server

Mon, 01 Jan 0001 00:00:00 +0000

Microsoft SQL Server (MSSQL) Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Microsoft SQL Server

Microsoft SQL Server is a relational database management system (RDBMS) developed by Microsoft. It is a software product that stores and retrieves data as requested by other software applications. It can run on various platforms, such as Windows, Linux, and Azure.

Monitoring Microsoft SQL Server

To holistically monitor Microsoft SQL Server, you need to track various aspects of its performance, health, and availability. Some of the key metrics to monitor include system level metrics about the impact of SQL Server such as:

Microsoft SQL Server monitoring checklist: the signals every production instance needs

Mon, 01 Jan 0001 00:00:00 +0000

Microsoft SQL Server monitoring checklist: the signals every production instance needs

Most SQL Server outages are not exotic. The transaction log fills because a backup job silently stopped. A sleeping session with an open transaction blocks forty other sessions until the worker pool runs dry. TempDB runs out of space and every database on the instance stalls at once. All of these are visible hours or days in advance if you collect the right signals. Most teams do not.

Microsoft SQL Server monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Microsoft SQL Server monitoring maturity model: from survival to expert

Most SQL Server outages are not exotic. The transaction log fills because nobody noticed log backups stopped. A head blocker sits idle with an open transaction while the worker thread pool drains. Error 825 appears in the error log for weeks before the disk actually fails. In each case, the signal was available; the monitoring just was not looking at it.

Microsoft Teams

Mon, 01 Jan 0001 00:00:00 +0000

Microsoft Teams

Mon, 01 Jan 0001 00:00:00 +0000

Microwave Data Systems SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Microwave Networks Incorporated SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mikom GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

MikroTik Licensing

Mon, 01 Jan 0001 00:00:00 +0000

Mikrotik Router

Mon, 01 Jan 0001 00:00:00 +0000

Mikrotik SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Milestone Systems A S SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mimosa Networks Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Minecraft

Mon, 01 Jan 0001 00:00:00 +0000

Minecraft Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Minecraft Monitoring

What Is Minecraft?

Minecraft is a sandbox video game that gained worldwide popularity due to its open-ended nature. It allows players to explore and create in a block-based world, using tools to build structures, gather resources, and craft items. With hundreds of millions of copies sold, Minecraft’s appeal spans across all age groups and makes it not just a game but a platform for creativity and learning.

Monitoring Minecraft With Netdata

Monitoring your Minecraft server is essential to ensure smooth gameplay and optimal performance. Netdata provides a robust solution for monitoring Minecraft using the Minecraft Exporter. By leveraging openmetrics, Netdata integrates seamlessly with any Prometheus exporter. This means you get automated dashboards, customizable alerts, and advanced analytics without needing to set up a separate Prometheus server or Grafana. This streamlined approach simplifies the monitoring process, allowing you to focus on what matters most—keeping your players happy.

MISCONF Redis is configured to save RDB snapshots - what it means and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

MISCONF Redis is configured to save RDB snapshots - what it means and how to fix it

Applications see MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk on every write. Reads still work, but SET, HSET, LPUSH, and all mutating commands are rejected.

Redis makes itself read-only when the last background save failed and stop-writes-on-bgsave-error is yes (the default). The error persists until a subsequent BGSAVE succeeds. Retrying writes will not help. The instance is protecting you from accepting writes that can never be persisted. To recover, fix the underlying persistence failure and clear the error state with a successful save.

Mission Critical Software Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mitel Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mitsubishi Electric Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Moca Multimedia Over Coax Alliance SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Modbus protocol

Mon, 01 Jan 0001 00:00:00 +0000

Modbus Protocol Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Modbus Protocol Monitoring

What Is Modbus Protocol?

The Modbus protocol is a messaging structure widely utilized in industrial automation systems to transmit data over serial lines between electronic devices. Originally developed in 1979 by Modicon (now Schneider Electric) to communicate with its PLCs, it has evolved to become a de facto standard in industries worldwide due to its simplicity and reliability.

Monitoring Modbus Protocol With Netdata

Effectively monitoring the Modbus protocol is crucial for maintaining the performance and reliability of industrial control systems. Netdata offers an intuitive way to monitor Modbus by integrating with an openmetrics (Prometheus) exporter. Specifically, the modbusrtu_exporter can be leveraged to collect vital metrics seamlessly.

MogileFS

Mon, 01 Jan 0001 00:00:00 +0000

MogileFS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

MogileFS Monitoring

What Is MogileFS?

MogileFS is a distributed file system designed for managing large volumes of files efficiently. It offers reliable file storage across multiple servers, enabling seamless scalability and redundancy. Ideal for infrastructures that require extensive data storage solutions, MogileFS ensures data replication and distribution, making it resilient against server failures.

Monitoring MogileFS With Netdata

To monitor MogileFS, Netdata utilizes a powerful tool known as an openmetrics (Prometheus) exporter. This mechanism allows Netdata to ingest data from any Prometheus exporter, providing users with automated and dynamic dashboards, notifications, and alerts—without the need for a dedicated Prometheus server or Grafana. The MogileFS Exporter is designed specifically to collect metrics that provide insights into the performance and health of MogileFS systems. These metrics are crucial for real-time monitoring, ensuring prompt detection and resolution of any issues.

Monet Systems Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB and Transparent Huge Pages: why THP must be disabled

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB and Transparent Huge Pages: why THP must be disabled

If you see a mongod startup warning about Transparent Huge Pages, or you are chasing tail latency spikes and memory fragmentation that do not correlate with cache pressure, ticket exhaustion, or slow queries, the cause may be THP. The standard advice has always been to disable it, but that is now version-dependent. MongoDB 8.0 upgraded TCMalloc to use per-CPU caches, which changes the recommendation for most x86_64 Linux deployments. For MongoDB 7.0 and earlier, disabling THP is still correct. For 8.0 on x86_64 Linux, THP should be enabled. Running the wrong configuration silently degrades throughput and increases latency.

MongoDB Authentication failed: credential rotation, brute force, and the log signal

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB Authentication failed: credential rotation, brute force, and the log signal

A spike of Authentication failed entries in the MongoDB log is abnormal: in a healthy deployment the baseline is near zero. The signal usually means one of four things: a brute-force attack, an application presenting stale credentials after rotation, an expired x.509 certificate, or an upstream LDAP or Kerberos directory issue. Distinguishing between these quickly determines whether you are facing a security incident or an impending application outage.

MongoDB balancer stuck and jumbo chunks: permanent imbalance and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB balancer stuck and jumbo chunks: permanent imbalance and how to fix it

One shard is hot while the others idle. sh.status() shows chunk counts skewed more than 20%. The balancer is either stopped or running without closing the gap. Until you fix the root cause, the imbalance persists.

Two failure modes cause this. The balancer itself can be disabled, restricted to a narrow window, or blocked by an unhealthy config server. Or the cluster has jumbo chunks: ranges that exceed the configured chunkSize but cannot split because too many documents share the exact same shard key value. MongoDB marks those chunks jumbo in config.chunks and the balancer skips them. That heavy chunk pins load and storage on a single shard and creates a floor on how balanced the cluster can become.

MongoDB cache too small: sizing the WiredTiger cache for your working set

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB cache too small: sizing the WiredTiger cache for your working set

When MongoDB latency doubles and disk read IOPS climb, operators usually check indexes and the query planner first. If db.currentOp() shows no runaway query and the slow query log is quiet, the culprit is often the WiredTiger cache.

WiredTiger maintains its own in-memory cache of uncompressed B-tree pages, separate from the OS page cache. MongoDB defaults the cache to max(0.5 * (RAM - 1 GB), 256 MB). That default works for a single mongod on a dedicated host, but it breaks down in containers, multi-tenant deployments, and during organic data growth. Once the working set exceeds the cache, reads fault to disk, pages are decompressed, and eviction threads compete with application threads for CPU.

MongoDB checkpoint duration climbing: diagnosing slow WiredTiger checkpoints

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB checkpoint duration climbing: diagnosing slow WiredTiger checkpoints

You notice transaction checkpoint most recent time (msecs) climbing past 10 seconds, then 30, then 50. It is trending upward, check after check, approaching the 60-second default checkpoint interval. When checkpoint duration meets or exceeds the interval, WiredTiger has no margin left. The next checkpoint starts late, dirty pages accumulate faster than they flush, and the journal can fill to the point where all new writes block until the checkpoint finishes. This is a common production failure mode that starts as a slow climb and ends as a write freeze.

MongoDB checkpoint stall write freeze: when all writes stop with no error

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB checkpoint stall write freeze: when all writes stop with no error

Writes time out or hang while mongod is running, TCP port 27017 is open, and reads still return results from cache. The MongoDB logs are quiet, but db.serverStatus().opcounters shows write counts frozen. This is a WiredTiger checkpoint stall: the checkpoint process fell behind, dirty pages accumulated, and new writes blocked. The freeze lasts until the current checkpoint completes. If the I/O bottleneck remains, queued writes flood through and the next checkpoint stalls again.

MongoDB chunk migration storms: moveChunk I/O pressure and range locks

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB chunk migration storms: moveChunk I/O pressure and range locks

Write latency spikes across multiple shards simultaneously. Queries time out while mongos logs show no election events. Donor shard primaries report growing queue depths, and config.changelog shows a wall of moveChunk.error entries interleaved with retries. The balancer is hammering the cluster instead of helping.

A single moveChunk operation is expensive. It copies documents to the recipient, enters a critical section with a shared range lock on the donor, then deletes orphaned documents. When the balancer triggers many migrations in quick succession, or individual migrations fail and retry in a tight loop, overlapping I/O pressure and lock contention compound into a cluster-wide latency event.

MongoDB connection churn: high totalCreated rate and thread creation overhead

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB connection churn: high totalCreated rate and thread creation overhead

db.serverStatus().connections can show low current and a rapidly climbing totalCreated. That mismatch is connection churn: connections open and close rapidly instead of being reused. MongoDB uses a thread-per-connection model, so each cycle costs roughly a megabyte of thread stack, scheduling overhead, and file descriptor work. The result is rising RSS, CPU contention, and latency spikes that do not correlate with the active connection count.

MongoDB connection refused at maxIncomingConnections: hitting the connection ceiling

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB connection refused at maxIncomingConnections: hitting the connection ceiling

Application logs show connection timeouts. MongoDB logs show connection refused or error accepting new connection. db.serverStatus().connections shows current well below the configured maximum. This disconnect means you are hitting a hard ceiling at the TCP accept layer, not experiencing gradual degradation.

MongoDB uses a one-thread-per-connection model. Each accepted connection consumes at least one file descriptor and a thread stack sized by the OS ulimit -s. While maxIncomingConnections sets the logical inbound cap, the OS file-descriptor limit (ulimit -n) usually enforces the actual ceiling. Rejections happen before the connection handshake completes, so serverStatus().connections.current never counts refused connections. Look at logs, ratios, and OS-level resource counts to find the real limit.

MongoDB connection storm spiral: reconnection floods after an election or deploy

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB connection storm spiral: reconnection floods after an election or deploy

Connection count on a primary jumps from 200 to 4,000 in under a minute. Resident memory climbs, query latencies double, and application logs fill with timeout errors. The slow query log shows nothing unusual. Individual queries are not the problem. The database is drowning in threads.

This is a connection storm spiral. A trigger event, usually a replica set election, application deploy, or network blip, invalidates existing connections across your application fleet. Every driver reconnects at once. Each new connection costs MongoDB a dedicated thread and roughly 1 MB of stack memory . The resulting RSS spike and ticket contention slow down operations already in flight, causing more timeouts, which drives even more reconnections. The feedback loop ends in OOM kill or unresponsiveness.

MongoDB disk full: emergency recovery when mongod can't write the journal

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB disk full: emergency recovery when mongod can’t write the journal

When the filesystem backing the data or journal directory crosses a critical threshold, WiredTiger cannot allocate new journal extents. If mongod crashes or restarts, recovery replays journal files since the last checkpoint and requires free headroom to create or extend files during that replay. On a full disk, mongod hangs in recovery without binding to port 27017.

If the node is a standalone, there is no replica to fail over to. If it is a secondary, cluster redundancy is reduced while the member is down. Recovery is complicated by a counterintuitive storage engine behavior: WiredTiger reclaims space internally after deletes, but does not automatically shrink data files or return bytes to the operating system. A volume that reads 99% full after a massive delete remains 99% full at the filesystem level.

MongoDB disk I/O saturation: correlating iostat with WiredTiger signals

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB disk I/O saturation: correlating iostat with WiredTiger signals

When opLatencies.writes climbs and globalLock.currentQueue grows, db.serverStatus().wiredTiger.transaction often shows the most recent checkpoint took 45 seconds. WiredTiger metrics tell you what is hurting, but they do not tell you why. The next question is whether the disk is actually saturated.

Disk I/O saturation surfaces as climbing journal sync latency, checkpoint duration exceeding the 60-second interval, application-thread evictions, and ticket exhaustion. The only way to separate a storage problem from a query problem is to correlate OS-level disk signals (iostat -x) with WiredTiger internal signals in the same time window. This guide shows how to do that safely during an incident.

MongoDB exceeded memory limit for $group — aggregation spills and allowDiskUse

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB exceeded memory limit for $group — aggregation spills and allowDiskUse

Application logs show error 16945, or an aggregation pipeline slows by an order of magnitude. In MongoDB, every aggregation stage not backed by an index is limited to 100 megabytes of RAM. When a stage exceeds this limit and disk spilling is not enabled, the operation fails immediately. If spilling is enabled, MongoDB writes temporary files to disk, which keeps the pipeline alive but adds unpredictable latency and extra I/O load.

MongoDB exposed to the internet without authentication: bindIp and the breach scenario

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB exposed to the internet without authentication: bindIp and the breach scenario

A mongod process bound to all interfaces with authentication disabled exposes every database to any host that can reach port 27017. If you are responding to a scan, PAGE, or audit, confirm the exposure, measure the blast radius, and eliminate the surface.

What this means

MongoDB’s net.bindIp controls which interfaces accept connections. Modern packages default bindIp to 127.0.0.1; exposure usually follows an explicit override to a wildcard such as 0.0.0.0 or ::. Without authentication, any reachable host can list databases, read or write documents, and execute administrative commands.

MongoDB flow control throttling writes: when the primary slows itself down

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB flow control throttling writes: when the primary slows itself down

Write throughput on the primary has dropped by 30% or more. Application logs show intermittent write latency spikes, but the primary’s CPU, memory, and disk metrics look healthy. There are no elections, cache pressure warnings, or obvious errors. Check replication: secondaries are lagging. The primary is not sick; it is throttling itself.

MongoDB 4.2 introduced flow control, a ticket-based admission mechanism that caps the primary’s write rate to keep secondaries from falling off the oplog. When isLagged is true, the primary artificially limits throughput. The fix is rarely on the primary. Look at the replication pipeline: a slow secondary, an oplog window that is too small, or a topology change that distorts majority commit lag.

MongoDB globalLock.currentQueue growing: operations queuing for the storage engine

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB globalLock.currentQueue growing: operations queuing for the storage engine

globalLock.currentQueue.total is climbing and not stabilizing after a burst. Under WiredTiger, this metric reflects waits at the intent lock and ticket admission layers, not a single global mutex. The engine is receiving work faster than it can drain it.

If the total queue sustains above 20 and grows, the system is saturated. Queued operations become latency spikes, then connection pileups as applications retry, then a degradation spiral. The root cause is usually ticket exhaustion, a single hot collection, a long-running operation, or storage I/O degradation inflating ticket hold times.

MongoDB journal sync latency high: the storage signal that warns 60 seconds early

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB journal sync latency high: the storage signal that warns 60 seconds early

Application write latency spikes. Connections pile up. Look back 60 seconds and WiredTiger journal sync latency was likely already climbing. Every write with j:true or w:"majority" blocks until the journal buffer is fsynced to disk. When storage struggles, journal sync is the first domino to fall.

Journal sync latency is a storage subsystem signal, not a query or cache problem. The block device under mongod cannot absorb small sequential writes fast enough. The result is head-of-line delay for all durable writes, which cascades into ticket exhaustion and connection backlog.

MongoDB lock wait times: collection and metadata lock contention during DDL

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB lock wait times: collection and metadata lock contention during DDL

When p99 latency jumps and globalLock.currentQueue grows, check serverStatus().locks. If timeAcquiringMicros is climbing for Collection or Metadata, the cause is almost always DDL: createIndexes, dropIndexes, collMod, renameCollection, or similar commands that acquire exclusive collection, database, or metadata locks. WiredTiger uses document-level concurrency for ordinary reads and writes, so normal CRUD rarely blocks. A single schema change can serialize operations on a hot collection or across a database during peak traffic.

MongoDB long-running operations: finding and killing the query holding a ticket

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB long-running operations: finding and killing the query holding a ticket

Your application latency just spiked. opLatencies show reads and writes climbing. globalLock.currentQueue is no longer zero. You check db.serverStatus().wiredTiger.concurrentTransactions: available tickets are near zero, but throughput has not increased. An operation is holding a ticket without making progress.

A collection scan, an unbounded aggregation, or a stalled write can hold a WiredTiger read or write ticket for minutes. The default is 128 read and 128 write tickets in most versions, so one long-running operation can cascade into system-wide queuing, connection pileup, and application timeouts. Find it and kill it, but killing the wrong operation can crash a node or leave data inconsistent.

MongoDB long-running transactions: pinned snapshots and silent cache pressure

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB long-running transactions: pinned snapshots and silent cache pressure

MongoDB latency climbs while write throughput looks normal and the working set is unchanged. Check WiredTiger cache: the dirty ratio is rising and application threads have started evicting pages. The cause is often invisible in standard cache metrics – a multi-document transaction that opened a snapshot and never let it go.

Multi-document transactions pin WiredTiger snapshots until commit or abort. That snapshot prevents the storage engine from evicting old document versions. A single transaction left open for minutes can silently pin enough pages to push the cache into eviction pressure. An inactive-but-open transaction is worse than an active one because it holds the snapshot indefinitely with no forward progress.

MongoDB Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB Monitoring

What Is MongoDB?

MongoDB is a leading NoSQL database platform designed for flexibility, scalability, and performance. It is used to store documents in a flexible, JSON-like format, which makes it perfect for handling large volumes of unstructured data. For more insights, check out MongoDB’s official site.

Monitoring MongoDB With Netdata

Netdata provides comprehensive monitoring for MongoDB, allowing users to gain real-time insights into their MongoDB servers. By utilizing the MongoDB monitoring tool from Netdata, users can track critical metrics and enhance their troubleshooting capabilities.

MongoDB monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB monitoring checklist: the signals every production cluster needs

Production MongoDB failures are preceded by signals that are visible but often unmonitored: climbing dirty cache ratio, shrinking oplog window, or ticket counts approaching zero. This guide organizes essential signals into four monitoring levels. Use them to audit instrumentation or triage gaps during an incident.

Each level builds on the previous one. If you are missing a survival signal, instrument it before adding expert metrics. The thresholds below are drawn from the MongoDB serverStatus() and rs.status() contract and from operational patterns observed across WiredTiger deployments.

MongoDB monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB monitoring maturity model: from survival to expert

If you run MongoDB in production, you need a progression: what to watch first to know the database is alive, what to add next to know why it is slowing down, and what to track finally to predict a cascade before it becomes an outage.

Treat the levels below as gates, not a shopping list. Automate and alert on Level 1 before you build Level 2 dashboards. If you are instrumenting Level 4 but lack a reliable page for disk space or member state, you have inverted your priorities.

MongoDB no primary / election storm: repeated elections and write outages

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB no primary / election storm: repeated elections and write outages

Applications log “not primary” errors. rs.status() shows a different PRIMARY than thirty seconds ago. MongoDB logs repeat "Starting an election" and "Stepping down". Each election costs 2-12 seconds of write unavailability. More than two in ten minutes is an election storm.

This pattern is more dangerous than a single failover because it creates rolling write outages that do not self-stabilize. Drivers reconnect, retry buffers fill, and application latency degrades even when a primary exists. Root causes usually fall into three categories: the primary is too slow to answer heartbeats, the network is dropping or delaying packets between members, or a misconfigured priority is forcing a healthy primary to step down.

MongoDB not master error: writes hitting a non-primary node after failover

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB not master error: writes hitting a non-primary node after failover

A node restart, network partition, or planned stepdown triggers a MongoDB election. Seconds later, application logs show NotWritablePrimary (code 10107) or the legacy string not master and slaveOk=false. Writes fail against a node that used to be PRIMARY, even though the cluster has elected a new one.

This guide covers how to find the root cause and stop it from recurring.

MongoDB not primary and secondaryOk=false: reading from a secondary and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB not primary and secondaryOk=false: reading from a secondary and how to fix it

Your application logs show NotPrimaryNoSecondaryOk (code 13435) with the message "not master and slaveOk=false". Metrics show read failures against a specific host. The mongod process is running, replica set heartbeats are clean, and replication lag looks normal. The cluster is not down. The error is a routing decision: a client sent a read to a replica set member that is not the primary, without declaring that reading from a non-primary is acceptable.

MongoDB noTimeout cursors causing cache pressure: pinned snapshots and silent eviction stalls

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB noTimeout cursors causing cache pressure: pinned snapshots and silent eviction stalls

When wiredTiger.cache.bytes currently in the cache climbs, the dirty ratio trends toward 20%, and read latencies spike without a single slow query in the log, check metrics.cursor.open.noTimeout.

Each noTimeout cursor pins a WiredTiger snapshot indefinitely. Old document versions cannot be evicted while that snapshot is open, so the cache fills with unreachable history until background eviction falls behind and application threads are forced to clean up. The result is a silent cache pressure cascade that looks like a capacity problem but is actually a cursor lifecycle problem.

MongoDB OOM-killed by the kernel: RSS, cache sizing, and oom_score_adj

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB OOM-killed by the kernel: RSS, cache sizing, and oom_score_adj

You find mongod gone. The replica set has no primary. Applications time out. MongoDB logs show no graceful shutdown. Instead, dmesg shows Out of memory: Killed process 12345 (mongod). The Linux OOM killer has reaped the process. MongoDB is a frequent target because its resident set size is usually the largest on the host.

An OOM kill is not a MongoDB bug. It is the kernel freeing RAM by terminating the highest-scoring process. mongod’s RSS is dominated by the WiredTiger cache, plus roughly 1 MB per connection, plus roughly 500 MB to 1 GB of internal overhead for indexes, session buffers, and stack. When that sum comes within 1 GB of total RAM, the node is in the danger zone. The kill is abrupt: no stepdown, no replica set coordination, and after restart the cache must warm again.

MongoDB operation exceeded time limit (MaxTimeMSExpired): maxTimeMS and killed operations

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB operation exceeded time limit (MaxTimeMSExpired): maxTimeMS and killed operations

Error code 50, MaxTimeMSExpired, means the server killed an operation that exceeded its processing budget. Raising the timeout without fixing the root cause turns acute failures into chronic resource exhaustion. The operation was already pathologically slow; maxTimeMS ended it before it consumed more resources or held locks and tickets indefinitely.

maxTimeMS sets a cumulative processing budget in milliseconds. MongoDB enforces it using the same interrupt mechanism as killOp, terminating the operation only at designated interrupt points. Idle time between cursor batches does not count toward the limit, and on direct connections network latency is excluded from the server-side clock. On sharded clusters, however, latency between mongos and shard mongod instances counts against the limit. Distinguish a true MaxTimeMSExpired from a client-side socket timeout, where the client gives up before the server responds.

MongoDB oplog window collapse: secondaries falling off and forced full resync

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB oplog window collapse: secondaries falling off and forced full resync

A secondary transitions to RECOVERING and logs “too stale to catch up.” The oplog window compresses from 48 hours to 90 minutes while replication lag on one secondary climbs steadily. These are the signatures of oplog window collapse: a write surge turns over the oplog faster than secondaries can consume it, and the safety margin between window and lag evaporates.

MongoDB oplog window too small: sizing the oplog for your write volume

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB oplog window too small: sizing the oplog for your write volume

The oplog window is the only thing standing between a routine secondary restart and a multi-hour full initial sync. It is a fixed-size capped collection that stores a variable amount of history. As your write volume grows, the window compresses. Most teams size the oplog once during initial deployment and never look at it again. Six months later, a routine maintenance window turns into an incident because the secondary fell off the oplog, entered RECOVERING, and forced a resync that saturated the remaining nodes.

MongoDB page faults high: working set exceeding memory after warmup

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB page faults high: working set exceeding memory after warmup

Hard page faults long after startup mean the active data set exceeds resident memory. On Linux, extra_info.page_faults counts major faults: the OS read data from disk because the page was missing from both the WiredTiger cache and the OS page cache. A brief spike after restart is normal during warmup, but sustained faults mean the working set does not fit. On EBS gp3, 50 faults per second can degrade latency. On NVMe, hundreds per second may be tolerable, but neither is free. Confirm the cause, distinguish warmup from pressure, and reduce the fault rate without guessing.

MongoDB pages evicted by application threads: when eviction becomes user latency

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB pages evicted by application threads: when eviction becomes user latency

Query p99 latency doubles or triples, but iostat is not saturated and the slow query log shows no single offender. The signal is in db.serverStatus().wiredTiger.cache: pages evicted by application threads has moved from zero to a sustained nonzero rate.

This metric marks the moment when WiredTiger’s dedicated eviction workers fall behind and application threads are drafted to do the work. Any sustained nonzero rate is abnormal. Once application threads evict, they perform page reconciliation and disk I/O inline with the request handler thread, directly inflating user-visible latency. The companion counter pages selected for eviction unable to be evicted means eviction is stalled and the cache is effectively frozen.

MongoDB replica set member unhealthy: reading rs.status() states

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB replica set member unhealthy: reading rs.status() states

rs.status() output is perspective-dependent and easy to misread. A member can show health: 1 while RECOVERING and unable to serve reads, or appear UNKNOWN from one node yet SECONDARY from another because of an asymmetric firewall rule. health: 0 alone does not mean the process is dead, and health: 1 alone does not mean the node is healthy.

This guide maps stateStr values to failure modes and shows how to distinguish transient startup states, replication lag spirals, and network partitions.

MongoDB replication lag: detection, diagnosis, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB replication lag: detection, diagnosis, and fixes

Replication lag is the delay between the primary’s latest oplog entry and a secondary’s last applied entry. When lag grows faster than the oplog window, the secondary cannot catch up and requires a full initial sync, which can take hours to days. Until that happens, every second of lag erodes failover safety and read consistency.

What this means

MongoDB replicates by having secondaries tail the primary’s oplog, a capped collection in local.oplog.rs. The primary records every write with a timestamp; the secondary fetches entries and applies them locally. Lag is the delta between the primary’s optimeDate and the secondary’s optimeDate from rs.status().

MongoDB rollback after failover: silent data loss and the rollback directory

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB rollback after failover: silent data loss and the rollback directory

A replica set member in ROLLBACK state, or an application reporting vanished documents after failover, means a former primary held writes that never reached a majority. When that node rejoins, MongoDB erases the divergent history and writes the removed data to files under <dbPath>/rollback/. The application may have received acknowledgment for those writes. With w:1, acknowledgment meant only that the primary applied the write. It did not guarantee replication to a majority or survival through failover. That is silent data loss.

MongoDB RSS growing without cache growth: leaks, threads, and tcmalloc fragmentation

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB RSS growing without cache growth: leaks, threads, and tcmalloc fragmentation

db.serverStatus().mem.resident climbs while WiredTiger cache utilization stays flat and the host is not swapping. Virtual memory is larger than RSS by design and is not an alert target. Only RSS reflects physical memory pressure. When RSS grows without cache growth, the problem lives outside the storage engine.

This pattern points to one of three areas: tcmalloc heap retention and fragmentation, per-connection thread stack accumulation, or unbounded internal allocations from cursors, plan caches, or aggregation pipelines. Each connection reserves roughly 1 MB of stack space, so a connection storm can add gigabytes of RSS in minutes. TCMalloc caches freed memory in per-thread or per-CPU arenas, which inflates RSS independently of the WiredTiger cache.

MongoDB scanned objects ratio high: documents examined vs returned and wasted work

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB scanned objects ratio high: documents examined vs returned and wasted work

A high scanned-objects-to-returned-documents ratio means MongoDB examines far more documents than it returns. The alert usually fires as an Atlas query targeting event or a chart showing metrics.queryExecutor.scannedObjects climbing faster than metrics.document.returned. A ratio near 1:1 is efficient. A ratio above 1000:1 means the database examines roughly a thousand documents for every one it returns. Wasted work consumes read tickets, pushes unnecessary data through the WiredTiger cache, and often precedes latency spikes or ticket exhaustion.

MongoDB scatter-gather queries: when mongos fans out to every shard

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB scatter-gather queries: when mongos fans out to every shard

In a sharded MongoDB cluster, mongos reads chunk-to-shard mappings from the config servers and routes each query to the smallest set of shards that can satisfy it. When the predicate includes the shard key, mongos targets one shard or a bounded subset. When it does not, mongos broadcasts the query to every shard that owns chunks for the collection and merges the results. This is a scatter-gather query.

MongoDB sharding hot shard: one shard saturating while others idle

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB sharding hot shard: one shard saturating while others idle

Application latency climbs while aggregate CPU, I/O, and connection counts look healthy. Drill into individual shards and one node is pinned near 95% utilization while peers sit near 20%. sh.status() shows an even chunk distribution. You have a hot shard.

A hot shard occurs when one member of a sharded cluster receives a disproportionate share of read or write traffic. The cause is usually an access-pattern mismatch against the shard key, not a chunk-count imbalance. The overloaded shard saturates its WiredTiger cache and exhausts read or write tickets, pushing operations into queues. Because other shards are idle, aggregate cluster metrics hide the problem until the hot shard cascades into a latency spike or availability event.

MongoDB silent index regression: when a dropped index quietly becomes a collection scan

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB silent index regression: when a dropped index quietly becomes a collection scan

Read latency on the primary doubles while connection counts and write throughput stay flat. There are no election events or cache pressure alerts. Traffic is unchanged. Yet p99 read latency climbs until operations time out.

The slow query log shows queries that used to finish in milliseconds now taking seconds. The plans show COLLSCAN. An index that existed last week is gone, or the query planner switched to a less efficient index after a cache invalidation. Because queries still return correct results, the regression is silent until it becomes an outage.

MongoDB slow query COLLSCAN: collection scans and the missing index

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB slow query COLLSCAN: collection scans and the missing index

Queries that used to return in tens of milliseconds now breach application timeouts. Read latency climbs while write throughput stays flat. In the MongoDB slow query log, you see planSummary: "COLLSCAN" attached to operations that should be indexed. A collection scan reads documents that will never be returned, wastes disk I/O, floods the WiredTiger cache with irrelevant data, and holds read tickets until the whole instance cascades into cache pressure.

MongoDB storage not reclaimed after delete: WiredTiger, compact, and resync

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB storage not reclaimed after delete: WiredTiger, compact, and resync

You deleted a large portion of a collection, and db.collection.stats() confirms the logical size dropped. df -h on the data volume does not. The filesystem blocks were not returned. WiredTiger maintains internal free-space lists and reclaims pages for future writes rather than shrinking files. Dropping a database or collection removes the underlying files and returns space to the OS immediately; this article addresses the case where documents are deleted but the collection files remain. This guide explains how to verify the state and the two operational paths to return blocks to the OS: the compact command and replica set member resync.

MongoDB swapping: why mongod must never swap and how to tune the OS

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB swapping: why mongod must never swap and how to tune the OS

Application timeouts climb. MongoDB latency jumps from milliseconds to seconds or minutes, yet mongod is still running and accepting connections. CPU is low, disk I/O is not saturated, and the MongoDB log shows no errors. The process has not crashed. It has entered swap death. When the Linux kernel evicts mongod pages to swap, the database continues to function at roughly 1/1000th of normal speed. MongoDB relies on the WiredTiger cache and OS page cache to remain resident in RAM.

MongoDB ticket exhaustion: WiredTiger read/write tickets and queued operations

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB ticket exhaustion: WiredTiger read/write tickets and queued operations

Your application times out while the OS shows idle CPU and disk utilisation looks survivable. The MongoDB log shows no obvious errors, yet operations stall. The likely cause is WiredTiger ticket exhaustion: the storage engine has run out of read or write concurrency tokens, and new work queues behind slow operations. Confirm ticket starvation, find the root cause, and fix it without raising the ticket limit.

MongoDB TLS certificate expiry: rotating certs without dropping connections

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB TLS certificate expiry: rotating certs without dropping connections

An expired TLS certificate does not degrade gracefully. In MongoDB, client drivers, replica set members, and mongos routers validate certificates during every TLS handshake. When the server certificate’s notAfter date passes, OpenSSL rejects the handshake immediately. Existing connections persist until they close, but nothing reconnects once they do. The result is a sudden cascade: secondaries miss heartbeats and transition to DOWN, application connection pools exhaust, and monitoring probes fail. During an active incident, check certificate dates first. For preventive work, your rotation strategy depends on whether you run MongoDB 5.0 or later.

MongoDB Too many open files: file descriptor exhaustion and ulimit tuning

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB Too many open files: file descriptor exhaustion and ulimit tuning

Connection timeouts appear in application logs. MongoDB logs show Too many open files or error accepting new connection. New secondaries fail to sync, or a stable node rejects connections after restart. The mongod process hit the OS file descriptor limit. The failure is often silent until a dependent system breaks.

FD exhaustion is not always about connection count. WiredTiger maintains open file descriptors for data files, indexes, and journals. A dense deployment with thousands of collections can hold tens of thousands of FDs in steady state. A connection surge, index build, or restarted node rebuilding caches can push the process over the limit. MongoDB then cannot accept new connections, open new data files, or continue replication.

MongoDB too stale to catch up: secondary stuck in RECOVERING and how to resync

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB too stale to catch up: secondary stuck in RECOVERING and how to resync

You check rs.status() during an incident and see a member stuck in RECOVERING with an errmsg reading error RS102 too stale to catch up. The node is alive but will never transition back to SECONDARY on its own. Its last replicated oplog entry is older than the oldest entry still available on the primary, so the history it needs has already been overwritten. Incremental replication is impossible from this state. The only path forward is a full initial sync, which on large datasets can take hours to days, adds significant read load to the sync source, and leaves the cluster with reduced redundancy until it completes. If the stale member is a voting node, you are now one failure away from losing majority. This guide covers how to confirm the condition, identify why the secondary fell off, and recover without pushing the remaining cluster members into the same trap.

MongoDB unused indexes: $indexStats, write amplification, and safe removal

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB unused indexes: $indexStats, write amplification, and safe removal

Every insert and delete updates all indexes on a MongoDB collection; every update that modifies an indexed field updates every index containing that field. An unused index consumes WiredTiger cache space, increases write latency, and amplifies disk I/O. Over time, redundant indexes accumulate from schema migrations, abandoned query patterns, and exploratory tuning.

The $indexStats aggregation stage exposes per-index operation counts, but counters reset on mongod restart and internal operations can inflate values. This guide covers detecting truly unused indexes, validating redundancy, and removing them without causing query regressions.

MongoDB w:majority write concern timeout (wtimeout): replication lag and at-risk writes

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB w:majority write concern timeout (wtimeout): replication lag and at-risk writes

Your application logs show write concern timeouts, or worse, the driver returned an error that the application swallowed and moved on. The write succeeded on the primary, but the cluster could not confirm it across a majority of data-bearing members before the deadline expired. If the primary crashes right now, that data is gone. This article explains how to diagnose the root cause, whether it is replication lag, a down secondary, or network saturation, and what to do before the next primary failure.

MongoDB WiredTiger cache dirty ratio high: the leading indicator nobody watches

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB WiredTiger cache dirty ratio high: the leading indicator nobody watches

Cache fill at 70% looks safe, but if dirty ratio is climbing past 15%, a latency spike is already forming. Dirty ratio measures modified pages not yet flushed to disk. While fill ratio tells you how much cache is in use, dirty ratio tells you how fast the storage engine is falling behind. It often leads checkpoint stalls and eviction-driven latency spikes by minutes.

MongoDB WiredTiger cache pressure cascade: eviction stalls and latency spikes

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB WiredTiger cache pressure cascade: eviction stalls and latency spikes

Latency jumps from milliseconds to seconds for both reads and writes. The slow query log shows no single offender, but connection count climbs as clients retry and timeout. This is the cache pressure cascade. It starts in the storage engine and becomes a self-reinforcing spiral through replication, admission control, and connection handling. This guide covers the mechanism, confirmation under pressure, and how to stop it.

MongoDB WriteConflict errors: optimistic concurrency retries under contention

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB WriteConflict errors: optimistic concurrency retries under contention

WriteConflict exceptions (error 112) in application logs, or unexplained write latency spikes, point to document-level contention under WiredTiger optimistic concurrency control. Outside of transactions, MongoDB retries single-document writes internally; the client sees slower responses rather than errors. Inside multi-document transactions, MongoDB aborts immediately and returns error 112. Either way, the root cause is concurrent writers targeting the same document.

What this means

WiredTiger uses optimistic concurrency control at the document level. Two concurrent writes to the same document do not block indefinitely; one proceeds and the other encounters a conflict.

Monit

Mon, 01 Jan 0001 00:00:00 +0000

Monit Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Monit Monitoring

What Is Monit?

Monit is a small Open Source utility for managing and monitoring Unix systems. It conducts automatic maintenance and repair and can engage in stage-based escalation if there are problems.

Monitoring Monit With Netdata

Integrating the Monit monitoring tool with Netdata provides real-time, comprehensive monitoring of system services. With Netdata, you can visualize all the data that Monit collects, helping you monitor Monit and optimize your infrastructure more effectively. To see it in action, check out our live demo.

Monitoring 101

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring Kubernetes across vanilla, EKS, GKE, AKS, k3s, and rke2

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring Kubernetes across vanilla, EKS, GKE, AKS, k3s, and rke2

Every Kubernetes cluster exposes the same core control plane components, but what you can actually scrape, query, and alert on depends entirely on how the distribution packages those components. An operator migrating between vanilla kubeadm, Amazon EKS, Google GKE, Azure AKS, k3s, and rke2 needs to know which metrics endpoints exist, which are reachable without extra tooling, and which are gated behind vendor lock-in or architectural quirks. This article maps the native monitoring surface of each variant so you can adapt your collection strategy without discovering gaps during an incident.

Monitoring MIG-partitioned NVIDIA GPUs: per-instance metrics

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring MIG-partitioned NVIDIA GPUs: per-instance metrics

On a MIG-enabled GPU (A100, A30, H100 and later), the physical card is partitioned into isolated GPU instances, each with its own SMs, memory partition, and failure domain. Every monitoring habit built around whole-GPU metrics breaks here: aggregate utilization, aggregate memory, and even some health counters describe the card, not the tenant. An instance can be OOM while nvidia-smi shows healthy free memory on the card.

Monitoring overlay tunnels (IPsec/GRE/VXLAN): the signals that matter

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring overlay tunnels (IPsec/GRE/VXLAN): the signals that matter

Overlay tunnels share the underlay’s physical path but add their own failure modes: encapsulation overhead, separate control and data planes, and type-specific state machines. Monitoring only the tunnel interface’s link state is the fundamental trap. An interface reporting “UP” can still be forwarding into a black hole because the peer is unreachable, the SA has expired, or the underlay is dropping fragments.

Monitoring route origin and AS-path changes for hijack detection

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring route origin and AS-path changes for hijack detection

BGP route hijacks do not announce themselves. A prefix legitimately originated by AS2906 starts appearing in the global table with origin AS65001. Traffic that should reach your infrastructure follows a different path, gets blackholed, or lands on an interception point. The control-plane signals are there, but they are distributed across route collectors, RPKI validators, and per-prefix state that most monitoring stacks do not track at the granularity needed.

Monnit Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Moser Baer AG SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

mosquitto

Mon, 01 Jan 0001 00:00:00 +0000

Mosquitto Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Mosquitto Monitoring

What Is Mosquitto?

Mosquitto is a lightweight MQTT broker, designed to facilitate IoT communications efficiently. As a central hub for sending and receiving messages, Mosquitto ensures that devices can publish and subscribe to channels seamlessly, making it a critical component for IoT infrastructure and applications.

Monitoring Mosquitto With Netdata

Monitoring Mosquitto can enhance your IoT message transport and system performance. By using Netdata, you can effectively monitor Mosquitto with ease. Netdata utilizes an openmetrics (Prometheus) exporter to gather and ingest data from the Mosquitto exporter. This allows users to enjoy automated dashboards and alerts, without the need for a standalone Prometheus server or Grafana setup.

Motorola SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Moxa Technologies Co Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Mpb Communications Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

MQTT Blackbox

Mon, 01 Jan 0001 00:00:00 +0000

MQTT Blackbox Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

MQTT Blackbox Monitoring

What Is MQTT Blackbox?

MQTT Blackbox is a specialized monitoring technique designed to test and track the performance of MQTT message transport using blackbox testing methods. It leverages the MQTT Blackbox Exporter to simulate client interactions and analyze the reliability and efficiency of message brokers in real-time.

Monitoring MQTT Blackbox With Netdata

Netdata excels at comprehensive MQTT Blackbox monitoring by utilizing an openmetrics (prometheus) exporter. With Netdata, you can effortlessly ingest data from any Prometheus exporter, eliminating the need for a Prometheus server or Grafana. This integration provides users with automated dashboards, real-time alerts, and more, making it an incredibly efficient tool for monitoring MQTT Blackbox.

Mrv Communications In Reach Product Division SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

MS Exchange

Mon, 01 Jan 0001 00:00:00 +0000

mtail

Mon, 01 Jan 0001 00:00:00 +0000

Mtail Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Mtail Monitoring

What Is Mtail?

Mtail is a vital tool used for extracting and parsing log data. Developed by Google, Mtail assists in monitoring and showcasing real-time logs. This is particularly beneficial for DevOps, SREs, and IT professionals who need to ensure the smooth operation of systems through log data metrics. Mtail acts as a log data extractor tailored to plug gaps in monitoring solutions, perfect for complex environments.

Monitoring Mtail With Netdata

Netdata offers a seamlessly integrated environment to monitor mtail using the OpenMetrics (Prometheus) exporter. Netdata’s advanced platform can ingest data from any Prometheus exporter, providing automated dashboards, alerts, and visualizations without the need for a Prometheus server or Grafana. This attribute makes Netdata an exceptional mtail monitoring tool.

Mts Allstream Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Multiple drives failing at once: it's the power supply, not the drives

Mon, 01 Jan 0001 00:00:00 +0000

Multiple drives failing at once: it’s the power supply, not the drives

Three, five, or twelve drives light up at once. Spin_Retry_Count has jumped on every HDD in a shelf. Unsafe Shutdown counts are climbing across every NVMe device. A few drives may have dropped off the bus and come back. The instinct is to start filing RMAs.

Stop. When the same symptom appears on many drives at the same time, the probability that every drive independently decided to fail in the same hour approaches zero. The root cause is almost certainly external: a degrading PSU, an overloaded PDU, a failing UPS battery, or aggregate spin-up inrush sagging the 12V rail because staggered spin-up is disabled.

Mylex Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

MySQL

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Aborted_connects and Aborted_clients climbing: diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Aborted_connects and Aborted_clients climbing: diagnosis

You notice Aborted_connects or Aborted_clients climbing on a production MySQL instance. Because these are cumulative counters, a steady upward slope means something is actively failing or dropping connections. In a busy system, a rising Aborted_connects rate can trigger host blocking via max_connect_errors, suddenly preventing legitimate clients from connecting. A rising Aborted_clients rate usually shows up as application-side exceptions about closed connections, forcing retries that can cascade into connection exhaustion. The two counters track completely different failure modes: Aborted_connects counts connection attempts that never finished authentication; Aborted_clients counts connections that authenticated but then died unexpectedly. Treating them as the same metric leads to wrong fixes. Restarting the network stack will not fix a credential rotation bug, and increasing max_connections will not fix a client killed by wait_timeout.

MySQL adaptive hash index latch contention: high CPU, low throughput

Mon, 01 Jan 0001 00:00:00 +0000

MySQL adaptive hash index latch contention: high CPU, low throughput

Burning CPU while barely answering queries: OS user time near 100%, application latency spiking, and Questions flat or falling. Threads_connected is high but Threads_running stays low, and disk I/O is quiet. This pattern matches an Adaptive Hash Index (AHI) latch storm.

InnoDB builds an in-memory hash index over frequently accessed B-tree pages to speed lookups. Since MySQL 5.7, the latch is partitioned into innodb_adaptive_hash_index_parts (default 8). Under high concurrency, especially point lookups on a hot index, threads can collide on the same partition latch and spin instead of executing. CPU saturates while throughput collapses.

MySQL authentication failure spike: brute force vs broken credential rotation

Mon, 01 Jan 0001 00:00:00 +0000

MySQL authentication failure spike: brute force vs broken credential rotation

You get paged because Aborted_connects is climbing. The error log shows a wall of authentication failures, application dashboards are yellow, and someone in security is asking if this is an attack. Before you block IP addresses or rotate passwords, you need to know which failure mode you are dealing with. External brute force and internal broken credential rotation produce the same MySQL status counters, but the fixes are opposite. One requires closing the network window and flushing blocked hosts. The other requires finding the application instance that missed the new secret.

MySQL binary logs filling the disk: expiry, lagging replicas, and purge

Mon, 01 Jan 0001 00:00:00 +0000

MySQL binary logs filling the disk: expiry, lagging replicas, and purge

You get a disk-full alert on the MySQL primary. df -h shows the data partition at 95%, and du points to /var/lib/mysql/binlog.* consuming hundreds of gigabytes. Writes are about to fail.

Binary logs are append-only. MySQL rotates to a new file at max_binlog_size (default 1 GB), but rotation does not delete old files. Deletion only happens via automatic expiry or PURGE BINARY LOGS. MySQL refuses to delete any binlog file that a connected replica has not yet consumed. In MySQL 5.7, the default expiry is never (expire_logs_days = 0). In 8.0, the default is 30 days, but automatic purge cannot remove files that a connected replica still needs. A replica lagging past the expiry window therefore causes unbounded growth.

MySQL connection exhaustion: detection, diagnosis, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

MySQL connection exhaustion: detection, diagnosis, and prevention

ERROR 1040 (HY000): Too many connections. Health checks fail. Users cannot sign in. If you configured an admin account correctly, a break-glass session may still get through on the reserved slot, and SELECT 1 returns. MySQL is running, but the connection pool is a wall.

This is a hard cliff. One moment queries flow; the next, every new TCP handshake to port 3306 is rejected. The server does not queue connections. Whether the root cause is a connection leak, a retry storm, or genuine overload from slow queries, the symptom is the same: Threads_connected has reached max_connections.

MySQL Created_tmp_disk_tables: temp tables spilling to disk

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Created_tmp_disk_tables: temp tables spilling to disk

Queries that GROUP BY, ORDER BY, DISTINCT, or materialize derived results build internal temporary tables. When those tables exceed memory limits, MySQL spills them to disk. The status counter Created_tmp_disk_tables tracks this, and its ratio to Created_tmp_tables is the signal to watch during a latency incident.

A low ratio is normal. A high ratio means queries pay a random-disk penalty for intermediate results. You see that as rising query latency, increased disk I/O on the data directory, and connection pile-up. This guide covers how to read the ratio, find the offending patterns, and fix them without just adding RAM.

MySQL ERROR 1040 (HY000): Too many connections - causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1040 (HY000): Too many connections - causes and fixes

Once Threads_connected reaches max_connections, MySQL returns ERROR 1040 (HY000) to every new connection attempt before authentication. The server is not down; it is full.

This error is usually a symptom. Connections may be leaking from the application, a cache stampede may have flooded the pool, or slow queries may be holding slots open longer than expected. MySQL reserves one extra connection for users with CONNECTION_ADMIN (or the deprecated SUPER privilege). If that slot is free, an operator can still connect. If an app user has taken it, you may need to restart.

MySQL ERROR 1045 (28000): Access denied for user - diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1045 (28000): Access denied for user - diagnosis

When you see ERROR 1045 (28000): Access denied for user 'app'@'10.0.0.5' (using password: YES), the connection never reached the query parser. MySQL rejected it during authentication, incremented Aborted_connects, and returned SQLSTATE 28000. The error always includes the effective user and host as seen by the server. This is your first and most important clue.

This error has four root causes in production: wrong credentials, user@host mismatch, authentication plugin incompatibility, and host-level blocking from accumulated connection errors. A brute-force probe and a misconfigured deploy look identical from the server side, so diagnosis hinges on correlating the error pattern with connection metrics and the host cache.

MySQL ERROR 1205: Lock wait timeout exceeded; try restarting transaction

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1205: Lock wait timeout exceeded; try restarting transaction

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction appears when one transaction holds an InnoDB row lock so long that another transaction exhausts innodb_lock_wait_timeout (default 50 seconds). Unlike a deadlock, which InnoDB resolves automatically, a timeout means a blocker is still active. The root cause is almost always a long-running transaction or hot row contention. Find it before it cascades into a wider outage.

MySQL ERROR 1213: Deadlock found when trying to get lock; try restarting transaction

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1213: Deadlock found when trying to get lock; try restarting transaction

Your application logs show ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction. One transaction was rolled back; the other completed normally. InnoDB broke a circular lock wait by selecting the cheaper transaction as the victim. A few deadlocks per hour in high-concurrency OLTP is normal. A sustained storm is not: it means transactions are colliding under load.

MySQL ERROR 1290: --read-only option so it cannot execute this statement

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1290: –read-only option so it cannot execute this statement

A write that fails with ERROR 1290 (HY000): The MySQL server is running with the --read-only option so it cannot execute this statement is a routing or topology problem, not a query problem. The server refuses the statement because read_only or super_read_only is enabled. You are likely sending writes to a replica, a recently promoted primary that never cleared its read-only flag, or a node correctly enforcing super_read_only. Confirm the instance role and fix the write path.

MySQL ERROR 2006/2013: MySQL server has gone away -- causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 2006/2013: MySQL server has gone away – causes and fixes

ERROR 2006 (CR_SERVER_GONE_ERROR) and ERROR 2013 (CR_SERVER_LOST) produce the messages “MySQL server has gone away” and “Lost connection during query.” They often appear together, but they indicate different failures. ERROR 2006 means the server closed a connection the client believed was valid. ERROR 2013 means the connection dropped while a query was executing or results were transferring.

Treating them identically leads to incorrect fixes. Raising max_connections does not stop idle timeouts. Tuning timeouts does not fix an OOM kill. Before changing configuration, determine whether the MySQL process restarted. A stable process points to timeout or packet mismatches. A restarted process points to a crash or memory exhaustion.

MySQL ERROR 24 (HY000): Too many open files -- raising open_files_limit

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 24 (HY000): Too many open files – raising open_files_limit

ERROR 24 (HY000): Too many open files means the OS refused a file descriptor request because the mysqld process hit its hard ceiling. Clients may see Can't create/write to file with errno: 24. Tables that were accessible moments ago fail to open, and new connections may be rejected even though Threads_connected is well below max_connections.

MySQL holds file descriptors for cached tables, client connections, binary logs, relay logs, and on-disk temporary tables. When combined demand exceeds the OS-enforced limit, the server returns errno 24. Unlike connection exhaustion, which is gated by max_connections, file descriptor exhaustion is gated by open_files_limit and the OS limits that enforce it. Raising open_files_limit in my.cnf is not enough if systemd, PAM, or the shell ulimit blocks the request.

MySQL FLUSH TABLES WITH READ LOCK stall: backups that freeze the server

Mon, 01 Jan 0001 00:00:00 +0000

MySQL FLUSH TABLES WITH READ LOCK stall: backups that freeze the server

Your application suddenly cannot write. Threads_connected climbs toward max_connections, but Questions flatlines. The last change was a backup job that started ten minutes ago. The culprit is almost always FLUSH TABLES WITH READ LOCK (FTWRL), and the damage is caused not by the lock itself but by what happens while the server waits to acquire it.

FTWRL is triggered by mysqldump --master-data, mysqldump --lock-tables, and similar tools that need a consistent logical backup. It attempts to close all open tables and acquire a global read lock. While it waits for a long-running query to finish, new writes are already blocked. The result is a whole-server freeze: the query rate drops to zero, connections pile up, and the application sees cascading timeouts.

MySQL full table scans: Handler_read_rnd_next and the missing index

Mon, 01 Jan 0001 00:00:00 +0000

MySQL full table scans: Handler_read_rnd_next and the missing index

You get paged because query latency is spiking. Slow_queries is climbing. Handler_read_rnd_next has jumped five-fold over its baseline and keeps rising. Connections, CPU, and buffer pool hit ratio look fine. The culprit is usually a new full table scan after a deploy, schema change, or query pattern shift.

Handler_read_rnd_next increments when the storage engine reads the next row during a table scan or sorted retrieval. In a healthy OLTP system, Handler_read_key dominates and Handler_read_rnd_next stays flat relative to query volume. When the ratio inverts, you are looking at a query plan regression, a missing index, or an optimizer decision gone wrong.

MySQL gap locks and next-key locks: surprising deadlocks under REPEATABLE READ

Mon, 01 Jan 0001 00:00:00 +0000

MySQL gap locks and next-key locks: surprising deadlocks under REPEATABLE READ

MySQL deadlocks in production frequently involve transactions that modify apparently unrelated rows. Two concurrent UPDATEs with narrow WHERE clauses collide, or an UPDATE blocks every INSERT into the table. The cause is usually REPEATABLE READ combined with InnoDB next-key locking and a missing or non-unique index.

Under REPEATABLE READ, the default in MySQL, InnoDB does not lock only matching rows. To prevent phantom reads, it locks the gaps between index entries. When no suitable index exists, or when the optimizer scans a non-unique index, a targeted UPDATE escalates into a range lock covering far more of the table than the predicate suggests.

MySQL Got error 28 from storage engine / No space left on device — recovery

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Got error 28 from storage engine / No space left on device — recovery

When Got error 28 from storage engine appears, the underlying filesystem is at 100%. MySQL needs writable space for InnoDB data files, redo logs, binary logs, temporary tables, relay logs, and slow query logs. Writes fail immediately. Affected threads enter a retry loop, logging warnings until an operator frees space or kills the operation.

The most dangerous scenario is the redo log filling up. If checkpoint age approaches capacity, InnoDB forces aggressive synchronous flushing. Under sustained pressure this can trigger an unclean shutdown and crash recovery on restart. Replicas are similarly vulnerable: a full relay log partition stops the I/O thread and lets lag grow unboundedly, risking binlog expiry on the source before catch-up.

MySQL GTID errant transactions: detecting replication divergence

Mon, 01 Jan 0001 00:00:00 +0000

MySQL GTID errant transactions: detecting replication divergence

Before a planned failover, or after an orchestrator aborts with a GTID consistency error, replication can appear healthy: SHOW REPLICA STATUS reports both threads running, Seconds_Behind_Source is near zero, and the error log is quiet. Yet comparing GTID sets between source and replica reveals mismatching numbers. Errant transactions are GTIDs present in a replica’s gtid_executed set that the source never generated.

Ordinary lag resolves as the replica catches up. Errant transactions represent true divergence. Promoting a replica with extra GTIDs propagates those transactions into the new source and downstream replicas, creating split-brain that is expensive to reverse. This guide covers distinguishing benign lag from dangerous divergence, pinpointing offending GTIDs, and deciding between empty-transaction injection and a full rebuild.

MySQL InnoDB buffer pool hit ratio collapse: the cliff edge

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB buffer pool hit ratio collapse: the cliff edge

Your OLTP queries were running in single-digit milliseconds. Now every query is taking seconds, the disk subsystem is saturated, Threads_running is climbing toward max_connections, and the buffer pool hit ratio, which sat at 99.9% for months, just fell through 95% and keeps dropping.

This is the InnoDB buffer pool cliff edge. When the working set exceeds the buffer pool, pages are evicted before they can be reused. Every miss becomes a physical disk read. The degradation is non-linear: 99.9% to 99% is a slow bleed, 99% to 95% is rapid, and below 95% disk saturation, uniform latency inflation, and connection exhaustion turn a capacity problem into an availability incident.

MySQL InnoDB checkpoint age: the redo log capacity signal nobody watches

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB checkpoint age: the redo log capacity signal nobody watches

Checkpoint age is the distance between the current InnoDB log sequence number (LSN) and the last checkpoint LSN. It measures how much of the circular redo log is occupied by changes not yet flushed to data files. When this age approaches redo log capacity, InnoDB escalates flushing aggressiveness until it runs out of options and stalls all writes.

Standard MySQL does not expose Innodb_checkpoint_age as a status variable. Depending on version, you must parse SHOW ENGINE INNODB STATUS, enable a disabled-by-default INNODB_METRICS counter, or compute the delta from MySQL 8.0.30+ redo status variables. MySQL 8.0.30 replaced innodb_log_file_size and innodb_log_files_in_group with innodb_redo_log_capacity.

MySQL InnoDB history list length growing: purge lag explained

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB history list length growing: purge lag explained

If query latency climbs across every table and no single query stands out, check the InnoDB history list length. It tracks unpurged undo records: MVCC debt. When purge falls behind, every consistent read walks longer version chains, and the slowdown is global. Unlike a slow query, this debt is invisible to conventional query analysis.

In MySQL 8.0, the programmatic source for this metric is trx_rseg_history_len in information_schema.INNODB_METRICS. Despite being labeled status_counter in the metrics table, trx_rseg_history_len behaves as a gauge: it rises when purge lags and falls when purge catches up.

MySQL InnoDB redo log checkpoint stall: when all writes freeze

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB redo log checkpoint stall: when all writes freeze

InnoDB redo log checkpoint stalls freeze all writes synchronously when checkpoint age reaches the end of the circular redo log. INSERTs, UPDATEs, and DELETEs that complete in milliseconds hang for seconds. Threads_running climbs while the Questions rate collapses. Read queries may still return initially, but soon every connection waits for a write lock or commit acknowledgment. Then throughput recovers just as abruptly. This is not a disk failure, deadlock, or runaway query. It is MySQL’s write path running out of runway.

MySQL InnoDB row lock contention: finding who blocks whom

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB row lock contention: finding who blocks whom

Queries that normally finish in milliseconds take seconds. Threads_running climbs while Questions stalls. SHOW PROCESSLIST shows active threads, yet the database is frozen. That is the shape of InnoDB row lock contention: transactions are waiting to release row-level locks, and the queue is growing.

Row lock contention differs from a metadata lock cascade. Metadata locks block DDL and DML at the table level and live in performance_schema.metadata_locks. Row locks are held by open InnoDB transactions and block at the row, gap, or next-key level. Rising Innodb_row_lock_current_waits alongside rising Threads_running signals a live contention crisis. The goal is to identify the blocker, the waiter, and the lock footprint so you can break the chain without guessing.

MySQL innodb_buffer_pool_size tuning: 60-80% of RAM and when that breaks

Mon, 01 Jan 0001 00:00:00 +0000

MySQL innodb_buffer_pool_size tuning: 60-80% of RAM and when that breaks

The 60-80% rule works for a bare-metal host running only MySQL. In containers, shared hardware, and high-connection-count environments, it is a hazard. Size the pool against aligned allocation, cgroup limits, per-connection memory, and OS headroom. Do not size it as a percentage of total RAM.

What the buffer pool costs

InnoDB caches data and index pages in a fixed-size pool. Every read and write touches it. When the working set fits entirely in memory, queries avoid disk. When it does not, InnoDB evicts pages and disk reads dominate latency.

MySQL Innodb_buffer_pool_wait_free > 0: buffer pool memory pressure

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Innodb_buffer_pool_wait_free > 0: buffer pool memory pressure

Innodb_buffer_pool_wait_free increments when InnoDB must synchronously flush dirty pages to make room for new reads. A sustained nonzero rate means queries are waiting on disk writes before they can proceed. Unlike the buffer pool hit ratio, which can stay above 99% while the system stalls, wait_free confirms the buffer pool is operating at its limit.

This is the Buffer Pool Cliff pattern: once the working set exceeds available clean pages, performance degrades non-linearly, disk I/O saturates, and threads pile up.

MySQL innodb_deadlock_detect=OFF: when deadlock detection becomes the bottleneck

Mon, 01 Jan 0001 00:00:00 +0000

MySQL innodb_deadlock_detect=OFF: when deadlock detection becomes the bottleneck

At extreme concurrency, InnoDB’s deadlock detector can become the bottleneck. It traverses the wait-for graph on every lock enqueue. That cost is usually negligible until thousands of concurrent writing transactions hit a small set of hot rows. When traversal itself saturates CPU, throughput collapses even though disks are idle and the buffer pool is warm.

innodb_deadlock_detect=OFF removes that traversal. Instead of instant deadlock detection, InnoDB relies exclusively on innodb_lock_wait_timeout to break cycles. The blocked transaction waits, holding its locks, until the timeout expires. This is not a generic performance tuning knob. It is a narrow, high-risk tradeoff that converts instant rollback into delayed timeout, and it is appropriate only when you have confirmed through testing that deadlock detection is the bottleneck.

MySQL Innodb_log_waits > 0: the log buffer is too small (not a checkpoint stall)

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Innodb_log_waits > 0: the log buffer is too small (not a checkpoint stall)

InnoDB increments Innodb_log_waits every time a thread waits because the log buffer is full. A sustained nonzero rate means the buffer cannot absorb your write burst, or the storage layer cannot drain it fast enough.

This is not a checkpoint stall. Checkpoint stalls happen when the on-disk redo log fills and InnoDB forces synchronous dirty-page flushing. Innodb_log_waits measures in-memory log buffer pressure only. Confusing the two leads to tuning redo log file size while the real problem persists.

MySQL innodb_redo_log_capacity sizing: how big should the redo log be

Mon, 01 Jan 0001 00:00:00 +0000

MySQL innodb_redo_log_capacity sizing: how big should the redo log be

An undersized InnoDB redo log is a common cause of MySQL write stalls. When redo generation outpaces dirty-page flushing, checkpoint age advances until InnoDB enters synchronous flushing and user writes block.

Since MySQL 8.0.30, innodb_redo_log_capacity replaces innodb_log_file_size and innodb_log_files_in_group. Resizing no longer requires a restart, but the sizing logic is unchanged: capacity must absorb peak write rates without pushing checkpoint age into the synchronous-flush zone.

MySQL InnoDB: Database page corruption — detection and recovery

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB: Database page corruption — detection and recovery

Your error log shows InnoDB: Database page corruption on disk or a failed file read. MySQL may abort the connection, mark the tablespace read-only, or crash. Page corruption is a storage-layer failure, not a query bug: a 16KB InnoDB page failed checksum validation.

The critical distinction is between secondary index corruption, which is rebuildable without data loss, and clustered index corruption, which affects the table data itself. Detect the failure, classify the damage, extract what you can, and rebuild safely.

MySQL long-running transactions: detecting and killing the silent blocker

Mon, 01 Jan 0001 00:00:00 +0000

MySQL long-running transactions: detecting and killing the silent blocker

Threads_running climbs. A DDL operation that should take seconds is stuck for minutes. Storage grows steadily with no matching data increase. The cause is often a single idle transaction in INNODB_TRX, holding row locks and an MVCC read view long after its last query finished. It does not appear in the slow query log and may have no active query string. Left alone, it blocks InnoDB purge, inflates the history list, and can trigger a metadata lock cascade that fills the connection pool.

MySQL metadata lock cascade: how one ALTER TABLE freezes a whole table

Mon, 01 Jan 0001 00:00:00 +0000

MySQL metadata lock cascade: how one ALTER TABLE freezes a whole table

You run an ALTER TABLE to add an index. Seconds later, health checks fail for queries that touch only that table. Other tables work fine. CPU and disk are idle. The connection pool fills. SHOW ENGINE INNODB STATUS shows no row lock waits. The culprit is a metadata lock cascade, not InnoDB contention.

This happens when DDL requests an exclusive metadata lock (MDL) on a table already protected by a shared MDL held by a long-running or idle transaction. The DDL waits. Subsequent DML on that table queues behind it. The queue grows until the connection pool exhausts and the application times out. Because the outage is isolated to one table and leaves no trace in InnoDB lock metrics, it is easy to misdiagnose as a network blip or a runaway query.

MySQL Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Monitoring

What Is MySQL?

MySQL is a widely-used open-source relational database management system. It is a crucial component in web application architecture and is known for its reliability, scalability, and performance. MySQL supports a variety of database-driven applications by enabling efficient storage, retrieval, and management of data.

Monitoring MySQL With Netdata

Netdata offers real-time, insightful metrics and monitoring capabilities required for effective MySQL monitoring. With Netdata, you gain access to detailed visualizations and alerts to ensure that your MySQL server operates optimally. This is achieved via Netdata’s MySQL monitoring tool which seamlessly integrates and enables deep analytics of your MySQL instances.

MySQL monitoring checklist: the signals every production instance needs

Mon, 01 Jan 0001 00:00:00 +0000

MySQL monitoring checklist: the signals every production instance needs

Most production incidents involving MySQL stem from missing observability into specific internal subsystems. Connection saturation, buffer pool thrashing, and replication lag can degrade a healthy cluster into an outage if the right signals are not monitored.

This checklist serves as a tiered operational reference for senior engineers and SREs. It defines the exact metrics needed to keep MySQL instances available, fast, and reliable, scaling from basic survival metrics to expert-level internals. Use it to validate existing dashboards or baseline monitoring for a new deployment.

MySQL monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

MySQL monitoring maturity model: from survival to expert

Most MySQL monitoring stacks grow reactively. A team starts with a liveness check and disk alerting. An incident exposes a gap, a new signal gets added, and the cycle repeats. The result is uneven coverage: strong in areas that caused past outages, blind in areas that will cause the next one.

This article maps MySQL monitoring into four capability levels. Use it as a self-assessment, not a checklist to max out. Every level includes signals the playbook identifies as operationally important, along with the failure modes they catch. If you are missing an entire category at your current level, that is your highest-priority gap.

MySQL online DDL still blocking: ALGORITHM, LOCK, and the copy phase

Mon, 01 Jan 0001 00:00:00 +0000

MySQL online DDL still blocking: ALGORITHM, LOCK, and the copy phase

Online DDL in MySQL is not lock-free. Even with ALGORITHM=INPLACE or ALGORITHM=INSTANT, and even with LOCK=NONE, every online DDL operation passes through a brief window where it upgrades its metadata lock (MDL) to exclusive. That window is short, but it is real, and it is the single most common reason an “online” schema change still stalls a busy production table.

MySQL OOM-killed: buffer pool, per-connection buffers, and the kernel killer

Mon, 01 Jan 0001 00:00:00 +0000

MySQL OOM-killed: buffer pool, per-connection buffers, and the kernel killer

MySQL disappears. Application logs fill with connection timeouts. Uptime resets to near zero. In the kernel log: oom-kill: task mysqld and its anonymous RSS. Orchestrators mark the pod OOMKilled (exit code 137). To the application, this is a crash. To the kernel, mysqld was the largest memory consumer and the system ran out.

OOM kills are gradual, then catastrophic. Memory pressure builds as connections open, temp tables materialize, and dirty pages accumulate. The buffer pool is the obvious consumer, but the killer often enters through the back door: a connection burst multiplies per-thread buffers, a container limit sits too close to the buffer pool size, or Transparent Huge Pages block reclamation. Map all allocators to prevent recurrence.

MySQL Opened_tables climbing: table_open_cache and open-files pressure

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Opened_tables climbing: table_open_cache and open-files pressure

Steadily climbing Opened_tables with Open_tables pinned near table_open_cache means the instance is churning table file descriptors instead of reusing them. Each cache miss opens a table, consumes a file descriptor, and adds metadata lock overhead. The first symptom is usually sporadic query latency during peak traffic. Left unchecked, the instance exhausts its file descriptor limit and returns errors like Too many open files or refuses connections.

MySQL privilege change auditing: GRANT, REVOKE, and unexpected SUPER

Mon, 01 Jan 0001 00:00:00 +0000

MySQL privilege change auditing: GRANT, REVOKE, and unexpected SUPER

Unexpected privilege changes are high-impact, low-frequency events. A spike in Com_grant, an unknown account with SUPER, or a GRANT ALL outside a change window can signal privilege escalation, misconfigured automation, or migration debt. This guide covers how to detect, diagnose, and remove unauthorized grants and deprecated SUPER privileges in production.

What this means

MySQL increments status counters for GRANT, REVOKE, CREATE USER, ALTER USER, and DROP USER. The counters are cumulative; compute deltas over fixed intervals instead of reading absolute values.

MySQL purge lag from an idle transaction: the slow bleed

Mon, 01 Jan 0001 00:00:00 +0000

MySQL purge lag from an idle transaction: the slow bleed

Your OLTP workload slows over hours. No single query dominates the slow log. CPU and disk I/O are not saturated. Yet read latency climbs, throughput drifts downward, and the undo tablespace grows. The culprit is often a single idle transaction that opened a read view and never closed it. Under InnoDB’s default REPEATABLE READ isolation, that transaction pins the purge boundary. Every subsequent write appends to the undo log. The history list length grows without bound, and all MVCC reads traverse longer version chains.

MySQL query plan regression: when a query gets slow after a deploy or ANALYZE

Mon, 01 Jan 0001 00:00:00 +0000

MySQL query plan regression: when a query gets slow after a deploy or ANALYZE

A query that completed in milliseconds yesterday now times out. The deploy was clean, or maybe someone ran ANALYZE TABLE during maintenance. Slow_queries climbs, Handler_read_rnd_next jumps, and the same application code suddenly kills the database. The optimizer has changed how it executes one or more queries, and the new plan is far more expensive.

Plan regressions are sharp step-changes, not gradual growth. They correlate tightly with specific events: a schema migration, an index drop, a bulk load that triggered auto-recalc, a manual ANALYZE TABLE, or a MySQL version upgrade. The cause is usually stale or shifted statistics, an optimizer behavior change, or a missing index the optimizer can no longer use.

MySQL relay log filling the replica disk: Relay_Log_Space and recovery

Mon, 01 Jan 0001 00:00:00 +0000

MySQL relay log filling the replica disk: Relay_Log_Space and recovery

When SHOW REPLICA STATUS shows Seconds_Behind_Source climbing and Relay_Log_Space growing, the SQL thread is not applying events as fast as the IO thread fetches them. Normally MySQL purges each relay log after the SQL thread finishes it, so total disk usage stays bounded. When apply cannot keep up, files accumulate. If the partition fills, the IO thread stops. No new events arrive, lag becomes unbounded, and if the source purges binary logs before recovery, the replica requires a full resync.

MySQL Replica_IO_Running / Replica_SQL_Running not Yes: replication stopped

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Replica_IO_Running / Replica_SQL_Running not Yes: replication stopped

SHOW REPLICA STATUS\G (MySQL 8.0+) or SHOW SLAVE STATUS\G (MySQL 5.7) shows Replica_IO_Running or Replica_SQL_Running is not Yes. The IO thread may hang in Connecting, or the SQL thread may stop with an error. Replication no longer makes progress. Lag grows without bound, and if the source purges binary logs before recovery, you face a full rebuild.

Unlike many MySQL problems, a stopped replication thread does not self-heal. The SQL thread halts on the first apply error and waits for operator intervention. The IO thread retries automatically, but only up to the configured retry limit. Seconds_Behind_Source can show 0 even when the IO thread is disconnected and the SQL thread has consumed the relay log, so lag alone is not a reliable health signal.

MySQL replication lag spiral: Seconds_Behind_Source growing without bound

Mon, 01 Jan 0001 00:00:00 +0000

MySQL replication lag spiral: Seconds_Behind_Source growing without bound

You check SHOW REPLICA STATUS and see Seconds_Behind_Source is not just elevated; it is climbing. Five minutes ago it was 60. Now it is 120. The replica is not catching up; it is falling further behind.

This is a replication lag death spiral. The SQL apply thread (or threads) cannot replay events as fast as the source generates them. Relay logs accumulate. Lag compounds because longer apply queues mean larger effective transactions, longer lock holds, and greater exposure to blocking queries on the replica.

MySQL Seconds_Behind_Source is unreliable: measuring real replication lag

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Seconds_Behind_Source is unreliable: measuring real replication lag

You run SHOW REPLICA STATUS and see Seconds_Behind_Source = 0. Both replication threads say Yes. You assume the replica is current. During a source write burst, you fail over and discover hours of missing transactions. This happens because Seconds_Behind_Source is not a real-time lag measurement. It is a timestamp diff between the replica’s clock and the timestamp of the event the SQL thread is currently applying. When the SQL thread reaches the end of the relay log, the metric snaps to zero even if the I/O thread has not fetched newer events. When the SQL thread stops, it returns NULL, which most lag alerts ignore. When the source was idle and then bursts, it jumps to the idle duration even though real lag is minimal. With parallel replication, it oscillates based on worker timing. For failover-critical decisions, you need a different measurement strategy.

MySQL Select_full_join > 0: joins running without an index

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Select_full_join > 0: joins running without an index

Select_full_join is climbing. In an OLTP system, this counter should stay at zero. Any sustained nonzero rate means at least one query is executing a join without a usable index, effectively performing a full cross-product scan on the joined table. The damage scales with table size. A single unindexed join against a million-row table can stall an otherwise healthy instance.

This is not a buffer pool problem or a connection storm. It is a query correctness issue. The optimizer cannot find an index to resolve the join predicate, so it reads every row of the referenced table for each row of the driving table. In MySQL 8.0.20 and later, the execution engine uses hash join for these cases (block nested loop was removed), but the underlying pathology remains identical: missing index, catastrophic read amplification.

MySQL semi-synchronous replication stall: commits hanging on ACK

Mon, 01 Jan 0001 00:00:00 +0000

MySQL semi-synchronous replication stall: commits hanging on ACK

Commits suddenly take 10 seconds or more and then time out. On the MySQL primary, Threads_running climbs while the commit rate flatlines. A moment later, commits resume, but Rpl_semi_sync_source_no_tx is ticking upward. The primary is waiting for a semi-synchronous replica to acknowledge receipt of binlog events, and that ACK is not arriving fast enough or at all.

Confirm the stall, find the missing ACK, and decide whether to fix the replica path or temporarily fall back to asynchronous replication.

MySQL slow after restart: buffer pool warm-up and the cold cache

Mon, 01 Jan 0001 00:00:00 +0000

MySQL slow after restart: buffer pool warm-up and the cold cache

You restart MySQL for maintenance, crash recovery, or a failover. Seconds later, query latency jumps from sub-millisecond to tens or hundreds of milliseconds. Disk read I/O saturates. The buffer pool hit ratio that normally sits above 99% has collapsed. There are no runaway queries, no lock waits, and no checkpoint stalls. The server is simply cold.

After any restart, the InnoDB buffer pool starts empty. Every data page read misses memory and goes to disk, so Innodb_buffer_pool_reads spikes. If the adaptive hash index is enabled, it is empty too. The server may accept client connections long before the pool is ready to serve production traffic. The issue looks like a capacity crisis, but it is usually transient and expected. Distinguishing a cold cache from a true capacity problem keeps you from chasing the wrong fixes during an incident.

MySQL slow commits with idle CPU: redo and binlog fsync pressure

Mon, 01 Jan 0001 00:00:00 +0000

MySQL slow commits with idle CPU: redo and binlog fsync pressure

Slow commits or transaction timeouts with idle CPU usually mean the bottleneck is in the durability path, not the query engine.

With innodb_flush_log_at_trx_commit=1 and sync_binlog=1, every COMMIT triggers an InnoDB redo log fsync() and a binary log fsync() before returning to the client. These are sequential, latency-bound operations. When storage cannot drain fsync requests as fast as the server generates them, commits queue up while the CPU waits.

MySQL slow queries: from Slow_queries to the slow log to the fix

Mon, 01 Jan 0001 00:00:00 +0000

MySQL slow queries: from Slow_queries to the slow log to the fix

When Slow_queries climbs but the slow log is empty, or long_query_time is still at the default 10 seconds, you have an instrumentation gap. Slow_queries increments regardless of whether slow_query_log is ON, so a rising counter with no log detail is a dead end.

This guide moves from the status counter to slow log configuration, then to performance_schema digests, and finally to resource signals that explain why queries slowed. The goal is to help you decide in minutes whether you face a query plan regression, lock contention, buffer pool pressure, or a configuration gap.

MySQL Sort_merge_passes climbing: filesort and sort_buffer_size

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Sort_merge_passes climbing: filesort and sort_buffer_size

Sort_merge_passes climbing fast is the signature of filesort spilling to disk. You notice it in SHOW GLOBAL STATUS, along with rising disk I/O on the tmpdir partition and a growing Handler_read_rnd. Queries that returned in milliseconds now take seconds.

The absolute value of this cumulative counter is meaningless. Rate of change is what matters. A rapid climb within hours means sorts that used to stay in memory are now writing chunks to disk and merging them in multiple passes. Each pass costs disk I/O and adds latency. Active threads pile up behind slow sorts, driving Threads_running higher and the Questions rate lower.

MySQL stuck in InnoDB crash recovery: why startup hangs after an unclean shutdown

Mon, 01 Jan 0001 00:00:00 +0000

MySQL stuck in InnoDB crash recovery: why startup hangs after an unclean shutdown

After an unclean shutdown, mysqld may look healthy: the process is running, port 3306 accepts connections, and SHOW GLOBAL STATUS LIKE 'Uptime' counts up. Yet applications time out, SELECT 1 hangs, and the error log prints InnoDB recovery messages. InnoDB is replaying the redo log to make data files consistent, rolling back incomplete transactions, merging the change buffer, and running purge. Until that finishes, queries are rejected.

MySQL tablespace fragmentation: reclaiming space with OPTIMIZE TABLE

Mon, 01 Jan 0001 00:00:00 +0000

MySQL tablespace fragmentation: reclaiming space with OPTIMIZE TABLE

InnoDB tablespaces fragment after bulk deletes, updates, and purge cycles. Logical data volume shrinks, but the on-disk file retains allocated extents because InnoDB returns freed pages to its internal free lists rather than the OS. For file-per-table .ibd tables, that unused space is visible as DATA_FREE in information_schema.TABLES and can be reclaimed with OPTIMIZE TABLE.

Tables inside the shared system tablespace ibdata1 do not shrink at the OS level. Freed space returns to InnoDB’s internal free lists, but ibdata1 never releases extents back to the filesystem. If the goal is reclaiming disk space on a full volume, OPTIMIZE TABLE against tables in ibdata1 is ineffective. The rest of this guide covers identifying reclaimable tables, running the rebuild safely, and verifying the result.

MySQL Threads_connected vs Threads_running: which one to actually alert on

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Threads_connected vs Threads_running: which one to actually alert on

Threads_connected counts every client holding an open socket, including idle pooled connections. Threads_running counts only threads actively executing a statement. Paging on the first while ignoring the second is a common postmortem mistake. Threads_running relative to CPU cores is the real load signal. The gap between the two metrics diagnoses failure modes. Connection count still deserves a ticket, but rarely a page.

MySQL Threads_created climbing: thread cache churn and missing pooling

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Threads_created climbing: thread cache churn and missing pooling

When the rate of Threads_created climbs to tens or hundreds of new threads per minute, the thread cache is not absorbing your connection churn. In a pooled deployment this rate should stay near zero. Every cache miss pays the full cost of OS thread creation and initialization, plus a TLS handshake if require_secure_transport is enabled. The first visible symptom is usually connection latency spikes or CPU time diverted to thread management.

MySQL TLS and encryption monitoring: unencrypted connections and weak auth plugins

Mon, 01 Jan 0001 00:00:00 +0000

MySQL TLS and encryption monitoring: unencrypted connections and weak auth plugins

MySQL 8.0 generates self-signed certificates on startup and enables TLS by default, but this does not guarantee encrypted traffic. Clients must explicitly request TLS over TCP, and misconfigurations often leave sessions in plaintext without errors. Long-lived accounts frequently remain on mysql_native_password despite its deprecation in MySQL 8.0.34, while caching_sha2_password requires secure transport or RSA key exchange on first connection, causing sporadic authentication failures when client drivers lack encryption configuration. These risks are invisible to standard availability monitoring. This guide covers the status variables, Performance Schema tables, and account audits needed to detect unencrypted connections and weak authentication plugins in production.

MySQL undo tablespace growing: ibdata1 bloat and undo truncation

Mon, 01 Jan 0001 00:00:00 +0000

MySQL undo tablespace growing: ibdata1 bloat and undo truncation

Disk space alert: /var/lib/mysql/ibdata1 is 40 GB larger than last week, or the undo_*.ibu files in MySQL 8.0 are consuming terabytes. Table sizes from information_schema do not justify the growth. This is almost always undo log accumulation from a blocked purge thread.

Undo logs let InnoDB reconstruct older row versions for MVCC and transaction rollback. The purge thread deletes them when no active transaction needs them. If one long-running transaction holds a read view, purge stops. History list length then grows linearly with write rate, and the undo tablespace expands to hold the backlog. In MySQL 5.7, that space lands in the system tablespace ibdata1, which never shrinks. In MySQL 8.0, separate undo tablespaces can truncate automatically, but only after the history list drains and the purge thread frees rollback segments. Until then, the disk fills.

MySQL Waiting for table metadata lock: diagnosing the DDL stall

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Waiting for table metadata lock: diagnosing the DDL stall

You run SHOW PROCESSLIST and see a wall of threads in Waiting for table metadata lock. An ALTER TABLE that should finish in seconds hangs for minutes. Queries against one table stop returning, the application connection pool drains, and CPU and disk look calm. This is a metadata lock (MDL) stall. It is a queueing failure, not resource exhaustion. Left alone, it cascades into connection exhaustion and a partial outage where every other table continues to work normally.

MySQL: got a packet bigger than 'max_allowed_packet' bytes - causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

MySQL: got a packet bigger than ‘max_allowed_packet’ bytes - causes and fixes

ER_NET_PACKET_TOO_LARGE (MySQL error 1153) drops the connection immediately. The payload crossed a limit on the client, the server, or a replica. A common mistake is tuning only the server: a source set to 256 MB still aborts a 20 MB payload if the mysql client is at its default 16 MB. Replicas add their own hard and soft ceilings. This guide shows how to identify which boundary failed, raise it without a restart where possible, and prevent recurrence.

Mystrotv SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nag LLC SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nagios Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Nagios Monitoring

What Is Nagios?

Nagios is a powerful, enterprise-grade IT infrastructure monitoring solution. It enables organizations to identify and rectify problems related to networks, systems, and applications by alerting users of the fault instances to ensure your IT systems are running efficiently. With a strong reputation in the field of network monitoring and performance management, Nagios has been a go-to solution for IT admins and engineers wanting to monitor critical IT infrastructure components.

Nagios Plugins and Custom Scripts

Mon, 01 Jan 0001 00:00:00 +0000

Nagios SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

named not responding on port 53: total outage versus UDP-works-TCP-fails

Mon, 01 Jan 0001 00:00:00 +0000

named not responding on port 53: total outage versus UDP-works-TCP-fails

named can be alive (pgrep -x named returns a PID) yet not answering on the serving interface. Process liveness checks pass, but clients experience timeouts and retries that propagate to every service depending on DNS.

Two diagnostic forks determine the fix path. The first: total outage (both UDP and TCP fail on the serving interface) versus UDP-works-TCP-fails. A total outage means the main DNS path is broken. A UDP-works-TCP-fails condition leaves large responses, DNSSEC-heavy answers, and zone transfers at risk while UDP-only health checks stay green. The second fork: loopback success versus serving-interface success. A query that succeeds against 127.0.0.1 confirms the daemon processes queries, not that real clients can reach it.

Nasuni Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nasuni Filer

Mon, 01 Jan 0001 00:00:00 +0000

NAT and session-table exhaustion: catching it before connections fail

Mon, 01 Jan 0001 00:00:00 +0000

NAT and session-table exhaustion: catching it before connections fail

New connections fail while existing ones keep working. Applications report “connection refused” or timeouts. Open SSH sessions stay alive, but new SSH attempts hang. Your monitoring shows the firewall or NAT gateway is up, interfaces are healthy, and CPU is normal. The session or NAT translation table is full.

Session-table exhaustion is a cliff-edge failure. The table degrades gracefully until it hits its limit, then every new connection is denied. Existing flows continue because their entries are already in the table. The symptom pattern is distinctive but easy to misdiagnose as application failure, DNS issues, or upstream provider problems, because the applications are the ones reporting errors.

Nateks Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

National Standardization Committee Of Radio Television SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

NATS

Mon, 01 Jan 0001 00:00:00 +0000

NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check

Mon, 01 Jan 0001 00:00:00 +0000

NATS /healthz explained: js-server-only vs js-enabled-only vs the bare check

A NATS server restarts with a large JetStream store. The process is fine, clients will be served shortly, but your pager fires and Kubernetes kills the pod before recovery finishes. The server starts recovering again, the probe fails again, and you are in a restart loop that looks like an outage but is a health check misconfiguration.

The cause is almost always the same: a liveness probe or a PAGE alert pointed at the bare /healthz endpoint on a JetStream-enabled server. On such a server, bare /healthz does not answer “is the process alive?” It runs the full JetStream health suite, and that suite legitimately fails for minutes while the server replays and recovers its assets after a restart.

NATS Authentication Timeout: clients that connect but never finish the handshake

Mon, 01 Jan 0001 00:00:00 +0000

NATS Authentication Timeout: clients that connect but never finish the handshake

Your NATS server log shows lines like this, and clients are complaining they cannot connect:

[ERR] 10.2.3.14:51824 - cid:1042 - Authentication Timeout

The confusing part: the TCP connection succeeded. The client reached the server. But the authentication handshake never finished, so the server sent -ERR 'Authentication Timeout' and closed the connection. This is not a bad-credentials problem. It is a timing problem: credentials (or the TLS handshake that must precede them) never arrived within the server’s auth window. For the related case where credentials arrive but are rejected, the log string is Authorization Violation and the diagnosis is different.

NATS Authorization Violation: authentication failures and credential rotation

Mon, 01 Jan 0001 00:00:00 +0000

NATS Authorization Violation: authentication failures and credential rotation

Your NATS server log is filling with Authorization Violation entries, clients are failing to connect, and the error message tells you almost nothing. That is deliberate: NATS keeps auth error messages vague so they do not leak information to attackers. The side effect is that the same log line covers a typo’d password, an expired user JWT, a bad credential file deployed to a fleet, and someone port-scanning your cluster from the internet.

NATS connection churn: a stable connection count hiding constant reconnects

Mon, 01 Jan 0001 00:00:00 +0000

NATS connection churn: a stable connection count hiding constant reconnects

Your NATS dashboard shows 4,000 client connections. It showed 4,000 an hour ago, and 4,000 yesterday. Everything looks stable. Meanwhile, clients are connecting and disconnecting hundreds of times per minute. Every reconnect burns CPU on protocol handshakes (and TLS handshakes, if enabled), the server logs fill with connect and disconnect events, and your auth system processes a constant stream of authentication attempts.

NATS connection storm: reconnect thundering herd after a network event

Mon, 01 Jan 0001 00:00:00 +0000

NATS connection storm: reconnect thundering herd after a network event

A network device fails and recovers. A load balancer health check flaps. A rolling restart drops a node. For a few seconds, every client attached to a NATS server loses its connection. Then the network heals, and every one of those clients tries to reconnect in the same second.

Each reconnect is not cheap: TCP handshake, optional TLS handshake, protocol negotiation, authentication, and a full resubscribe of every subscription the client held. Multiply that by hundreds or thousands of clients arriving simultaneously and you have a connection storm: a sharp spike in connections and total_connections, a CPU spike dominated by TLS handshakes if encryption is enabled, memory climbing as per-connection state is allocated, and in the worst case the server hitting max_connections or the OS file descriptor limit and rejecting clients. Rejected clients retry. Now you have a reject-and-retry loop on top of the storm.

NATS consumer stalled at MaxAckPending: delivery stops until messages are acked

Mon, 01 Jan 0001 00:00:00 +0000

NATS consumer stalled at MaxAckPending: delivery stops until messages are acked

Your JetStream consumer was processing messages fine, and then deliveries just stopped. No error on the client. No error in the server log. The stream keeps growing, CPU and memory are normal, and the consumer application is still connected. Everything is green except the one thing that matters: no messages are moving.

This is the most common cause of “JetStream consumer stopped receiving messages,” and it is silent by design. When a consumer’s count of delivered-but-unacknowledged messages (num_ack_pending) reaches its configured MaxAckPending limit, the server stops delivering new messages to that consumer. No error is raised, no advisory is emitted. Delivery pauses until pending messages are acknowledged, negatively acknowledged, or expire past AckWait.

NATS context deadline exceeded: JetStream publish and request timeouts

Mon, 01 Jan 0001 00:00:00 +0000

NATS context deadline exceeded: JetStream publish and request timeouts

Your application logs fill up with nats: context deadline exceeded. It shows up on js.Publish(), on js.StreamInfo(), on consumer fetches, sometimes on nats CLI commands like nats stream view. The NATS server is running, /healthz returns ok, and core NATS pub/sub traffic is flowing fine. Yet every JetStream operation hangs until the client’s context fires.

This error is a client-side deadline expiring. The client sent a JetStream API request and no response came back before the deadline. The request is not being rejected; it is not being answered at all. Somewhere between the client and the JetStream subsystem, the request is stuck, and the server is usually still healthy enough to look innocent.

NATS crash loop: unexpected uptime resets and repeated restarts

Mon, 01 Jan 0001 00:00:00 +0000

NATS crash loop: unexpected uptime resets and repeated restarts

Your NATS server’s /varz uptime keeps resetting. The server was up for 4 minutes, then 2 minutes, then 6. Clients reconnect repeatedly, JetStream streams flap between unavailable and recovering, and every restart replays the WAL from scratch. This is a crash loop, and the fix depends entirely on why the process is dying.

The uptime field on /varz is the fastest confirmation. It reports time since process start as a NATS-specific duration string with y/d/h/m/s suffixes (for example “1d2h3m4s”), not a standard Go duration. When that value drops between scrapes, the process restarted. A single restart may be maintenance. More than 3 restarts in 30 minutes is a crash loop and needs an owner.

NATS file descriptor exhaustion: too many open files and the ulimit cliff

Mon, 01 Jan 0001 00:00:00 +0000

NATS file descriptor exhaustion: too many open files and the ulimit cliff

Clients suddenly cannot connect. The server process is alive, CPU is fine, memory is fine, but every new connection fails and the logs are full of too many open files. Existing clients keep working, which makes it look like a network problem until you count sockets.

This is file descriptor exhaustion, and it is a cliff-edge failure. There is no graceful degradation: the moment the process hits its OS ulimit -n, accept() starts failing, JetStream cannot open new storage files, and cluster routes cannot establish. It is one of the most common NATS incidents, because the default ulimit -n of 1024 on Linux is far too low for a production message broker, while the default max_connections in NATS is 65536. The OS limit is the real ceiling, and it is usually the one nobody set.

NATS gateway disconnected: cross-cluster traffic cut in a supercluster

Mon, 01 Jan 0001 00:00:00 +0000

NATS gateway disconnected: cross-cluster traffic cut in a supercluster

Subscribers in cluster A have stopped receiving messages published in cluster B. Publishers see no errors. Every server’s /healthz returns ok, CPU and memory look normal, and yet a whole class of traffic has silently stopped flowing between two sites. In a NATS supercluster, this is the signature of a gateway disconnection: the outbound gateway connection to the remote cluster is missing, so no cross-cluster forwarding happens at all.

NATS goroutine leak: connection cleanup that never completes

Mon, 01 Jan 0001 00:00:00 +0000

NATS goroutine leak: connection cleanup that never completes

The symptom is a steady, one-directional climb. Goroutine count goes up day after day, RSS follows it, and nothing on the traffic side explains it: client connections are flat, routes are flat, throughput is flat. Then, weeks later, the server gets OOM-killed or starts showing GC and scheduling overhead that has nothing to do with message load.

A NATS goroutine leak is connection (or subsystem) cleanup that never completes. A goroutine is spawned to handle a connection, a timer, or a Raft loop; the work ends, but the goroutine never exits. It sits blocked on a channel receive that will never fire, a lock that will never be released, or a retry loop with nothing to retry. Each one is cheap individually, so the leak is invisible until it is not: at roughly 4-8KB of stack per goroutine, 100k leaked goroutines is about 400-800MB of RSS.

NATS high CPU: subject matching, TLS, GC, and the container GOMAXPROCS trap

Mon, 01 Jan 0001 00:00:00 +0000

NATS high CPU: subject matching, TLS, GC, and the container GOMAXPROCS trap

A NATS server that is pegged on CPU usually looks worse than it is, or better than it is, depending on what you are measuring. The cpu field in /varz is process CPU where 100 means one full core, so on a 16-core host a value of 800 is only 50% busy. Operators regularly page themselves on a number that is not actually saturation, or dismiss real saturation because the number “only” reads 300 on a 2-core container.

NATS insufficient storage / maximum bytes exceeded: JetStream publishes rejected

Mon, 01 Jan 0001 00:00:00 +0000

NATS insufficient storage / maximum bytes exceeded: JetStream publishes rejected

Your producers start throwing JetStream publish errors: “insufficient storage”, “maximum bytes exceeded”, or “maximum messages exceeded”. The server itself looks healthy: /healthz returns ok, connections are stable, CPU and memory are normal. Writes to one or more streams are failing anyway.

This is a storage-limit failure, not a server failure. JetStream enforces limits at three levels: per-stream (max_bytes, max_msgs, max_age), per-account storage quotas, and the server-wide JetStream storage reservation. When any of them is hit, the discard policy decides the failure mode. With DiscardNew, new publishes are rejected and the publisher sees an error. With DiscardOld, the server silently evicts the oldest messages to make room, which is data loss for any consumer that has not caught up.

NATS JetStream AckWait tuning: matching the ack timeout to processing time

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream AckWait tuning: matching the ack timeout to processing time

AckWait is the timer the JetStream server starts when it delivers a message to a consumer. If the client does not ack before the timer expires, the server redelivers the message. That is the entire mechanism, and most “JetStream consumer is stuck” incidents that are not crashes come down to this timer being mismatched to the actual processing time of the work behind it.

NATS JetStream API errors: reading the /jsz api.errors counter without false alarms

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream API errors: reading the /jsz api.errors counter without false alarms

You opened /jsz and saw api.errors in the thousands, or an alert fired because the counter moved. The first question is not “what broke” but “is this counter telling me something real.” The JetStream API error counter is cumulative, coarse-grained, and incremented by perfectly healthy client behavior. Alerting on its absolute value or on any movement at all is a guaranteed false-alarm generator.

NATS JetStream consumer lag growing: falling behind the stream

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream consumer lag growing: falling behind the stream

The NATS server is healthy. /healthz returns ok, throughput looks normal, CPU and memory are fine. But one stream keeps growing, and the consumer attached to it is not keeping up. num_pending climbs minute after minute, and every dashboard that only watches server-level metrics shows green.

Consumer lag requires per-consumer polling, not a single server endpoint, so most monitoring setups never see it. The result: a stream with millions of pending messages, effectively down while the server looks healthy.

NATS JetStream consumer stopped receiving messages: the diagnostic tree

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream consumer stopped receiving messages: the diagnostic tree

A JetStream consumer that stops receiving messages almost never raises an error. The stream keeps accepting publishes, the health endpoint stays green, and your application simply goes quiet. The signals that explain this live per-consumer, not per-server, which is why server-level dashboards show nothing.

There are only a handful of mechanisms that produce this symptom, and each has a distinct fingerprint in consumer state. This article is the diagnostic tree. Start at the top with nats consumer info (or /jsz?consumers=true), read five counters, and the tree tells you which failure you are in and where to go next.

NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream disabled unexpectedly: the persistence subsystem failed to come up

The server is up. Clients connect, core NATS routes messages, /healthz?js-server-only=true returns ok. But /jsz reports disabled: true on a node where JetStream should be running, and every stream, consumer, and KV bucket backed by this node is gone or degraded. Publishers that rely on persistence fail; request-reply still works, which is why this gets noticed late.

This state means one of two things: JetStream failed to initialize at startup, or it initialized and was later shut down by the server itself. In both cases the root cause is almost always below the NATS layer: the storage directory, the disk underneath it, or the configuration. The server rarely recovers on its own, and a blind restart can make a corrupt store worse.

NATS JetStream disk I/O stall: the disk has space but is too slow

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream disk I/O stall: the disk has space but is too slow

JetStream publishes are timing out or creeping upward in latency. df -h shows plenty of free space. The /jsz endpoint shows storage nowhere near its reserved limits, yet api.inflight sits high and api.errors keeps climbing. On the host, iowait is elevated and disk latency looks bad.

This is the JetStream disk I/O stall pattern: the disk has space but is too slow. It is a different failure from storage exhaustion, and it is frequently misdiagnosed because the obvious capacity checks all pass.

NATS JetStream meta cluster leader flapping: admin operations that keep failing

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream meta cluster leader flapping: admin operations that keep failing

Stream and consumer administration is failing, but existing JetStream traffic mostly continues. Creating a stream times out, consumer updates return errors, and retries sometimes succeed only to fail again a minute later.

This pattern points at the JetStream meta Raft group. The meta group manages cluster-wide JetStream metadata: stream and consumer create, update, and delete operations. Existing streams use separate Raft groups for replicated data, so the data plane can remain available while the administrative plane is unstable.

NATS JetStream mirror and source lag: stale replicas and DR recovery point

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream mirror and source lag: stale replicas and DR recovery point

A JetStream stream configured as a mirror of another stream, or sourcing from one or more upstream streams, keeps a local copy that is only as fresh as its sync connection. Stream info reports two fields per mirror or source: lag, the number of messages the local copy is behind, and active, the time since the last sync activity. When those numbers move the wrong way, the copy is going stale.

NATS JetStream no leader / cluster not currently available: writes blocked

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream no leader / cluster not currently available: writes blocked

Your publishers are failing with nats: no leader for stream or JetStream cluster not currently available, or every JetStream API call times out with no responders on $JS.API. Stream info shows an empty cluster.leader. Writes stay paused until the affected Raft group elects a leader again.

This is a quorum problem, not a load problem. JetStream uses Raft for consensus: one meta group manages all JetStream metadata and admin operations, and each replicated stream (and consumer) has its own Raft group. A group without a leader blocks everything that depends on it. If the meta group is leaderless, all JetStream admin operations fail cluster-wide. If only a stream’s group is leaderless, that stream stops accepting messages while the rest of JetStream keeps working.

NATS JetStream not enabled for account: persistence calls failing on a core server

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream not enabled for account: persistence calls failing on a core server

Your application connects to NATS successfully, then fails on the first persistence call: stream creation, consumer creation, or a publish that waits for an ack. The client error is one of two strings: nats: jetstream not enabled for account (err_code 10039) or nats: jetstream not enabled (err_code 10076). Core pub/sub keeps working the whole time, which is what makes this confusing during an incident.

NATS JetStream Raft election storm: leaders flapping and writes pausing repeatedly

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream Raft election storm: leaders flapping and writes pausing repeatedly

Your JetStream cluster is electing leaders over and over. Logs alternate rapidly between Stepping down and JetStream cluster new leader. Publishes intermittently time out, stream and consumer management calls error out, and some streams briefly report no leader at all. Then it settles for a few minutes and starts again.

This is a Raft election storm. It is not a clean failover and it is not usually a software bug. It is a symptom of resource starvation or latency somewhere in the cluster, and it feeds itself: elections consume CPU and I/O, which delays heartbeats further, which triggers more elections.

NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream redelivery loop: num_redelivered climbing and messages reprocessed

A JetStream consumer’s num_redelivered counter is climbing and your application is processing the same messages over and over. Nothing is lost, but nothing completes either: deliveries happen, acks do not, and the server keeps trying again.

This is a redelivery loop. The server delivers a message, the consumer fails to acknowledge it within AckWait, the server redelivers it, and each pass increments num_redelivered. Every redelivery is wasted work: duplicate side effects if your handler is not idempotent, duplicate load on downstream systems, and a num_ack_pending count that never drains.

NATS JetStream replica lag: a non-current replica that would lose data on failover

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream replica lag: a non-current replica that would lose data on failover

A replicated JetStream stream shows current: false with a non-zero lag for one of its replicas in nats stream info. Everything still works: publishes succeed, consumers receive messages, no alerts fire on server health. But the replication factor you configured is not the replication factor you have. If the leader fails right now, a failover to the lagging replica loses every message the leader acknowledged but the replica has not yet written.

NATS JetStream retention: limits vs interest vs workqueue and the /dev/null stream

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream retention: limits vs interest vs workqueue and the /dev/null stream

JetStream’s retention policy decides when a stored message is deleted, and the failure modes are asymmetric: get it wrong in one direction and storage grows until publishes are rejected; get it wrong in the other and every message you publish is deleted on arrival while the server reports itself perfectly healthy.

This article covers the three policies (limits, interest, workqueue), the exact conditions under which each one deletes a message, the silent /dev/null failure mode, and the checks that confirm your streams are retaining what you think they are. It assumes a working mental model of streams, consumers, and acks. If not, start with how NATS actually works in production.

NATS JetStream storage exhaustion spiral: stalled consumers that starve retention

Mon, 01 Jan 0001 00:00:00 +0000

NATS JetStream storage exhaustion spiral: stalled consumers that starve retention

JetStream storage is climbing steadily. Not a spike, a slope. Then publishes start failing with API errors, producers back up, and the pipeline degrades. The server looks healthy: connections stable, CPU and memory fine. The disk is filling anyway.

This is the JetStream storage exhaustion spiral: a deadlock where one or more consumers have stalled (usually pinned at MaxAckPending) on a stream using interest or workqueue retention. The server cannot delete a message until every interested consumer has acknowledged it, so a single dead consumer makes its entire backlog undeletable. Messages accumulate, storage approaches the configured limit, and new publishes are rejected. The system is stuck: consumers must process messages to free space, but the consumers are the broken part.

NATS leaf node disconnected: an edge server isolated from the hub

Mon, 01 Jan 0001 00:00:00 +0000

NATS leaf node disconnected: an edge server isolated from the hub

A leaf node connection is the single TCP session that ties an edge NATS server to your hub cluster. When it drops, the edge site keeps running locally, but it is cut off from the rest of the messaging fabric. Subscribers on the edge stop receiving messages published at the hub, and subscribers at the hub stop receiving anything published at the edge.

NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall

Mon, 01 Jan 0001 00:00:00 +0000

NATS Maximum Connections Exceeded: new clients rejected at the max_connections wall

New clients cannot connect to your NATS server. Existing connections keep working. Client logs show the protocol error -ERR 'Maximum Connections Exceeded' and then the connection closes. The server itself looks healthy: /healthz returns ok, messages still flow for connected clients, and nothing crashed.

This is the max_connections wall. The server has reached its configured connection limit and is rejecting every new connection during the handshake. There is no queueing and no graceful degradation. One slot short of the limit, everything works. At the limit, every new client is turned away.

NATS Maximum Payload Violation: messages rejected for exceeding max_payload

Mon, 01 Jan 0001 00:00:00 +0000

NATS Maximum Payload Violation: messages rejected for exceeding max_payload

A publisher calls publish, the call fails, and the NATS server log shows a Maximum Payload Violation. Seconds later the same client reconnects, publishes again, and gets disconnected again. The server is not broken: it is enforcing the configured max_payload limit, which defaults to 1 MB. The application, meanwhile, is in a publish-disconnect-reconnect loop and its messages are not flowing.

The protocol behavior is strict. When a client sends a message whose payload exceeds max_payload, the server responds with -ERR 'Maximum Payload Violation' and closes the connection. Client libraries that auto-reconnect come straight back and repeat the offense, which is why a single oversized publish looks like connection churn rather than a clean, one-time error.

NATS memory growth and OOM: reading RSS past the Go GC sawtooth

Mon, 01 Jan 0001 00:00:00 +0000

NATS memory growth and OOM: reading RSS past the Go GC sawtooth

The nats-server process is climbing toward its container memory limit. Grafana shows a jagged line that spikes to nearly double the baseline and drops back, except lately the drops are getting shallower and the floor keeps rising. Then the pod restarts, uptime resets to a few minutes, and the cycle repeats. If JetStream is enabled, the numbers look even worse, and half of what you see is not memory the server is actually holding.

NATS messages published but not received: subject mismatches and cross-server gaps

Mon, 01 Jan 0001 00:00:00 +0000

NATS messages published but not received: subject mismatches and cross-server gaps

The publisher’s Publish() call returns no error. The server is healthy. The subscriber receives nothing. This is almost never a server bug. It is how core NATS works: fire-and-forget routing against an in-memory subject tree, with no persistence, no delivery guarantee, and no signal when a message matches zero subscriptions.

In core NATS, a message published to a subject with zero matching subscribers is silently discarded. No error, no log line, no metric. The only server-level evidence is an asymmetry between in_msgs and out_msgs, which most teams never graph.

NATS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NATS Monitoring

What Is NATS?

NATS is a high-performance messaging system that enables distributed applications to generate and consume microservices in a seamless manner. It is widely used in various industries for cloud, IoT, and edge applications due to its lightweight and scalable architecture.

Monitoring NATS With Netdata

Netdata offers a cutting-edge NATS monitoring tool that provides real-time insights into the performance and health of your NATS servers. By leveraging Netdata’s capabilities, organizations can keep their messaging infrastructure efficient and reliable. For an interactive experience, you can try out our Live Demo.

NATS monitoring checklist: the signals every production server needs

Mon, 01 Jan 0001 00:00:00 +0000

NATS monitoring checklist: the signals every production server needs

A NATS server can pass every health check and still be losing messages. Core NATS drops messages to slow or absent subscribers by design, JetStream consumers can stall while every server-level metric looks green, and a node can be partitioned from its cluster while its local health endpoint returns 200. “Is the process running?” is the start of a monitoring strategy, not the end of one.

NATS monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

NATS monitoring maturity model: from survival to expert

Most NATS outages are not caused by a lack of metrics. The server exposes a rich monitoring API on port 8222. The failures happen because teams collect the wrong tier of signals for the failures they actually experience. A /healthz check tells you the process is alive. It tells you nothing about a consumer stalled at MaxAckPending, a route connection backing up with pending bytes, or a Raft meta cluster electing a new leader every ninety seconds.

NATS no responders available for request: request-reply into the void

Mon, 01 Jan 0001 00:00:00 +0000

NATS no responders available for request: request-reply into the void

Your service just started throwing nats: no responders available for request. The client got an answer back from the server almost instantly, and the answer was: nobody is listening on that subject. This is the request-reply counterpart to NATS’s silent message loss: instead of the request vanishing and the client waiting out its timeout, the server short-circuits the call and fails fast with a 503 status in the reply headers.

NATS pending bytes growing: catching a slow consumer before it is disconnected

Mon, 01 Jan 0001 00:00:00 +0000

NATS pending bytes growing: catching a slow consumer before it is disconnected

By the time slow_consumers in /varz increments, the damage is done. The server has declared a connection slow, and in core NATS the default response is to disconnect it. Messages buffered for that client are dropped, not queued, not retried. The client library auto-reconnects, resubscribes, immediately falls behind on the same backlog, and you are in a churn loop.

NATS Permissions Violation: authorized clients publishing or subscribing out of scope

Mon, 01 Jan 0001 00:00:00 +0000

NATS Permissions Violation: authorized clients publishing or subscribing out of scope

A Permissions Violation entry in the NATS server log means an authenticated client tried to publish or subscribe to a subject outside its permitted scope. The server rejected the operation, logged the event with the subject, client IP, and account, and sent an -ERR back to the client. The connection stays open. This is not an authentication failure: the credentials were accepted, but the permissions attached to them do not cover what the client tried to do.

NATS route disconnected: a missing cluster route means a partition

Mon, 01 Jan 0001 00:00:00 +0000

NATS route disconnected: a missing cluster route means a partition

One of your NATS servers shows fewer routes than it should. In a full-mesh cluster of N servers, every server should hold N-1 route connections, one to each peer. When that count drops, you do not have a degraded link. You have a partition.

The impact is asymmetric and easy to underestimate. The isolated server still accepts client connections, still reports healthy on /healthz, and still routes messages locally. But subscribers connected to it stop receiving messages published on the other side of the partition, and publishers on it vanish from the rest of the cluster’s view. If you run JetStream with replicated streams, Raft groups whose members span the partition can lose quorum, which turns a messaging gap into write failures.

NATS route RTT high: inter-server latency that triggers Raft elections

Mon, 01 Jan 0001 00:00:00 +0000

NATS route RTT high: inter-server latency that triggers Raft elections

You are looking at a JetStream cluster that keeps electing new leaders. The /jsz meta cluster leader changes every few minutes, stream writes intermittently fail with API errors, and nats server list or /routez shows round-trip times between servers far above what a same-datacenter cluster should produce. Server CPU and memory look fine. Nothing has crashed. Yet the cluster cannot hold a stable leader.

NATS route slow consumer: inter-server forwarding backing up cluster-wide

Mon, 01 Jan 0001 00:00:00 +0000

NATS route slow consumer: inter-server forwarding backing up cluster-wide

Your NATS server’s slow_consumer_stats.routes counter just went non-zero, or /routez shows a pending_size that keeps climbing on one route. This is not the same problem as a slow client. A route is the TCP connection that carries inter-server traffic between two NATS servers in a cluster. When it backs up, every subscriber reachable through that peer falls behind or stops receiving messages entirely, and in a JetStream cluster the degradation can extend to Raft heartbeat timing and leader stability.

NATS server not responding: healthz failing and the process down or hung

Mon, 01 Jan 0001 00:00:00 +0000

NATS server not responding: healthz failing and the process down or hung

Your probe against the NATS monitoring port is failing: /healthz returns non-200, or the port does not answer at all. Clients may still be connected, reconnecting in a storm, or already failing over to other cluster nodes.

Do not start with a restart. A crashed process, a hung event loop, lame-duck shutdown, resource exhaustion, and JetStream recovery can all look like “NATS is down,” and the first safe action is different for each. Restarting too early destroys evidence and can turn a recoverable hang into data loss.

NATS silent message loss: zero-subscriber drops and the in/out asymmetry

Mon, 01 Jan 0001 00:00:00 +0000

NATS silent message loss: zero-subscriber drops and the in/out asymmetry

Your NATS server is healthy. Health checks pass, connection counts are stable, there are no slow consumers, no errors in the logs. And yet a downstream service insists it has not received a single message in the last hour, while the producer’s metrics show it published thousands.

Both are telling the truth. In core NATS, a message published to a subject with zero subscribers is discarded at the moment of publish. The publisher gets no error. The server writes no log line. No counter records the drop. The message simply does not exist anymore.

NATS slow consumer breakdown: clients vs routes vs gateways and blast radius

Mon, 01 Jan 0001 00:00:00 +0000

NATS slow consumer breakdown: clients vs routes vs gateways and blast radius

Your NATS alert fired: slow_consumers is incrementing. You open /varz, see a non-zero counter, and start hunting for a misbehaving client. That is the right move only some of the time. The aggregate slow_consumers counter mixes four very different kinds of events, and two of them mean your cluster fabric itself is backing up, not a single subscriber.

NATS slow consumer detected: the write buffer overflowed and messages were dropped

Mon, 01 Jan 0001 00:00:00 +0000

NATS slow consumer detected: the write buffer overflowed and messages were dropped

Your NATS server log shows Slow Consumer Detected, or your client logged nats: slow consumer, messages dropped. In core NATS (no JetStream), that line means messages were dropped. Not queued, not retried, not parked somewhere for later.

The server buffers outbound messages per connection, and when a subscriber cannot drain its write buffer fast enough, the server sheds load by dropping messages for that subscriber or disconnecting it outright. Delivery in core NATS is at-most-once, and the slow subscriber is not told what was lost.

NATS stalled clients and stale connections: half-dead sockets and write-path distress

Mon, 01 Jan 0001 00:00:00 +0000

NATS stalled clients and stale connections: half-dead sockets and write-path distress

Two counters in /varz tell you about connections that are unhealthy but not yet dead: stalled_clients and stale_connections. Neither one means a connection has been dropped. That is the point: both describe states that precede visible failure. Stalled clients are on the way to becoming slow consumers; stale connections are sockets that look open but whose peer has stopped responding.

NATS stream max_bytes: one stream hitting its limit while the server has room

Mon, 01 Jan 0001 00:00:00 +0000

NATS stream max_bytes: one stream hitting its limit while the server has room

Publishers to one JetStream stream start getting rejected with a “maximum bytes exceeded” error. You check the server: JetStream storage is at 40% of reserved_storage, the disk has hundreds of gigabytes free, and every server-level dashboard is green. Nothing about the server looks full.

What happened: max_bytes is a per-stream limit, independent of server-level and account-level JetStream quotas. One stream reached its own configured ceiling and started enforcing its discard policy while every other stream kept working. If your monitoring only watches aggregate JetStream storage (/jsz without parameters), this failure is invisible until publishers start erroring.

NATS subscription leak: unbounded subscription growth and subject-trie bloat

Mon, 01 Jan 0001 00:00:00 +0000

NATS subscription leak: unbounded subscription growth and subject-trie bloat

The symptom is a number that only goes up. You open /varz on a NATS server and subscriptions is higher than it was yesterday, higher than it was last week, climbing for days while connections stays flat. Memory follows the same slope. Nothing is erroring. No slow consumers, no restarts, no complaints from applications. The server just keeps getting heavier.

That is a subscription leak: an application (or a bug) is registering subscriptions faster than it removes them. Every subscription lives in the server’s subject trie and in per-connection tracking state. Left alone, the leak consumes memory, slows subject matching, and in a cluster propagates interest across routes so that one leaky client inflates the subject trie on every server.

NATS TLS certificate expiry: an expired cert that locks out every new connection

Mon, 01 Jan 0001 00:00:00 +0000

NATS TLS certificate expiry: an expired cert that locks out every new connection

Every new client connection to your NATS server is failing at the TLS handshake with a verification error, typically x509: certificate has expired or is not yet valid. Clients that were already connected still work. That is what makes this failure mode deceptive: the server passes every process-level check while refusing all new work.

If you run mutual TLS on cluster routes and gateways, the blast radius is bigger. Route and gateway connections also fail to establish, so servers that restart or reconnect after a network event cannot rejoin the cluster. A certificate that expires during an unrelated incident can turn a recoverable blip into a full partition.

NATS write_deadline and buffer sizing: tuning how long the server waits on a slow writer

Mon, 01 Jan 0001 00:00:00 +0000

NATS write_deadline and buffer sizing: tuning how long the server waits on a slow writer

A client gets disconnected with Slow Consumer Detected in the server log. Someone raises write_deadline, the disconnects stop, and the ticket gets closed. Two weeks later the same client is back in the log, and now the server is also showing memory growth because the larger buffer window lets more messages pile up per connection. This is the standard lifecycle of a write_deadline tuning mistake.

Nature Remo E lite devices

Mon, 01 Jan 0001 00:00:00 +0000

Nature Remo E lite devices Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Nature Remo Monitoring

What Is Nature Remo?

Nature Remo E lite devices are innovative smart home solutions designed to enhance home automation and energy management. These devices enable you to control your home environment, focusing on convenience and efficiency, by integrating seamlessly into your smart home ecosystem.

Monitoring Nature Remo With Netdata

To monitor Nature Remo, Netdata employs an openmetrics (Prometheus) exporter, allowing the ingestion of data from any Prometheus exporter. This setup ensures you get automated dashboards, real-time alerts, and a comprehensive monitoring experience without needing a Prometheus server or Grafana. Get the community exporter here to easily set up Nature Remo monitoring.

Nbase Switch Communication SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

NEC BGP

Mon, 01 Jan 0001 00:00:00 +0000

Nec Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

NEC Univerge

Mon, 01 Jan 0001 00:00:00 +0000

Neoteris Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

NET Framework

Mon, 01 Jan 0001 00:00:00 +0000

Net Insight AB SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Net Snmp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Net To Net Technologies SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Net Track GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Net-SNMP Host

Mon, 01 Jan 0001 00:00:00 +0000

net.inet.icmp.stats

Mon, 01 Jan 0001 00:00:00 +0000

net.inet.ip.stats

Mon, 01 Jan 0001 00:00:00 +0000

net.inet.tcp.states

Mon, 01 Jan 0001 00:00:00 +0000

net.inet.tcp.stats

Mon, 01 Jan 0001 00:00:00 +0000

net.inet.udp.stats

Mon, 01 Jan 0001 00:00:00 +0000

net.inet6.icmp6.stats

Mon, 01 Jan 0001 00:00:00 +0000

net.inet6.ip6.stats

Mon, 01 Jan 0001 00:00:00 +0000

net.isr

Mon, 01 Jan 0001 00:00:00 +0000

Netapp

Mon, 01 Jan 0001 00:00:00 +0000

Netapp ONTAP API

Mon, 01 Jan 0001 00:00:00 +0000

NetApp ONTAP API Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NetApp ONTAP API Monitoring

What Is NetApp ONTAP API?

NetApp ONTAP API, renowned for its robust data management capabilities, is a storage operating system designed to efficiently manage and protect petabytes of data. Its performance directly impacts storage efficiency, making it essential for technical professionals, such as DevOps, IT admins, and SREs, to keep an attentive eye on its metrics. Monitoring the performance and health of the NetApp ONTAP API is critical in ensuring data integrity and optimizing storage functionalities.

NetApp Solidfire

Mon, 01 Jan 0001 00:00:00 +0000

NetApp Solidfire Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NetApp Solidfire Monitoring

What Is NetApp Solidfire?

NetApp Solidfire is a high-performing storage solution designed to manage data efficiently across your storage infrastructure. It provides seamless data handling capabilities that ensure optimal resource allocation and management. Solidfire is instrumental in handling workloads with demanding storage and performance requirements.

Monitoring NetApp Solidfire With Netdata

When it comes to monitoring NetApp Solidfire, Netdata takes advantage of an openmetrics (Prometheus) exporter. The NetApp Solidfire Exporter is utilized to gather pertinent metrics, which are essential for monitoring and enhancing overall system performance.

Netatmo sensors

Mon, 01 Jan 0001 00:00:00 +0000

Netatmo Sensors Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Netatmo Sensors Monitoring

What Are Netatmo Sensors?

Netatmo sensors are a range of smart home devices designed to monitor environmental conditions such as temperature, humidity, and air quality. These IoT devices play a crucial role in home automation and energy management, enabling users to adapt their environments for comfort and efficiency.

Monitoring Netatmo Sensors With Netdata

Effective tools for monitoring Netatmo Sensors are essential for ensuring optimal performance and timely insights. Netdata provides a comprehensive solution for monitoring Netatmo Sensors without the complexity of setting up separate Prometheus servers or Grafana dashboards. By using an openmetrics (Prometheus) exporter, Netdata can ingest data from the Netatmo exporter, offering automated dashboards and alerts to keep track of your smart home devices’ performance in real time.

Netbotz SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

NetBox

Mon, 01 Jan 0001 00:00:00 +0000

Netcomm Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Netdata at AWS Cloud Days 2023

Mon, 01 Jan 0001 00:00:00 +0000

Ralph Meijer, VP of Technology at Netdata, joined a panel at AWS Cloud Days Athens 2023 on October 10. The session, “Running Managed Containers in AI-Powered Startups,” ran from 16:45 to 17:15 and focused on the operational side of containerized workloads in fast-moving companies.

The discussion covered what it actually takes to run containers in production when your team is small and your infrastructure is changing weekly. Observability came up repeatedly – specifically the gap between what managed container services give you out of the box and what you actually need to debug a problem at 2 AM. Ralph brought Netdata’s perspective: auto-detection of containers, per-second metrics without manual configuration, and the ability to monitor ephemeral workloads that spin up and disappear before traditional tools finish their first collection cycle.

Netdata at Civo Navigate 2023

Mon, 01 Jan 0001 00:00:00 +0000

Netdata sent two speakers to Civo Navigate Europe 2023 in London on September 5–6. Ralph Meijer, VP of Technology, opened with “Opinionated Observability” at noon on day one – a talk about the design trade-offs behind monitoring defaults and why those choices matter more than most vendors admit. The argument: when your tool ships with good opinions about what to collect, how to visualize it, and when to alert, you remove an entire class of configuration toil.

Netdata at Conf42 Cloud Native 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis spoke at Conf42 Cloud Native 2024 (virtual, March 2024) with “Practical AI with Machine Learning for Observability in Netdata.” The talk was a technical walkthrough of how Netdata applies unsupervised machine learning to metrics – not as a feature checkbox, but as a way to surface problems that static thresholds miss.

The key insight Costa presented: individual anomalies on individual metrics are often noise. A CPU spike on one node, a latency bump on one service – these happen constantly and mean nothing on their own. But when anomalies converge across multiple metrics and services simultaneously, that convergence is a strong signal that something unusual is actually happening. As he put it: “The power of ML becomes evident when seemingly noisy anomalies converge across various services, serving as indicators of something exceedingly unusual.”

Netdata at Conf42 DevOps 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis spoke at Conf42 DevOps 2024 on January 25 with a talk titled “Observability Standardization: the elephant is still in the room!” The event was virtual, so the audience was global.

The premise of the talk was blunt: the industry talks endlessly about open standards and protocol compatibility for observability, but the real question is whether we actually need custom monitoring solutions for every infrastructure scenario. OpenTelemetry, Prometheus exposition format, StatsD, SNMP – there is no shortage of standards. The problem is not the number of protocols. It is that the underlying data models, collection patterns, and retention strategies are so different across tools that “standardization” often just means another translation layer rather than true interoperability.

Netdata at Conf42 DevOps 2025

Mon, 01 Jan 0001 00:00:00 +0000

Shyam Sreevalsan spoke at Conf42 DevOps 2025 on January 23 (virtual) about the future of observability. The talk covered three threads: AI-driven anomaly detection, real-time processing at the edge, and predictive analytics as a replacement for reactive alerting.

The central argument was that monitoring is shifting from “tell me when something breaks” to “tell me before something breaks.” Proactive monitoring requires two things that most traditional tools lack: per-second data collection (because you cannot predict what you cannot see) and ML models that run continuously rather than on a query schedule. Shyam walked through how Netdata handles both – distributed agents collecting every second, unsupervised ML training on each metric individually, and anomaly correlation that surfaces coordinated deviations before they become incidents.

Netdata at Data Centre World London 2025

Mon, 01 Jan 0001 00:00:00 +0000

Netdata exhibited at Data Centre World London 2025, March 12–13 at ExCeL London, at Booth DC476 in the co-located Tech Show London event. We ran live infrastructure monitoring demos throughout both days and held a LEGO draw that produced two winners.

The audience here was different from a developer conference. Data centre operators and infrastructure managers care about things like power monitoring, cooling efficiency, network throughput, and server fleet health – all at scale, all in real time. The conversations at the booth centered on three recurring pain points: the cost of existing observability tools (especially per-GB ingestion pricing), the complexity of maintaining multiple monitoring systems for different parts of the stack, and the time wasted on manual troubleshooting when dashboards show 60-second averages that hide the actual problem.

Netdata at DevOops Athens Meetup 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis spoke at the DevOops Athens Meetup in December 2024, telling the origin story of Netdata. The project was born out of a production incident – a real problem that existing monitoring tools failed to diagnose quickly enough. That experience shaped everything that followed: the insistence on per-second granularity, the zero-configuration philosophy, the decision to run monitoring at the edge rather than depending on a remote backend.

For a local meetup in Athens, the tone was more personal than a typical conference talk. Costa could talk about the early days of the project, the decisions that felt risky at the time, and how those choices played out over the years. The audience was Athens-based DevOps practitioners – people who work in the Greek tech ecosystem and, in many cases, had been following the project since its early days.

Netdata at DevOps Meetup Finland 2024

Mon, 01 Jan 0001 00:00:00 +0000

Shyam Sreevalsan presented “Opinionated Observability” at the DevOps Finland Meetup in May 2024. The talk picked up a thread from Ralph Meijer’s earlier presentation at Civo Navigate and adapted it for a meetup audience – shorter, more conversational, and grounded in specific examples.

The core question: what happens when you move from collecting metrics to actually visualizing and alerting on them? That transition is where most monitoring setups break down. You install an agent, metrics flow into a database, and then you spend weeks building dashboards, tuning thresholds, and configuring notification rules. Shyam walked through the trade-offs involved in making those decisions for the user – good defaults for visualization, sensible alert thresholds out of the box, and the risks of being too opinionated versus not opinionated enough.

Netdata at DevOps Summer Retreat Latvia 2024

Mon, 01 Jan 0001 00:00:00 +0000

Shyam Sreevalsan spoke at the DevOps & Agile Summer Retreat in Latvia in July 2024. His talk, “The Future of DevOps: The Next 10 Years,” was an attempt to lay out a technical roadmap for where the discipline is heading – not vague predictions, but specific shifts already underway.

Three themes anchored the talk. First, AI-driven automation: not replacing engineers, but handling the repetitive diagnostic and remediation tasks that consume on-call time. Second, DevSecOps becoming the default rather than the exception – security as a continuous, embedded practice rather than a gate at the end of a pipeline. Third, decentralized autonomous teams with decentralized tools and workflows, where each team owns its full stack including observability, rather than relying on a central platform team to configure monitoring for them.

Netdata at DevOpsDays Geneva 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis spoke at DevOpsDays Geneva 2024 with a talk that had one of the more specific titles we have brought to a conference: “A powerful logs management solution we all have and use, but we underestimate: systemd-journal.” We also had a booth.

The argument was simple. Every Linux system running systemd already has a structured, indexed, binary log store built in. Most teams ignore it and ship logs to an external system immediately, paying for ingestion, storage, and query infrastructure that duplicates something the OS already provides. Costa walked through how systemd-journal works, what it stores, how Netdata integrates with it for log exploration and correlation with metrics, and where it falls short (multi-node aggregation being the obvious gap).

Netdata at DEVworld Amsterdam 2025

Mon, 01 Jan 0001 00:00:00 +0000

Netdata exhibited at DEVworld Conference in Amsterdam in February 2025, at Booth 18E. No talk at this one – just the team at the booth, running live demos and talking to developers.

DEVworld attracts a broad developer audience, not just infrastructure or DevOps specialists. That meant explaining Netdata to people who might not have thought much about monitoring beyond “is my app up or down.” The live demos worked well for this: showing real-time, per-second dashboards to someone who has only ever seen delayed, aggregated metrics makes the difference immediately obvious.

Netdata at FOCS 2026 — The Future of Cybersecurity Summit

Mon, 01 Jan 0001 00:00:00 +0000

Netdata is at the 7th Future of Cybersecurity Summit (FOCS) 2026, organized by the PIKOM Cybersecurity Chapter in Petaling Jaya, Malaysia, alongside our regional partner Cloud Engine Digital (CED). CED is an exhibition partner at FOCS and brings expertise in cloud infrastructure, security, and data engineering across Southeast Asia. This year’s theme is “Cybersecurity 2030: Building Resilience in an AI-Driven, Borderless World,” and the agenda covers AI-driven security operations, predictive risk management, and the regulatory landscape across the region.

Netdata at FOSDEM 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis presented in the Observability Devroom at FOSDEM 2024 in Brussels, starting at 16:30. The talk covered the journey of Netdata – how it started, the architectural decisions that shaped it, and the challenges of building a distributed monitoring system in the open.

FOSDEM is not a vendor event. The audience is developers and maintainers who care about how things work, not how they are marketed. Costa’s talk leaned into that: the early design goal of per-second granularity, the choice to run everything at the edge rather than relying on a centralized backend, and the ongoing tension between keeping the project simple for individual users while scaling it for organizations with thousands of nodes.

Netdata at FOSSCOMM 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis spoke at FOSSCOMM 2024, held November 9–10 at the University of Macedonia in Thessaloniki. FOSSCOMM is Greece’s largest open-source conference, and for Netdata – a project with Greek roots – it was a homecoming of sorts.

The talk, “Netdata: Open Source, Distributed Observability Pipeline – Journey and Challenges,” covered the full arc: how the project started, the architectural decisions that defined it, the community that grew around it, and the ongoing challenges of maintaining a large open-source codebase that millions of nodes depend on. Costa did not shy away from the hard parts – the difficulty of balancing open-source community expectations with commercial product development, and the engineering cost of supporting an agent that runs on everything from a Raspberry Pi to a 256-core production server.

Netdata at Gartner IOCS Las Vegas 2025

Mon, 01 Jan 0001 00:00:00 +0000

Netdata exhibited at the Gartner IT Infrastructure, Operations & Cloud Strategies Conference (IOCS) 2025, December 9–11 at The Venetian in Las Vegas. We were at Booth #636 in the Solution Village: Operations section, running live demos every 30 minutes and meeting with IT leaders from enterprises and mid-market organizations.

The highlight was Costa Tsaousis’s talk on December 11 at 12:30 PM PST in Theater 3: “Why 80% of Organizations Will Overspend for Observability in 2026.” The thesis was direct. Legacy observability tools have a broken economic model – they charge by data volume, which means the more you monitor, the more you pay. That creates a perverse incentive to sample, average, and reduce data, which in turn creates blind spots. Organizations end up paying more for less visibility. Costa walked through how sampling at 15–60 second intervals masks the very spikes and anomalies that cause incidents, and how that hidden cost – longer MTTR, more outages, more manual investigation – often exceeds the monitoring bill itself.

Netdata at GITEX AI ASIA 2026

Mon, 01 Jan 0001 00:00:00 +0000

Netdata is at GITEX AI ASIA 2026, April 9–10 at Marina Bay Sands in Singapore, together with our regional partner Cloud Engine Digital (CED). GITEX AI ASIA is the region’s largest tech and AI event, drawing enterprise leaders, startups, and investors from over 110 countries.

We’re showing live demos of Netdata’s real-time monitoring—per-second granularity across infrastructure, from Kubernetes clusters to bare metal servers. For teams running AI and ML workloads, this means capturing GPU utilization spikes, training pipeline anomalies, and inference latency fluctuations that 15–60 second sampling intervals simply cannot see.

Netdata at Howard Conference and Expo 2026

Mon, 01 Jan 0001 00:00:00 +0000

Netdata exhibited at the Howard Conference and Expo “Game On” event, February 24–26 at the Grand Hotel Marriott Resort in Fairhope, Alabama. Constantine Nikitiadis and Stuart McMurran were on the ground, running demos and talking with IT leaders attending the conference.

The demos focused on two things: per-second granularity versus the 15–60 second sampling that most monitoring tools default to, and AI-assisted troubleshooting that turns hours of dashboard investigation into minutes of directed analysis. For an audience of IT practitioners managing diverse infrastructure – bare metal, VMs, cloud, some Kubernetes – the “works everywhere” message landed well. One agent, one install, coverage across the full stack.

Netdata at India DevOps Show 2025

Mon, 01 Jan 0001 00:00:00 +0000

Netdata was a Co-Presenting Partner at the 7th Edition India DevOps Show on May 23, 2025 at the Holiday Inn in Mumbai. Satyadeep Ashwathnarayana, Constantine Nikitiadis, and Shyam Sreevalsan represented the team.

The talk – “Stop Building Dashboards. Start Solving Problems.” – challenged the default approach to observability that most DevOps teams fall into: install a monitoring tool, spend weeks building dashboards, and then stare at those dashboards during incidents trying to find the problem. The argument is that dashboards are a means, not an end. If your monitoring tool can surface the problem directly – through anomaly detection, automated correlation, and AI-driven investigation – then the dashboard becomes a verification step, not a diagnostic one.

Netdata at India DevOps Show 2026

Mon, 01 Jan 0001 00:00:00 +0000

Netdata participated as a Silver Partner at the 10th Edition India DevOps Show on February 13, 2026 at the Aloft ORR Hotel in Bengaluru. Shyam Sreevalsan and Constantine Nikitiadis represented the team, meeting with DevOps practitioners and tech leaders from across India.

The focus was on showing how per-second metrics change the way teams respond to deployments and incidents. When you can see the impact of a deployment within seconds rather than waiting for the next polling interval, rollback decisions get faster and more confident. We demonstrated Netdata’s AI-assisted troubleshooting – how it analyzes anomalies across the stack and produces actionable investigation reports in minutes rather than the hours of manual dashboard-digging that most teams are used to.

Netdata at ObservabilityCon 2023

Mon, 01 Jan 0001 00:00:00 +0000

Netdata had a booth at ObservabilityCon 2023 in London, hosted by Grafana Labs. No talk this time – just the team at a booth, ready for deep-dive conversations about how different observability tools fit together.

The event drew people who are already invested in open-source observability, many of them running Grafana alongside other tools. That made for pointed, technical conversations. A common thread: people were interested in how Netdata handles high-resolution metrics collection at the edge without requiring a heavy backend. The idea of collecting per-second data, running ML-based anomaly detection locally, and then feeding results into Grafana dashboards resonated with teams already comfortable in that ecosystem.

Netdata at Open Source Monitoring Conference 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis opened the final day of the Open Source Monitoring Conference (OSMC) 2024 in Nuremberg on November 21, speaking from 9:30 to 10:00 at the Jacobi venue. His talk, “Netdata: Open Source, Distributed Observability Pipeline – Journey and Challenge,” covered the project’s evolution and the architectural decisions that set it apart from other open-source monitoring tools.

OSMC runs November 19–21 and is one of the more established events in the European IT monitoring space. The attendees are people who run Icinga, Checkmk, Prometheus, Zabbix, and similar tools in production. They know monitoring. What they wanted to hear was how Netdata’s distributed approach – agents collecting and processing data at the edge, with no mandatory centralized storage – compares to the centralized architectures they are used to.

Netdata at Open Source Observability Day 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis spoke at Open Source Observability Day 2024 on October 24. The talk focused on the evolution of Netdata and the broader challenges facing open-source observability tooling.

The tension Costa addressed is one that every open-source monitoring project faces: how do you keep things simple for a developer who just wants to monitor a few servers, while also scaling to organizations with thousands of nodes and complex compliance requirements? Netdata’s answer has been a distributed architecture – agents at the edge doing the heavy lifting, with optional cloud coordination – but that comes with its own set of challenges around consistency, aggregation, and user experience.

Netdata at OpenConf 2025

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis spoke at OpenConf 2025, held November 21–22 at Dais Events in Athens. The talk, “Practical AI and Machine Learning for Observability in Netdata,” presented a specific framing of anomaly detection that Costa has been developing across multiple conferences: ML as an advisor, not just an alert trigger.

The distinction matters. Most ML-in-monitoring implementations boil down to “replace static thresholds with dynamic ones.” Netdata’s approach is different. Multiple independent ML models run on each node, each trained on a single metric’s behavior. When one model flags an anomaly, that is information but not necessarily action. When dozens of models across multiple services flag anomalies simultaneously, that convergence is a strong signal. The system acts as an advisor – surfacing unusual patterns, predicting potential failures, and detecting early signs of security breaches – rather than firing off yet another alert.

Netdata at QBITS 2025

Mon, 01 Jan 0001 00:00:00 +0000

Netdata sponsored QBITS 2025 in Montreal, April 8–10. Shyam Sreevalsan (VP - Product & Strategy) and Stuart McMurran (Enterprise Sales) represented the team at this Quadbridge event, which brings together IT leaders from across Canada and the US.

The audience was heavily tilted toward infrastructure decision-makers – CTOs, VPs of IT, directors of operations. These are people who sign off on monitoring tool purchases and live with the consequences. Three pain points came up in nearly every conversation at the booth: cost (observability bills that scale unpredictably with data volume), complexity (too many tools, too many dashboards, too much configuration), and manual troubleshooting (spending hours correlating data across systems during an incident).

Netdata at SREcon 2023

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis, Netdata’s founder and CEO, joined a panel at SREcon23 Americas on October 11, 2023 in San Francisco. The session, “Open-source Development as a Full-time Pursuit,” ran from 16:50 to 17:30 and brought together maintainers who build open-source infrastructure tooling as their day job – not as a side project.

The panel dug into the realities of sustaining an open-source project when it is also the foundation of a company. Topics included funding models, the tension between community contributions and product roadmap, and the practical challenge of keeping a project healthy when you have both volunteer contributors and a paid engineering team pulling in potentially different directions.

Netdata at SREday London 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis spoke at SREday London 2024 on September 19–20 with “Practical AI with Machine Learning for Observability in Netdata.” The event ran a promo code – COSTA10 – for 10% off tickets, which brought some extra traffic our way.

The talk was tailored for an SRE audience. These are people who live in dashboards, write alert rules, and get paged at night. They are skeptical of ML claims because they have seen too many tools that promise “intelligent alerting” and deliver more noise. Costa focused on the mechanics: how Netdata trains unsupervised models per metric at the edge, why anomaly convergence across metrics matters more than any single anomaly score, and how this translates to fewer false positives in practice.

Netdata at stackconf 2024

Mon, 01 Jan 0001 00:00:00 +0000

Costa Tsaousis spoke at stackconf 2024 in Berlin on June 19, in a 14:30–15:00 slot. The talk traced the history of Netdata from its inception: the original goals, the architectural bets, and what held up over time.

The starting point was straightforward. When Costa began building Netdata, the goal was high-resolution metrics – per-second granularity, not the 10- or 60-second averages that were standard at the time. That required a fundamentally different collection architecture: lightweight agents that process data locally, real-time visualization that does not depend on a query round-trip to a central database, and auto-detection of services so that adding a new node does not require writing configuration files.

Netdata at Tech Show London 2026

Mon, 01 Jan 0001 00:00:00 +0000

Netdata is exhibiting at Tech Show London 2026, March 4–5 at ExCeL London. We are at Booth F223 in the Cloud & AI Infrastructure zone. Constantine Nikitiadis and Shyam Sreevalsan are running the booth, with 1:1 meetings available for anyone who wants dedicated time.

The booth features live monitoring demos comparing per-second granularity against the 10–60 second averages that most tools deliver. For teams running AI and cloud infrastructure, that difference is not academic – GPU utilization spikes, model training anomalies, and container scheduling events happen on sub-second timescales that traditional monitoring simply misses. We are also showing Netdata’s AI-powered infrastructure observability: how unsupervised ML at the edge catches anomalies across your stack without requiring manual threshold configuration.

Netdata at WeAreDevelopers World Congress 2024

Mon, 01 Jan 0001 00:00:00 +0000

Netdata was at WeAreDevelopers World Congress 2024 in Berlin, July 17–19. We had a booth in Hall A at stand A_S10, and Costa Tsaousis gave a talk on “Practical AI with Machine Learning in Observability.”

The booth was packed for most of the event. We ran live demos of Netdata’s real-time dashboards, had a spin-the-wheel game, and did a prize draw. The wheel brought people in; the demos kept them. Developers who stopped by expecting a quick spin ended up watching per-second metrics streaming across a live infrastructure and asking how the anomaly detection works under the hood.

Netdata at WeAreDevelopers World Congress 2025

Mon, 01 Jan 0001 00:00:00 +0000

Netdata returned to WeAreDevelopers World Congress in Berlin as a partner for the 2025 edition, July 9–11. Costa Tsaousis gave “Practical AI with Machine Learning in Observability” – a talk he has been refining throughout 2024 and into 2025, each time sharpening the examples and responding to questions from previous audiences.

The booth was busy all three days. WeAreDevelopers draws tens of thousands of developers, and the crowd skews younger and more curious than a typical infrastructure conference. Many visitors had not thought deeply about monitoring before – they knew they needed it, but had not compared tools or architectures. Live demos of Netdata running on real infrastructure, showing per-second metrics with zero configuration, gave them a baseline to compare against whatever they try next.

Netdata Enterprise Agent

Mon, 01 Jan 0001 00:00:00 +0000

Netdata Mobile App

Mon, 01 Jan 0001 00:00:00 +0000

Netdata Pricing: Free Up to 5 Nodes | From $4.50/node

Mon, 01 Jan 0001 00:00:00 +0000

Netdata Streaming Topology

Mon, 01 Jan 0001 00:00:00 +0000

Netdata usage survey

Mon, 01 Jan 0001 00:00:00 +0000

Netdata vs Chronosphere | Monitoring Tools Comparison

Mon, 01 Jan 0001 00:00:00 +0000

Netdata provides edge-native observability with per-second granularity, automatic ML anomaly detection, and transparent pricing—eliminating PromQL learning curves and SaaS-only limitations that challenge Chronosphere users.

Netdata vs KloudFuse | Monitoring Tools Comparison

Mon, 01 Jan 0001 00:00:00 +0000

Discover why Netdata delivers instant infrastructure visibility with per-second granularity and transparent pricing, while KloudFuse requires days of setup and quote-based costs.

Netdata vs Microsoft SCOM | Monitoring Tools Comparison

Mon, 01 Jan 0001 00:00:00 +0000

Netdata vs Site24x7 | Monitoring Tools Comparison

Mon, 01 Jan 0001 00:00:00 +0000

Netdata: Monitoring and troubleshooting transformed

Mon, 01 Jan 0001 00:00:00 +0000

Netfilter

Mon, 01 Jan 0001 00:00:00 +0000

NetFlow

Mon, 01 Jan 0001 00:00:00 +0000

NetFlow storage sizing: how much disk your flow collector really needs

Mon, 01 Jan 0001 00:00:00 +0000

NetFlow storage sizing: how much disk your flow collector really needs

Flow records arrive at thousands to tens of thousands per second, and every record hits disk. The bottleneck is almost always disk throughput or capacity, not CPU.

This article covers the math: raw record sizes, effective storage after columnar compression, the capacity formula with worked examples, IOPS considerations, and the operational pitfalls that make disks fill faster than the formula predicts. The guidance applies to NetFlow v5/v9, IPFIX, and sFlow collectors using ClickHouse or similar columnar backends.

NetFlow v9/IPFIX template desync: flows decoded wrong or dropped after a reboot

Mon, 01 Jan 0001 00:00:00 +0000

NetFlow v9/IPFIX template desync: flows decoded wrong or dropped after a reboot

You rebooted a router or upgraded its firmware. Minutes later, your flow collector shows a gap or anomaly. The exporter is still sending data: UDP packet counters are nonzero and climbing. But decoded flow records are zero, suspiciously low, or the field values are shifted and garbled.

This is NetFlow v9 or IPFIX template desync. The collector holds cached template definitions that no longer match what the exporter is sending. Until it receives and caches the correct templates, it either drops records silently or misinterprets the byte layout, producing garbage fields.

NetFlow vs sFlow vs IPFIX: what they measure and how each one fails

Mon, 01 Jan 0001 00:00:00 +0000

NetFlow vs sFlow vs IPFIX: what they measure and how each one fails

Flow telemetry protocols are often lumped together as “flow data,” but they measure fundamentally different things. NetFlow and IPFIX build stateful flow records by tracking conversations in device memory. sFlow captures random packet samples without maintaining any flow state. This architectural split determines not only what you can see but how the data breaks when something goes wrong.

Netgear

Mon, 01 Jan 0001 00:00:00 +0000

Netgear Access Point

Mon, 01 Jan 0001 00:00:00 +0000

Netgear Readynas

Mon, 01 Jan 0001 00:00:00 +0000

Netgear SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Netgear Switch

Mon, 01 Jan 0001 00:00:00 +0000

Netline SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Netpartner S R O SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Netquest Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Netrake Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Netreality Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Netscaler SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Netscreen Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Netstar Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Network Alchemy Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Network Appliance Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Network Connections

Mon, 01 Jan 0001 00:00:00 +0000

Network interfaces

Mon, 01 Jan 0001 00:00:00 +0000

Network monitoring checklist: the signals every production network needs

Mon, 01 Jan 0001 00:00:00 +0000

Network monitoring checklist: the signals every production network needs

This checklist covers the signals production networks need, organized by detection priority and mapped to maturity levels from survival to expert.

An NPM stack is a federation of collectors, parsers, enrichment services, storage tiers, and an analytics core. Most production incidents are not “the network broke” but “a collector’s UDP buffer dropped packets,” “the NetFlow v9 template cache went stale after a device reboot,” or “the polling worker pool fell behind and now a healthy device looks down.” The checklist is organized to surface those failure modes, not just the top-level symptoms.

Network statistics

Mon, 01 Jan 0001 00:00:00 +0000

Network Subsystem

Mon, 01 Jan 0001 00:00:00 +0000

Network Technologies Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Networth Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

New Oak Communications Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

New Relic

Mon, 01 Jan 0001 00:00:00 +0000

Newbridge Networks Corporation SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Newtec Cy SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nextcloud Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Nextcloud Monitoring

What Is Nextcloud?

Nextcloud is a widely-used, self-hosted productivity platform that offers a suite of client-server software for creating and using file hosting services. It is designed to allow users to share and collaborate on documents, manage files, and streamline communication all within the cloud. As a cornerstone of cloud services and computing, ensuring the optimal performance of your Nextcloud servers is critical for scalability and efficient operations.

Nextcloud servers

Mon, 01 Jan 0001 00:00:00 +0000

NextDNS

Mon, 01 Jan 0001 00:00:00 +0000

NextDNS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NextDNS Monitoring

What Is NextDNS?

NextDNS is a powerful DNS resolver and security platform, tailored to provide efficient DNS management and enhance security. It’s an essential tool for IT teams looking to optimize their network infrastructure by offering better privacy, security, and performance.

Monitoring NextDNS With Netdata

Monitor NextDNS effortlessly with Netdata, which employs an OpenMetrics (Prometheus) exporter for seamless integration. This enables you to gather crucial DNS resolver and security metrics, helping you manage your domain’s name system more efficiently. One of the standout features of Netdata is its ability to ingest data from any Prometheus exporter, providing automated dashboards and alerts without requiring a dedicated Prometheus server or Grafana setup.

Nextnet SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

NFS Client

Mon, 01 Jan 0001 00:00:00 +0000

NFS Server

Mon, 01 Jan 0001 00:00:00 +0000

NGINX

Mon, 01 Jan 0001 00:00:00 +0000

NGINX $request_time vs $upstream_response_time: isolating where latency lives

Mon, 01 Jan 0001 00:00:00 +0000

NGINX $request_time vs $upstream_response_time: isolating where latency lives

P95 $request_time doubles. The default assumption is upstream degradation, so you scale the backend, tune the database, and add instances. The latency barely moves. The bottleneck was a slow mobile client, proxy temp-file disk I/O, or a large request body on a lossy network. This is the most common nginx misdiagnosis.

$request_time measures the full cycle from the first byte read from the client to the last byte sent to the client. It includes reading the request, waiting for the upstream, and writing the response. $upstream_response_time measures only the backend portion, from establishing the upstream connection to receiving the last byte of the response body. The gap between them is where client-side, network, and nginx-internal delays live. To avoid chasing phantom backend problems, log both variables and compare them.

nginx 413 Request Entity Too Large: client_max_body_size explained

Mon, 01 Jan 0001 00:00:00 +0000

nginx 413 Request Entity Too Large: client_max_body_size explained

A 413 Request Entity Too Large after adding client_max_body_size 50m to nginx.conf usually means a more specific context still overrides it, or the upstream application rejects the body after nginx accepts it. The directive applies to http, server, and location blocks. Raising it at the edge only moves the failure deeper if the rest of the stack is not adjusted. This guide covers how the directive inherits, when nginx fires the 413, how proxy_request_buffering changes the failure mode, and why you must verify every hop including the upstream application.

nginx 499 status code: why clients close connections before the response

Mon, 01 Jan 0001 00:00:00 +0000

nginx 499 status code: why clients close connections before the response

Status 499 in nginx access logs means the client closed the TCP connection before nginx finished responding. It is an nginx-specific code that never reaches the client, so it is easy to dismiss. In practice, a 499 surge is an early warning: users or intermediaries abandon requests before upstreams officially time out and before 5xx errors spike. Ignore 499s and you usually see 502s or 504s minutes later.

nginx 500 Internal Server Error: how to diagnose it

Mon, 01 Jan 0001 00:00:00 +0000

nginx 500 Internal Server Error: how to diagnose it

A 500 from nginx tells you something failed in the request path, but not whether the failure originated in your application, FastCGI/uWSGI backend, or nginx itself. When 500s spike during an incident, first determine which side of the nginx boundary is breaking.

Unlike 502 Bad Gateway or 504 Gateway Time-out, which point upstream, a 500 can be an application bug passed through by nginx, a configuration error, a permission failure, or resource exhaustion inside an nginx worker.

NGINX 502 Bad Gateway: Causes And How To Fix It

Mon, 01 Jan 0001 00:00:00 +0000

A 502 Bad Gateway means the upstream server returned an invalid response, refused the connection, or terminated before completing the response. Unlike 504, which signals upstream slowness, 502 means the upstream never produced a valid response or nginx could not reach it.

Start with the error log. A single line like connect() failed (111: Connection refused) tells you the upstream is not listening. A line like upstream prematurely closed connection tells you the backend died mid-request. Match the exact message to the root cause.

nginx 503 Service Temporarily Unavailable: causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

nginx 503 service temporarily unavailable: causes and fixes

A 503 Service Temporarily Unavailable from nginx does not always mean the upstream application is broken. A healthy nginx process returns 503 by design when rate limits reject traffic, or when every backend in an upstream block is unavailable. The same status code covers three different failure paths: intentional throttling, upstream exhaustion, or resource saturation on the nginx host. The fix depends on which path the request took.

NGINX 504 Gateway Time-Out: Causes & Fixes

Mon, 01 Jan 0001 00:00:00 +0000

A 504 Gateway Time-out means nginx reached the upstream but the upstream did not finish its response before proxy_read_timeout expired. The default is 60 seconds. Unlike a 502 Bad Gateway, which means nginx never established a valid upstream connection, a 504 means the connection succeeded but the response did not complete in time.

This guide covers isolating slow upstreams via access log variables, tuning timeouts and retries, and distinguishing 504 from 502.

NGINX access log performance: buffering, sampling, and the event loop

Mon, 01 Jan 0001 00:00:00 +0000

NGINX access log performance: buffering, sampling, and the event loop

Every NGINX worker is a single-threaded event loop. By default, finishing a request triggers an immediate, synchronous write of the access log line. Under normal load the cost is negligible. Under incident conditions, when error rates and log volume spike, that synchronous write becomes a compounding failure: disk I/O stalls the event loop, responses slow down, clients time out, and the resulting errors and abandonments generate even more log lines.

NGINX active connections climbing: reading, writing, waiting explained

Mon, 01 Jan 0001 00:00:00 +0000

NGINX active connections climbing: reading, writing, waiting explained

When operators see Active connections climbing in stub_status, the first instinct is often to add capacity. That instinct is usually wrong. The stub_status module exposes exactly seven metrics, and the most useful of them is the breakdown of active connections into Reading, Writing, and Waiting. The absolute number of active connections is almost meaningless without the ratio between these three states. A server with 10,000 active connections where 9,000 are Waiting is healthy. A server with 500 active connections where 400 are Reading may be under a slowloris-style attack.

NGINX backend cascade failure: when slow upstreams take down everything

Mon, 01 Jan 0001 00:00:00 +0000

NGINX backend cascade failure: when slow upstreams take down everything

Users report timeouts. 502 Bad Gateway and 504 Gateway Time-out responses are climbing, and nginx error logs show upstream timeouts. On the nginx host, CPU and memory are normal, and the master process is alive. The proxy is healthy but out of connections.

This is a backend cascade failure. One slow upstream causes nginx workers to hold connections open while waiting for responses, consuming finite worker_connections slots. As slots fill, new requests cannot be forwarded. Traffic concentrates on the remaining healthy backends, which overload and slow down. Eventually every backend times out or fails health checks, and nginx returns 502/504 to all clients while the proxy process remains up.

nginx connect() failed (111: Connection refused) while connecting to upstream

Mon, 01 Jan 0001 00:00:00 +0000

nginx connect() failed (111: Connection refused) while connecting to upstream

HTTP 502 Bad Gateway and the error connect() failed (111: Connection refused) while connecting to upstream mean nginx reached the upstream IP, but the target port actively refused the TCP connection. The backend is either not running, not listening on the interface nginx expects, or a firewall is blocking the port.

This is distinct from upstream timed out (110: Connection timed out). A timeout means the TCP SYN never received a response, usually because a firewall silently dropped the packet or the host is unreachable. Errno 111 means the network path is open but no process is accepting connections. The error log includes the upstream address, such as upstream: "fastcgi://127.0.0.1:9000". Read that line first to isolate the exact backend.

NGINX connection exhaustion: detection, diagnosis, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

NGINX connection exhaustion: detection, diagnosis, and prevention

Users see connection timeouts while load balancer health checks and the NGINX stub_status endpoint still return HTTP 200. New connections are silently dropped. Connection exhaustion is a cliff-edge failure: once the limit is hit, there is no graceful degradation. Connections are refused at the kernel level, or accepted into the TCP backlog but discarded by NGINX because no worker has a free connection slot.

NGINX DNS resolution failures on dynamic upstreams: 502s and resolver_timeout

Mon, 01 Jan 0001 00:00:00 +0000

NGINX DNS resolution failures on dynamic upstreams: 502s and resolver_timeout

What this means

Intermittent 502 Bad Gateway responses that only hit locations using a variable in proxy_pass, such as proxy_pass http://$backend;, point to dynamic DNS resolution failure. Static upstream locations are unaffected. dig from the host may succeed instantly while nginx logs show 502s with latency spikes clustering at exactly 30 seconds, the default resolver_timeout.

nginx resolves upstream hostnames through two paths.

NGINX dropped connections: the accepts vs handled gap

Mon, 01 Jan 0001 00:00:00 +0000

NGINX dropped connections: the accepts vs handled gap

Users report intermittent connection timeouts. Your HTTP 5xx rate is flat. The error log is quiet. Something is dropping traffic before it ever becomes a request.

On every NGINX instance, the stub_status page exposes two cumulative counters: accepts and handled. When accepts grows faster than handled, NGINX is taking connections from the kernel and then discarding them. This gap is a leading indicator of connection-slot or file-descriptor exhaustion. It often starts increasing minutes before the system hits the hard wall.

NGINX limit_req burst and nodelay tuning: rate limiting without blocking real users

Mon, 01 Jan 0001 00:00:00 +0000

NGINX limit_req burst and nodelay tuning: rate limiting without blocking real users

Most production nginx rate limiting configs fall into two camps: no burst at all, which rejects legitimate traffic during harmless spikes, or burst without nodelay, which queues real users into artificial delays that mimic upstream slowness. Neither is what you want. The limit_req module implements a leaky bucket at millisecond granularity, and the interaction between burst and nodelay determines whether a request is delayed, rejected, or forwarded immediately. Understanding that interaction, and sizing the shared memory zone to match your traffic profile, is the difference between rate limiting that protects upstreams and rate limiting that creates incidents during normal user behavior.

nginx limiting requests, excess -- understanding limit_req rejections

Mon, 01 Jan 0001 00:00:00 +0000

nginx limiting requests, excess – understanding limit_req rejections

When [error] ... limiting requests, excess appears in nginx error logs alongside 503 responses in access logs, determine whether you are under attack, misconfigured, or out of shared memory. The ngx_http_limit_req_module implements a leaky bucket rate limiter. Its interaction with burst, nodelay, and shared memory sizing determines whether you reject malicious traffic, delay legitimate users, or silently stop enforcing limits.

What it is and why it matters

limit_req is nginx’s request-level rate limiter. It uses a shared memory zone, configured via limit_req_zone, to track request rates per key, typically $binary_remote_addr. The zone is mapped into every worker process. When a request arrives, nginx checks the key’s current rate against the configured limit. Depending on burst and nodelay, it delays the request, rejects it, or processes it immediately.

NGINX listen queue overflow: somaxconn, backlog, and silent connection drops

Mon, 01 Jan 0001 00:00:00 +0000

NGINX listen queue overflow: somaxconn, backlog, and silent connection drops

Clients report intermittent connection timeouts. Your load balancer health checks pass. NGINX error logs are clean and access logs show no 5xx spikes. The issue is not in NGINX workers or upstream applications. It is in the kernel accept queue.

When the accept queue fills, the kernel drops new connections silently. NGINX never sees them, so it logs nothing. Evidence is client-side timeouts and the kernel counter TcpExtListenOverflows.

NGINX log disk full: when logging silently stops and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

NGINX log disk full: when logging silently stops and how to recover

During an incident, upstream latency climbs and 5xx errors appear. You run tail -f /var/log/nginx/error.log and the cursor sits there. The last entry is hours old. Requests still return 200s. The server is serving, but it has stopped logging. Disk utilization shows /var/log at 100 percent.

This is the NGINX log disk-full failure mode. The process never signals the client or the operator that it can no longer write diagnostics. Visibility evaporates when it is most needed.

NGINX Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NGINX Monitoring

What Is NGINX?

NGINX is a high-performance web server, reverse proxy server, and load balancer designed to handle a large number of concurrent connections efficiently. Its modular architecture allows it to be extended with additional features, making it a versatile component in modern application infrastructures.

Monitoring NGINX With Netdata

When it comes to monitoring NGINX, Netdata offers an intuitive and real-time monitoring solution. With Netdata’s NGINX monitoring tool, you can observe your server’s metrics in real time, use interactive charts, and detect any anomalies swiftly.

NGINX monitoring checklist: the signals every production server needs

Mon, 01 Jan 0001 00:00:00 +0000

NGINX monitoring checklist: the signals every production server needs

NGINX is an event-driven, single-threaded-per-worker process. Most production failures follow predictable patterns: connection exhaustion, backend cascades, file descriptor limits, or silent kernel-level drops. This article maps the signals that expose those failures into four cumulative maturity levels: Survival, Operational, Mature, and Expert. Use it to audit your current coverage or to justify instrumentation work before the next incident.

Each level adds depth. Survival answers “Is it up?” Operational answers “Is it healthy?” Mature adds leading indicators. Expert adds the signals you instrument after your third postmortem. The tables below list each signal, why it matters, and the threshold that should trigger a response.

NGINX monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

NGINX monitoring maturity model: from survival to expert

nginx exposes exactly seven scalars through stub_status. Latency distributions, upstream health, cache efficiency, and kernel-level drops live in access logs, error logs, or OS counters. Teams that collect only the stub_status numbers assume they have visibility. They do not.

This article defines four monitoring maturity levels. Level 1 tells you if nginx is alive. Level 2 tells you if it is healthy. Level 3 gives you leading indicators of saturation. Level 4 exposes the blind spots that only appear after repeated incidents. Use these levels to audit your current coverage and decide which signals to add next.

nginx no live upstreams while connecting to upstream: what it means

Mon, 01 Jan 0001 00:00:00 +0000

nginx no live upstreams while connecting to upstream: what it means

When nginx logs no live upstreams while connecting to upstream, every server in the affected upstream block is marked unavailable. The proxied request has no eligible backend, so nginx returns 502 Bad Gateway. This is not an nginx defect; it signals that all backends have failed open-source nginx’s passive health checks, or a network partition has made them unreachable from the nginx host.

NGINX old worker processes accumulating after reload: worker_shutdown_timeout

Mon, 01 Jan 0001 00:00:00 +0000

NGINX old worker processes accumulating after reload: worker_shutdown_timeout

After nginx -s reload, worker process count climbs above worker_processes. Memory and file descriptor usage grow. The workers are not crashing; they are old generations waiting for long-lived connections to close.

This pattern is normal in small doses. During a reload, the master keeps old workers alive until active connections drain. Without a shutdown deadline, a single WebSocket, gRPC stream, or long-polling connection can pin an old worker indefinitely. In environments that reload frequently, such as Kubernetes ingress controllers reacting to endpoint changes, accumulation becomes a resource leak that can exhaust memory or file descriptors.

NGINX Plus

Mon, 01 Jan 0001 00:00:00 +0000

NGINX Plus Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NGINX Plus Monitoring

What Is NGINX Plus?

NGINX Plus is a premium version of NGINX, offering additional features such as advanced load balancing, reliability, security, and flexibility to deploy applications. It is widely used for web serving, reverse proxying, caching, load balancing, media streaming, and more.

Monitoring NGINX Plus With Netdata

Netdata provides a robust NGINX Plus monitoring tool that gives real-time insight into the performance and health of your NGINX Plus servers. With the ability to visualize key metrics and diagnose performance issues swiftly, Netdata becomes an invaluable tool for any DevOps, SRE, or IT professional managing NGINX Plus instances. To learn more, see the NGINX Plus collector documentation.

NGINX proxy buffer spill to disk: proxy_buffers and temp file latency

Mon, 01 Jan 0001 00:00:00 +0000

NGINX proxy buffer spill to disk: proxy_buffers and temp file latency

You notice some proxied requests are crawling. Upstream logs show sub-50 ms response times. The network path is clean. The nginx error log is quiet. In the access log, however, $request_time is ten times larger than $upstream_response_time. For large responses, this gap is the signature of proxy buffer spill.

When an upstream response exceeds the memory buffers allocated by proxy_buffers, nginx writes the overflow to a temporary file under proxy_temp_path and reads it back later. Because the log message for this event is emitted at debug level only, the delay is silent. Standard upstream monitoring gives no hint; the delay hides entirely inside nginx.

NGINX proxy buffer tuning: proxy_buffers, proxy_buffer_size, and busy buffers

Mon, 01 Jan 0001 00:00:00 +0000

NGINX proxy buffer tuning: proxy_buffers, proxy_buffer_size, and busy buffers

When proxy_buffering is on (the default), NGINX absorbs the upstream response in memory before sending it to the client. This shields backends from slow clients and enables compression, but three directives control it: proxy_buffer_size for headers, proxy_buffers for the body, and proxy_busy_buffers_size for the in-flight flush window. Misconfiguration causes 502s, silent disk spills, and reload failures.

The defaults are modest: eight body buffers of one memory page each, and one header page (typically 4K or 8K). That works for static sites and small JSON, but it fails for modern workloads: APIs with large JWT tokens in headers, bulk exports returning multi-megabyte JSON, and Server-Sent Events streams. Undersized body buffers spill to disk. Undersized header buffers return 502. Invalid proxy_busy_buffers_size values prevent NGINX from starting or reloading.

NGINX proxy cache hit rate is low: measuring and improving it

Mon, 01 Jan 0001 00:00:00 +0000

NGINX proxy cache hit rate is low: measuring and improving it

Low NGINX proxy cache hit rate shows up as upstream CPU climbing or origin traffic higher than expected. Access logs show MISS and BYPASS where you expect HIT. When caching fails, every request reaches the backend, adding latency and load. The symptom is usually a gradual slide from 85% to 40% over a day, or a collapse to zero after a deployment or restart. Root cause is often configuration drift: a new header, a changed query parameter, or a keys_zone sized for last year’s traffic. Diagnose by measuring which cache status dominates, then trace that status back to the directive or upstream behavior that produces it.

NGINX proxy_cache not caching: why responses bypass the cache

Mon, 01 Jan 0001 00:00:00 +0000

NGINX proxy_cache not caching: why responses bypass the cache

After enabling proxy_cache and defining the cache path, upstreams still take every hit. Access logs show $upstream_cache_status as BYPASS or MISS, hit rate stays near zero, and nothing appears in the error log. NGINX applies a strict request-phase and response-phase decision tree before anything enters the cache. If any condition fails, the response is never stored. The symptom looks like upstream overload, but the root cause is usually a directive, a header, or a missing validity window.

NGINX rate limiting returns 503 not 429: limit_req_status explained

Mon, 01 Jan 0001 00:00:00 +0000

NGINX rate limiting returns 503 not 429: limit_req_status explained

When limit_req rejects a request, nginx returns 503 by default. This is the same code used for genuine capacity exhaustion, so a 503 spike pages the on-call rotation even when the upstream is healthy and the infrastructure is fine. Changing limit_req_status to 429 is one directive, but the implications for alerting, monitoring, and client behavior are not trivial.

What it is and why it matters

The limit_req_status directive sets the HTTP status code returned when limit_req rejects a request. The companion directive limit_conn_status does the same for connection limits enforced by limit_conn. Both default to 503.

nginx recv() failed (104: Connection reset by peer) while reading from upstream

Mon, 01 Jan 0001 00:00:00 +0000

nginx recv() failed (104: Connection reset by peer) while reading from upstream

You tail the nginx error log and see this:

[error] ... recv() failed (104: Connection reset by peer) while reading response header from upstream

The client gets a 502 Bad Gateway. The error is not a configuration syntax problem, and the service was working five minutes ago. This message means the upstream server sent a TCP RST while nginx was mid-read on an upstream connection. The reset originates from the backend or from a network middlebox between nginx and the backend. It does not come from nginx itself, and it does not come from the client.

NGINX reload not applying config: why old workers keep serving

Mon, 01 Jan 0001 00:00:00 +0000

NGINX reload not applying config: why old workers keep serving

You pushed a config change, ran nginx -s reload, and moved on. Hours later, the new certificate is not being served, the updated upstream is not receiving traffic, or the tightened rate limit never took effect. NGINX did not stop running, but the reload never applied. This is the silent rollback: when a reload fails validation, the master keeps the previous configuration active and old workers continue serving. Even when validation passes, old workers can remain alive for hours if long-lived connections prevent them from draining and worker_shutdown_timeout is not set. This guide shows how to confirm the failure, find the root cause, and prevent config drift from going undetected.

NGINX slow requests: from access log to root cause

Mon, 01 Jan 0001 00:00:00 +0000

NGINX slow requests: from access log to root cause

Elevated $request_time in access logs does not mean the upstream is slow. The variable measures the full lifecycle: from reading the first client byte through sending the last response byte. That includes client upload, upstream wait, nginx processing, and client download. Blaming the backend by reflex is the most common nginx latency mistake.

To split the time accurately, confirm your log_format includes $request_time, $upstream_response_time, $upstream_connect_time, and $upstream_header_time.

NGINX slowloris and slow-client attacks: detection and mitigation

Mon, 01 Jan 0001 00:00:00 +0000

NGINX slowloris and slow-client attacks: detection and mitigation

You check stub_status and the Reading count is triple its normal value and not dropping. Requests per second has collapsed to near zero. Active connections are climbing toward worker_connections * worker_processes while worker CPU stays idle. This is not a backend slowdown. It is a slowloris or slow-client attack: connections open faster than they complete, and NGINX waits for data that arrives one byte at a time.

NGINX SSL certificate expired: detection and emergency renewal

Mon, 01 Jan 0001 00:00:00 +0000

NGINX SSL certificate expired: detection and emergency renewal

An expired SSL certificate on NGINX is an immediate outage, not gradual degradation. Browsers and API clients reject the connection at the TLS handshake, often before NGINX logs anything useful. The fix is rarely as simple as running a renewal script again. Verify what is on disk, confirm the running configuration is using it, and force a reload so workers load the new certificate.

NGINX SSL session cache: improving TLS resumption and cutting CPU

Mon, 01 Jan 0001 00:00:00 +0000

NGINX SSL session cache: improving TLS resumption and cutting CPU

TLS handshakes are the most CPU-intensive routine operation in nginx. A worker terminating RSA-2048 TLS can handle only a few hundred full handshakes per second, compared to tens of thousands of plain HTTP requests. In production, every reconnecting client that repeats a full handshake wastes CPU and adds latency. The ssl_session_cache directive exists to eliminate that waste by allowing session resumption across connections. Yet many configurations either omit it, under-size it, or misunderstand how TLS 1.3 changes resumption behavior. This article explains the mechanism, sizing, and the signals that tell you whether your cache is working.

nginx SSL_do_handshake() failed — diagnosing TLS handshake errors

Mon, 01 Jan 0001 00:00:00 +0000

nginx SSL_do_handshake() failed — diagnosing TLS handshake errors

You see [crit] or [error] entries for SSL_do_handshake() failed in /var/log/nginx/error.log. The message might involve a client connecting to nginx, or nginx connecting to an upstream server. These errors mean the TLS negotiation never completed, so no application data was exchanged. The impact ranges from a few browsers showing security warnings to all proxied traffic returning 502 Bad Gateway.

Before chasing certificates, determine which side of a connection is failing. nginx logs two variants: while SSL handshaking to client and while SSL handshaking to upstream. The first affects visitors hitting your server directly. The second affects nginx as a reverse proxy calling a backend over HTTPS. The symptoms, root causes, and fixes are different.

NGINX SSL/TLS handshake CPU saturation: detection and tuning

Mon, 01 Jan 0001 00:00:00 +0000

NGINX SSL/TLS handshake CPU saturation: detection and tuning

NGINX latency climbs while requests per second flatline. Worker processes are pinned near 100% CPU, yet active connections are nowhere near the worker_connections limit. Access logs show fast upstream response times, but $request_time is an order of magnitude larger. The bottleneck is not the network, disk, or backends: it is the TLS handshake.

When workers burn CPU on cryptography, the single-threaded event loop has no time left for request processing. Every new TCP connection that requires a full SSL handshake adds asymmetric crypto workload. If clients do not resume sessions and the connection rate is high, throughput collapses even though the machine has plenty of idle connection slots. This guide shows how to detect, diagnose, and tune for SSL handshake CPU saturation.

NGINX Unit

Mon, 01 Jan 0001 00:00:00 +0000

NGINX Unit Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NGINX Unit Monitoring

NGINX Unit monitoring is crucial for ensuring the optimal performance and reliability of web applications. With the capabilities offered by the NGINX Unit, a dynamic application server, it’s possible to handle various configurations for multiple languages seamlessly. Using robust monitoring tools like the Netdata platform can provide real-time insights into your NGINX Unit, allowing IT engineers, DevOps, and SRE teams to swiftly detect issues and optimize their infrastructure.

NGINX upstream keepalive: eliminating per-request TCP and TLS overhead

Mon, 01 Jan 0001 00:00:00 +0000

NGINX upstream keepalive: eliminating per-request TCP and TLS overhead

Every proxied request that opens a fresh TCP connection burns latency on the handshake and, if the upstream uses HTTPS, on TLS negotiation. At low volume this cost is invisible. At production throughput it becomes a measurable tax on every request, and at extreme scale it can exhaust the kernel’s ephemeral port range and bury the host in TIME_WAIT sockets. For HTTPS upstreams, the CPU cost of TLS handshakes across thousands of requests per second consumes worker cycles that could be spent proxying traffic.

nginx upstream prematurely closed connection while reading response header

Mon, 01 Jan 0001 00:00:00 +0000

nginx upstream prematurely closed connection while reading response header

upstream prematurely closed connection while reading response header from upstream means the upstream server closed the TCP socket while nginx was still reading response headers. This produces a 502 Bad Gateway. Unlike a timeout, the upstream actively terminated the connection.

The root cause is typically on the backend: a crash, worker recycle, request size limit, or stale keepalive connection the backend closed while nginx tried to reuse it. nginx retries the request on another backend only if proxy_next_upstream includes error (the default for idempotent methods). Retries improve availability but do not fix the underlying issue.

nginx upstream sent too big header while reading response header from upstream

Mon, 01 Jan 0001 00:00:00 +0000

nginx upstream sent too big header while reading response header from upstream

The error log contains upstream sent too big header while reading response header from upstream. Clients receive 502 Bad Gateway. This is not an upstream crash or network timeout. It is a hard size limit: an upstream server is sending response headers larger than the fixed buffer nginx allocates for reading them, so nginx aborts the request and returns 502.

nginx upstream timed out (110: Connection timed out) while connecting/reading

Mon, 01 Jan 0001 00:00:00 +0000

nginx upstream timed out (110: Connection timed out) while connecting/reading

upstream timed out (110: Connection timed out) in the nginx error log usually surfaces to clients as a 504 Gateway Timeout. The suffix after the error string tells you which phase failed: connecting, sending, or reading. That phase determines whether you are looking at a dead backend, a network partition, or a retry storm hiding the real problem.

The defaults are unforgiving. proxy_connect_timeout, proxy_send_timeout, and proxy_read_timeout all default to 60 seconds, and proxy_next_upstream implicitly retries on error and timeout. Retries can mask the root cause while exhausting upstream capacity.

NGINX VTS

Mon, 01 Jan 0001 00:00:00 +0000

NGINX VTS Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NGINX VTS Monitoring

What Is NGINX VTS?

NGINX VTS (Virtual Traffic Status) is a third-party module for NGINX, a high-performance web server and reverse proxy. The VTS module provides detailed traffic status and metrics crucial for web server monitoring and management. It enables users to keep track of the overall health and performance of their NGINX server instances.

Monitoring NGINX VTS With Netdata

Using Netdata as an NGINX VTS monitoring tool allows you to collect and visualize vital metrics in real time, offering unprecedented visibility into your web server’s performance. With Netdata, you can monitor NGINX VTS effortlessly by leveraging its extensive integration capabilities and live data streaming features.

NGINX worker_connections and worker_processes: sizing for real traffic

Mon, 01 Jan 0001 00:00:00 +0000

NGINX worker_connections and worker_processes: sizing for real traffic

NGINX defaults leave most CPU cores idle and exhaust quickly under load. The real limit is often the OS file descriptor ceiling, which silently overrides the directive.

Sizing these parameters means understanding the capacity chain: kernel queue, connection slot, file descriptor limit, event loop. This guide provides concrete rules for static and proxy workloads and the signals that reveal when headroom has disappeared.

NGINX worker_rlimit_nofile: setting file descriptor limits correctly

Mon, 01 Jan 0001 00:00:00 +0000

NGINX worker_rlimit_nofile: setting file descriptor limits correctly

nginx file descriptor exhaustion is a silent failure mode. Existing connections continue to be served, but new connections queue in the kernel backlog and are eventually dropped. The client sees a timeout, while nginx error logs may show nothing until the limit is hit.

worker_rlimit_nofile raises the per-worker file descriptor limit above conservative operating system defaults. It does not operate in isolation. It sits inside a layered stack of kernel parameters, systemd service limits, container runtime defaults, and PAM session policies. A value written into nginx.conf is only effective if the master process inherits a hard limit at least as high, and the kernel ceiling permits it. If any layer caps the limit below your intended value, workers inherit that lower ceiling and your tuning is silently ignored.

nginx: a client request body is buffered to a temporary file — what it means

Mon, 01 Jan 0001 00:00:00 +0000

nginx: a client request body is buffered to a temporary file — what it means

You are tailing the nginx error log during a latency investigation and see the line: a client request body is buffered to a temporary file. It is logged at [warn], not [error], so it is easy to ignore. But the message means a request body has exceeded the in-memory buffer and nginx is now writing that data to disk. On a busy reverse proxy or file-upload endpoint, this behavior can add hundreds of milliseconds or seconds of latency before your upstream application receives the payload. It also consumes file descriptors and disk I/O capacity without ever surfacing as a 5xx status code.

nginx: bind() to 0.0.0.0:80 failed (98: Address already in use)

Mon, 01 Jan 0001 00:00:00 +0000

nginx: bind() to 0.0.0.0:80 failed (98: Address already in use)

The error log shows [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use) and the master process exits. During a reload, old workers keep running with the previous configuration, so users may not notice immediately. During system boot or a manual start, the service is down.

Error 98 is EADDRINUSE. The nginx master binds listening sockets before forking workers. If the kernel reports port 80 is occupied, nginx cannot start or apply the new configuration. The holder might be a different service, a stale nginx master after a crash, or another nginx instance. Inside the same configuration, conflicting socket options for the same address:port can also trigger the error.

nginx: configuration file test failed - finding the syntax error

Mon, 01 Jan 0001 00:00:00 +0000

nginx: configuration file test failed - finding the syntax error

An nginx -t or nginx -s reload ending with nginx: configuration file /path/to/nginx.conf test failed means the configuration tree is syntactically invalid or references a missing file. The master process rejects the change, so the running server continues with the previous working configuration. That prevents an outage, but your intended change is silently inactive. Read the exact error message, map it to the real source, fix it, and validate with nginx -t before reloading.

nginx: could not build server_names_hash -- server_names_hash_bucket_size

Mon, 01 Jan 0001 00:00:00 +0000

nginx: could not build server_names_hash – server_names_hash_bucket_size

Running nginx -t or nginx -s reload stops with [emerg] could not build the server_names_hash, you should increase server_names_hash_bucket_size: 32. Alternatively, nginx starts but logs [warn] could not build optimal server_names_hash, you should increase either server_names_hash_max_size: 512 or server_names_hash_bucket_size: 64.

This error means your configuration has exceeded a limit in nginx’s server name hashing logic. nginx builds this hash at configuration load time, not at request time. If the build fails, the configuration test fails. If the build succeeds but is suboptimal, nginx starts with degraded lookup performance. Both directives, server_names_hash_bucket_size and server_names_hash_max_size, are valid only inside the http context.

nginx: no resolver defined to resolve - dynamic upstream DNS

Mon, 01 Jan 0001 00:00:00 +0000

nginx: no resolver defined to resolve - dynamic upstream DNS

502 Bad Gateway responses paired with no resolver defined to resolve example.com in the error log mean proxy_pass uses a variable - for example, proxy_pass http://$backend; - and the enclosing context has no resolver directive.

With a literal proxy_pass, nginx resolves the upstream hostname once at startup or reload and caches the result indefinitely. It never queries DNS again until restart or reload. With a variable-based proxy_pass, nginx resolves the hostname at request time through its internal async resolver. Without a resolver directive, the lookup fails immediately and returns 502.

nginx: too many open files - diagnosing file descriptor exhaustion

Mon, 01 Jan 0001 00:00:00 +0000

nginx: too many open files - diagnosing file descriptor exhaustion

After a traffic spike, the error log shows accept4() failed (24: Too many open files), then goes silent. Existing connections still serve, but new ones cannot land.

File descriptor exhaustion is a hard failure. Once the limit is hit, nginx cannot accept new connections, open upstream sockets, or write to the error log. Default OS limits of 1024 are too low for production reverse proxies. Each proxied request consumes at least two FDs, and idle keepalive connections hold them indefinitely. The effective limit is the lower of worker_rlimit_nofile and the OS hard limit enforced by systemd or the container runtime.

nginx: worker_connections are not enough — causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

nginx: worker_connections are not enough — causes and fixes

Your error log shows worker_connections are not enough while connecting to upstream. New clients time out while existing connections may still work. This is a hard capacity cliff: once a worker exhausts its connection slots, it cannot accept new connections until a slot frees. The default limit is 512 per worker, not 1024, and in reverse-proxy mode each request consumes at least two slots. Raising the number is often the first reaction, but if a slow backend is holding connections open, the slots will fill again no matter how high you set the limit.

NIC RSS misconfiguration: one CPU core silently dropping your telemetry

Mon, 01 Jan 0001 00:00:00 +0000

NIC RSS misconfiguration: one CPU core silently dropping your telemetry

Your flow collector has 16 CPU cores, but one is pinned at 100% while the other 15 sit idle. NIC receive drop counters are climbing. UDP socket buffer errors (Udp_RcvbufErrors) are incrementing. Your bandwidth charts show traffic declining during what is actually a traffic spike. The box looks underpowered, so you start sizing a bigger one. The real problem: Receive Side Scaling (RSS) is funneling every inbound packet to a single receive queue serviced by a single CPU core. No amount of additional cores or RAM fixes this until RSS distributes interrupts across them.

Nice Systems Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nimble Storage SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nokia BGP

Mon, 01 Jan 0001 00:00:00 +0000

Nokia Formerly Alcatel Lucent SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nokia Formerly Coriant R D GmbH SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nokia Formerly Infinera Corp SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nokia Formerly Lumentis AB SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nokia Formerly Transmode Systems AB SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nokia Networks Formerly Nokia Siemens Networks SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nokia Service Router OS

Mon, 01 Jan 0001 00:00:00 +0000

Nokia SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nomad Containers

Mon, 01 Jan 0001 00:00:00 +0000

Nomadix SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Non-Uniform Memory Access

Mon, 01 Jan 0001 00:00:00 +0000

Normalizing syslog severity across vendors: why 'critical' isn't critical

Mon, 01 Jan 0001 00:00:00 +0000

Normalizing syslog severity across vendors: why ‘critical’ isn’t critical

RFC 5424 defines eight syslog severity levels, numbered 0 through 7: Emergency, Alert, Critical, Error, Warning, Notice, Informational, and Debug. Every major network vendor implements the same numeric scale. The integer that means “Critical” on a Cisco router means “Critical” on a Juniper switch.

But the severity a device assigns to a given event is not standardized. A BGP session reset might arrive as severity 5 (Notice) from one vendor and severity 3 (Error) from another. A hardware alarm that one platform logs as Critical (2), another logs as Alert (1) or Warning (4). Same operational condition, different severity label, same RFC.

Northern Telecom Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Novell SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Novelsat SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

NRPE daemon

Mon, 01 Jan 0001 00:00:00 +0000

NRPE daemon Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NRPE daemon Monitoring

What Is NRPE daemon?

The Nagios Remote Plugin Executor (NRPE) daemon is an integral part of system and network monitoring, widely utilized to execute remote commands and scripts. This daemon facilitates the collection of key performance metrics from remote systems, providing insights into system health and performance. It is especially crucial for environments relying on Nagios for extensive monitoring capabilities, allowing for seamless integration of remote checks into the central monitoring framework.

Nsc Communications Siberia Ltd SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nsc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

NSD

Mon, 01 Jan 0001 00:00:00 +0000

NSD Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NSD Monitoring

What Is NSD?

NSD is an authoritative DNS name server developed by NLnet Labs. It is renowned for its high performance, robust security measures, and support for various DNS standards. NSD is a cornerstone for businesses looking for a reliable DNS solution to ensure smooth and secure network communication. You can find more in-depth information on the NSD official documentation.

Monitoring NSD With Netdata

Monitoring NSD effectively is crucial to maintaining optimal server performance and securing uptime. Netdata offers a comprehensive NSD monitoring tool that easily integrates with your system to provide in-depth insights into your DNS server’s performance. By leveraging Netdata, you can continuously monitor NSD metrics in real time, enabling swift diagnosis and resolution of potential issues.

Nsi Software SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

ntfy

Mon, 01 Jan 0001 00:00:00 +0000

NTP drift on network devices: the silent killer of event correlation

Mon, 01 Jan 0001 00:00:00 +0000

NTP drift on network devices: the silent killer of event correlation

Clock drift on network devices produces no visible symptom. The device stays up, interfaces carry traffic, BGP sessions remain Established, SNMP keeps responding. The damage surfaces hours or days later, in a postmortem where two devices’ timestamps disagree by hundreds of milliseconds and the analyst cannot reconstruct the event sequence. Every cross-device correlation in the monitoring stack depends on accurate, monotonic time across every collector and every polled device.

NTPd

Mon, 01 Jan 0001 00:00:00 +0000

NTPd Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NTPd Monitoring

What Is NTPd?

NTPd stands for Network Time Protocol daemon, an essential component for time synchronization across computer networks. By utilizing the NTP protocol, NTPd ensures that timekeeping is accurate and consistent across systems, which is crucial for numerous applications and services.

Monitoring NTPd With Netdata

When you monitor NTPd with Netdata, you gain real-time insights into your time synchronization processes. The NTPd monitoring tool by Netdata offers comprehensive visibility into NTPd’s operational metrics, helping you to ensure your network’s timing accuracy.

NUMA Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Nutanix Inc SNMP Traps

Mon, 01 Jan 0001 00:00:00 +0000

Nvidia

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer

Your job dies with a CUDA allocation or mapping error, but nvidia-smi shows gigabytes of framebuffer free. You check for fragmentation, restart the job, and it fails again at the same place. The resource that ran out is not VRAM. It is the BAR1 aperture, the PCIe-mapped window the CPU uses to reach GPU memory directly.

BAR1 exhaustion is a distinct failure mode from framebuffer OOM. A GPU can have most of its framebuffer free and still refuse new mappings because every process, IPC handle, and GPUDirect RDMA registration consumes space in a small, shared aperture. It is most common in multi-process, MPS, containerized, and multi-tenant environments where many processes map GPU memory at once.

NVIDIA BGP

Mon, 01 Jan 0001 00:00:00 +0000

Nvidia Cumulus Linux Switch

Mon, 01 Jan 0001 00:00:00 +0000

Nvidia Data Center GPU Manager (DCGM)

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA DCGM exporter on Kubernetes: mapping GPU metrics to pods

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA DCGM exporter on Kubernetes: mapping GPU metrics to pods

On a bare-metal GPU node, attribution is simple: nvidia-smi shows you a PID, you look up the process, done. On Kubernetes there are three layers between you and that answer. The device plugin assigns GPUs to pods, the kubelet tracks those assignments, and dcgm-exporter reads GPU state through NVML/DCGM, which has no idea what a pod is. Unless the exporter explicitly joins these two views, your GPU metrics are half-blind: you can see GPU 3 on node 17 at 98% utilization, but not which workload is responsible.

NVIDIA driver and CUDA version mismatch: subtle failures and crashes

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA driver and CUDA version mismatch: subtle failures and crashes

The loud version of this problem is easy: a job dies at startup with CUDA driver version is insufficient for CUDA runtime version, and the fix is a driver upgrade. The version that ruins your week is quieter. A kernel update goes out fleet-wide, the NVIDIA module rebuilds on most nodes but not all, and now a subset of training jobs crash with kernel-launch failures or, worse, run to completion with subtly wrong results. Nothing in your dashboards looks broken: GPUs up, memory fine, temperatures normal.

NVIDIA Fabric Manager not running: NVSwitch GPUs lose NVLink

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Fabric Manager not running: NVSwitch GPUs lose NVLink

Your multi-GPU training job hangs during NCCL initialization, or fails with cudaErrorSystemNotReady (error 802) the moment a process touches CUDA. Each GPU shows up in nvidia-smi with normal temperature, memory, and utilization. You burn hours on NCCL debug logs, InfiniBand checks, and application-level bisection before someone runs systemctl status nvidia-fabricmanager and finds the service failed two days ago after a reboot.

Nvidia GPU

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU clock throttle reasons: why the GPU isn't running at full speed

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU clock throttle reasons: why the GPU isn’t running at full speed

A training job that used to finish an epoch in 40 minutes now takes 90. nvidia-smi shows 100% GPU utilization, memory usage looks normal, and the workload is running, just slowly. This is the classic throttling symptom: the GPU is executing kernels at reduced clock speeds, so everything takes longer while utilization stays high.

The mistake most teams make here is guessing. They check temperature, see 82C, and debate whether that is “too hot.” They see power draw pinned at the limit. None of that says why the clocks are down. The GPU already knows, and it reports the answer directly through the clock event reasons bitmask.

NVIDIA GPU configuration drift: ECC, persistence, power limits, compute mode

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU configuration drift: ECC, persistence, power limits, compute mode

A GPU that passes every health check can still be misconfigured. ECC turned off, persistence mode lost after a reboot, a power limit set below default, compute mode flipped to Exclusive. None of these produce an XID error or a crashed job on their own. They produce silent data corruption, unexplained throttling, scheduling failures, and monitoring gaps that surface days later as a different incident.

NVIDIA GPU ECC disabled: the silent data-corruption risk

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU ECC disabled: the silent data-corruption risk

A datacenter GPU with ECC disabled does not crash when memory goes bad. It keeps running, keeps returning answers, and keeps writing checkpoints, while single-bit errors corrupt whatever it is computing. There is no error counter incrementing, no XID in dmesg, no page in the middle of the night. The usual discovery path is a training run that diverges to NaN on one specific node, or an inference service returning subtly wrong results, days or weeks after someone flipped the setting.

NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate

You opened nvidia-smi -q -d ECC because something looked off, and now you are staring at four sets of counters with non-zero values and no idea which ones matter. Maybe a monitoring alert fired on “ECC errors present”. Maybe a training run produced NaN loss on one node and someone told you to check ECC. Either way, the raw output does not tell you the two things you need to know: has data been corrupted, and is this GPU dying.

NVIDIA GPU fan at 0%: fan failure on air-cooled cards

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU fan at 0%: fan failure on air-cooled cards

nvidia-smi shows Fan: 0% and the GPU is sitting at 70C under load. Either the fan is dead, or the card is doing exactly what its firmware told it to do. Telling those two apart quickly is the whole job: the wrong guess in one direction means a cooked GPU, the wrong guess in the other means a pointless RMA.

NVIDIA GPU HBM (memory) temperature: the thermal limit most teams miss

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU HBM (memory) temperature: the thermal limit most teams miss

Most GPU fleets monitor one temperature: temperature.gpu, the die temperature. Alerts, dashboards, and throttle runbooks are built around it. On datacenter GPUs with HBM (A100, H100, H200), that is only half the picture, and for memory-bound work it is often the wrong half.

The HBM stacks sit on the same package as the die but have a worse heat path. Under LLM inference, embedding-heavy workloads, and memory-bound training, HBM temperature commonly runs 10 to 20 C hotter than the die. HBM also has its own throttle point. When it trips, the GPU reduces memory bandwidth with no Xid, no error, and no log entry most teams watch. The job keeps running. It just gets slower while die temperature still looks healthy.

NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU

HBM failure on a datacenter GPU is almost never a surprise if you are watching the right signals. It is a staged degradation: single-bit ECC errors rise over days or weeks, the GPU burns through its self-repair capacity (row remaps on Ampere and later, page retirement on pre-Ampere), the first uncorrectable double-bit error lands, and eventually the GPU can no longer repair itself or falls off the PCIe bus entirely.

NVIDIA GPU HW Power Brake Slowdown: the chassis is cutting GPU power

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU HW Power Brake Slowdown: the chassis is cutting GPU power

You are looking at a GPU that is running, reachable, not overheating, and still delivering a fraction of its normal throughput. nvidia-smi -q -d PERFORMANCE shows HW Power Brake Slowdown : Active (and usually HW Slowdown : Active alongside it), SM clocks are at half their rated speed or lower, and the temperature looks fine. The workload did not change. The GPU did not change. Something outside the GPU decided it gets less power.

NVIDIA GPU memory leak: framebuffer usage climbing without a plateau

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU memory leak: framebuffer usage climbing without a plateau

Your training job has been running for six hours. nvidia-smi showed 40 GiB used after warmup. Then 45. Then 52. The curve has not flattened, and at this rate the job dies with CUDA out of memory sometime tomorrow, taking a day of checkpoint progress with it.

GPU memory is a cliff resource: there is no swap and no graceful degradation. When an allocation fails, it fails immediately. The saving grace is that a true leak is visible hours before it kills you, as sustained linear growth in framebuffer usage. The hard part is telling that apart from the many things that look like a leak but are not, because ML frameworks deliberately fill memory and hold it.

NVIDIA GPU memory-bandwidth bound: high DRAM activity, idle SMs

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU memory-bandwidth bound: high DRAM activity, idle SMs

Your training job or inference service is running, nvidia-smi shows high GPU utilization, power draw looks healthy, and throughput is far below what the hardware should deliver. The profiling metrics show the pattern: the device memory interface is active nearly 100% of cycles while the streaming multiprocessors sit mostly idle. The GPU is memory-bandwidth bound. It is not short on compute. It is waiting on HBM.

Nvidia GPU Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

What is Nvidia GPU?

Nvidia GPU (Graphic Processing Unit) is a specialized electronic circuit designed to rapidly process and manipulate graphics data. Nvidia GPUs are typically found in high-end gaming computers, workstations, and servers. They are used to power complex video games and other graphics-intensive tasks.

Monitoring Nvidia GPU with Netdata

The prerequisites for monitoring Nvidia GPU with Netdata are to have a system with an Nvidia GPU and Netdata installed on your system.

Nvidia GPU Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Nvidia GPU Monitoring

What Is Nvidia GPU?

Nvidia GPUs are specialized processing units designed by Nvidia primarily for graphics rendering, though they are widely used in computational tasks such as deep learning, scientific simulations, and cryptocurrency mining. Nvidia’s advanced GPU technology empowers applications to perform complex tasks efficiently.

Monitoring Nvidia GPU With Netdata

Netdata provides real-time monitoring for Nvidia GPUs by leveraging the nvidia-smi CLI tool. This setup allows you to keep an eye on various performance metrics, ensuring optimal operation and helping diagnose potential issues as they occur.

NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs

Most GPU monitoring setups fail in one of two ways. Either they collect a handful of nvidia-smi counters and page on the wrong things (raw temperature, memory percentage, idle PCIe downgrade), or they collect everything and alert on nothing meaningful. Both failure modes come from the same root cause: no shared vocabulary for which signals matter, at what fidelity, and with what alert semantics.

NVIDIA GPU monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU monitoring maturity model: from survival to expert

Most GPU fleets sit at one of two extremes: nothing beyond “does nvidia-smi still respond,” or a wall of dashboards nobody wired to an alert. Neither survives a real incident. A GPU falling off the bus at 3 a.m., a training job running 5x slower because one card is thermally throttled, an accelerating ECC error rate that becomes silent data corruption next week: all detectable, but only if you collect the right signals at the right fidelity.

NVIDIA GPU NVLink down or degraded: a link that dropped in multi-GPU training

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU NVLink down or degraded: a link that dropped in multi-GPU training

A link that drops mid-training is one of the nastiest GPU failures to catch, because the job usually does not crash. It hangs at a synchronization barrier, or it keeps running at a fraction of its normal step rate. Bandwidth between GPUs collapses from hundreds of GB/s (600 GB/s on A100, 900 GB/s on H100) to PCIe speeds, and every AllReduce now waits on the slowest path in the collective.

NVIDIA GPU NVLink errors: CRC and replay errors on GPU-to-GPU links

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU NVLink errors: CRC and replay errors on GPU-to-GPU links

Your training job is running. No crashes, no CUDA errors, no OOM. But step time is up roughly 20%, and the slowdown shows up exactly during the collective phases: AllReduce, gradient sync. The team has already blamed the data loader, shuffled hyperparameters, and re-run the job twice. The actual problem is one degraded NVLink cable retransmitting corrupted flits, and nothing in the default monitoring stack is looking at it.

NVIDIA GPU PCIe link running below maximum: x16 silently training at x8

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU PCIe link running below maximum: x16 silently training at x8

The job runs. Health checks pass. nvidia-smi shows the GPU, temperature is fine, ECC is clean, and utilization looks normal. Yet training throughput is half of what it should be, or one GPU in an eight-GPU node drags every collective operation. A common root cause is a PCIe link that negotiated below its capability: a Gen4 x16 card silently running at Gen3, or worse, at x8 or x4.

NVIDIA GPU PCIe replay counter rising: physical-layer signal integrity

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU PCIe replay counter rising: physical-layer signal integrity

The PCIe replay counter on one of your GPUs is climbing. Not a burst at boot, but a sustained rate of hundreds or thousands per second. Nothing has crashed yet: training still runs, nvidia-smi still answers, and there are no XID errors in dmesg. That is exactly why this signal gets ignored, and exactly why it should not be.

A PCIe replay is a link-layer retransmission. The GPU (or the root complex, switch, or retimer on the path) received a packet that failed its CRC check and asked for it again. Occasional replays are normal PCIe behavior. A sustained high rate means bits are being corrupted on the wire often enough that retransmission is routine, not exceptional. That is a physical-layer signal integrity problem: a cable, connector, slot, riser, or an overheating retimer or PCIe switch somewhere between the CPU and the GPU.

NVIDIA GPU PCIe straggler: one slow link throttling multi-GPU training

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU PCIe straggler: one slow link throttling multi-GPU training

Your distributed training job is running. No errors, no crashes, no XIDs. But step time jumped 30-60% and never came back, and scaling from 4 GPUs to 8 GPUs bought almost nothing. Per-GPU utilization shows one GPU a few points below the rest, and its PCIe throughput is consistently below its peers.

That is the PCIe straggler pattern. One GPU’s link negotiated at a lower generation or narrower width than it should have: Gen3 instead of Gen4, x8 instead of x16, or worse. That GPU now feeds data and exchanges gradients at half (or less) of its peers’ bandwidth. In synchronous data-parallel training, every collective waits for the slowest participant, so the whole job runs at the speed of the worst link.

NVIDIA GPU Remapping Failure Occurred: HBM spare rows exhausted (Ampere+)

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU Remapping Failure Occurred: HBM spare rows exhausted (Ampere+)

You ran nvidia-smi -q -d ROW_REMAPPER (or your monitoring did) and saw Remapping Failure Occurred : Yes. This is not a driver problem, not a configuration problem, and not something a reboot clears. It is the GPU telling you that its hardware self-repair budget is spent.

On Ampere and later GPUs (A100, H100, H200, Blackwell), the memory subsystem does not retire whole pages the way Volta and Turing did. Instead, when HBM rows start failing, the GPU remaps them to a finite pool of spare rows built into each DRAM bank. While spares remain, the GPU heals itself transparently: correctable ECC events trigger remaps, XID 63 appears in the kernel log as an informational note, and workloads never notice.

NVIDIA GPU retired pages: page retirement, pending retirements, and end of life

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU retired pages: page retirement, pending retirements, and end of life

Retired pages are the permanent record of a GPU’s memory hardware degrading. Every retirement means the driver found a framebuffer page it could no longer trust and removed it from the allocatable pool for good. The counts persist in the GPU’s InfoROM across reboots and driver reloads, which makes them one of the few genuinely cumulative hardware health signals nvidia-smi exposes.

NVIDIA GPU single-bit ECC error rate rising: the leading indicator of HBM failure

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU single-bit ECC error rate rising: the leading indicator of HBM failure

A GPU in your fleet has started logging corrected ECC errors faster than it used to. Maybe you noticed a bump in ecc.errors.corrected.volatile.total during a routine check, or Xid 92 events started appearing in dmesg. Nothing is broken yet: jobs run, loss curves look normal, nvidia-smi reports a healthy card. This is exactly the moment most teams get wrong. They look at the absolute count, decide it is small, and move on.

NVIDIA GPU SW Power Cap: performance capped by the power limit

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU SW Power Cap: performance capped by the power limit

nvidia-smi reports SW Power Cap: Active under Clocks Event Reasons, the workload is running slower than it should, and the GPU is not hot. This is the power-limit throttle: the GPU’s power scaling algorithm is holding clocks below what the workload requested because the board is consuming as much power as it is allowed to.

Two things make this symptom confusing. First, SW Power Cap is not always a fault. A GPU at full load sitting exactly at its power limit is doing the maximum work its TDP allows, and datacenter operators often cap GPUs below the factory limit on purpose. Second, the number that actually matters is enforced.power.limit, not power.limit, and the two can disagree. Reading the wrong field sends you chasing a misconfiguration that does not exist, or missing one that does.

NVIDIA GPU temperature too high: die temperature and thermal limits

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU temperature too high: die temperature and thermal limits

A GPU reporting 85 °C is not necessarily a GPU with a temperature problem. Under sustained load, datacenter GPUs like the A100 and H100 are designed to run hot, and 80 °C is routine under full load. The number alone tells you almost nothing.

What matters is where the temperature sits relative to the SKU’s own thermal limits, whether the GPU has started throttling, and whether the pattern is one hot GPU or a whole chassis running hot. A GPU at 85 °C with no throttle reasons active is fine. A GPU at 75 °C with hw_thermal_slowdown active has a cooling system that is already failing.

NVIDIA GPU Tensor Cores idle: mixed precision not being used

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU Tensor Cores idle: mixed precision not being used

Your training job is running, nvidia-smi shows high SM utilization, power draw looks healthy, and step time is 5 to 10 times worse than the hardware spec says it should be. DCGM profiling shows Tensor Core Active pinned at or near zero. The GPU is busy, but it is doing the work on CUDA cores instead of Tensor Cores.

Nothing errors. Nothing crashes. The job just runs at a fraction of the throughput you paid for, and because SM utilization still reads 90%+, it looks healthy in every dashboard that only tracks utilization.gpu.

NVIDIA GPU thermal cascade in dense servers: one hot GPU heats its neighbours

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU thermal cascade in dense servers: one hot GPU heats its neighbours

Three of the eight GPUs in a node are thermal throttling. Training step time has doubled. The instinct is to suspect the cooling system as a whole, or the workload. But line up per-GPU temperatures and one unit stands out: GPU 5 is running 12 degrees hotter than everything else, and the GPUs downstream of it in the airflow path are the ones throttling.

NVIDIA GPU thermal throttling: clocks dropping under heat

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU thermal throttling: clocks dropping under heat

Training throughput fell 25% overnight. No errors in the application logs. nvidia-smi shows every GPU at 100% utilization. Nothing looks broken, and yet the job is measurably slower.

This is what NVIDIA GPU thermal throttling looks like in practice: the GPU keeps working, keeps reporting full utilization, and silently delivers 10-40% less real work because its SM clocks have dropped. Utilization measures time busy, not work done. A kernel running at half clock speed still occupies the SMs for the whole sampling window, so utilization.gpu stays pinned at 100% while tokens/sec or samples/sec collapse.

NVIDIA GPU utilization low during training: a data-pipeline bottleneck

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA GPU utilization low during training: a data-pipeline bottleneck

Your training job is running, the loss is decreasing, and nvidia-smi shows the GPU at 20-40% utilization, or oscillating between 0% and 100% in a sawtooth. The job finishes, so it looks healthy. It is not: you are paying for a GPU that is idle most of the time because the host cannot feed it.

Low utilization during active training is almost never a GPU fault. It means the GPU is starved: waiting on CPU preprocessing, disk or network data loading, pageable-memory copies across PCIe, or a synchronization barrier. The hardware is fine; the pipeline upstream of it is the bottleneck.

Nvidia Mellanox Switchx

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps

A training node accepts a job, and the first CUDA call takes two seconds before any kernel runs. The next job lands, and it happens again. Your dashboard shows the GPU bouncing between P0 and P8 all day, clocks collapsing to idle between launches, power draw dropping toward zero. It looks like throttling. It looks like the GPU is restarting constantly. It is neither: the driver is unloading GPU state every time the last CUDA context exits, and reinitializing it on the next one.

NVIDIA Xid 13: Graphics Engine Exception

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid 13: Graphics Engine Exception

You found a line like this in the kernel log:

NVRM: Xid (PCI:0000:65:00.0): 13, pid='<unknown>', name=<unknown>, Graphics Engine Exception.

Or, more likely, you found the application-side symptom first: a training job died with a generic CUDA error, a run produced NaN loss, or an inference service started returning garbage, and only after digging into dmesg did Xid 13 show up.

Xid 13 is the most ambiguous Xid code NVIDIA emits. It means the graphics engine raised an exception while executing work. The three candidate causes are a bad CUDA kernel (out-of-bounds access, illegal instruction), a driver bug, or genuine hardware degradation. Most isolated occurrences are software. The operational problem is telling which case you are in, because the correct response ranges from “file a bug against the application” to “RMA the GPU”.

NVIDIA Xid 31: GPU memory page fault (invalid address)

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid 31: GPU memory page fault (invalid address)

You found NVRM: Xid (PCI:0000:xx:00.x): 31, ... in the kernel log, usually right after a training job or inference process died with CUDA error: an illegal memory access was encountered. The GPU raised a memory page fault: some unit on the chip accessed a virtual address that was not mapped to valid GPU memory.

The important thing to know up front: Xid 31 almost always means the application is wrong, not the GPU. Accessing freed GPU memory, bad pointer arithmetic, an out-of-bounds index in a kernel. These produce exactly this fault. The exception is when the same GPU produces Xid 31 across multiple unrelated applications. Then the suspicion flips to hardware.

NVIDIA Xid 43: GPU stopped processing (the GPU hang)

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid 43: GPU stopped processing (the GPU hang)

You see NVRM: Xid (PCI:0000:XX:00.0): 43, ... GPU stopped processing in dmesg, and the job on that GPU has gone silent. CUDA calls do not return. nvidia-smi either hangs outright or returns a stale view of the card: utilization frozen at the last sampled value, memory allocation unchanged, power draw flat. The GPU has not disappeared from the bus the way it does with Xid 79. It is still enumerated and still answering the driver at some level, but the compute engine is wedged and nothing it was running will complete.

NVIDIA Xid 48: Double Bit ECC Error

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid 48: Double Bit ECC Error

You found NVRM: Xid (PCI:....): 48, Double Bit ECC Error in dmesg, a training job just died, or a monitoring alert fired on a new uncorrected ECC error. This error is unambiguous: data corruption has occurred in GPU memory, and whatever was running on that GPU when it happened cannot be trusted.

The CUDA context that triggered the error is typically killed, recent outputs (weights, checkpoints, inference results) may be corrupt, and the GPU needs a reset or node reboot before it returns to clean service. There is no “watch and see” path. The correct response is to drain the GPU, verify whether the faulting memory was retired or remapped, validate or discard recent work, and schedule replacement.

NVIDIA Xid 63 and 64: ECC page retirement and row-remap events

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid 63 and 64: ECC page retirement and row-remap events

You found a line like this in the kernel log:

NVRM: Xid (PCI:0000:10:1c): 63, pid=1896, Row Remapper: New row marked for remapping, reset gpu to activate.

Or the less welcome variant, Xid 64. Both come from the GPU’s memory self-repair machinery: ECC detected a bad region of HBM or GDDR memory, and the driver tried to permanently remove it from service. Xid 63 means the repair was recorded successfully. Xid 64 means the recording failed, which is a different and more serious situation.

NVIDIA Xid 74: NVLink error

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid 74: NVLink error

You found NVRM: Xid 74 in dmesg. The driver detected an error on an NVLink connection between this GPU and another GPU or an NVSwitch. The confusing part: nothing crashed. Training is still running. What actually happened is that the link’s error recovery machinery fired - CRC failures, packet replays, link recovery events - and the driver logged it.

Xid 74 is the classic silent straggler. Retransmissions eat NVLink bandwidth, and collective operations like AllReduce synchronize at the speed of the slowest participant. Step time creeps up, throughput drops, and nobody gets paged because nothing “failed.” The job is just slower, and the GPU-hour bill is higher.

NVIDIA Xid 79: GPU has fallen off the bus

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid 79: GPU has fallen off the bus

You will usually find this one by accident: a training job crashes with an opaque CUDA error, nvidia-smi hangs or shows ERR! for one device, and when you check dmesg you find NVRM: Xid (PCI:0000:XX:00): 79, GPU has fallen off the bus. The GPU is gone. Not throttled, not wedged, not holding stale contexts. The PCIe link between the host and the GPU has failed, and the device is no longer reachable at all.

NVIDIA Xid 92: high single-bit ECC error rate

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid 92: high single-bit ECC error rate

You found a line like this in the kernel log:

NVRM: Xid (PCI:0000:3b:00.0): 92, pid='<unknown>', name=<unknown>, High single-bit ECC error rate

Nothing crashed. No job failed. nvidia-smi still works, the GPU is still training or serving, and there is no Xid 48 or 79 demanding an immediate reboot. That is exactly the point of Xid 92: the driver has seen the correctable single-bit ECC error rate cross its internal threshold. Nothing is broken yet. Something will be.

NVIDIA Xid 94 and 95: contained vs uncontained ECC errors (Ampere+)

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid 94 and 95: contained vs uncontained ECC errors (Ampere+)

You are looking at a kernel log line like NVRM: Xid (PCI:0000:01:00): 94, pid=7194, Contained: ... or ...: 95, pid=7062, Uncontained: ... on an A100, H100, or newer GPU, and you need to know two things fast: how bad is it, and what do you drain.

Both Xid 94 and Xid 95 are uncorrectable (double-bit class) ECC error events. The distinction NVIDIA draws, starting with the A100, is containment: whether the corruption was confined to the faulting application context or may have spread across everything running on the GPU. That distinction drives the entire response.

NVIDIA Xid errors: reading NVRM Xid messages in the kernel log

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA Xid errors: reading NVRM Xid messages in the kernel log

You found a line like this in the kernel log, or a user reported a dead training job and you went looking:

NVRM: Xid (PCI:0000:41:00): 79, pid=1432, name=python, GPU has fallen off the bus.

Xid messages are the NVIDIA driver’s primary error-reporting channel. Every significant GPU fault, from a correctable memory event to a catastrophic PCIe link failure, is printed by the NVRM kernel module as an Xid code in the kernel log. Not in nvidia-smi output. Not in a sysfs file. In the log stream, alongside everything else the kernel has to say.

nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver

Mon, 01 Jan 0001 00:00:00 +0000

nvidia-smi hangs or is unresponsive: a wedged GPU or a stuck driver

You run nvidia-smi and nothing comes back. No output, no error, no exit code. Ctrl-C does nothing, and kill -9 does nothing either. This is a different class of failure from “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver”, where the driver at least answers and tells you it is broken. A hang means the driver is stuck inside a kernel call, usually waiting on GPU hardware that will never respond.

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver

Mon, 01 Jan 0001 00:00:00 +0000

NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver

You run nvidia-smi and get:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

This error means the userspace side of the driver (NVML, the library nvidia-smi talks to) cannot reach the kernel side. Either the nvidia kernel module is not loaded, it is loaded but built for a different kernel than the one running, the kernel refused to load it, or the GPU itself is gone from the PCIe bus. nvidia-smi exits non-zero and prints the message to stderr; the error string is the reliable signal, not the exact exit code.

NVMe ASPM latency spikes: PCIe power states adding first-request latency

Mon, 01 Jan 0001 00:00:00 +0000

NVMe ASPM latency spikes: PCIe power states adding first-request latency

You are chasing tail latency on an NVMe-backed service. The p99 graph shows sporadic spikes: tens of microseconds added to reads that should complete in well under 100us. The spikes do not correlate with load, temperature, media errors, queue depth, or garbage collection. SMART is clean. The block layer looks clean. The one pattern you can find: the slow I/Os tend to be the first request after the device sat idle.

NVMe available spare below threshold: critical warning bit 0 and end-of-life wear

Mon, 01 Jan 0001 00:00:00 +0000

NVMe available spare below threshold: critical warning bit 0 and end-of-life wear

Your monitoring fired on an NVMe drive: critical warning bit 0 is set, or available_spare has dropped to (or below) the vendor threshold. The drive still works. Reads and writes complete, latency looks fine, nothing in the application layer is complaining. That is exactly what makes this signal easy to ignore and expensive to ignore.

Bit 0 means the controller’s pool of spare NAND blocks, the reserve it uses to transparently replace failed cells, has fallen below the safety margin the vendor baked into the firmware. The drive is telling you it is approaching end-of-life. It is not telling you it has failed.

NVMe Available Spare below threshold: the spare block pool is running out

Mon, 01 Jan 0001 00:00:00 +0000

NVMe Available Spare below threshold: the spare block pool is running out

When smartctl -A /dev/nvme0n1 reports Available Spare below the Available Spare Threshold, the drive’s internal spare block pool is running low. The controller is reporting less reserved NAND capacity than it considers safe for continued reliable operation.

Available Spare is the percentage of reserved NAND blocks remaining for replacing worn or failed blocks. It starts at 100% and decreases monotonically as the drive consumes spares. When it reaches 0%, the next bad block causes permanent data loss for that block’s data: the drive has no spare to remap to.

NVMe available spare declining: watching the wear trajectory before the threshold

Mon, 01 Jan 0001 00:00:00 +0000

NVMe available spare declining: watching the wear trajectory before the threshold

The drive reported available_spare at 100% for two years. This quarter it reads 94%, and last month it was 96%. Nothing has alerted: critical_warning is zero, I/O is clean, latency is normal. The question is whether you are watching normal aging or the early edge of a failure curve. The answer is almost never in the current value. It is in the rate of change.

NVMe controller reset loop: repeated resets from a firmware hang

Mon, 01 Jan 0001 00:00:00 +0000

NVMe controller reset loop: repeated resets from a firmware hang

Your kernel log shows the same cycle over and over: an I/O command stalls until the 30-second io_timeout fires, the NVMe driver resets the controller, I/O briefly recovers, then the whole thing hangs again. Each cycle costs your applications 5 to 30 seconds of stalled I/O. Databases time out, requests fail, clusters rebalance, and minutes later it happens again.

This is the controller firmware hang loop. The firmware hits a bug triggered by a specific command sequence, queue depth, I/O size, or power state transition. The kernel detects the timeout, resets the controller, and replays the in-flight commands. The same trigger recurs, so the controller hangs again. The loop can continue indefinitely.

NVMe controller state not live: reading resetting, deleting, and dead from sysfs

Mon, 01 Jan 0001 00:00:00 +0000

NVMe controller state not live: reading resetting, deleting, and dead from sysfs

The fastest way to answer “is this NVMe device actually usable right now” is a single file: /sys/class/nvme/nvmeX/state. It is the kernel NVMe driver’s own view of the controller, exposed as one word: live, resetting, connecting, deleting, dead, or new. No SMART parsing, no log pages, no vendor tooling. If the state is not live, I/O to that device is stalled, failing, or about to.

NVMe Critical Warning bits: decoding the health-log bitmask

Mon, 01 Jan 0001 00:00:00 +0000

NVMe Critical Warning bits: decoding the health-log bitmask

The NVMe Critical Warning byte is the single most important health signal on an NVMe drive. Unlike ATA SMART’s scattered attribute IDs, this one byte consolidates five distinct critical conditions into a compact bitmask. Most operators who find this article have just seen a non-zero value from smartctl and need to know which bit is set and what to do about it.

NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask

Mon, 01 Jan 0001 00:00:00 +0000

NVMe critical_warning is nonzero: decoding the SMART critical warning bitmask

Your monitoring fired because critical_warning in the NVMe SMART log is nonzero. Or you ran nvme smart-log /dev/nvme0 during an investigation and saw something like critical_warning : 0x04 where you expected 0x00. You need two things fast: which bit is set, and how bad it is.

The most common mistake here is treating critical_warning != 0 as one alert with one severity. It is a bitmask of six independent conditions with wildly different severity. Bit 3 means the drive has gone read-only and is refusing writes: a page-right-now outage. Bit 0 means spare capacity is below the vendor threshold: a procurement ticket, not a 3 a.m. incident. Bit 1 may be a transient thermal event during a backup run that clears itself.

NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus

Mon, 01 Jan 0001 00:00:00 +0000

NVMe device disappeared: nvme0: Removing and a drive that fell off the PCIe bus

You are looking at a host where an NVMe drive was present and is now gone. The kernel log shows some version of this sequence: a pcieport: AER: Uncorrectable (Fatal) line, possibly followed by nvme nvme0: controller is down; will reset: CSTS=0xffffffff, then nvme nvme0: Removing (or nvme nvme0: Removing after probe failure status: -19), and often nvme0n1: detected capacity change from X to 0. After that, /dev/nvme0n1 no longer exists and /sys/class/nvme/nvme0 is empty.

NVMe devices

Mon, 01 Jan 0001 00:00:00 +0000

NVMe drive in read-only mode: critical warning bit 3 and rejected writes

Mon, 01 Jan 0001 00:00:00 +0000

NVMe drive in read-only mode: critical warning bit 3 and rejected writes

Your monitoring fires a PAGE: the NVMe device has set critical_warning bit 3 (0x08), meaning the drive has autonomously placed its media in read-only mode. Every write command the host sends is now rejected by the drive’s firmware. Reads still work. Writes do not.

This is not transient, not load-dependent, and not cleared by a reboot or controller reset. The drive has decided, based on its own internal assessment, that it can no longer safely accept writes, typically because it has run out of spare NAND blocks to remap failing cells into. It has switched itself into data-preservation mode so you can get your data off before it dies completely.

NVMe endurance runway: projecting time-to-replacement from wear signals

Mon, 01 Jan 0001 00:00:00 +0000

NVMe endurance runway: projecting time-to-replacement from wear signals

NVMe wear-out is one of the few storage failures that usually announces itself months in advance. The announcement is quiet: a monotonically rising percentage_used, a slow drift in available_spare, maybe the first few media_errors. If you only look at current values, you find out when the drive crosses a threshold. If you trend the rates, you can procure before the drive becomes an incident.

NVMe error log entries growing: num_err_log_entries beyond media errors

Mon, 01 Jan 0001 00:00:00 +0000

NVMe error log entries growing: num_err_log_entries beyond media errors

Your NVMe alert fired because num_err_log_entries is climbing. You pull the SMART log, and media_errors is zero. The drive looks healthy, but something is writing error entries at a steady, sometimes alarming, rate.

num_err_log_entries is a superset counter: it counts every error the controller records, including admin command errors, I/O command errors, internal controller errors, and thermal events. media_errors counts only data integrity failures against the NAND. A rising num_err_log_entries with flat media_errors points away from failing flash and toward firmware, driver, or command-level issues. In a large share of cases, it points at your own monitoring stack.

NVMe firmware version tracking: catching the bugs vendors do not advertise

Mon, 01 Jan 0001 00:00:00 +0000

NVMe firmware version tracking: catching the bugs vendors do not advertise

The NVMe controller is an embedded computer running proprietary firmware that owns your data path: the flash translation layer, wear leveling, garbage collection, error correction, and the queue interface to the host. When that firmware has a bug, the symptoms show up as reset loops, erratic latency, premature wear, or in the worst cases silent data loss. Controller firmware bugs are more common than vendors admit, and they are usually triggered by specific command sequences or power-state transitions, so two identical drives on different firmware revisions can behave like different hardware.

NVMe high I/O latency: reading block-layer latency and the outliers that matter

Mon, 01 Jan 0001 00:00:00 +0000

NVMe high I/O latency: reading block-layer latency and the outliers that matter

Latency complaints about NVMe arrive as “the database is slow” or “queries time out randomly.” By the time the report reaches you, the mean latency from iostat often looks fine, because NVMe mean latency is almost always fine. The damage comes from the tail: the occasional 10 ms or 1 s outlier that breaks a request deadline and cascades into application timeouts.

NVMe Media and Data Integrity Errors incrementing: confirmed NAND corruption

Mon, 01 Jan 0001 00:00:00 +0000

NVMe Media and Data Integrity Errors incrementing: confirmed NAND corruption

The NVMe SMART/Health Information log exposes a field called “Media and Data Integrity Errors.” When this value is zero, the controller has never returned data that failed integrity verification. When it increments, the controller detected an unrecovered data integrity error on data retrieved from NAND flash. These are errors such as uncorrectable ECC failures, CRC checksum failures, or LBA tag mismatches that exceeded the controller’s internal correction layer.

NVMe media placed in read-only mode: Critical Warning bit 3

Mon, 01 Jan 0001 00:00:00 +0000

NVMe media placed in read-only mode: Critical Warning bit 3

When smartctl reports Critical Warning: 0x08 on an NVMe drive, the controller has locked the media into read-only mode. The drive is refusing all writes to protect existing data. This is a hardware-enforced decision made by the drive firmware. Reboots, firmware updates, and format commands will not clear it.

The most common cause is endurance exhaustion: the spare block pool is depleted and the controller locks writes rather than risk corruption. Read-only transitions can also result from firmware bugs, thermal events, or electrical issues. An unexpected transition on a drive well below its rated endurance warrants forensic review.

NVMe media_errors increasing: uncorrectable data-integrity errors on NAND

Mon, 01 Jan 0001 00:00:00 +0000

NVMe media_errors increasing: uncorrectable data-integrity errors on NAND

Your monitoring shows the media_errors counter on an NVMe drive going up. In nvme smart-log output the field is labelled media_errors (or “Media and Data Integrity Errors” in some nvme-cli versions), and it is the one SMART field you should never explain away: each increment is a read or write where the controller could not maintain data integrity even after its internal ECC and retry mechanisms were exhausted.

NVMe Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

NVMe Monitoring

What Is NVMe?

NVMe, or Non-Volatile Memory Express, is a storage protocol designed to capitalize on the low latency and internal parallelism of solid-state drives (SSDs). It is widely embraced for boosting storage performance and is crucial for applications demanding high data throughput and speed. NVMe allows for direct CPU communication, drastically reducing data transfer overhead compared to legacy protocols like SATA.

Monitoring NVMe With Netdata

Monitoring NVMe is essential for maintaining optimal performance, ensuring data integrity, and identifying potential hardware issues. Netdata provides a robust set of monitoring tools tailored for NVMe devices that allow you to track their health and performance metrics in real time.