Academies on Netdata

What Is Edge Monitoring? Edge Observability Explained

Thu, 25 Jun 2026 00:00:00 +0000

Edge monitoring means running the full observability pipeline - collection, storage, anomaly detection, alerting, and dashboards - on or near the monitored device itself, instead of shipping all raw telemetry to a central system first. Each node in the fleet collects and processes its own metrics locally, and only forwards what is needed upstream. This model is essential for distributed fleets of thousands of devices where centralizing all telemetry is economically and technically impractical.

NetFlow vs sFlow vs IPFIX: Differences & When To Use

Wed, 24 Jun 2026 00:00:00 +0000

NetFlow, sFlow, and IPFIX are three network traffic export protocols that give visibility into traffic flows without requiring full packet capture. NetFlow and IPFIX aggregate packets into flow records on the device before exporting them, providing exact (if unsampled) accounting. sFlow takes a fundamentally different approach: it exports every Nth packet header plus interface counters, making it stateless and scalable but statistical rather than exact. IPFIX is the IETF-standardized evolution of NetFlow v9, adding vendor-neutral extensibility through template-based information elements.

Network Topology Mapping Explained

Wed, 24 Jun 2026 00:00:00 +0000

Network topology mapping is the process of discovering and visualizing how devices, endpoints, and services connect across a network. It captures both physical (Layer 2) relationships - which switch port links to which device - and logical (Layer 3) relationships - how subnets, routes, and autonomous systems reach each other. A topology map can range from a hand-drawn diagram to a live, continuously updated graph built from protocol data such as LLDP, CDP, ARP, FDB, OSPF, and BGP.

SNMP Monitoring Guide: How To Monitor Network Devices

Wed, 24 Jun 2026 00:00:00 +0000

To monitor SNMP devices you enable an SNMP agent on the device, open UDP/161 from the monitoring host, point your monitoring tool at the device with the correct credentials, and poll interface and health counters on a fixed interval. SNMP works by having a central manager query an agent that exposes numeric objects (OIDs) defined in MIBs, with optional push notifications (traps) for event-driven signals. The bulk of day-to-day network monitoring (interface utilization, errors, port state, CPU, memory) comes from periodic polling of those OIDs.

SNMP Traps vs Polling: What Is The Difference?

Wed, 24 Jun 2026 00:00:00 +0000

SNMP polling is a pull model where a monitoring server periodically queries a device over UDP port 161 to read metrics and state. SNMP traps are a push model where the device itself sends an unsolicited notification over UDP port 162 the instant an event occurs. The two are not competing choices: mature network monitoring uses polling for continuous metrics and traps for immediate event alerts.

What is SNMP polling?

Polling is the classic request-response model at the heart of SNMP-based monitoring. A central manager (your monitoring system) sends a GET, GETNEXT, or GETBULK request to an SNMP agent running on a router, switch, UPS, or other networked device. The agent responds with the requested values.

What Is NetFlow? Network Flow Monitoring Explained

Wed, 24 Jun 2026 00:00:00 +0000

NetFlow is a network protocol, originally developed by Cisco, for collecting metadata about IP traffic flows as they pass through a router, switch, or firewall. A flow is a unidirectional sequence of packets that share the same key fields - classically the 5-tuple of source IP, destination IP, source port, destination port, and protocol. NetFlow does not capture packet payloads; it exports compact metadata records that a collector stores and analyzes, giving you visibility into who is talking to whom, how much bandwidth they are using, and when.

Elasticsearch Yellow Cluster: Unassigned Shards Fix

Sun, 07 Sep 2025 00:00:00 +0000

You run a health check on your production cluster and the result comes back: status: yellow. It’s not the dreaded red status, so your application is likely still serving requests, but this is a critical warning sign. An Elasticsearch yellow cluster status is a direct indication that your data’s high availability is compromised. While all your primary shards are active, one or more replica shards have failed to be assigned to a node. Ignoring this warning can lead to data loss if another node fails, or worse, it could be a symptom of a network partition risking an Elasticsearch split-brain.

Docker Layer Caching in CI Pipelines Cut Build Times by 70 %

Sat, 06 Sep 2025 00:00:00 +0000

You push a one-line code change, and your CI/CD pipeline kicks off. You grab a coffee, come back, and it’s still running, stuck on the docker build step. Slow container builds are a silent productivity killer, delaying feedback, slowing down deployments, and frustrating developers. In an ephemeral CI environment where every job starts with a clean slate, Docker often has to rebuild your entire application image from scratch, every single time.

Linux Cgroups V2 Memory Throttling & OOM Fix

Fri, 05 Sep 2025 00:00:00 +0000

Your critical service is lagging. Users are complaining about timeouts. You check your orchestration platform and see the dreaded OOMKilled status on a container. You dive into the node’s logs (dmesg) and confirm it: the kernel’s Out-of-Memory (OOM) killer has claimed another victim. The immediate fix is easy—restart the container, maybe give it more memory—but the real question remains unanswered: why did it happen? Was it a sudden memory leak, a traffic spike, or something more subtle?

Designing Error Budget Policies For SLOs At Scale

Thu, 04 Sep 2025 00:00:00 +0000

In every engineering organization, there’s a constant, fundamental tension: the push to ship new features versus the need to maintain a stable, reliable service. Move too fast, and you risk outages that erode user trust. Move too slowly, and you risk being outpaced by the competition. For years, this balancing act was managed by intuition, late-night heroics, and tense priority meetings. Site Reliability Engineering (SRE) offers a better way: the error budget.

Consul Service Discovery Failures: Causes & Fixes

Wed, 03 Sep 2025 00:00:00 +0000

It’s a scenario that keeps DevOps and SRE teams up at night: your application logs fill with connection errors, services start failing, and you realize a critical component can’t find the database it depends on. The culprit? A breakdown in your service discovery mechanism. For many, that mechanism is HashiCorp Consul, the backbone of modern microservice architectures. When Consul falters, your entire ecosystem can become unstable.

Understanding how to diagnose these failures is crucial. The problem often lies deep within the operational layers—agent communication issues, misconfigured health checks, or disruptions in the gossip protocol that maintains cluster state. In this guide, we’ll dissect the most common causes of Consul service discovery failures, providing you with the tools to troubleshoot and resolve them. More importantly, we’ll show you how to shift from a reactive, fire-fighting mode to a proactive one, using comprehensive monitoring to build a truly resilient Consul deployment.

Fix NGINX 502/504 Double-Proxy Errors With CloudFront

Tue, 02 Sep 2025 00:00:00 +0000

You’ve set up a modern, resilient architecture: NGINX as your robust origin server and Cloudflare as your global CDN and security layer. Then, it happens. A visitor reports seeing a dreaded white screen with “502 Bad Gateway” or “504 Gateway Timeout.” The immediate question is, who’s to blame? Is Cloudflare having an edge_error, or is your NGINX origin server failing? This confusion is a common pitfall in a double_reverse_proxy setup where requests pass through multiple layers.

PostgreSQL 17 Cardinality Estimation & Tuning Tips

Mon, 01 Sep 2025 00:00:00 +0000

You’ve seen it before: a query that runs in milliseconds on your staging server takes minutes to execute in production. Or a seemingly simple JOIN causes the query_planner to choose a disastrously slow Nested Loop over a much faster Hash Join. In almost every case, the root cause of these performance mysteries is not a bug in PostgreSQL, but a flaw in its understanding of your data. This is the challenge of cardinality_estimation. The planner is only as smart as the statistics it’s given, and when its row_estimation is wrong, the consequences are severe.

Using FOR UPDATE SKIP LOCKED For Queue Workflows

Fri, 29 Aug 2025 00:00:00 +0000

One of the most common and powerful patterns in modern application development is the job queue. Whether you’re sending emails, processing images, or running complex calculations, offloading tasks to background workers is essential for building responsive and scalable systems. Many developers reach for dedicated queueing software like RabbitMQ or Redis, but for many use cases, your primary PostgreSQL database already has all the tools you need to build a robust, transactional, and incredibly performant job_queue_postgres.

SSL/TLS Handshake Failures Causing NGINX 502 Errors

Wed, 27 Aug 2025 00:00:00 +0000

You’ve set up NGINX as a reverse proxy, pointing it to a secure upstream service. You refresh your browser, and instead of your application, you’re greeted by a stark “502 Bad Gateway” page. You check your upstream service; it’s running perfectly. Puzzled, you turn to the NGINX error logs and find a cryptic message that seems to complicate things even further: SSL_do_handshake() failed.

This scenario is frustratingly common. A 502 error suggests a problem with the upstream server, but in many cases, NGINX itself is failing to establish a secure connection with that server. The true culprit is an SSL/TLS handshake failure, often masked by this generic HTTP error code. Understanding the root cause is key to a quick resolution, preventing prolonged downtime and tedious debugging sessions.

How Autovacuum Causes PostgreSQL Deadlocks

Tue, 26 Aug 2025 00:00:00 +0000

You’ve meticulously optimized your application queries. Your transaction logic is sound. Yet, under heavy load, your system seizes up, logging the dreaded “deadlock detected” error. You dig into the logs, expecting to find two application transactions locked in a deadly embrace, but instead, you find a surprising culprit: one of the participants is the PostgreSQL autovacuum process. How can a routine maintenance task, designed to keep the database healthy, be the cause of a production-stopping postgres_deadlock?

Real-World PostgreSQL Deadlock Examples & Fixes

Mon, 25 Aug 2025 00:00:00 +0000

It’s 3 AM. The pager screams. Your application is throwing a cascade of errors, and users are reporting that the system is completely frozen. You dive into the logs and see the same ominous message repeating over and over: ERROR: deadlock detected. A PostgreSQL deadlock is one of the most abrupt and disruptive failures a database can experience. It’s not a performance degradation; it’s a hard stop where two or more transactions are locked in a fatal embrace, each waiting for a resource the other holds.

Redis Sentinel Failover & Split-Brain Recovery Guide

Sun, 24 Aug 2025 00:00:00 +0000

You’re on call. An alert fires—your Redis master node is unreachable. Your heart rate quickens. You’ve set up Redis Sentinel for high availability, but is it working? Did the failover succeed? Or worse, are you now in a Redis cluster split-brain situation where two nodes think they’re the master, leading to data inconsistency and eventual loss? In these critical moments, blindly trusting the automation isn’t enough; you need to verify what’s happening.

Fix NGINX 503 Errors From Misconfigured limit_req

Sat, 23 Aug 2025 00:00:00 +0000

You’ve done the responsible thing. To protect your application from abusive bots and prevent any single user from overwhelming your services, you’ve implemented rate limiting in NGINX. You add the limit_req_zone and limit_req directives, push the configuration, and watch. But instead of seeing a drop in malicious traffic, your monitoring dashboards light up with a sea of red. A massive 503 Service Unavailable spike appears, and legitimate users are complaining they can’t access your site. Your shield has become a weapon turned against yourself.

Prometheus Alertmanager: Noise Reduction Rules

Fri, 22 Aug 2025 00:00:00 +0000

It’s 3 AM, and a torrent of PagerDuty notifications floods your phone. A single network partition has triggered a cascade, making every service instance that can’t reach the database fire its own individual alert. You’re drowning in hundreds of notifications, all pointing to the same root cause. This is alert fatigue, a critical problem for any SRE or on-call engineer. When you’re constantly bombarded with low-signal noise, you risk becoming desensitized, potentially overlooking the one critical alert that signals a major incident.

Tune NGINX Timeouts: Eliminate Upstream Errors

Thu, 21 Aug 2025 00:00:00 +0000

The 504 Gateway Timeout error is a familiar and frustrating sight for anyone managing web applications. It signifies a breakdown in communication, but not between the user and your server. Instead, it means NGINX, acting as your trusty reverse proxy, gave up waiting for a response from a backend, or “upstream,” service. This could be your Node.js application, a PHP-FPM process, or a Python microservice.

While your first instinct might be to blame the application, the root cause is often a simple configuration mismatch. NGINX has built-in timers to protect itself from unresponsive backends. When a backend process takes too long—longer than NGINX is configured to wait—NGINX proactively closes the connection and serves the dreaded 504 error. Understanding and tuning directives like proxy_read_timeout and fastcgi_read_timeout is the key to resolving these issues and building a more resilient infrastructure.

Linux Load Average Spikes: IO Wait & Bottlenecks

Tue, 19 Aug 2025 00:00:00 +0000

An alert fires: “High load average on production server.” Your heart rate quickens. You SSH into the machine and run a command like top, only to be confused. The CPU usage is hovering at 10%, but the load average is sky-high. What’s going on? If the CPU isn’t busy, what is the system “loaded” with? This common scenario highlights one of the most misunderstood metrics in Linux performance troubleshooting: the system load average.

Kubernetes CrashLoopBackOff: Causes & Quick Fixes

Mon, 18 Aug 2025 00:00:00 +0000

You’ve deployed your application to Kubernetes, but a quick check on your pods reveals the dreaded CrashLoopBackOff status. Your pod is stuck in a restart loop, and your service is down. This isn’t an error itself, but a status indicating that Kubernetes is trying to start a container, it crashes, and Kubernetes waits an exponentially increasing amount of time before trying again. This back-off mechanism prevents a faulty pod from overwhelming the cluster with constant restart attempts.

Jenkins Declarative Pipeline: Parallelism Fixes

Sun, 17 Aug 2025 00:00:00 +0000

You’ve embraced the power of Jenkins Declarative Pipelines, and now you’re looking to take the next step in CI/CD optimization: running stages in parallel. The promise is alluring—slashing your build times by running tests, linting, and security scans simultaneously. You eagerly refactor your Jenkinsfile, commit the change, and watch your pipeline trigger. But instead of blazing-fast execution, your build gets stuck, silently hanging with no obvious error.

This frustrating scenario is often caused by two insidious problems: executor starvation and node label mismatch. While the parallel step in Jenkins is incredibly powerful, using it naively can lead to deadlocked pipelines and misconfigured jobs that fail in subtle ways. Understanding how Jenkins allocates resources is the key to unlocking true parallelism without the headaches. This guide will walk you through these common pitfalls and provide the best practices to avoid them.

Buffering keepalive and Hidden 500-Series Errors in NGINX

Sat, 16 Aug 2025 00:00:00 +0000

Your NGINX instance is humming along, handling traffic flawlessly. Then, a sudden traffic spike or a deployment of a new feature that handles larger files occurs, and your monitoring dashboards light up with 500-series errors. You check the NGINX error logs, but find nothing conclusive—just generic 502 Bad Gateway or 504 Gateway Timeout messages that don’t point to a root cause. These are the hidden, frustrating errors that often stem not from your application code, but from poorly tuned NGINX buffering and keepalive settings.

Fix Helm Chart Rollback & Pending-Upgrade Failures

Fri, 15 Aug 2025 00:00:00 +0000

It’s a scenario that every Kubernetes operator dreads. A production deployment has gone wrong, you confidently initiate a Helm rollback, and then… nothing. The command hangs, and a quick check reveals the dreaded pending-rollback status. Your application is now in a broken state, and you’re blocked from deploying any new fixes. This is more than a minor inconvenience; it’s a critical failure that can leave your services unstable and your deployment pipeline paralyzed.

Fix NGINX 502 Bad Gateway In Kubernetes Ingress

Wed, 13 Aug 2025 00:00:00 +0000

You’ve deployed your application to Kubernetes, and everything seems fine until a user reports the dreaded 502 Bad Gateway or 504 Gateway Timeout. In a traditional VM setup, your first step would be to SSH into the server and check the NGINX configuration and logs. But in the ephemeral, abstracted world of Kubernetes, the problem is rarely that simple. A gateway error from the NGINX Ingress Controller is often a symptom of a deeper issue within the cluster’s networking or health management systems.

Blue-Green And Canary Deployments With NGINX

Tue, 12 Aug 2025 00:00:00 +0000

The moment of truth arrives. You’ve tested the new version of your application, the container image is pushed, and the deployment pipeline is ready. You click “deploy,” and a wave of anxiety hits. Will this be a smooth, zero-downtime rollout, or will your dashboards soon light up with 502 Bad Gateway and 504 Gateway Timeout errors? For many teams using advanced deployment strategies like Blue-Green or Canary, this fear is all too real.

Docker Compose Networking: Service Discovery & Ports

Sun, 10 Aug 2025 00:00:00 +0000

You’ve meticulously crafted your docker-compose.yml file, your services build correctly, and the startup process finishes without a single error. Yet, when your application tries to connect to its database, you get a “Connection refused” or “Host not found” error. This is a classic and frustrating scenario in Docker Compose networking. Your containers are running, but they’re isolated in their own digital worlds, unable to communicate.

These issues almost always boil down to two core mysteries: service discovery failed events, where containers can’t find each other, and docker compose port conflict problems, where port mappings are misunderstood or misconfigured. To build reliable containerized applications, you must look beyond the docker-compose.yml file and understand the virtual network that Docker creates. This guide will demystify that network, providing you with the tools and techniques to troubleshoot and resolve these common connectivity challenges.

Metric Cardinality In Observability: Strategies

Sun, 10 Aug 2025 00:00:00 +0000

It’s a story familiar to any SRE or DevOps engineer. You add a seemingly innocuous label to a key metric—user_id, request_id, container_id—to gain deeper insight. Suddenly, your monitoring bill skyrockets, your Prometheus_TSDB instance starts gasping for memory, and dashboards slow to a crawl. You have just triggered a label_explosion, the single biggest challenge in modern metrics-based observability: metrics_cardinality.

High cardinality isn’t an edge case; it’s the new normal in a world of microservices, containers, and complex user interactions. As the number of unique time series grows into the millions or even billions, it places immense pressure on monitoring systems, impacting storage_costs, query performance, and the fundamental ability to scale.

Early Deadlock Detection With pg_stat_kcache & eBPF

Sat, 09 Aug 2025 00:00:00 +0000

The dreaded deadlock. For anyone managing a high-traffic PostgreSQL database, it’s a familiar nemesis. The database logs a “deadlock detected” message, a transaction is unceremoniously aborted, and your application has to handle the fallout. This reactive cycle is frustrating; by the time you’re alerted, the damage is already done. Traditional postgres_metrics and monitoring tools are excellent at telling you that a deadlock occurred, but they fall short of explaining the subtle conditions that led to it.

GitLab Runner Executor Failures: Docker & Kubernetes

Thu, 24 Jul 2025 00:00:00 +0000

You’ve been there. You push your code, a new CI/CD pipeline kicks off in GitLab, and you wait for that satisfying green checkmark. Instead, you get a dreaded red ‘X’. The pipeline failed. Digging into the job logs, you find a cryptic message: “ERROR: Job failed (system failure): prepare environment: exit code 1”. The culprit is often the GitLab Runner executor—the very engine responsible for running your jobs—failing in its environment.

Apache Kafka Consumer Lag: Troubleshooting & Fixes

Wed, 23 Jul 2025 00:00:00 +0000

You’ve built a powerful, real-time data pipeline with Apache Kafka, but suddenly, things grind to a halt. The dreaded consumer lag is exploding, alerts are firing, and your downstream applications are starved for data. This scenario is all too common for teams running Kafka at scale. Often, the culprit is a subtle interplay between consumer group rebalancing, partition assignment, and suboptimal consumer configurations. Understanding these mechanics is not just about fixing a problem; it’s about building resilient, high-throughput streaming systems from the ground up.

Apache mod_rewrite Debugging & Log Analysis

Wed, 23 Jul 2025 00:00:00 +0000

You’ve crafted what seems like a perfect rewrite rule, uploaded it to your .htaccess file, and refreshed your browser, only to be met with a “Too many redirects” error. You’re stuck in an Apache redirect loop, a frustrating and all-too-common problem when working with Apache mod_rewrite. This powerful module can manipulate URLs in almost any way imaginable, but its complexity means that a small mistake in a regular expression or a missing condition can lead to unexpected behavior, broken links, or infinite loops.

AWS ELB 5xx Surge Analysis: Health & Draining

Wed, 23 Jul 2025 00:00:00 +0000

The alert notification chimes, and your dashboard lights up with an ELB 5xx error rate spike. Your application, running behind an AWS Application Load Balancer (ALB), is suddenly returning 502, 503, or 504 errors. This is a high-stakes scenario where every second of downtime impacts users. The load balancer is often just the messenger; the real culprit lies somewhere in your backend infrastructure. Is a target group unhealthy? Are your connections timing out? Is a misconfigured deployment process wreaking havoc?

25 NGINX Directives to Audit Before Opening a Support Ticket

Wed, 09 Jul 2025 00:00:00 +0000

You’ve built a robust application, but it’s sluggish or throwing intermittent errors. Your first instinct might be to blame the code, but often the culprit lies in the intricate web of your NGINX configuration. Before you spend hours debugging your application or drafting a support ticket, a thorough NGINX config check can save you time and reveal simple yet impactful misconfigurations.

An NGINX configuration audit isn’t just for troubleshooting; it’s a proactive step toward better performance, tighter security, and greater stability. Many common issues related to performance bottlenecks, security vulnerabilities, or proxy errors stem from overlooked or poorly optimized directives. This checklist covers 25 essential directives you should review. Working through this list can help you pinpoint problems, implement best practices, and gain a deeper understanding of how NGINX operates.

Short vs Long Transactions In PostgreSQL Benchmarks

Wed, 09 Jul 2025 00:00:00 +0000

Every developer working with a database faces a fundamental design choice: how to structure their transactions. Should a complex business operation, involving multiple database writes, be wrapped in a single, all-or-nothing transaction? Or is it better to break it down into a series of smaller, faster commits? On the surface, the long_running_transaction seems safer, guaranteeing perfect atomicity. But as traffic grows, a mysterious performance degradation can set in. Latency spikes, throughput drops, and users start seeing errors. The culprit is often not the queries themselves, but the invisible contention they create.

How Can Generative AI (Gen AI) Be Used In Cybersecurity