Network Monitoring Operations Guides on Netdata

ARP cache staleness: when IP-to-MAC mapping goes bad

Mon, 01 Jan 0001 00:00:00 +0000

ARP cache staleness: when IP-to-MAC mapping goes bad

Hosts on the same subnet stop reaching each other after a VM live-migrates, a container gets rescheduled, or a firewall fails over. ICMP works from some hosts but not others. TCP sessions hang or reset. The data plane is healthy, but the ARP cache on one or more hosts holds a stale IP-to-MAC mapping.

ARP cache staleness is the gap between when a MAC address changes and when every interested host learns about the change. On Linux, this gap is governed by the neighbor (NUD) state machine and its timing parameters. On Windows Vista and later, the neighbor cache follows the same RFC 4861 model. Both platforms default to roughly the same reachable time window: about 15 to 45 seconds before an entry transitions to a stale state, followed by a probe sequence that adds several more seconds before resolution or eviction.

Asymmetric routing: why your path and latency measurements lie

Mon, 01 Jan 0001 00:00:00 +0000

Asymmetric routing: why your path and latency measurements lie

Your monitoring says the path is fine. Ping latency is normal, traceroute shows a clean route, and interface counters look healthy. But applications are slow, TCP sessions stall or reset, and users are complaining. Your tools are measuring only half the path.

In asymmetric routing, traffic from host A to host B takes one path (P1) while return traffic from B to A takes a different path (P2). When P2 is degraded, congested, or broken, your measurements average the healthy forward path with the impaired return path. Every acknowledgment and response is fighting through a bad route while the aggregate looks acceptable.

Audit log gaps: detecting syslog/trap tampering or loss

Mon, 01 Jan 0001 00:00:00 +0000

Audit log gaps: detecting syslog/trap tampering or loss

An audit log gap is any period where expected syslog messages or SNMP traps from a network device fail to arrive at the collector. UDP syslog on port 514 and SNMP traps on port 162 are fire-and-forget transports with no delivery guarantee. The kernel silently drops datagrams when socket buffers fill, and the application layer never sees the loss. TCP syslog can stall under collector backpressure. Most gaps are operational: network loss, device-side buffer overflow, or logging subsystem failure. The difficulty is distinguishing those from deliberate log suppression after compromise.

BGP flapping: why a peer keeps resetting and how to find the cause

Mon, 01 Jan 0001 00:00:00 +0000

BGP flapping: why a peer keeps resetting and how to find the cause

A BGP peer cycling between Established and Idle is sending a specific signal. The session tears down because one side sent a NOTIFICATION message, and that message carries an error code and subcode that pinpoints the cause. Most monitoring watches only the FSM state (up or down) and ignores the NOTIFICATION payload, so the operator sees flapping without knowing why.

BGP NOTIFICATION and Cease messages: what each subcode is telling you

Mon, 01 Jan 0001 00:00:00 +0000

BGP NOTIFICATION and Cease messages: what each subcode is telling you

A BGP NOTIFICATION in your router log is a peer telling you why it tore down the session. The message carries an error code and an error subcode. Those two numbers tell you whether you are looking at a maintenance window, a route leak, a prefix-limit hit, a CPU-starved control plane, or a BFD-triggered teardown.

Cease (code 6) is the most common NOTIFICATION. Its subcodes, defined in RFC 4486 and extended by RFC 8538 and RFC 9384, hold most of the diagnostic value. Codes 2 through 5 appear less often but point to distinct failure classes: parameter mismatch, malformed updates, hold-timer expiry, and FSM errors.

BGP RIB and FIB growth: monitoring route-table size before it bites

Mon, 01 Jan 0001 00:00:00 +0000

BGP RIB and FIB growth: monitoring route-table size before it bites

The global BGP routing table grows every year. In 2026, the IPv4 default-free zone sits at approximately 940,000 prefixes, with IPv6 adding roughly 190,000 more. These numbers increase steadily, and the hardware that programs forwarding decisions from them has finite capacity. When that capacity runs out, new routes do not get installed in the forwarding plane. Traffic to affected destinations blackholes. The BGP session stays Established the entire time.

BGP route leak and hijack: the detection signals and alerts that matter

Mon, 01 Jan 0001 00:00:00 +0000

BGP route leak and hijack: the detection signals and alerts that matter

A BGP route leak or hijack does not tear down your session. The peering stays Established, keepalives flow, and the session “up” indicator stays green. What changes is which prefixes your network believes are reachable, through which origin AS, and via what path. Traffic is silently misrouted or blackholed while the session looks healthy.

BGP has no built-in authentication of route ownership. Any AS can announce any prefix. Whether other networks accept the announcement depends on their filtering, and filtering is inconsistently deployed. Approximately 50% of routable IP prefixes carry a Route Origin Authorization (ROA), and only about 6.5% of Internet users sit behind networks that actively reject RPKI-invalid routes. That gap is where leaks and hijacks propagate globally before anyone notices.

BGP session Established but stale: detecting silent route loss

Mon, 01 Jan 0001 00:00:00 +0000

BGP session Established but stale: detecting silent route loss

Your BGP session to a transit provider or iBGP peer is Established, but destinations are unreachable. The RIB is missing prefixes from that peer, or the routes it has are stale. No NOTIFICATION was sent, no session flap occurred, and your monitoring trusts the FSM state.

This is the “Established but stale” pattern. KEEPALIVEs are still exchanged at the TCP level, but the UPDATE exchange has stopped. The peer stopped sending routes, a middlebox is silently dropping UPDATE packets, or Graceful Restart is holding the session open after the remote side went down.

Cold-start topology: why your map is incomplete after a collector restart

Mon, 01 Jan 0001 00:00:00 +0000

Cold-start topology: why your map is incomplete after a collector restart

You restart your flow collector or topology engine for a routine upgrade. The process comes back up cleanly. The dashboard loads. But the topology map is half-empty, endpoint positions are wrong, and within minutes someone pages you asking why a security investigation points to the wrong switch port.

The root cause is not a bug. After a restart, the topology inference engine has no cached neighbor tables, no FDB entries, no ARP data, and potentially no flow templates. It must rebuild all of these from live polling and flow data before it can produce a reliable view. The window between restart and first complete topology ranges from a few minutes to over 30 minutes, depending on poll cadence, template refresh intervals, and which sources your topology engine fuses.

Collector CPU and TSDB write-queue saturation: the capacity signals

Mon, 01 Jan 0001 00:00:00 +0000

Collector CPU and TSDB write-queue saturation: the capacity signals

When a network monitoring collector saturates, the first visible symptom is rarely high collector CPU. It is traffic charts showing a decline during a traffic spike, an SNMP poll cycle drifting past its configured interval, or unexplained gaps in flow data. The degradation sits one to three subsystems downstream of the actual bottleneck, which is why collector-side incidents are frequently misdiagnosed.

Correlating cloud VPC flow logs with on-prem NetFlow

Mon, 01 Jan 0001 00:00:00 +0000

Correlating cloud VPC flow logs with on-prem NetFlow

Cloud flow logs and on-prem flow records share the 5-tuple concept but diverge in nearly every dimension that matters for correlation: transport, latency, sampling, timestamps, topology, and NAT visibility. Cloud providers emit VPC flow logs via push to object storage with implicit sampling and aggregation intervals measured in minutes. On-premises devices export NetFlow v5/v9, IPFIX, or sFlow over UDP with configurable sampling and near-real-time delivery.

Device control-plane CPU saturation: when SNMP polling causes the spike

Mon, 01 Jan 0001 00:00:00 +0000

Device control-plane CPU saturation: when SNMP polling causes the spike

The router’s control-plane CPU is pinned at 95%. SNMP polls are timing out. BGP sessions are approaching hold-time expiry. The instinct is to blame the device or suspect an attack, but the monitoring system itself is frequently the source of the load.

The control-plane CPU handles everything that is not hardware-forwarded packet switching: the SNMP agent, BGP, OSPF, STP, the CLI, syslog, AAA, and management interfaces. When SNMP polling saturates this CPU, every control-plane function degrades at once. The symptoms look like a device problem. The cause is often on the collector side.

Device memory pressure: control-plane memory pools and leaks

Mon, 01 Jan 0001 00:00:00 +0000

Device memory pressure: control-plane memory pools and leaks

A network device alerts at 85% control-plane memory utilization. Or worse: it does not alert, and you discover the problem when BGP sessions start dropping from hold-time expiry, SNMP stops responding, or the device reboots itself. Control-plane memory exhaustion degrades every process on the route processor: routing protocols, CLI, SNMP, management interfaces, logging.

“High memory utilization” on a network device is ambiguous. Some platforms cache aggressively and report 97% used under normal operation. Some pools are expected to sit near zero free. A genuine memory leak may take weeks to manifest, making it easy to dismiss the trend until the device crashes at 3 a.m.

Flow export-to-ingest latency: why your NetFlow data is minutes behind

Mon, 01 Jan 0001 00:00:00 +0000

Flow export-to-ingest latency: why your NetFlow data is minutes behind

Flow export-to-ingest latency accumulates across a pipeline: the exporter’s active timeout, the UDP transport path, the kernel socket buffer, the collector’s parser, and the storage write queue. Each stage can add seconds or minutes, and each has a different fix.

The most common cause is the active timeout default on most network devices: 30 minutes. Long-lived flows (VPN tunnels, database connections, bulk transfers) are not exported until the timer expires. The collector is not slow and the network is not congested. The device is behaving as configured. But if you need near-real-time visibility, a 30-minute export delay is indistinguishable from broken telemetry.

Interface discards with low utilization: diagnosing ifInDiscards/ifOutDiscards

Mon, 01 Jan 0001 00:00:00 +0000

Interface discards with low utilization: diagnosing ifInDiscards/ifOutDiscards

ifOutDiscards is climbing on a critical uplink. Utilization sits at 35%. No CRC errors, no input errors, no physical-layer alarms. The link is up and passing traffic, but something is silently dropping packets, and your averaged utilization metrics are not telling you why.

The gap between what the counters show and what the silicon is doing comes down to two things: the averaging window on utilization, and the fact that discards happen at buffer-queue granularity, not at link-rate granularity.

Interface flapping: link up/down storms and their blast radius

Mon, 01 Jan 0001 00:00:00 +0000

Interface flapping: link up/down storms and their blast radius

Interface flapping is when a network interface oscillates rapidly between up and down states. Each transition generates a linkDown/linkUp trap pair, syslog entries, and an STP topology change notification. At low rates this is operational noise. At high rates it becomes a multi-layer failure: the trap receiver overflows, the syslog pipeline saturates, STP reconvergence flushes MAC tables across the VLAN, and the monitoring platform reports misleading availability because the poll interval is slower than the flap cadence.

Interface input/output errors: finding the bad link with ifInErrors/ifOutErrors

Mon, 01 Jan 0001 00:00:00 +0000

Interface input/output errors: finding the bad link with ifInErrors/ifOutErrors

A switch port reports 47,000 input errors and climbing. The interface is up and traffic is flowing. Your monitoring fired an alert on ifInErrors crossing threshold. Now you need to determine whether this is a dirty fiber, a dying SFP, a duplex mismatch, buffer exhaustion, or a counter artifact from an interface flap.

ifInErrors (.1.3.6.1.2.1.2.2.1.14) and ifOutErrors (.1.3.6.1.2.1.2.2.1.20) are aggregate counters. They tell you something is wrong, but not what. The counter is a sum of multiple error types: CRC, alignment, runts, giants, overruns, frame errors, and on some platforms, input drops. A frame that arrives with both a CRC error and a runt condition increments ifInErrors by exactly 1, not 2. You cannot reconcile the sub-counters against the aggregate by simple addition.

Interface saturation: measuring utilization against ifHighSpeed correctly

Mon, 01 Jan 0001 00:00:00 +0000

Interface saturation: measuring utilization against ifHighSpeed correctly

Interface utilization is one of the most frequently miscomputed network metrics. The formula is simple: divide throughput by capacity, multiply by 100. But the SNMP objects for throughput and capacity have different precision, units, and failure modes. An interface that is genuinely saturated can report 0% utilization. A healthy link can report 300%.

The root cause is almost always the denominator. ifSpeed (IF-MIB .1.3.6.1.2.1.2.2.1.5) is a 32-bit Gauge that caps at 4,294,967,295 bps, approximately 4.29 Gbps. For any link faster than that, ifSpeed saturates at its maximum value and utilization computed against it is wrong. The correct denominator is ifHighSpeed (.1.3.6.1.2.1.31.1.1.1.15), which reports speed in units of 1,000,000 bps (Mbps) and has no practical upper bound. A value of 10000 means 10 Gbps; 100000 means 100 Gbps.

License expiry silently disabling features: monitor days-to-expiry

Mon, 01 Jan 0001 00:00:00 +0000

License expiry silently disabling features: monitor days-to-expiry

Your firewall dashboard shows green. Interfaces are up, CPU and memory are normal, traffic is flowing. But at 09:00, someone reports VPN connections failing, IPS no longer blocking threats, or URL filtering not enforcing policy. A feature license expired at midnight, and the device silently stopped performing the licensed function without raising a visible alarm.

The device stays up, counters keep incrementing, throughput looks normal. The license-expiry message in syslog is low severity and gets buried under routine noise. By the time someone notices, the feature has been disabled for hours.

Locating endpoints behind NAT and wireless: the positioning problem

Mon, 01 Jan 0001 00:00:00 +0000

Locating endpoints behind NAT and wireless: the positioning problem

Endpoint positioning maps a MAC address or IP to a specific switch port, access point, or VLAN. It underpins security investigations, access control enforcement, and day-to-day troubleshooting. When the endpoint sits behind a NAT boundary, the Layer 2 and Layer 3 signals that topology engines rely on (FDB entries, ARP tables, flow records) all report the NAT device’s identity, not the endpoint behind it. The endpoint becomes operationally invisible upstream.

Microbursts: catching sub-second congestion that minute averages hide

Mon, 01 Jan 0001 00:00:00 +0000

Microbursts: catching sub-second congestion that minute averages hide

Your switches are dropping packets. The utilization charts say everything is fine. Interface counters show moderate load, error rates are clean, and no congestion alert has fired. But applications report retransmissions, latency spikes, and intermittent connectivity. The problem resolved between polls.

A microburst is a short, intense spike of traffic that fills a switch egress queue faster than the queue can drain. The burst may last 50 milliseconds or less, but during that window the queue overflows and packets are tail-dropped. By the time your SNMP poller arrives 60 or 300 seconds later, the burst is over, the queue has drained, and interface utilization has been averaged down to an unremarkable number.

Monitoring overlay tunnels (IPsec/GRE/VXLAN): the signals that matter

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring overlay tunnels (IPsec/GRE/VXLAN): the signals that matter

Overlay tunnels share the underlay’s physical path but add their own failure modes: encapsulation overhead, separate control and data planes, and type-specific state machines. Monitoring only the tunnel interface’s link state is the fundamental trap. An interface reporting “UP” can still be forwarding into a black hole because the peer is unreachable, the SA has expired, or the underlay is dropping fragments.

Monitoring route origin and AS-path changes for hijack detection

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring route origin and AS-path changes for hijack detection

BGP route hijacks do not announce themselves. A prefix legitimately originated by AS2906 starts appearing in the global table with origin AS65001. Traffic that should reach your infrastructure follows a different path, gets blackholed, or lands on an interception point. The control-plane signals are there, but they are distributed across route collectors, RPKI validators, and per-prefix state that most monitoring stacks do not track at the granularity needed.

NAT and session-table exhaustion: catching it before connections fail

Mon, 01 Jan 0001 00:00:00 +0000

NAT and session-table exhaustion: catching it before connections fail

New connections fail while existing ones keep working. Applications report “connection refused” or timeouts. Open SSH sessions stay alive, but new SSH attempts hang. Your monitoring shows the firewall or NAT gateway is up, interfaces are healthy, and CPU is normal. The session or NAT translation table is full.

Session-table exhaustion is a cliff-edge failure. The table degrades gracefully until it hits its limit, then every new connection is denied. Existing flows continue because their entries are already in the table. The symptom pattern is distinctive but easy to misdiagnose as application failure, DNS issues, or upstream provider problems, because the applications are the ones reporting errors.

NetFlow storage sizing: how much disk your flow collector really needs

Mon, 01 Jan 0001 00:00:00 +0000

NetFlow storage sizing: how much disk your flow collector really needs

Flow records arrive at thousands to tens of thousands per second, and every record hits disk. The bottleneck is almost always disk throughput or capacity, not CPU.

This article covers the math: raw record sizes, effective storage after columnar compression, the capacity formula with worked examples, IOPS considerations, and the operational pitfalls that make disks fill faster than the formula predicts. The guidance applies to NetFlow v5/v9, IPFIX, and sFlow collectors using ClickHouse or similar columnar backends.

NetFlow v9/IPFIX template desync: flows decoded wrong or dropped after a reboot

Mon, 01 Jan 0001 00:00:00 +0000

NetFlow v9/IPFIX template desync: flows decoded wrong or dropped after a reboot

You rebooted a router or upgraded its firmware. Minutes later, your flow collector shows a gap or anomaly. The exporter is still sending data: UDP packet counters are nonzero and climbing. But decoded flow records are zero, suspiciously low, or the field values are shifted and garbled.

This is NetFlow v9 or IPFIX template desync. The collector holds cached template definitions that no longer match what the exporter is sending. Until it receives and caches the correct templates, it either drops records silently or misinterprets the byte layout, producing garbage fields.

NetFlow vs sFlow vs IPFIX: what they measure and how each one fails

Mon, 01 Jan 0001 00:00:00 +0000

NetFlow vs sFlow vs IPFIX: what they measure and how each one fails

Flow telemetry protocols are often lumped together as “flow data,” but they measure fundamentally different things. NetFlow and IPFIX build stateful flow records by tracking conversations in device memory. sFlow captures random packet samples without maintaining any flow state. This architectural split determines not only what you can see but how the data breaks when something goes wrong.

Network monitoring checklist: the signals every production network needs

Mon, 01 Jan 0001 00:00:00 +0000

Network monitoring checklist: the signals every production network needs

This checklist covers the signals production networks need, organized by detection priority and mapped to maturity levels from survival to expert.

An NPM stack is a federation of collectors, parsers, enrichment services, storage tiers, and an analytics core. Most production incidents are not “the network broke” but “a collector’s UDP buffer dropped packets,” “the NetFlow v9 template cache went stale after a device reboot,” or “the polling worker pool fell behind and now a healthy device looks down.” The checklist is organized to surface those failure modes, not just the top-level symptoms.

NIC RSS misconfiguration: one CPU core silently dropping your telemetry

Mon, 01 Jan 0001 00:00:00 +0000

NIC RSS misconfiguration: one CPU core silently dropping your telemetry

Your flow collector has 16 CPU cores, but one is pinned at 100% while the other 15 sit idle. NIC receive drop counters are climbing. UDP socket buffer errors (Udp_RcvbufErrors) are incrementing. Your bandwidth charts show traffic declining during what is actually a traffic spike. The box looks underpowered, so you start sizing a bigger one. The real problem: Receive Side Scaling (RSS) is funneling every inbound packet to a single receive queue serviced by a single CPU core. No amount of additional cores or RAM fixes this until RSS distributes interrupts across them.

Normalizing syslog severity across vendors: why 'critical' isn't critical

Mon, 01 Jan 0001 00:00:00 +0000

Normalizing syslog severity across vendors: why ‘critical’ isn’t critical

RFC 5424 defines eight syslog severity levels, numbered 0 through 7: Emergency, Alert, Critical, Error, Warning, Notice, Informational, and Debug. Every major network vendor implements the same numeric scale. The integer that means “Critical” on a Cisco router means “Critical” on a Juniper switch.

But the severity a device assigns to a given event is not standardized. A BGP session reset might arrive as severity 5 (Notice) from one vendor and severity 3 (Error) from another. A hardware alarm that one platform logs as Critical (2), another logs as Alert (1) or Warning (4). Same operational condition, different severity label, same RFC.

NTP drift on network devices: the silent killer of event correlation

Mon, 01 Jan 0001 00:00:00 +0000

NTP drift on network devices: the silent killer of event correlation

Clock drift on network devices produces no visible symptom. The device stays up, interfaces carry traffic, BGP sessions remain Established, SNMP keeps responding. The damage surfaces hours or days later, in a postmortem where two devices’ timestamps disagree by hundreds of milliseconds and the analyst cannot reconstruct the event sequence. Every cross-device correlation in the monitoring stack depends on accurate, monotonic time across every collector and every polled device.

RPKI invalid routes: monitoring route origin validation

Mon, 01 Jan 0001 00:00:00 +0000

RPKI invalid routes: monitoring route origin validation

RPKI (Resource Public Key Infrastructure) route origin validation is a strong signal for detecting BGP hijacks and route leaks before they reroute traffic. A route classified Invalid by RPKI is, with high confidence, not originated by the authorized AS. Accepting one is a security event.

The monitoring problem: RPKI validation state is not exposed through any standard SNMP MIB. BGP4-MIB (RFC 4273) predates RPKI entirely. CISCO-BGP4-MIB has no validation-state column. There is no portable OID to count invalid routes across a multi-vendor estate. Every platform exposes this through its own CLI, proprietary MIB extensions, or BMP (RFC 7854).

SD-WAN tunnel up but degraded: when the control plane lies

Mon, 01 Jan 0001 00:00:00 +0000

SD-WAN tunnel up but degraded: when the control plane lies

The orchestrator shows your SD-WAN tunnel as UP. Control connections to vSmart or vBond are healthy. OMP sessions are Established. But users at the far end report slow applications, dropped voice calls, or timeouts.

The control plane reports a healthy tunnel while the data plane is degraded with packet loss, latency spikes, or silent traffic drops. Interface counters show UP/UP because the degradation is on the underlay path or inside the encapsulated data plane, not on the local interface.

sFlow sampling rate: why your traffic totals are off by 1000x

Mon, 01 Jan 0001 00:00:00 +0000

sFlow sampling rate: why your traffic totals are off by 1000x

Your sFlow-derived bandwidth charts show a 10G link carrying 12 Mbps. SNMP counters on the same interface show 8.4 Gbps. The switch is not broken and the collector is not dropping packets. The analytics pipeline is summing raw sampled bytes without multiplying by the sampling rate.

sFlow is not NetFlow. It does not maintain a flow cache on the device, aggregate bytes per conversation, and export summary totals. sFlow exports individual packet samples, one per datagram, each carrying the packet’s header data and metadata about the sampling process. The collector is responsible for turning those samples into traffic estimates through multiplication. When that multiplication is missing, every chart, alert, capacity plan, and billing report built on the data is wrong by the sampling factor.

Silent UDP flow data loss: why your NetFlow collector is dropping records

Mon, 01 Jan 0001 00:00:00 +0000

Silent UDP flow data loss: why your NetFlow collector is dropping records

Your flow analytics show traffic declining on multiple exporters simultaneously. SNMP interface counters say traffic is rising. No device alarms, no exporter config changes, no visible network events. The most likely cause: your collector is silently dropping UDP datagrams at the kernel socket buffer boundary.

UDP has no delivery guarantee. When the socket receive buffer fills, the kernel discards incoming datagrams silently. No error is logged. The only signal is UdpRcvbufErrors in /proc/net/snmp, a counter most teams do not monitor. During a traffic spike or DDoS, your charts may show “normal” or declining traffic while actual packet rates are significantly higher.

SNMP authentication-failure spikes: misconfiguration vs reconnaissance

Mon, 01 Jan 0001 00:00:00 +0000

SNMP authentication-failure spikes: misconfiguration vs reconnaissance

SNMP authentication-failure traps are one of the few security signals built into the network monitoring stack. When they spike, the question is never “is something wrong?” - it is “is this a broken poller or someone probing my devices?” The answer changes the response from a quiet config fix to a security incident.

The authenticationFailure trap (OID 1.3.6.1.6.3.1.1.5.5) fires whenever an SNMP agent receives a protocol message that is not properly authenticated. On SNMPv2c, that means a wrong community string. On SNMPv3, it means a wrong username, wrong auth protocol, wrong auth password, or wrong privacy password. The trap is defined in SNMPv2-MIB and every compliant agent can generate it, but many vendors ship with it disabled by default. If you have never explicitly enabled it (for example, snmp-server enable traps snmp authentication on Cisco IOS), you may have no signal at all.

SNMP counter discontinuity after reboot: bogus rate spikes explained

Mon, 01 Jan 0001 00:00:00 +0000

SNMP counter discontinuity after reboot: bogus rate spikes explained

A 1-gigabit interface shows 40 terabits per second on your dashboard right after a switch reboot. The traffic never happened. The chart is lying because of how SNMP counters and rate calculations interact.

SNMP interface counters (ifInOctets, ifHCInOctets, ifOutOctets, and friends) are monotonically increasing integers. Your monitoring platform does not read current bandwidth from the device. It subtracts the previous counter value from the current one, divides by elapsed time, and reports the result as a rate. When a counter resets to zero after a reboot or wraps past its maximum, that subtraction produces a physically impossible number. If your alerting or billing pipeline acts on it, you have a problem.

SNMP counter rollover: fake traffic spikes from 32-bit counters

Mon, 01 Jan 0001 00:00:00 +0000

SNMP counter rollover: fake traffic spikes from 32-bit counters

A bandwidth chart suddenly shows a multi-terabit spike on a 10G interface. The on-call engineer investigates and finds the link was nearly idle. The spike is a math artifact: a 32-bit SNMP counter wrapped from near its maximum value (4,294,967,295) back to zero between two polls, and the collector’s differencing algorithm produced a nonsensical delta.

Depending on how the collector handles the wrap, the symptom differs: a fake spike (when the negative delta is treated as unsigned) or a fake traffic drop to zero (when treated as signed and clamped). Both hide real traffic patterns and train operators to ignore chart anomalies, including genuine ones.

SNMP poll response latency: diagnosing a slow poller

Mon, 01 Jan 0001 00:00:00 +0000

SNMP poll response latency: diagnosing a slow poller

SNMP poll response latency is the round-trip time from your collector’s GET or GETBULK request to the device’s response. When it climbs, rate calculations lose accuracy, worker threads hold their slots longer than expected, and the poller falls behind schedule. Healthy devices start appearing stale or unreachable.

The most common misdiagnosis is “the network is slow.” On a LAN, an SNMP GET to sysUpTime should return in single-digit milliseconds. When the same device takes 2 to 5 seconds to respond, ICMP to the same target will usually confirm the path is fine. The bottleneck is almost always the device’s SNMP agent, the collector’s scheduler design, or a specific OID family that triggers expensive computation on the device CPU.

SNMP poller falling behind: the polling-storm cascade and how to catch it

Mon, 01 Jan 0001 00:00:00 +0000

SNMP poller falling behind: the polling-storm cascade and how to catch it

A single slow device is all it takes. Your SNMP poller queue drifts past its scheduled interval. Within minutes, 30 devices show as DOWN in your NMS dashboard. Every one of them responds to ping. The network is fine; your poller is the problem.

Scheduler fall-behind is the most common false “device down” trigger in network monitoring. When a poller cannot complete its collection cycle within the configured interval, every subsequent cycle inherits the debt. Devices that are reachable and healthy appear DOWN because their next poll slot arrives late relative to the alerting threshold. The cascade is self-reinforcing: missed polls generate retries, retries consume worker threads, fewer workers means slower polls for all other devices, more devices time out, and queue depth grows unboundedly.

SNMP timeouts and retries: why devices show as down when they aren't

Mon, 01 Jan 0001 00:00:00 +0000

SNMP timeouts and retries: why devices show as down when they aren’t

SNMP runs over UDP port 161, a transport with no delivery guarantee. When your monitoring platform reports that devices are down, the first question is not “why is the network broken” but “is this actually a network problem, or is my polling stack the problem.” SNMP timeout and retry behavior is one of the most common causes of false-positive “device down” alerts, and it is also one of the most misdiagnosed.

SNMP trap receiver dropping traps: silent UDP/162 loss

Mon, 01 Jan 0001 00:00:00 +0000

SNMP trap receiver dropping traps: silent UDP/162 loss

When SNMP traps silently disappear, the first place to look is rarely the device. SNMP traps are push-based UDP datagrams on port 162. The kernel buffers them, and the receiver application (typically snmptrapd or a commercial collector) must drain that buffer faster than it fills. If it does not, the kernel silently drops datagrams and increments a counter the application never sees. No error is logged, and no alert fires.

SNMP v2c vs v3: monitoring coverage and security trade-offs

Mon, 01 Jan 0001 00:00:00 +0000

SNMP v2c vs v3: monitoring coverage and security trade-offs

The choice between SNMPv2c and SNMPv3 is rarely about whether v3 is more secure. It is. The real question is what you give up operationally when you move to v3, what breaks during migration, and where v2c remains the pragmatic default because the cost of v3 exceeds the risk it mitigates on a given segment.

If you need the full network monitoring signal catalogue for context, see the network monitoring checklist.

SNMPv3 authentication failures: authorizationError and usmStats decoded

Mon, 01 Jan 0001 00:00:00 +0000

SNMPv3 authentication failures: authorizationError and usmStats decoded

SNMPv3 authentication failures surface as authorizationError (errorStatus 16, errorIndex 0) on the manager side, with no indication of which step of the User-based Security Model (USM) state machine failed. The manager reports “auth failed” but not whether the username was unknown, the HMAC digest mismatched, the packet arrived outside the time window, or the engine ID was never discovered.

The agent knows exactly what went wrong. Every SNMPv3 engine maintains six read-only Counter32 statistics under the usmStats subtree (RFC 3414). Each failed inbound packet increments exactly one counter before the agent responds with a Report PDU. Manager-side libraries translate that Report PDU into authorizationError, discarding the counter value that pinpoints the root cause.

Stale FDB/MAC tables: why endpoint location is wrong

Mon, 01 Jan 0001 00:00:00 +0000

Stale FDB/MAC tables: why endpoint location is wrong

Your topology platform says endpoint aa:bb:cc:dd:ee:ff is on switch port Gi1/0/24. Your security team sends someone to that port. The endpoint is not there. It moved hours ago, or it went offline, or it vMotioned to a different host. The FDB entry was stale and the platform presented it as current.

The Forwarding Database (FDB), also called the MAC address table or CAM table, maps MAC addresses to switch ports. Topology inference engines use FDB data, cross-referenced with ARP tables and CDP/LLDP neighbor data, to deduce where endpoints are physically connected. The inference is probabilistic. It degrades as input data freshness degrades.

STP topology-change storms: reconvergence cascades explained

Mon, 01 Jan 0001 00:00:00 +0000

STP topology-change storms: reconvergence cascades explained

A topology-change notification (TCN) is not itself a failure. STP generates one every time a non-edge port transitions up or down. That is normal during maintenance, link recovery, or device boot. The problem is what happens next. When TCNs fire repeatedly, or when a single TCN hits a large Layer 2 domain with thousands of MAC addresses, the protocol’s designed response becomes a self-inflicted traffic event.

Syslog parser backpressure: when one chatty device stalls the pipeline

Mon, 01 Jan 0001 00:00:00 +0000

Syslog parser backpressure: when one chatty device stalls the pipeline

A single device floods your syslog collector. The parser thread pool saturates, queues fill, and UDP datagrams start dropping at the kernel socket buffer. Critical messages from other devices, including BGP NOTIFICATIONS and hardware alarms, are silently lost. The dashboard shows a normal or slightly elevated syslog rate because dropped packets never reach the application layer.

The collector process is still running. The network is fine. The failure is inside the ingestion pipeline, at the seam between the kernel socket buffer and the parser, where backpressure builds and has nowhere to go.

Temperature, fan, and PSU monitoring: predicting hardware failure

Mon, 01 Jan 0001 00:00:00 +0000

Temperature, fan, and PSU monitoring: predicting hardware failure

Environmental sensors on network devices are the earliest leading indicators of hardware failure. Temperature trends, fan state changes, and PSU status transitions often precede field-replaceable unit failures by hours or days. The data is not hard to collect, but the MIB landscape is fragmented across vendors, thresholds vary by platform, and inherited polling templates frequently target deprecated OIDs. A template that worked on a Catalyst 3560 can silently return nothing on a Catalyst 8500.

Trap and syslog flood from link flaps: surviving the storm

Mon, 01 Jan 0001 00:00:00 +0000

Trap and syslog flood from link flaps: surviving the storm

A single bad SFP starts flapping. Within seconds, your trap receiver is processing hundreds of linkDown/linkUp pairs per second, your syslog pipeline is drowning in LINK-3-UPDOWN messages, and STP topology change notifications are cascading across the L2 domain. The kernel socket buffer on UDP 162 overflows, and the root-cause hardware alarm is as likely to be dropped as any other datagram in the flood.

Udp_RcvbufErrors: tuning kernel receive buffers for flow, trap, and syslog collectors

Mon, 01 Jan 0001 00:00:00 +0000

Udp_RcvbufErrors: tuning kernel receive buffers for flow, trap, and syslog collectors

Udp_RcvbufErrors is incrementing on your flow collector. Flow charts show traffic declining during what is actually a traffic spike. The kernel is receiving datagrams from exporters but the socket receive buffer is full, so it drops them silently. No application-level counter moves. No error log fires. The dashboards lie downward while the real traffic goes upward.

Flow collectors (NetFlow v5/v9, IPFIX, sFlow), SNMP trap receivers (UDP 162), and syslog receivers (UDP 514) all depend on UDP socket buffers. When the buffer overflows, the kernel increments Udp_RcvbufErrors in /proc/net/snmp and discards the datagram. The application never sees it.

Vendor API 429 throttling: Meraki, Cato, and PAN-OS rate limits

Mon, 01 Jan 0001 00:00:00 +0000

Vendor API 429 throttling: Meraki, Cato, and PAN-OS rate limits

Meraki, Cato, or PAN-OS API-polled devices go dark in your dashboard while ICMP and SNMP to the same devices return healthy responses. Every device sourced from the same vendor API flatlines at the same timestamp. The cause: your collector exhausted the vendor API rate-limit budget and is now receiving HTTP 429 instead of data.

This pattern is frequently misdiagnosed because the symptom (devices appearing “down”) sits two layers above the cause (rate limit exhausted). It surfaces most often during incidents, when teams tighten polling intervals for faster data, or silently when multiple tools share a single API key without coordination.

Vendor API latency and pagination: monitoring pull-mode collection

Mon, 01 Jan 0001 00:00:00 +0000

Vendor API latency and pagination: monitoring pull-mode collection

For SD-WAN controllers, cloud-managed networking, and modern firewall platforms, vendor API pull-mode collection is now the primary telemetry path. Operators depend on HTTPS calls to Meraki, Cato, PAN-OS, RESTCONF, and gRPC endpoints to learn tunnel state, license validity, session counts, and topology. Each API has its own authentication model, rate-limit budget, and pagination semantics. A collector that ignores these constraints will silently lose data, get throttled, or report healthy when the payload is empty.

Vendor API silent data gap: HTTP 200 with an empty payload

Mon, 01 Jan 0001 00:00:00 +0000

Vendor API silent data gap: HTTP 200 with an empty payload

Your SD-WAN controller dashboard shows flat lines. The Meraki organization API has not updated in twenty minutes. The PAN-OS firewall telemetry stopped at 03:00. Your collector logs show zero errors, every request returned HTTP 200, and no 5xx or timeout appears anywhere. But the data is gone.

The API endpoint is reachable, the TCP connection succeeds, the HTTP status code says OK, and the response body is empty, null, or contains an error wrapped inside a success envelope. Your collector accepted the response as valid because it checked the status code and nothing else. Many API adapters treat a 200 with an empty payload as “no data to report” rather than “the API is broken.” Charts go flat, but no error fires. If the API is your only telemetry source for an SD-WAN overlay or a cloud-managed firewall estate, you are blind without knowing it.