Network monitoring checklist: the signals every production network needs

This checklist covers the signals production networks need, organized by detection priority and mapped to maturity levels from survival to expert.

An NPM stack is a federation of collectors, parsers, enrichment services, storage tiers, and an analytics core. Most production incidents are not “the network broke” but “a collector’s UDP buffer dropped packets,” “the NetFlow v9 template cache went stale after a device reboot,” or “the polling worker pool fell behind and now a healthy device looks down.” The checklist is organized to surface those failure modes, not just the top-level symptoms.

The federation at a glance

When a signal is missing, stale, or wrong, the fault is usually one or two subsystems upstream of the dashboard. The typical NPM stack includes:

• Time synchronization substrate (NTP/PTP). Every cross-collector correlation depends on accurate, monotonic time across collectors, polled devices, and API endpoints.
• Polling transport. ICMP, UDP/161 (SNMP), TCP/22 (SSH/CLI scrape), and HTTPS (vendor APIs) reaching each managed endpoint.
• SNMP polling engine. A scheduler fanning OID requests across devices with timeouts, counter tables, and a device state machine (UP / STALE / UNKNOWN / DOWN).
• Flow collection subsystem. NetFlow v5/v9 and IPFIX collectors with template caches, sampling-rate awareness, and flow record storage. sFlow is sample-datagram oriented, not template-flow, and has a different failure profile.
• Topology inference engine. Fuses CDP/LLDP neighbor tables, FDB entries, ARP tables, STP state, and routing tables to derive Layer-2 and Layer-3 topology.
• BGP monitoring subsystem. Active or passive sessions tracking FSM state, prefix announcements, AS-path changes, and RPKI validity.
• Syslog and trap ingestion. UDP/TCP/TLS listeners with parser backpressure, facility/severity handling, and deduplication.
• Vendor API integration layer. Pull-mode clients for SD-WAN controllers, cloud platforms, and modern firewalls, each with their own auth, rate limit, and pagination semantics.
• Storage tiers. Counter TSDB (downsampled for long retention), full-resolution flow store, topology graph DB, raw syslog store, and event/alert log.

Signal domains by detection priority

The domains below are ordered by detection priority: the earliest surfacing of real issues with the best signal-to-noise comes first. Within each domain, the most operationally critical signals are listed first.

Availability

Errors

Saturation

Internal state, replication, and correctness

Latency, throughput, and security

Monitoring maturity levels

These levels are sequential and cumulative. Each level includes everything below it.

flowchart TD
    L4["L4 Expert
BMP, RPKI integrity, per-VRF,
sampling-rate forensics"] --> L3["L3 Mature
UDP drops, RSS, flow end-to-end loss,
NTP on devices, topology confidence"]
    L3 --> L2["L2 Operational
CPU and memory, traps, licenses,
topology discovery, API status"]
    L2 --> L1["L1 Survival
sysUpTime, ifOperStatus, BGP state,
utilization, errors, syslog, trap port"]

L1: survival

The absolute minimum to know if the network is alive and not on fire:

• SNMP reachability (sysUpTime GET) for every critical-path device
• Interface operational status (ifOperStatus) for critical interfaces
• BGP FSM state for critical eBGP and iBGP peers
• Interface utilization (ifHCInOctets / ifHCOutOctets vs ifHighSpeed) for top-10 interfaces
• Interface error counters (ifInErrors, ifOutErrors) for critical interfaces
• Syslog severity 0-3 (EMERG through ERR) forwarded from critical devices
• Flow collector port listening (UDP 2055 for NetFlow, 6343 for sFlow, 4739 for IPFIX)
• Trap receiver bound on UDP 162

A team at L1 catches hard outages. Nothing else.

L2: operational

Everything in L1, plus:

• All interfaces for status, utilization, errors
• All BGP peers for FSM state and prefix count
• SNMP poll latency and timeout rate per device
• Device control-plane CPU and memory
• Flow records received per second; syslog source count
• License days-to-expiry for all licensed features
• Temperature, fan, power supply state
• Topology discovery (CDP/LLDP)
• STP root bridge identity and topology change count
• Vendor API HTTP status for SD-WAN and cloud
• SNMP authentication failure rate
• ColdStart/warmStart detection with alerting

A team at L2 has visibility into most failures. They still miss silent failures, license cliffs, and topology staleness.

L3: mature

Everything in L2, plus:

• UDP socket buffer drops (Udp_RcvbufErrors) on flow, trap, and syslog collectors
• NIC RX/TX drops and RSS IRQ distribution
• Collector CPU (per-core, %soft) and disk space
• TSDB write queue depth and series cardinality
• Flow export-to-ingest latency and sampling rate consistency
• Flow exporter drop rate (device-side) and inbound-vs-exported comparison
• Interface counter discontinuity detection
• NAT/session table utilization
• Vendor API request latency, error rate, and rate-limit remaining
• Active path probes (IPSLA/TWAMP/HTTP) on critical paths
• Cross-collector time skew and NTP offset on monitored devices
• Topology view consistency and inference confidence score
• BGP route advertisement vs reception symmetry
• Poller poll cycle duration vs configured interval
• ARP cache entry count and staleness
• RPKI/ROA validation state for all BGP sessions
• BGP NOTIFICATION Cease subcode parsing (RFC 4486, RFC 8538, RFC 9384)
• Configuration drift detection
• Endpoint positioning orphan rate

A team at L3 catches most incidents in their early stages.

L4: expert

Everything in L3, plus the signals operators add after multiple major incidents:

• BMP (RFC 7854) for Adj-RIB-In visibility (pre-policy and post-policy routes). BGP4-MIB bgp4PathAttrTable only reflects best-path routes; Adj-RIB-In entries are not accessible over SNMP.
• BGP AS-path baseline deviation detection for own prefixes and upstreams
• RPKI validator health monitoring; alert on “Unknown” rate changes (signals validator outage)
• Sub-prefix hijack detection: alert when a more-specific appears without a less-specific in the RIB
• Smart License and vendor license server reachability monitored continuously
• Per-VRF and per-tenant isolation: BGP RIB size, flow volume, license utilization tracked per VRF
• Per-priority-queue discard counters (vendor QoS MIBs) revealing QoS queue saturation behind moderate utilization
• CoPP (control-plane policer) drop counters
• NIC per-queue drop counters via ethtool -S (rx_missed_errors, rx_no_dma_resources)
• /proc/net/softnet_stat for kernel packet processing backpressure
• FDB/ARP entry freshness (time since last refresh, computed from polling deltas)
• Flow template cache hit/miss ratio for NetFlow v9/IPFIX
• License grace-period state with feature-specific counter validation (IPS drops at 0 when traffic flows after license expiry)
• Asymmetric routing detection (forward vs reverse probe comparison)

The signals most teams miss

These are the systematic blind spots that keep causing incidents:

1. UDP socket buffer drops are not monitored. Udp_RcvbufErrors is the number one missed signal in flow collection. Charts show declining traffic during incidents that are actually traffic spikes. Production flow collectors need net.core.rmem_max raised to 16 MB or higher, tuned to actual ingress volume.

2. License expiry is monitored only when too late. A licensed feature (IPS, VPN, threat prevention) silently disables at midnight. The device stays up. The syslog message is low severity and buried. Users notice at 09:00.

3. NTP drift on monitored devices is not watched. Two devices 200ms apart on the same flap produce records that do not correlate. Postmortems fail to reconstruct events because timestamps are seconds apart.

4. BGP “Established but stale” is not detected. The FSM reports Established but UPDATE exchange stopped. Graceful Restart keeps the FSM green while the session is gone. Track bgpPeerInUpdates rate and the timestamp of last received prefix.

5. Trap receiver drops are invisible. During a trap flood, the highest-priority trap (root cause) is statistically the most likely to be dropped. There is no per-source drop counter.

6. Vendor API silent failures are not detected. HTTP 200 with empty payload is treated as “no data” rather than “API is broken.” PAN-OS returns <response status="error"> inside HTTP 200.

7. NetFlow v9/IPFIX template desync is invisible. After a device reboot or upgrade, templates arrive on a 5-30 minute interval. Until then, all data records are silently discarded.

8. Sampling rate normalization is skipped. sFlow analytics report raw counts without scaling. Bandwidth charts are wrong by the sampling factor (often 1:1000 or worse).

9. 32-bit counter rollover is treated as a real spike. ifInOctets wraps in approximately 3.4 seconds at 10G line rate. Naive differencing produces terabit spikes or negative utilization.

10. Poller fall-behind is not detected. The scheduler oversubscribes devices, retries compound, control-plane CPU spikes, and healthy devices appear “down.” The platform is the problem; the network is not.

How Netdata helps

Netdata collects many of the collector-side signals in this checklist that most NPM platforms miss:

• SNMP data collection polls sysUpTime, ifOperStatus, ifHCInOctets/ifHCOutOctets, ifInErrors/ifOutErrors, ifInDiscards/ifOutDiscards, and device CPU/memory with configurable intervals down to 1 second.
• Linux system plugins expose Udp_RcvbufErrors, NIC RX/TX drops from /proc/net/dev, per-core softirq from /proc/softirqs, and /proc/net/softnet_stat for kernel packet processing backpressure.
• Network interface metrics include per-NIC ring buffer drops and ethtool -S counters like rx_missed_errors, so you can distinguish NIC-level drops from socket-buffer drops.
• Cross-layer correlation lets you join rising Udp_RcvbufErrors with rising flow receive rate and rising collector CPU in a single view, which is the diagnostic chain for silent UDP flow loss.
• NTP metrics surface offset and drift on collectors and, where SNMP exposes hrSystemDate, on monitored devices.

Related guides

• Silent UDP flow data loss: why your NetFlow collector is dropping records