MongoDB Operations Guides on Netdata

How MongoDB actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How MongoDB actually works in production: a mental model for operators

A write latency spike in MongoDB is rarely a query problem. It is more often a storage journal stall, cache eviction cascade, ticket exhaustion, or replication flow-control throttle. Operators who treat MongoDB as a monolithic document database often tune indexes while the real bottleneck is a WiredTiger checkpoint falling behind.

This guide is a runtime mental model for the subsystems that matter in production: the WiredTiger storage engine, replication state machine, thread-per-connection execution model, and sharding coordination layer. Understanding how these compete for memory, disk, tickets, and connections lets you read signals correctly instead of guessing.

MongoDB and Transparent Huge Pages: why THP must be disabled

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB and Transparent Huge Pages: why THP must be disabled

If you see a mongod startup warning about Transparent Huge Pages, or you are chasing tail latency spikes and memory fragmentation that do not correlate with cache pressure, ticket exhaustion, or slow queries, the cause may be THP. The standard advice has always been to disable it, but that is now version-dependent. MongoDB 8.0 upgraded TCMalloc to use per-CPU caches, which changes the recommendation for most x86_64 Linux deployments. For MongoDB 7.0 and earlier, disabling THP is still correct. For 8.0 on x86_64 Linux, THP should be enabled. Running the wrong configuration silently degrades throughput and increases latency.

MongoDB Authentication failed: credential rotation, brute force, and the log signal

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB Authentication failed: credential rotation, brute force, and the log signal

A spike of Authentication failed entries in the MongoDB log is abnormal: in a healthy deployment the baseline is near zero. The signal usually means one of four things: a brute-force attack, an application presenting stale credentials after rotation, an expired x.509 certificate, or an upstream LDAP or Kerberos directory issue. Distinguishing between these quickly determines whether you are facing a security incident or an impending application outage.

MongoDB balancer stuck and jumbo chunks: permanent imbalance and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB balancer stuck and jumbo chunks: permanent imbalance and how to fix it

One shard is hot while the others idle. sh.status() shows chunk counts skewed more than 20%. The balancer is either stopped or running without closing the gap. Until you fix the root cause, the imbalance persists.

Two failure modes cause this. The balancer itself can be disabled, restricted to a narrow window, or blocked by an unhealthy config server. Or the cluster has jumbo chunks: ranges that exceed the configured chunkSize but cannot split because too many documents share the exact same shard key value. MongoDB marks those chunks jumbo in config.chunks and the balancer skips them. That heavy chunk pins load and storage on a single shard and creates a floor on how balanced the cluster can become.

MongoDB cache too small: sizing the WiredTiger cache for your working set

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB cache too small: sizing the WiredTiger cache for your working set

When MongoDB latency doubles and disk read IOPS climb, operators usually check indexes and the query planner first. If db.currentOp() shows no runaway query and the slow query log is quiet, the culprit is often the WiredTiger cache.

WiredTiger maintains its own in-memory cache of uncompressed B-tree pages, separate from the OS page cache. MongoDB defaults the cache to max(0.5 * (RAM - 1 GB), 256 MB). That default works for a single mongod on a dedicated host, but it breaks down in containers, multi-tenant deployments, and during organic data growth. Once the working set exceeds the cache, reads fault to disk, pages are decompressed, and eviction threads compete with application threads for CPU.

MongoDB checkpoint duration climbing: diagnosing slow WiredTiger checkpoints

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB checkpoint duration climbing: diagnosing slow WiredTiger checkpoints

You notice transaction checkpoint most recent time (msecs) climbing past 10 seconds, then 30, then 50. It is trending upward, check after check, approaching the 60-second default checkpoint interval. When checkpoint duration meets or exceeds the interval, WiredTiger has no margin left. The next checkpoint starts late, dirty pages accumulate faster than they flush, and the journal can fill to the point where all new writes block until the checkpoint finishes. This is a common production failure mode that starts as a slow climb and ends as a write freeze.

MongoDB checkpoint stall write freeze: when all writes stop with no error

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB checkpoint stall write freeze: when all writes stop with no error

Writes time out or hang while mongod is running, TCP port 27017 is open, and reads still return results from cache. The MongoDB logs are quiet, but db.serverStatus().opcounters shows write counts frozen. This is a WiredTiger checkpoint stall: the checkpoint process fell behind, dirty pages accumulated, and new writes blocked. The freeze lasts until the current checkpoint completes. If the I/O bottleneck remains, queued writes flood through and the next checkpoint stalls again.

MongoDB chunk migration storms: moveChunk I/O pressure and range locks

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB chunk migration storms: moveChunk I/O pressure and range locks

Write latency spikes across multiple shards simultaneously. Queries time out while mongos logs show no election events. Donor shard primaries report growing queue depths, and config.changelog shows a wall of moveChunk.error entries interleaved with retries. The balancer is hammering the cluster instead of helping.

A single moveChunk operation is expensive. It copies documents to the recipient, enters a critical section with a shared range lock on the donor, then deletes orphaned documents. When the balancer triggers many migrations in quick succession, or individual migrations fail and retry in a tight loop, overlapping I/O pressure and lock contention compound into a cluster-wide latency event.

MongoDB connection churn: high totalCreated rate and thread creation overhead

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB connection churn: high totalCreated rate and thread creation overhead

db.serverStatus().connections can show low current and a rapidly climbing totalCreated. That mismatch is connection churn: connections open and close rapidly instead of being reused. MongoDB uses a thread-per-connection model, so each cycle costs roughly a megabyte of thread stack, scheduling overhead, and file descriptor work. The result is rising RSS, CPU contention, and latency spikes that do not correlate with the active connection count.

MongoDB connection refused at maxIncomingConnections: hitting the connection ceiling

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB connection refused at maxIncomingConnections: hitting the connection ceiling

Application logs show connection timeouts. MongoDB logs show connection refused or error accepting new connection. db.serverStatus().connections shows current well below the configured maximum. This disconnect means you are hitting a hard ceiling at the TCP accept layer, not experiencing gradual degradation.

MongoDB uses a one-thread-per-connection model. Each accepted connection consumes at least one file descriptor and a thread stack sized by the OS ulimit -s. While maxIncomingConnections sets the logical inbound cap, the OS file-descriptor limit (ulimit -n) usually enforces the actual ceiling. Rejections happen before the connection handshake completes, so serverStatus().connections.current never counts refused connections. Look at logs, ratios, and OS-level resource counts to find the real limit.

MongoDB connection storm spiral: reconnection floods after an election or deploy

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB connection storm spiral: reconnection floods after an election or deploy

Connection count on a primary jumps from 200 to 4,000 in under a minute. Resident memory climbs, query latencies double, and application logs fill with timeout errors. The slow query log shows nothing unusual. Individual queries are not the problem. The database is drowning in threads.

This is a connection storm spiral. A trigger event, usually a replica set election, application deploy, or network blip, invalidates existing connections across your application fleet. Every driver reconnects at once. Each new connection costs MongoDB a dedicated thread and roughly 1 MB of stack memory . The resulting RSS spike and ticket contention slow down operations already in flight, causing more timeouts, which drives even more reconnections. The feedback loop ends in OOM kill or unresponsiveness.

MongoDB disk full: emergency recovery when mongod can't write the journal

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB disk full: emergency recovery when mongod can’t write the journal

When the filesystem backing the data or journal directory crosses a critical threshold, WiredTiger cannot allocate new journal extents. If mongod crashes or restarts, recovery replays journal files since the last checkpoint and requires free headroom to create or extend files during that replay. On a full disk, mongod hangs in recovery without binding to port 27017.

If the node is a standalone, there is no replica to fail over to. If it is a secondary, cluster redundancy is reduced while the member is down. Recovery is complicated by a counterintuitive storage engine behavior: WiredTiger reclaims space internally after deletes, but does not automatically shrink data files or return bytes to the operating system. A volume that reads 99% full after a massive delete remains 99% full at the filesystem level.

MongoDB disk I/O saturation: correlating iostat with WiredTiger signals

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB disk I/O saturation: correlating iostat with WiredTiger signals

When opLatencies.writes climbs and globalLock.currentQueue grows, db.serverStatus().wiredTiger.transaction often shows the most recent checkpoint took 45 seconds. WiredTiger metrics tell you what is hurting, but they do not tell you why. The next question is whether the disk is actually saturated.

Disk I/O saturation surfaces as climbing journal sync latency, checkpoint duration exceeding the 60-second interval, application-thread evictions, and ticket exhaustion. The only way to separate a storage problem from a query problem is to correlate OS-level disk signals (iostat -x) with WiredTiger internal signals in the same time window. This guide shows how to do that safely during an incident.

MongoDB exceeded memory limit for $group — aggregation spills and allowDiskUse

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB exceeded memory limit for $group — aggregation spills and allowDiskUse

Application logs show error 16945, or an aggregation pipeline slows by an order of magnitude. In MongoDB, every aggregation stage not backed by an index is limited to 100 megabytes of RAM. When a stage exceeds this limit and disk spilling is not enabled, the operation fails immediately. If spilling is enabled, MongoDB writes temporary files to disk, which keeps the pipeline alive but adds unpredictable latency and extra I/O load.

MongoDB exposed to the internet without authentication: bindIp and the breach scenario

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB exposed to the internet without authentication: bindIp and the breach scenario

A mongod process bound to all interfaces with authentication disabled exposes every database to any host that can reach port 27017. If you are responding to a scan, PAGE, or audit, confirm the exposure, measure the blast radius, and eliminate the surface.

What this means

MongoDB’s net.bindIp controls which interfaces accept connections. Modern packages default bindIp to 127.0.0.1; exposure usually follows an explicit override to a wildcard such as 0.0.0.0 or ::. Without authentication, any reachable host can list databases, read or write documents, and execute administrative commands.

MongoDB flow control throttling writes: when the primary slows itself down

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB flow control throttling writes: when the primary slows itself down

Write throughput on the primary has dropped by 30% or more. Application logs show intermittent write latency spikes, but the primary’s CPU, memory, and disk metrics look healthy. There are no elections, cache pressure warnings, or obvious errors. Check replication: secondaries are lagging. The primary is not sick; it is throttling itself.

MongoDB 4.2 introduced flow control, a ticket-based admission mechanism that caps the primary’s write rate to keep secondaries from falling off the oplog. When isLagged is true, the primary artificially limits throughput. The fix is rarely on the primary. Look at the replication pipeline: a slow secondary, an oplog window that is too small, or a topology change that distorts majority commit lag.

MongoDB globalLock.currentQueue growing: operations queuing for the storage engine

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB globalLock.currentQueue growing: operations queuing for the storage engine

globalLock.currentQueue.total is climbing and not stabilizing after a burst. Under WiredTiger, this metric reflects waits at the intent lock and ticket admission layers, not a single global mutex. The engine is receiving work faster than it can drain it.

If the total queue sustains above 20 and grows, the system is saturated. Queued operations become latency spikes, then connection pileups as applications retry, then a degradation spiral. The root cause is usually ticket exhaustion, a single hot collection, a long-running operation, or storage I/O degradation inflating ticket hold times.

MongoDB journal sync latency high: the storage signal that warns 60 seconds early

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB journal sync latency high: the storage signal that warns 60 seconds early

Application write latency spikes. Connections pile up. Look back 60 seconds and WiredTiger journal sync latency was likely already climbing. Every write with j:true or w:"majority" blocks until the journal buffer is fsynced to disk. When storage struggles, journal sync is the first domino to fall.

Journal sync latency is a storage subsystem signal, not a query or cache problem. The block device under mongod cannot absorb small sequential writes fast enough. The result is head-of-line delay for all durable writes, which cascades into ticket exhaustion and connection backlog.

MongoDB lock wait times: collection and metadata lock contention during DDL

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB lock wait times: collection and metadata lock contention during DDL

When p99 latency jumps and globalLock.currentQueue grows, check serverStatus().locks. If timeAcquiringMicros is climbing for Collection or Metadata, the cause is almost always DDL: createIndexes, dropIndexes, collMod, renameCollection, or similar commands that acquire exclusive collection, database, or metadata locks. WiredTiger uses document-level concurrency for ordinary reads and writes, so normal CRUD rarely blocks. A single schema change can serialize operations on a hot collection or across a database during peak traffic.

MongoDB long-running operations: finding and killing the query holding a ticket

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB long-running operations: finding and killing the query holding a ticket

Your application latency just spiked. opLatencies show reads and writes climbing. globalLock.currentQueue is no longer zero. You check db.serverStatus().wiredTiger.concurrentTransactions: available tickets are near zero, but throughput has not increased. An operation is holding a ticket without making progress.

A collection scan, an unbounded aggregation, or a stalled write can hold a WiredTiger read or write ticket for minutes. The default is 128 read and 128 write tickets in most versions, so one long-running operation can cascade into system-wide queuing, connection pileup, and application timeouts. Find it and kill it, but killing the wrong operation can crash a node or leave data inconsistent.

MongoDB long-running transactions: pinned snapshots and silent cache pressure

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB long-running transactions: pinned snapshots and silent cache pressure

MongoDB latency climbs while write throughput looks normal and the working set is unchanged. Check WiredTiger cache: the dirty ratio is rising and application threads have started evicting pages. The cause is often invisible in standard cache metrics – a multi-document transaction that opened a snapshot and never let it go.

Multi-document transactions pin WiredTiger snapshots until commit or abort. That snapshot prevents the storage engine from evicting old document versions. A single transaction left open for minutes can silently pin enough pages to push the cache into eviction pressure. An inactive-but-open transaction is worse than an active one because it holds the snapshot indefinitely with no forward progress.

MongoDB monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB monitoring checklist: the signals every production cluster needs

Production MongoDB failures are preceded by signals that are visible but often unmonitored: climbing dirty cache ratio, shrinking oplog window, or ticket counts approaching zero. This guide organizes essential signals into four monitoring levels. Use them to audit instrumentation or triage gaps during an incident.

Each level builds on the previous one. If you are missing a survival signal, instrument it before adding expert metrics. The thresholds below are drawn from the MongoDB serverStatus() and rs.status() contract and from operational patterns observed across WiredTiger deployments.

MongoDB monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB monitoring maturity model: from survival to expert

If you run MongoDB in production, you need a progression: what to watch first to know the database is alive, what to add next to know why it is slowing down, and what to track finally to predict a cascade before it becomes an outage.

Treat the levels below as gates, not a shopping list. Automate and alert on Level 1 before you build Level 2 dashboards. If you are instrumenting Level 4 but lack a reliable page for disk space or member state, you have inverted your priorities.

MongoDB no primary / election storm: repeated elections and write outages

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB no primary / election storm: repeated elections and write outages

Applications log “not primary” errors. rs.status() shows a different PRIMARY than thirty seconds ago. MongoDB logs repeat "Starting an election" and "Stepping down". Each election costs 2-12 seconds of write unavailability. More than two in ten minutes is an election storm.

This pattern is more dangerous than a single failover because it creates rolling write outages that do not self-stabilize. Drivers reconnect, retry buffers fill, and application latency degrades even when a primary exists. Root causes usually fall into three categories: the primary is too slow to answer heartbeats, the network is dropping or delaying packets between members, or a misconfigured priority is forcing a healthy primary to step down.

MongoDB not master error: writes hitting a non-primary node after failover

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB not master error: writes hitting a non-primary node after failover

A node restart, network partition, or planned stepdown triggers a MongoDB election. Seconds later, application logs show NotWritablePrimary (code 10107) or the legacy string not master and slaveOk=false. Writes fail against a node that used to be PRIMARY, even though the cluster has elected a new one.

This guide covers how to find the root cause and stop it from recurring.

MongoDB not primary and secondaryOk=false: reading from a secondary and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB not primary and secondaryOk=false: reading from a secondary and how to fix it

Your application logs show NotPrimaryNoSecondaryOk (code 13435) with the message "not master and slaveOk=false". Metrics show read failures against a specific host. The mongod process is running, replica set heartbeats are clean, and replication lag looks normal. The cluster is not down. The error is a routing decision: a client sent a read to a replica set member that is not the primary, without declaring that reading from a non-primary is acceptable.

MongoDB noTimeout cursors causing cache pressure: pinned snapshots and silent eviction stalls

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB noTimeout cursors causing cache pressure: pinned snapshots and silent eviction stalls

When wiredTiger.cache.bytes currently in the cache climbs, the dirty ratio trends toward 20%, and read latencies spike without a single slow query in the log, check metrics.cursor.open.noTimeout.

Each noTimeout cursor pins a WiredTiger snapshot indefinitely. Old document versions cannot be evicted while that snapshot is open, so the cache fills with unreachable history until background eviction falls behind and application threads are forced to clean up. The result is a silent cache pressure cascade that looks like a capacity problem but is actually a cursor lifecycle problem.

MongoDB OOM-killed by the kernel: RSS, cache sizing, and oom_score_adj

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB OOM-killed by the kernel: RSS, cache sizing, and oom_score_adj

You find mongod gone. The replica set has no primary. Applications time out. MongoDB logs show no graceful shutdown. Instead, dmesg shows Out of memory: Killed process 12345 (mongod). The Linux OOM killer has reaped the process. MongoDB is a frequent target because its resident set size is usually the largest on the host.

An OOM kill is not a MongoDB bug. It is the kernel freeing RAM by terminating the highest-scoring process. mongod’s RSS is dominated by the WiredTiger cache, plus roughly 1 MB per connection, plus roughly 500 MB to 1 GB of internal overhead for indexes, session buffers, and stack. When that sum comes within 1 GB of total RAM, the node is in the danger zone. The kill is abrupt: no stepdown, no replica set coordination, and after restart the cache must warm again.

MongoDB operation exceeded time limit (MaxTimeMSExpired): maxTimeMS and killed operations

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB operation exceeded time limit (MaxTimeMSExpired): maxTimeMS and killed operations

Error code 50, MaxTimeMSExpired, means the server killed an operation that exceeded its processing budget. Raising the timeout without fixing the root cause turns acute failures into chronic resource exhaustion. The operation was already pathologically slow; maxTimeMS ended it before it consumed more resources or held locks and tickets indefinitely.

maxTimeMS sets a cumulative processing budget in milliseconds. MongoDB enforces it using the same interrupt mechanism as killOp, terminating the operation only at designated interrupt points. Idle time between cursor batches does not count toward the limit, and on direct connections network latency is excluded from the server-side clock. On sharded clusters, however, latency between mongos and shard mongod instances counts against the limit. Distinguish a true MaxTimeMSExpired from a client-side socket timeout, where the client gives up before the server responds.

MongoDB oplog window collapse: secondaries falling off and forced full resync

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB oplog window collapse: secondaries falling off and forced full resync

A secondary transitions to RECOVERING and logs “too stale to catch up.” The oplog window compresses from 48 hours to 90 minutes while replication lag on one secondary climbs steadily. These are the signatures of oplog window collapse: a write surge turns over the oplog faster than secondaries can consume it, and the safety margin between window and lag evaporates.

MongoDB oplog window too small: sizing the oplog for your write volume

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB oplog window too small: sizing the oplog for your write volume

The oplog window is the only thing standing between a routine secondary restart and a multi-hour full initial sync. It is a fixed-size capped collection that stores a variable amount of history. As your write volume grows, the window compresses. Most teams size the oplog once during initial deployment and never look at it again. Six months later, a routine maintenance window turns into an incident because the secondary fell off the oplog, entered RECOVERING, and forced a resync that saturated the remaining nodes.

MongoDB page faults high: working set exceeding memory after warmup

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB page faults high: working set exceeding memory after warmup

Hard page faults long after startup mean the active data set exceeds resident memory. On Linux, extra_info.page_faults counts major faults: the OS read data from disk because the page was missing from both the WiredTiger cache and the OS page cache. A brief spike after restart is normal during warmup, but sustained faults mean the working set does not fit. On EBS gp3, 50 faults per second can degrade latency. On NVMe, hundreds per second may be tolerable, but neither is free. Confirm the cause, distinguish warmup from pressure, and reduce the fault rate without guessing.

MongoDB pages evicted by application threads: when eviction becomes user latency

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB pages evicted by application threads: when eviction becomes user latency

Query p99 latency doubles or triples, but iostat is not saturated and the slow query log shows no single offender. The signal is in db.serverStatus().wiredTiger.cache: pages evicted by application threads has moved from zero to a sustained nonzero rate.

This metric marks the moment when WiredTiger’s dedicated eviction workers fall behind and application threads are drafted to do the work. Any sustained nonzero rate is abnormal. Once application threads evict, they perform page reconciliation and disk I/O inline with the request handler thread, directly inflating user-visible latency. The companion counter pages selected for eviction unable to be evicted means eviction is stalled and the cache is effectively frozen.

MongoDB replica set member unhealthy: reading rs.status() states

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB replica set member unhealthy: reading rs.status() states

rs.status() output is perspective-dependent and easy to misread. A member can show health: 1 while RECOVERING and unable to serve reads, or appear UNKNOWN from one node yet SECONDARY from another because of an asymmetric firewall rule. health: 0 alone does not mean the process is dead, and health: 1 alone does not mean the node is healthy.

This guide maps stateStr values to failure modes and shows how to distinguish transient startup states, replication lag spirals, and network partitions.

MongoDB replication lag: detection, diagnosis, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB replication lag: detection, diagnosis, and fixes

Replication lag is the delay between the primary’s latest oplog entry and a secondary’s last applied entry. When lag grows faster than the oplog window, the secondary cannot catch up and requires a full initial sync, which can take hours to days. Until that happens, every second of lag erodes failover safety and read consistency.

What this means

MongoDB replicates by having secondaries tail the primary’s oplog, a capped collection in local.oplog.rs. The primary records every write with a timestamp; the secondary fetches entries and applies them locally. Lag is the delta between the primary’s optimeDate and the secondary’s optimeDate from rs.status().

MongoDB rollback after failover: silent data loss and the rollback directory

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB rollback after failover: silent data loss and the rollback directory

A replica set member in ROLLBACK state, or an application reporting vanished documents after failover, means a former primary held writes that never reached a majority. When that node rejoins, MongoDB erases the divergent history and writes the removed data to files under <dbPath>/rollback/. The application may have received acknowledgment for those writes. With w:1, acknowledgment meant only that the primary applied the write. It did not guarantee replication to a majority or survival through failover. That is silent data loss.

MongoDB RSS growing without cache growth: leaks, threads, and tcmalloc fragmentation

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB RSS growing without cache growth: leaks, threads, and tcmalloc fragmentation

db.serverStatus().mem.resident climbs while WiredTiger cache utilization stays flat and the host is not swapping. Virtual memory is larger than RSS by design and is not an alert target. Only RSS reflects physical memory pressure. When RSS grows without cache growth, the problem lives outside the storage engine.

This pattern points to one of three areas: tcmalloc heap retention and fragmentation, per-connection thread stack accumulation, or unbounded internal allocations from cursors, plan caches, or aggregation pipelines. Each connection reserves roughly 1 MB of stack space, so a connection storm can add gigabytes of RSS in minutes. TCMalloc caches freed memory in per-thread or per-CPU arenas, which inflates RSS independently of the WiredTiger cache.

MongoDB scanned objects ratio high: documents examined vs returned and wasted work

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB scanned objects ratio high: documents examined vs returned and wasted work

A high scanned-objects-to-returned-documents ratio means MongoDB examines far more documents than it returns. The alert usually fires as an Atlas query targeting event or a chart showing metrics.queryExecutor.scannedObjects climbing faster than metrics.document.returned. A ratio near 1:1 is efficient. A ratio above 1000:1 means the database examines roughly a thousand documents for every one it returns. Wasted work consumes read tickets, pushes unnecessary data through the WiredTiger cache, and often precedes latency spikes or ticket exhaustion.

MongoDB scatter-gather queries: when mongos fans out to every shard

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB scatter-gather queries: when mongos fans out to every shard

In a sharded MongoDB cluster, mongos reads chunk-to-shard mappings from the config servers and routes each query to the smallest set of shards that can satisfy it. When the predicate includes the shard key, mongos targets one shard or a bounded subset. When it does not, mongos broadcasts the query to every shard that owns chunks for the collection and merges the results. This is a scatter-gather query.

MongoDB sharding hot shard: one shard saturating while others idle

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB sharding hot shard: one shard saturating while others idle

Application latency climbs while aggregate CPU, I/O, and connection counts look healthy. Drill into individual shards and one node is pinned near 95% utilization while peers sit near 20%. sh.status() shows an even chunk distribution. You have a hot shard.

A hot shard occurs when one member of a sharded cluster receives a disproportionate share of read or write traffic. The cause is usually an access-pattern mismatch against the shard key, not a chunk-count imbalance. The overloaded shard saturates its WiredTiger cache and exhausts read or write tickets, pushing operations into queues. Because other shards are idle, aggregate cluster metrics hide the problem until the hot shard cascades into a latency spike or availability event.

MongoDB silent index regression: when a dropped index quietly becomes a collection scan

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB silent index regression: when a dropped index quietly becomes a collection scan

Read latency on the primary doubles while connection counts and write throughput stay flat. There are no election events or cache pressure alerts. Traffic is unchanged. Yet p99 read latency climbs until operations time out.

The slow query log shows queries that used to finish in milliseconds now taking seconds. The plans show COLLSCAN. An index that existed last week is gone, or the query planner switched to a less efficient index after a cache invalidation. Because queries still return correct results, the regression is silent until it becomes an outage.

MongoDB slow query COLLSCAN: collection scans and the missing index

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB slow query COLLSCAN: collection scans and the missing index

Queries that used to return in tens of milliseconds now breach application timeouts. Read latency climbs while write throughput stays flat. In the MongoDB slow query log, you see planSummary: "COLLSCAN" attached to operations that should be indexed. A collection scan reads documents that will never be returned, wastes disk I/O, floods the WiredTiger cache with irrelevant data, and holds read tickets until the whole instance cascades into cache pressure.

MongoDB storage not reclaimed after delete: WiredTiger, compact, and resync

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB storage not reclaimed after delete: WiredTiger, compact, and resync

You deleted a large portion of a collection, and db.collection.stats() confirms the logical size dropped. df -h on the data volume does not. The filesystem blocks were not returned. WiredTiger maintains internal free-space lists and reclaims pages for future writes rather than shrinking files. Dropping a database or collection removes the underlying files and returns space to the OS immediately; this article addresses the case where documents are deleted but the collection files remain. This guide explains how to verify the state and the two operational paths to return blocks to the OS: the compact command and replica set member resync.

MongoDB swapping: why mongod must never swap and how to tune the OS

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB swapping: why mongod must never swap and how to tune the OS

Application timeouts climb. MongoDB latency jumps from milliseconds to seconds or minutes, yet mongod is still running and accepting connections. CPU is low, disk I/O is not saturated, and the MongoDB log shows no errors. The process has not crashed. It has entered swap death. When the Linux kernel evicts mongod pages to swap, the database continues to function at roughly 1/1000th of normal speed. MongoDB relies on the WiredTiger cache and OS page cache to remain resident in RAM.

MongoDB ticket exhaustion: WiredTiger read/write tickets and queued operations

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB ticket exhaustion: WiredTiger read/write tickets and queued operations

Your application times out while the OS shows idle CPU and disk utilisation looks survivable. The MongoDB log shows no obvious errors, yet operations stall. The likely cause is WiredTiger ticket exhaustion: the storage engine has run out of read or write concurrency tokens, and new work queues behind slow operations. Confirm ticket starvation, find the root cause, and fix it without raising the ticket limit.

MongoDB TLS certificate expiry: rotating certs without dropping connections

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB TLS certificate expiry: rotating certs without dropping connections

An expired TLS certificate does not degrade gracefully. In MongoDB, client drivers, replica set members, and mongos routers validate certificates during every TLS handshake. When the server certificate’s notAfter date passes, OpenSSL rejects the handshake immediately. Existing connections persist until they close, but nothing reconnects once they do. The result is a sudden cascade: secondaries miss heartbeats and transition to DOWN, application connection pools exhaust, and monitoring probes fail. During an active incident, check certificate dates first. For preventive work, your rotation strategy depends on whether you run MongoDB 5.0 or later.

MongoDB Too many open files: file descriptor exhaustion and ulimit tuning

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB Too many open files: file descriptor exhaustion and ulimit tuning

Connection timeouts appear in application logs. MongoDB logs show Too many open files or error accepting new connection. New secondaries fail to sync, or a stable node rejects connections after restart. The mongod process hit the OS file descriptor limit. The failure is often silent until a dependent system breaks.

FD exhaustion is not always about connection count. WiredTiger maintains open file descriptors for data files, indexes, and journals. A dense deployment with thousands of collections can hold tens of thousands of FDs in steady state. A connection surge, index build, or restarted node rebuilding caches can push the process over the limit. MongoDB then cannot accept new connections, open new data files, or continue replication.

MongoDB too stale to catch up: secondary stuck in RECOVERING and how to resync

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB too stale to catch up: secondary stuck in RECOVERING and how to resync

You check rs.status() during an incident and see a member stuck in RECOVERING with an errmsg reading error RS102 too stale to catch up. The node is alive but will never transition back to SECONDARY on its own. Its last replicated oplog entry is older than the oldest entry still available on the primary, so the history it needs has already been overwritten. Incremental replication is impossible from this state. The only path forward is a full initial sync, which on large datasets can take hours to days, adds significant read load to the sync source, and leaves the cluster with reduced redundancy until it completes. If the stale member is a voting node, you are now one failure away from losing majority. This guide covers how to confirm the condition, identify why the secondary fell off, and recover without pushing the remaining cluster members into the same trap.

MongoDB unused indexes: $indexStats, write amplification, and safe removal

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB unused indexes: $indexStats, write amplification, and safe removal

Every insert and delete updates all indexes on a MongoDB collection; every update that modifies an indexed field updates every index containing that field. An unused index consumes WiredTiger cache space, increases write latency, and amplifies disk I/O. Over time, redundant indexes accumulate from schema migrations, abandoned query patterns, and exploratory tuning.

The $indexStats aggregation stage exposes per-index operation counts, but counters reset on mongod restart and internal operations can inflate values. This guide covers detecting truly unused indexes, validating redundancy, and removing them without causing query regressions.

MongoDB w:majority write concern timeout (wtimeout): replication lag and at-risk writes

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB w:majority write concern timeout (wtimeout): replication lag and at-risk writes

Your application logs show write concern timeouts, or worse, the driver returned an error that the application swallowed and moved on. The write succeeded on the primary, but the cluster could not confirm it across a majority of data-bearing members before the deadline expired. If the primary crashes right now, that data is gone. This article explains how to diagnose the root cause, whether it is replication lag, a down secondary, or network saturation, and what to do before the next primary failure.

MongoDB WiredTiger cache dirty ratio high: the leading indicator nobody watches

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB WiredTiger cache dirty ratio high: the leading indicator nobody watches

Cache fill at 70% looks safe, but if dirty ratio is climbing past 15%, a latency spike is already forming. Dirty ratio measures modified pages not yet flushed to disk. While fill ratio tells you how much cache is in use, dirty ratio tells you how fast the storage engine is falling behind. It often leads checkpoint stalls and eviction-driven latency spikes by minutes.

MongoDB WiredTiger cache pressure cascade: eviction stalls and latency spikes

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB WiredTiger cache pressure cascade: eviction stalls and latency spikes

Latency jumps from milliseconds to seconds for both reads and writes. The slow query log shows no single offender, but connection count climbs as clients retry and timeout. This is the cache pressure cascade. It starts in the storage engine and becomes a self-reinforcing spiral through replication, admission control, and connection handling. This guide covers the mechanism, confirmation under pressure, and how to stop it.

MongoDB WriteConflict errors: optimistic concurrency retries under contention

Mon, 01 Jan 0001 00:00:00 +0000

MongoDB WriteConflict errors: optimistic concurrency retries under contention

WriteConflict exceptions (error 112) in application logs, or unexplained write latency spikes, point to document-level contention under WiredTiger optimistic concurrency control. Outside of transactions, MongoDB retries single-document writes internally; the client sees slower responses rather than errors. Inside multi-document transactions, MongoDB aborts immediately and returns error 112. Either way, the root cause is concurrent writers targeting the same document.

What this means

WiredTiger uses optimistic concurrency control at the document level. Two concurrent writes to the same document do not block indefinitely; one proceeds and the other encounters a conflict.