MongoDB pages evicted by application threads: when eviction becomes user latency

Query p99 latency doubles or triples, but iostat is not saturated and the slow query log shows no single offender. The signal is in db.serverStatus().wiredTiger.cache: pages evicted by application threads has moved from zero to a sustained nonzero rate.

This metric marks the moment when WiredTiger’s dedicated eviction workers fall behind and application threads are drafted to do the work. Any sustained nonzero rate is abnormal. Once application threads evict, they perform page reconciliation and disk I/O inline with the request handler thread, directly inflating user-visible latency. The companion counter pages selected for eviction unable to be evicted means eviction is stalled and the cache is effectively frozen.

What this means

WiredTiger maintains an in-memory cache separate from the OS page cache. Dedicated eviction threads keep cache usage near eviction_target (default 80% of configured cache size) by removing clean and dirty pages. When cache fill reaches eviction_trigger (default 95%) or the dirty ratio reaches eviction_dirty_trigger (default 20%), the engine aggressively demands more eviction throughput.

If background eviction threads cannot free pages fast enough, WiredTiger throttles client operations and forces application threads themselves to evict pages before proceeding. This is synchronous I/O on the request handler thread. The thread must reconcile the page - converting it to on-disk format - and write it out before continuing with the original query or write.

The transition is sharp. A cache at 90% fill with 2% dirty may be comfortable. A cache at 85% fill with 18% dirty can be catastrophic if checkpoints and eviction cannot flush data to disk quickly enough. The dirty ratio is often the stronger leading indicator because dirty pages must be reconciled and written, while clean pages can be discarded immediately.

flowchart TD
    A[Cache reaches eviction_trigger 95%] --> B[Background eviction falls behind]
    B --> C[Application threads evict pages]
    C --> D[Inline I/O adds latency to ops]
    D --> E[Tickets held longer]
    E --> F[Operations queue]
    F --> G[Connections retry and pile up]
    G --> H[Memory pressure increases]
    H --> B

Common causes

Quick checks

// Cache fill and dirty ratios
var c = db.serverStatus().wiredTiger.cache;
var max = c["maximum bytes configured"];
print("Fill: " + (100 * c["bytes currently in the cache"] / max).toFixed(1) + "%");
print("Dirty: " + (100 * c["tracked dirty bytes in the cache"] / max).toFixed(1) + "%");

// App-thread evictions and stall counters
print("App evictions: " + c["pages evicted by application threads"]);
print("Eviction stalls: " + c["pages selected for eviction unable to be evicted"]);

// Available tickets (MongoDB <=7.x)
var t = db.serverStatus().wiredTiger.concurrentTransactions;
print("Read tickets available: " + t.read.available);
print("Write tickets available: " + t.write.available);
// For MongoDB 8.0+, check db.serverStatus().queues.execution instead

// Long-running operations
db.currentOp({ active: true, secs_running: { $gt: 10 } }).inprog.forEach(function(op) {
  print(op.opid + " | " + op.secs_running + "s | " + op.ns);
});

// Journal sync latency
var log = db.serverStatus().wiredTiger.log;
var syncOps = log["log sync operations"];
print("Avg log sync (us): " + (syncOps > 0 ? (log["log sync time duration (usecs)"] / syncOps).toFixed(0) : "N/A"));

// Checkpoint duration
var txn = db.serverStatus().wiredTiger.transaction;
print("Last checkpoint (ms): " + txn["transaction checkpoint most recent time (msecs)"]);

// noTimeout cursors
print("noTimeout cursors: " + db.serverStatus().metrics.cursor.open.noTimeout);

# OS storage latency and utilization
iostat -x 1 3

How to diagnose it

1. Confirm app-thread eviction. Sample db.serverStatus().wiredTiger.cache twice over a 10-second window and confirm pages evicted by application threads is increasing. If pages selected for eviction unable to be evicted is also increasing, eviction is stalled.

2. Determine whether pressure is from overall fill or dirty pages. Calculate fill ratio and dirty ratio from bytes currently in the cache and tracked dirty bytes in the cache against maximum bytes configured. Dirty ratio above 15% points to write throughput overwhelming checkpointing. Fill ratio above 95% with low dirty ratio points to a working set that exceeds cache capacity.

3. Check for long-running snapshots. Run db.currentOp() filtering for active operations older than 60 seconds. Inspect db.serverStatus().metrics.cursor.open.noTimeout for cursors holding old snapshots open. Check db.serverStatus().transactions.currentActive and db.currentOp() for multi-document transactions pinning versions.

4. Check storage health. Run iostat -x 1 5 on the host and look for %util near 100, await above 50 ms, or avgqu-sz showing device saturation. Correlate with wiredTiger.log sync latency and wiredTiger.transaction checkpoint duration. If journal sync latency spikes 30-60 seconds before application latency, the storage layer is the bottleneck.

5. Check ticket availability. In MongoDB 7.x and earlier, inspect wiredTiger.concurrentTransactions. In MongoDB 8.0+, inspect queues.execution . If available tickets are near zero while app-thread evictions are rising, the system is in a self-reinforcing saturation loop: slow operations hold tickets longer, which queues new operations and increases memory pressure.

6. Check replication and flow control. If db.serverStatus().flowControl.timeAcquiringMicros is high or growing, the primary is throttling writes to protect secondaries. App-thread evictions under flow control indicate the cache cannot keep up even with throttled ingestion.

Metrics and signals to monitor

Fixes

Kill long-running snapshots and cursors

Identify the operation in db.currentOp(), confirm it is a read or a cursor with noTimeout: true, and terminate it with db.killOp(opid). This immediately releases the snapshot and allows eviction to proceed.

WARNING: db.killOp() is disruptive. Killing a write operation may leave data partially updated if the application does not handle it idempotently. Prioritize killing read-only operations and noTimeout cursors first.

Reduce write throughput

If the dirty ratio is climbing because of a bulk import, migration, or batch job, pause or throttle the workload. This is often the fastest way to stop the cascade and let checkpoints catch up.

Address storage I/O degradation

If iostat shows elevated await or depleted cloud burst credits, storage is the bottleneck. In a replica set, consider stepping down the primary to move writes to a member with healthier disks. Do not kill the checkpoint process; let it complete. Tradeoff: brief write unavailability during election.

Resize the WiredTiger cache

If the cache is clearly undersized for the working set and the host has available RAM, increase the WiredTiger cache allocation via storage.wiredTiger.engineConfig.cacheSizeGB and restart the node. In containers, verify the cache limit reflects the container memory boundary, not the host RAM, and leave 1-2 GB of headroom for connections and internal overhead. Tradeoff: a larger cache reduces eviction pressure but increases OOM risk if the working set continues to grow.

Tune eviction parameters only as a last resort

Adjusting eviction_dirty_target or eviction thread counts can help specific bulk-load patterns, but incorrect values can destabilize the engine. Test changes in staging. Do not tune during an active incident without a rollback plan.

Prevention

• Monitor the dirty ratio, not just fill ratio. A cache at 75% fill with 2% dirty is healthy. A cache at 70% fill with 18% dirty is not. Alert on dirty ratio above 10%.
• Monitor application-thread evictions. They should remain at zero. Any sustained rate indicates a configuration or capacity issue.
• Monitor ticket availability. Available tickets dropping below 25% of total is an early warning that operations are queuing.
• Set explicit cache limits in containerized deployments. Auto-sizing against host RAM causes premature eviction in Kubernetes and Docker.
• Avoid long-running transactions and noTimeout cursors. Both pin snapshots and prevent eviction of old versions. Set application query timeouts and review currentOp regularly.
• Trend checkpoint duration and journal sync latency. These are leading indicators that give 30-60 seconds of warning before application latency spikes. A checkpoint duration approaching the 60-second default interval signals that the storage layer is falling behind.

How Netdata helps

Netdata correlates pages evicted by application threads with cache dirty ratio, fill ratio, and checkpoint duration. It surfaces available WiredTiger tickets alongside opLatencies to show whether latency spikes correlate with ticket exhaustion or cache pressure. It tracks journal sync latency as a leading indicator of storage degradation, and monitors currentOp longest-running operation age to catch snapshot holders before they trigger eviction stalls. Alerts on sustained nonzero application-thread eviction rates include correlation to noTimeout cursor counts and active transaction metrics.

Related guides

• How MongoDB actually works in production: a mental model for operators
• MongoDB WiredTiger cache pressure cascade: eviction stalls and latency spikes
• MongoDB monitoring checklist: the signals every production cluster needs
• MongoDB monitoring maturity model: from survival to expert