$ guides / activemq / activemq-store-write-latency ▌

Operations Guides

ActiveMQ KahaDB journal write latency: fsync as the persistent-throughput ceiling

A producer is sending persistent messages to ActiveMQ Classic and throughput is far below what the network, CPU, and broker configuration suggest should be possible. Sends are not failing. There are no exceptions. Each send() just takes a few milliseconds, or tens of milliseconds, and the aggregate rate caps out no matter how many producers you add.

This is almost always the KahaDB journal fsync. For persistent messages, ActiveMQ Classic writes the message to the KahaDB write-ahead journal and acknowledges the producer only after the fsync completes. That fsync is the critical path for every persistent send, and its latency sets a hard ceiling on persistent throughput. When the underlying storage gets slow, persistent messaging gets slower in direct proportion to the fsync latency.

There is no JMX metric for this. The broker does not tell you its journal writes are slow. You have to infer it from the block device underneath KahaDB, from producer-side send latency, or from diagnostic log lines, which is why this failure mode gets misdiagnosed as “the broker is slow” so often.

What this means

KahaDB is a write-ahead log store. The persistent send path: the transport thread reads the message off the wire, the broker appends it to the current journal file (db-*.log, 32MB each by default), and then it forces the data to durable media before acking the producer. That force operation is the fsync, and the producer’s send() call blocks until it returns.

With the default sync behavior of one fsync per journal write, a single producing thread can sustain at most 1 / fsync_latency sends per second:

Fsync latency	Max sends/sec per thread (approx)
0.5 ms	2000
2 ms	500
10 ms	100
50 ms	20

Broker-side batching and concurrent senders improve the aggregate number, but every send still pays for durability on the same device. When fsync latency doubles, the persistent throughput ceiling halves.

Operational latency bands:

<2 ms on w_await: healthy. Typical for local SSD/NVMe.
2-10 ms: acceptable. Busy SSD or local HDD.
10-50 ms: degraded. Persistent throughput is materially capped; investigate I/O contention.
>50 ms: critical. Persistent messaging is effectively throttled to tens of messages per second per producer thread.

flowchart LR
  P[Producer send] --> T[Transport thread]
  T --> J[Append to KahaDB journal]
  J --> F["fsync to block device (send blocks here)"]
  F --> A[Ack to producer]
  F --> D[Dispatch to consumers]
  S[Storage device: local SSD, SAN, or NFS] -. latency .-> F

Everything upstream of F (network, serialization, broker CPU) can be fast, and the send still waits on storage.

Common causes

Cause	What it looks like	First thing to check
Sync strategy `always` on slow storage	Steady per-send latency matching fsync latency; throughput ceiling that does not move with producer count	`journalDiskSyncStrategy` in the KahaDB persistence adapter config
I/O contention on the KahaDB device	`w_await` spikes correlate with backups, snapshots, co-located app writes, or other tenants	`iostat -xd` on the device, check what else writes to it
VM noisy neighbor	Latency spikes at irregular intervals, nothing on the guest explains them, `w_await` high with low guest-side throughput	Compare guest `iostat` with hypervisor/storage-side stats
NFS or shared-storage HA	Baseline write latency elevated by network round-trip; occasional multi-second stalls	`findmnt` output for the KahaDB directory; NFS server health
HDD instead of SSD/NVMe	Fsync consistently in the 2-10ms band even with low utilization	`/sys/block/<dev>/queue/rotational` (1 = rotational)
One-by-one store writes amplifying fsync count	Very high write IOPS relative to message rate, especially with slow consumers	Whether `concurrentStoreAndDispatchQueues` is enabled (see AMQ-7028 below)
Storage degradation	Latency climbs over days or weeks at constant workload, sometimes with device errors	`dmesg`, SMART data, array controller status

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Find the block device hosting KahaDB
df /opt/activemq/data/kahadb/ | awk 'NR==2{print $1}'

# 2. Watch write latency on that device (replace sda with your device)
#    Key columns: w_await (write latency), await (overall), %util
iostat -xd 1 10 /dev/sda

# 3. Is KahaDB on NFS?
findmnt -T /opt/activemq/data/kahadb/ -o TARGET,SOURCE,FSTYPE

# 4. Rotational or flash? (1 = HDD, 0 = SSD/NVMe)
cat /sys/block/sda/queue/rotational

# 5. Check the broker log for slow KahaDB access reports
grep -i "Slow KahaDB access" /opt/activemq/data/activemq.log | tail -20

The slow-access log line only appears if you have enabled the diagnostic threshold. Set the system property org.apache.activemq.store.kahadb.LOG_SLOW_ACCESS_TIME (value in milliseconds, e.g. 1500) on broker startup; the broker then logs lines like Slow KahaDB access: cleanup took 1277 whenever a store operation exceeds it.

Also confirm which sync strategy is configured in activemq.xml on the <kahaDB> persistence adapter:

grep -A5 "kahaDB" /opt/activemq/conf/activemq.xml | grep -i "sync\|journal"

Since ActiveMQ 5.14 the relevant setting is journalDiskSyncStrategy with values always (the default), periodic, and never. The older boolean enableJournalDiskSyncs is deprecated as of 5.14.

How to diagnose it

Establish the producer-side symptom. Measure per-send() latency for persistent messages from the application side. If it sits consistently in the tens of milliseconds and throughput per producer thread is the reciprocal of that latency, you are on the fsync critical path.
Measure the device. Run iostat -xd 1 on the KahaDB block device for a few minutes under load. Read w_await, not just await: the journal is write-dominated, so overall await can be masked by reads. Map the observed w_await to the latency bands above.
Rule out co-tenants. If KahaDB shares a device with the OS, broker logs, temp store, or other applications, any of them can inflate write latency. Check what else lives on the same partition and whether latency spikes line up with their activity (backup windows, log rotation, snapshot jobs).
Correlate with enqueue rate. Pull TotalEnqueueCount deltas from the broker MBean (or your monitoring) for the same window. If enqueue rate drops in lockstep with w_await spikes, storage latency is the constraint, not consumers, not flow control, not GC. Check that MemoryPercentUsage is not at 100% to rule out flow control masquerading as slowness.
Check the sync strategy. If journalDiskSyncStrategy is always (default) and the device cannot deliver fast fsyncs, you have found the ceiling. The question becomes whether you can make the device faster or whether you can safely relax the sync semantics.
For shared-storage HA, test the network path. NFS write latency includes the network round-trip plus the NFS server’s own disk latency. Compare w_await seen inside the client against latency on the NFS server itself. Also be aware that NFSv3 has a file-lock release problem after an abnormal master termination (the standby cannot acquire the lock); NFSv4 handles this with lock timeouts.
Look at write amplification. If write IOPS on the device vastly exceeds the message enqueue rate, each message may be producing more than one store write. One known case: concurrentStoreAndDispatchQueues=true combined with a high-latency filesystem and slow consumers causes one-by-one writes to the filesystem, multiplying the fsync penalty (tracked as AMQ-7028).

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`w_await` on the KahaDB device (`iostat -x`)	Direct proxy for journal fsync latency; no JMX equivalent exists	>10 ms sustained; >50 ms is critical
Producer-side persistent send latency	The user-visible expression of fsync latency	Rising mean or p99 with no flow control active
Enqueue rate (delta of `TotalEnqueueCount`)	Drops in lockstep with storage latency when fsync is the ceiling	Falls while producers are active and memory is not full
Write IOPS on the device vs enqueue rate	Reveals write amplification (multiple store writes per message)	IOPS many times higher than message rate
`%util` on the device	Shows whether the device is saturated or just slow	High util at modest throughput
`MemoryPercentUsage` (broker MBean)	Rules flow control in or out as the cause of slow sends	100% means blocked sends are flow control, not fsync
NFS server latency (if shared storage)	NFS adds network round-trip to every journal fsync	Server-side latency fine but client-side latency high
Slow KahaDB access log lines	Broker-internal confirmation that store operations exceeded the threshold	Repeated lines during the degraded window

The correlation that matters: w_await and enqueue rate plotted together. Enqueue falling exactly when write latency rises is the fingerprint of the fsync ceiling. Enqueue falling while w_await stays flat means look elsewhere: flow control, consumers, GC.

Fixes

Make the device faster

Put KahaDB on local SSD or NVMe. Journal writes are sequential, so flash helps less with raw write bandwidth than with random I/O, but fsync latency on flash is dramatically better than on spinning disk, and fsync latency is what you are buying. Dedicate the device or partition to KahaDB so broker logs, temp store, and OS activity cannot contend with journal writes. On virtualized hosts, verify latency from both inside the guest and on the hypervisor; noisy neighbors show up only on the physical side.

Adjust the sync strategy, with eyes open

journalDiskSyncStrategy has three settings, and each is a durability tradeoff, not a free performance knob:

always (default): fsync after every journal write. Full JMS durability. This is the ceiling described above.
periodic: sync at a configurable interval (journalDiskSyncInterval, default 1000 ms). Sends stop blocking on per-write fsync, and potential loss on broker failure is bounded to at most one interval of messages.
never: no explicit sync; the OS flushes on its own schedule. Equivalent to the old enableJournalDiskSyncs=false. Message loss can occur on broker failure.

Moving to periodic is the standard lever when the storage cannot deliver sub-2ms fsyncs and the business can tolerate a bounded loss window. Do not set never for messages you cannot afford to lose. Any change here requires a broker restart and should be validated with a failover/kill test before trusting it in production.

There is also a documented system property org.apache.activemq.kahaDB.files.skipMetadataUpdate=true that switches the journal force call from fsync() to fdatasync() semantics (FileChannel#force(false)), avoiding metadata flushes. Whether it actually helps depends on the JVM implementation: some JVMs map it to a real fdatasync, others fall back to fsync with no benefit. Measure before and after rather than assuming a win.

Fix the NFS path for shared-storage HA

If KahaDB sits on NFS for a shared-storage master/slave pair, every journal fsync carries network latency. Prefer a low-latency SAN or, if NFS is required, NFSv4 (which also fixes the NFSv3 lock-release problem after an abnormal master termination). Note that OCFS2 does not support the cluster-aware locking this setup needs; both brokers can believe they hold the master lock. If you can move HA off shared files entirely, do it; shared-storage locking is a chronic source of split-brain risk.

Reduce write amplification

If concurrentStoreAndDispatchQueues=true is set and you are on a slow filesystem with slow consumers, consider disabling it for affected destinations. The one-by-one write behavior in that mode multiplies the number of fsync-paying operations. Also check preallocation settings: on SSD, preallocationScope=entire_journal_async avoids delaying writes by preallocating in a background thread; on HDD the extra thread contention hurts, so keep the default.

Prevention

Alert on w_await, not just throughput. Ticket above 10 ms sustained on the KahaDB device; page only when elevated latency is confirmed by enqueue-rate collapse or send-latency impact. I/O noise from backups and snapshots without service impact should not wake anyone up.
Baseline the device. Record normal w_await and write IOPS under peak load so a 3x regression is obvious, not a matter of opinion during an incident.
Keep KahaDB on dedicated, fast storage. This is the single most effective preventive measure. Persistent messaging on shared HDD or congested NFS is a latency incident waiting for a traffic spike.
Capacity-plan against the ceiling. Know your measured fsync latency and compute the per-thread send ceiling from it. If the business requirement exceeds it, the answer is faster storage or a periodic sync window, decided in advance, not during an outage.
Include storage latency in broker change reviews. Any change to sync strategy, preallocation, or store-dispatch concurrency changes the fsync behavior. Test under realistic persistent load before rolling out.
Test the durability tradeoff you chose. If you run periodic, periodically verify with a kill test that actual loss on broker failure matches the configured interval.

How Netdata helps

Netdata collects per-disk latency, IOPS, and utilization from every block device at per-second resolution, so the w_await spike that caps your persistent throughput is visible at the granularity it actually happens at, not averaged away in one-minute samples.
Because Netdata also collects JMX metrics from the ActiveMQ broker on the same host, you can correlate TotalEnqueueCount rate drops with KahaDB device write latency on one dashboard and confirm (or rule out) the fsync ceiling in seconds.
Filesystem metrics on the KahaDB mount point catch the related failure mode of the partition filling up, which turns a latency problem into a write-failure and store-corruption problem.
NFS client and network interface metrics expose the network round-trip component of journal fsync latency in shared-storage HA deployments.
Alerting on disk latency with per-second data lets you set the 10 ms / 50 ms bands from this article as real thresholds tied to the specific KahaDB device, rather than inferring storage health from broker-side symptoms.
Historical retention means you can compare today’s fsync latency against last week’s baseline at the same traffic level, which is how you catch slow storage degradation before it becomes a throughput incident.