$ guides / zookeeper / zookeeper-packet-len-out-of-range ▌

Operations Guides

ZooKeeper "Packet len is out of range": jute.maxbuffer and oversized znodes

ZooKeeper “Packet len is out of range”: jute.maxbuffer and oversized znodes

Packet len <N> is out of range! looks like a network framing problem. It is not. The ZooKeeper client is telling you the server’s serialized response exceeded the client’s maximum deserialization buffer, and the client closed the connection rather than read a truncated packet.

On the server side, the same condition produces a terser log line: Len error. That appears when a client attempts a write whose payload exceeds the server’s configured buffer limit. Both sides are governed by one Java system property: jute.maxbuffer, which defaults to 0xfffff (1048575 bytes, just under 1 MB).

The limit is not just about large writes. A getChildren call on a parent with tens of thousands of children can produce a response that exceeds the limit even though every individual znode is tiny. Worse, an oversized znode already in the tree can prevent a follower from completing its sync after a restart, blocking ensemble recovery.

What this means

jute.maxbuffer caps the size of a single serialized ZooKeeper packet. It is enforced independently on the server and on every client, as a Java system property (-Djute.maxbuffer=<bytes>) with no zookeeper. prefix. The check happens in the JUTE serialization layer (ZooKeeper’s RPC format), before any application-level logic runs.

When the limit is hit, the two sides report it differently:

Server-side rejection (write too large). The server reads the incoming length prefix, sees it exceeds its jute.maxbuffer, and logs Len error. The connection is closed. The client sees a connection reset or a session disconnect.
Client-side rejection (response too large). The client reads the response length prefix, sees it exceeds its own jute.maxbuffer, and logs Packet len <N> is out of range! (or, on some client versions, Unreasonable length). The client closes the connection and enters its reconnect loop.

Because the check is on the serialized packet, not the logical operation, the trigger can be surprising. The two production scenarios are:

A single large znode. An application calls setData with a payload near or above 1 MB, or calls getData on a znode someone else wrote at that size.
A large getChildren response. The response contains every child name concatenated. A parent with tens of thousands of children can produce a response exceeding 1 MB even if each child znode stores zero bytes.

There is no chunking or pagination for ZooKeeper responses. The entire serialized response must fit in one buffer.

flowchart TD
    A[Client request] --> B{Operation type}
    B -->|setData / create| C[Server checks payload vs jute.maxbuffer]
    B -->|getData| D[Server serializes znode data]
    B -->|getChildren| E[Server serializes all child names]
    C -->|Exceeds limit| F[Server logs: Len error
closes connection]
    D --> G[Client checks response vs jute.maxbuffer]
    E --> G
    G -->|Exceeds limit| H[Client logs: Packet len out of range
closes connection]
    G -->|Within limit| I[Operation succeeds]
    C -->|Within limit| I

A third scenario, less common but more dangerous: an oversized znode already in the tree blocks a follower from syncing. When a follower restarts and needs to catch up, the leader may need to transfer the data tree. If the tree contains a znode whose serialized form exceeds the follower’s jute.maxbuffer, the sync fails and the follower cannot rejoin the ensemble.

Common causes

Cause	What it looks like	First thing to check
Large blob in a single znode	Client logs `Packet len <N> is out of range` after `setData` or `getData`; server logs `Len error` on writes	`zk_large_requests_rejected` incrementing
Huge `getChildren` response	Client logs `Packet len <N> is out of range` after listing children; no individual znode is large	Child count on the parent path
Client/server `jute.maxbuffer` mismatch	Server accepted a write that the client cannot read back; only some clients fail	`-Djute.maxbuffer` value on each JVM
Oversized znode blocking follower sync	Follower cannot rejoin after restart; buffer errors during sync	Leader logs during follower sync attempt
Version-specific parsing bug	`jute.maxbuffer` set to hex or large value triggers integer errors	ZooKeeper version (see Fixes below)

Quick checks

These commands require 4lw.commands.whitelist to include mntr (and dump where used). They are read-only and do not modify ZooKeeper state.

# Server-side rejection counter (available since ZooKeeper 3.6.0)
echo mntr | nc localhost 2181 | grep zk_large_requests_rejected

# Server-side Len error in logs
grep "Len error" /var/log/zookeeper/zookeeper.log | tail -20

# Client-side buffer errors in application logs
grep -r "Packet len.*is out of range" /var/log/your-application/ | tail -20

# Total data tree size and znode count
echo mntr | nc localhost 2181 | grep -E "zk_znode_count|zk_approximate_data_size"

# Verify jute.maxbuffer is set on the server JVM
ps -ef | grep QuorumPeerMain | grep -o -- '-Djute.maxbuffer=[0-9]*'

# Current connection state (are clients cycling?)
echo mntr | nc localhost 2181 | grep -E "zk_num_alive_connections|zk_connection_drop_count"

How to diagnose it

Determine which side is rejecting. If the server log shows Len error, the client is writing too much. If only the client log shows Packet len <N> is out of range, the server happily serialized a response that the client refuses to read. The fix differs.
Check zk_large_requests_rejected. This counter (via mntr since ZooKeeper 3.6.0) tracks server-side rejections. A non-zero rate confirms the server is turning away oversized writes. If it is zero but clients are still failing, the rejection is happening client-side: the response exceeds the client’s limit, not the server’s.
Identify the offending path. If a getChildren call is the trigger, find the parent with the most children. The dump four-letter command (if whitelisted) lists ephemeral nodes and sessions. For persistent children, walk the tree with zkCli.sh starting from suspected high-cardinality paths (service registries, task queues, consumer offset paths).
Check for version-specific bugs. Several ZooKeeper versions have jute.maxbuffer parsing bugs (see Fixes below). If you set the property but behavior does not change, verify your version is not affected.
Verify consistency across the ensemble and clients. jute.maxbuffer must be the same on every server and every client. A server with a higher limit than its clients creates an invisible trap: the server accepts a write, then every client that tries to read it fails.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`zk_large_requests_rejected`	Direct counter of server-side rejections (3.6.0+)	Any non-zero increment rate
`zk_znode_count`	Large trees produce large `getChildren` responses	Growth concentrated in one subtree
`zk_approximate_data_size`	Tracks total data stored; large blobs inflate this	Sudden jump indicates a large write
`zk_connection_drop_count`	Connections cycle when clients hit the buffer limit	Spike correlated with rejection events
`zk_outstanding_requests`	May climb if connections are rapidly cycling	Sustained non-zero with active rejection
`zk_num_alive_connections`	Drops when clients disconnect after buffer errors	Sharp decline matching error timestamps

Fixes

The right fix: move large data out of ZooKeeper

The ZooKeeper project does not recommend increasing jute.maxbuffer for production use. Large znodes cause latency spikes on every replica, reduce write throughput, and make leader-follower synchronization unpredictable (sometimes causing timeouts that destabilize the quorum). The correct fix is to stop storing large data in ZooKeeper.

If an application is storing blobs (configuration documents, serialized objects, media), move that data to an object store, database, or cache. Store only a reference or a small metadata payload in the znode.

If getChildren returns too many children

A parent with tens of thousands of children is a design problem, not a configuration problem. Options:

Shard the hierarchy. Break the flat list into bucketed subdirectories (for example, /registry/00/ through /registry/ff/ by hash prefix). Each getChildren call stays under the buffer limit.
Move the registry. High-cardinality membership data (service instances, task assignments) often belongs in a system designed for it (Redis, etcd, a database) rather than ZooKeeper.
Review ephemeral node patterns. Ephemeral nodes under a single parent are the classic trigger. If every client registers under /members/, that parent grows without bound.

If you must increase jute.maxbuffer

If changing the application is not immediately possible, you can raise the limit as a stopgap. Rules:

Set it on every server AND every client. A mismatch creates silent failures. The property has no zookeeper. prefix; it is -Djute.maxbuffer=<bytes>.
Restart all JVMs. The property is read at JVM startup. A rolling restart of the ensemble plus a restart of all client applications is required.
Know the risks. The ZooKeeper admin guide explicitly warns against this for production. Larger buffers mean larger packets on every write and every sync, increasing memory pressure, GC pauses, and sync times.
Avoid values at or above 1 GB. In versions before 3.8.5, 3.9.3, and 3.10.0, setting jute.maxbuffer to 1073741824 (1 GB) or higher caused an integer overflow in (maxBufferSize + extraMaxBufferSize), producing a negative number and triggering Unreasonable length errors even for small responses (ZOOKEEPER-4843).

If the client/server defaults mismatch

Before ZooKeeper 3.6.0, the client-side default was documented as 4 MB but actually enforced 1 MB due to a bug in BinaryInputArchive.checkLength() (fixed in ZOOKEEPER-3593). If you are running clients older than 3.6.0 alongside a newer server, clients may fail reading data the server considers valid. The fix is to upgrade clients to 3.6.0+ or set -Djute.maxbuffer explicitly on both sides.

Additional version-specific issues:

3.5.0 to 3.5.2. jute.maxbuffer was effectively ignored on the client side due to a parsing bug (fixed in 3.5.3, ZOOKEEPER-2517).
3.5.6. Setting jute.maxbuffer with a hexadecimal value (such as 0xfffff) threw a parseInt error because the code switched from Integer.getInteger() to Integer.parseInt() (fixed in 3.5.7, ZOOKEEPER-3667).

Prevention

Never store large data in ZooKeeper. This is the project’s own guidance. ZooKeeper is a coordination service, not a data store.
Monitor child counts on parent znodes. A slowly growing parent is the leading indicator of a future getChildren failure. Track zk_znode_count and investigate any subtree growing without bound.
Set jute.maxbuffer explicitly everywhere. Do not rely on defaults. Set the same value on every server and every client JVM so that a version upgrade or a new client library cannot silently introduce a mismatch.
Alert on zk_large_requests_rejected. This counter should be zero in steady state. Any non-zero rate means an application is misusing ZooKeeper or a configuration is wrong.
Upgrade past known bug versions. If you are running 3.5.x, ensure you are on 3.5.7+. If you need large buffer values, ensure you are on 3.8.5+, 3.9.3+, or 3.10.0+ to avoid the integer overflow.

How Netdata helps

Per-second collection of zk_large_requests_rejected shows the exact moment an application starts sending oversized requests, before clients begin cycling connections.
Correlation with zk_znode_count and zk_approximate_data_size shows whether the tree is growing toward a future buffer failure, even when no rejections have occurred yet.
Connection drop metrics (zk_connection_drop_count, zk_num_alive_connections) reveal the blast radius: how many clients are cycling because of buffer rejections.
Anomaly detection on connection drops and outstanding requests catches the secondary effects (reconnect storms, queue buildup) that follow a buffer rejection event, even when the rejection counter itself is not yet scraped.
JVM heap and GC pause metrics help you assess the risk of increasing jute.maxbuffer: if heap is already under pressure, a larger buffer will make things worse.