ZooKeeper “X is not executed because it is not in the whitelist”: four-letter-word commands blocked

The error string is exact. When you run echo mntr | nc localhost 2181 against a ZooKeeper 3.5.3+ server that has not been configured for it, the server replies:

mntr is not executed because it is not in the whitelist.

Same shape for ruok, isro, stat, conf, envi, cons, wchs, and the rest of the four-letter-word (4lw) command set. Only srvr works out of the box, because the bundled zkServer.sh status check depends on it.

The change is intentional. Since 3.5.3 the default whitelist for 4lw.commands.whitelist was narrowed from * (everything) to srvr only. The motivation is documented: wchp and wchc are CPU-intensive enough to be usable in denial-of-service attacks. The fix shipped in 3.4.10 and 3.5.3 was to ship the whitelist closed by default and require operators to open it explicitly.

The failure mode that hurts is silent. Health probes and metric scrapers that previously consumed mntr, ruok, or isro start receiving either the whitelist error string or nothing. Many scrapers do not treat that as a fetch failure; they parse the body as an empty payload and emit zero-valued metrics. From the dashboard’s point of view the cluster looks healthy at exactly the moment it has stopped being observed.

What this means

Four-letter-word commands are short text probes sent over the client port (default 2181). They predate the JMX layer and the HTTP AdminServer. The ones most monitoring stacks depend on are:

  • mntr - the monitoring dump. Emits all zk_* key/value pairs: zk_server_state, zk_avg_latency, zk_outstanding_requests, zk_zxid, zk_znode_count, and dozens of others.
  • ruok - shallow liveness. Returns imok if the JVM thread that handles the command is responsive. Does not confirm quorum, leadership, or that the data tree is loaded.
  • isro - read/write state. Returns rw if the node is in a functional ensemble, ro if it has lost quorum but is still serving stale reads.
  • stat and srvr - server summary (role, version, counters, latency). srvr is the default whitelist entry because the bundled scripts rely on it.
  • conf - parsed zoo.cfg.

When a 4lw command is not on the whitelist, the server writes the literal string “<cmd> is not executed because it is not in the whitelist.” back over the socket and closes the response. There is no log entry by default. There is no metric increment. The server itself is fine. Only your observability has stopped.

Three things make this painful after an upgrade:

  1. The error does not look like an error to a scraper. Telegraf’s ZooKeeper input logs unexpected line in mntr response: 'mntr is not executed because it is not in the whitelist.' and then stops emitting metrics. A pipeline that treats absent metrics as zero now shows a cluster sitting at zk_avg_latency = 0, zk_outstanding_requests = 0, zk_znode_count = 0. It looks healthy. It is unobserved.
  2. ruok returning nothing reads as “process down” to a naive probe, while isro returning nothing removes the only reliable signal that distinguishes a live read-write node from one that has lost quorum. The ro state - a node alive but unable to serve writes - is exactly the failure isro was built to expose.
  3. The change is version-gated and bites on upgrade. Pre-3.4.10 had no whitelist at all. 3.4.10 through 3.5.2 already had 4lw.commands.whitelist, but it defaulted to *. 3.5.3+ flips the default. Ensembles that ran for years without touching the property lose monitoring the day they cross that boundary.

Common causes

CauseWhat it looks likeFirst thing to check
Default whitelist on 3.5.3+Every 4lw except srvr returns the whitelist error stringgrep 4lw.commands.whitelist zoo.cfg returns nothing
Upgrade from 3.4.x or 3.5.0-3.5.2Monitoring was fine, broke immediately after upgradeecho srvr | nc localhost 2181 | grep Zookeeper to confirm new version
Distribution image ships closed whitelistContainerised ZK (Confluent images historically, some Helm charts) hits this on first deployInspect the active zoo.cfg mounted into the container
Restrictive list missing commandsmntr works, ruok does not (or vice versa)Diff the whitelist string against the set your scrapers call
Whitelist set on the wrong property name4lw.commands.whitelist is in zoo.cfg but the JVM was started with -Dzookeeper.4lw.commands.whitelist overriding itps -ef | grep QuorumPeerMain for -D overrides

Quick checks

# Confirm the symptom: which commands return the whitelist error?
for c in ruok mntr isro stat srvr conf cons envi wchs; do
  printf '%-6s -> ' "$c"
  echo "$c" | nc -w 2 localhost 2181 | head -1
done

# Confirm ZooKeeper version. srvr is the only 4lw on by default in 3.5.3+.
echo srvr | nc localhost 2181 | grep -E 'Zookeeper version|Mode'

# Read the effective whitelist from disk.
grep -E '4lw\.commands\.whitelist|zookeeper\.4lw\.commands\.whitelist' \
  /etc/zookeeper/conf/zoo.cfg /opt/zookeeper/conf/zoo.cfg 2>/dev/null

# Check whether the JVM overrides the file property.
ps -ef | grep '[Q]uorumPeerMain' | grep -o -- '-Dzookeeper.4lw.commands.whitelist=[^ ]*'

# Confirm the AdminServer is up (3.5+). Listens on 8080 by default.
curl -s --max-time 2 http://localhost:8080/commands/server_stats | head -c 200 ; echo

# Reproduce exactly what the scraper sees.
echo mntr | nc localhost 2181 | head -5

All of the above are read-only. None change ZK state.

How to diagnose it

  1. Reproduce the error from the host. echo mntr | nc localhost 2181 should produce the literal whitelist string. If it produces metrics, the problem is elsewhere (network ACL, wrong host, scraper auth).
  2. Confirm the ZooKeeper version. The whitelist default only changed in 3.5.3; if you are on 3.4.x or 3.5.0-3.5.2 and seeing the error, someone set an explicit restrictive whitelist.
  3. Read zoo.cfg for 4lw.commands.whitelist. The property uses dots (4lw.commands.whitelist) in the file and as a Java system property (zookeeper.4lw.commands.whitelist). The Java system property wins if both are set.
  4. Inspect the running process command line for -D overrides. A container image or a wrapper script can inject an override that bypasses zoo.cfg.
  5. Cross-check what each scraper actually calls. Common pattern: liveness probe uses ruok, readiness probe uses isro, Telegraf or a Prometheus exporter uses mntr, and operators run stat ad hoc. Each must be on the whitelist.
  6. Verify whether anything already consumes the AdminServer on port 8080. If a scraper already speaks HTTP, migrating it is cheaper than maintaining the whitelist.
flowchart TD
  A[4lw returns "not in the whitelist"] --> B{ZK version}
  B -->|3.4.x or 3.5.0-3.5.2| C[Explicit whitelist set - find it]
  B -->|3.5.3+| D[Default is srvr only]
  D --> E{Scraper needs}
  E -->|mntr, ruok, isro, stat| F[Add to 4lw.commands.whitelist]
  E -->|HTTP already supported| G[Move scraper to AdminServer :8080]
  F --> H[Rolling restart]
  G --> H

Metrics and signals to monitor

SignalWhy it mattersWarning sign
zk_server_state (from mntr)Confirms the node’s role. Without it you cannot tell leader from follower or detect split-brainMetric disappears from the scraper, or stays frozen at its last value
isro (rw / ro)The only 4lw that distinguishes a functional node from one serving stale reads with quorum lostReadiness probe silently green while the cluster is read-only
ruok (imok)Shallow liveness; not sufficient alone but consumed by load balancers and orchestratorsProbe reports the node down even though QuorumPeerMain is running
zk_outstanding_requestsLeading indicator of pipeline saturationStuck at zero because the scraper never sees the real value
zk_avg_latency, zk_max_latencyHeadline latency metricsFlatline at zero is the signature of a broken scrape, not a healthy cluster
zk_synced_followers, zk_pending_syncs (leader only)Replication health on the leaderAbsent from the leader’s scrape output
zk_looking_countElection frequency. Any non-zero outside maintenance is worth a ticketStale value hides repeated elections

| zk_uptime | Process uptime, useful for suppressing cold-start false positives | A frozen uptime hides restarts |

The pattern to alert on is not “metric crossed a threshold”. It is “metric source went away”. Treat a missing mntr scrape as a page, not a data gap.

Fixes

Two paths. Pick based on what your scrapers can speak.

Whitelist the commands your monitoring uses

Edit zoo.cfg:

4lw.commands.whitelist=srvr,ruok,isro,mntr,stat

Use a comma-separated list. Avoid * in production - it re-enables wchp and wchc, the CPU-intensive commands the whitelist exists to gate. If you need conf, envi, or cons for debugging, add them deliberately and remove them once the session ends; cons is expensive on ensembles with many client connections.

Apply with a rolling restart. There is no reload signal for 4lw.commands.whitelist; the property is read at startup. On a 3-node ensemble, restart one follower at a time, confirm it rejoins and mntr works, then proceed.

After the restart, verify each command individually:

for c in ruok mntr isro stat srvr; do
  printf '%-6s -> ' "$c"
  echo "$c" | nc -w 2 localhost 2181 | head -1
done

If you are running in a container, check where the config is mounted. Confluent images and some Helm charts historically did not set a whitelist and required the property to be added through the image’s own configuration mechanism.

Move scrapers to the AdminServer

ZooKeeper 3.5.0+ ships an embedded Jetty AdminServer on port 8080. The endpoint shape is /commands/<command> and the response is JSON. The equivalent of mntr is:

curl -s http://localhost:8080/commands/monitor | python3 -m json.tool | head -30

Other useful endpoints include /commands/server_stats (the srvr/stat equivalent) and /commands/environment.

The official documentation notes that 4lw commands are deprecated in favour of the AdminServer. No removal date has been announced and 4lw is still functional in current 3.8.x and 3.9.x releases, but new deployments should prefer HTTP where the scraper supports it.

The AdminServer has its own security surface. If you expose it beyond localhost, place it behind a firewall and keep current on patch releases. Recent AdminServer issues include an IP-authentication bypass fixed in 3.9.3 and a snapshot/restore permission check fixed in 3.9.4. Do not assume that closing the 4lw whitelist is the only thing that matters.

Do not silence the error

Pointing the health probe at srvr (which is always whitelisted) and calling it done is the same trap as using ruok as your only check: srvr confirms the JVM is up but says nothing about quorum, replication, or whether the node can serve writes. The fix is to whitelist the commands you need, not to downgrade your checks to the ones that already work.

Prevention

  • Set the whitelist explicitly in your base zoo.cfg template. Do not rely on the default. Treat it as part of cluster bootstrap, not a runtime tweak.
  • Pin the whitelist in version control alongside zoo.cfg. When someone upgrades the cluster, the whitelist moves with the config.
  • Include a 4lw sanity check in post-deploy smoke tests. Run ruok, mntr, and isro against every node and assert non-empty responses.
  • Alert on missing scrapes, not just on threshold crossings. A pipeline that silently emits zeros when the source disappears is a monitoring failure dressed up as a metric.
  • For new deployments, prefer the AdminServer over 4lw. The deprecation notice has been in the docs since 3.5.3-beta. Even if 4lw is never removed, the HTTP endpoint is easier to secure and scrape.
  • When upgrading across the 3.4.x-to-3.5.3+ boundary, treat the whitelist change as a breaking change in your release notes. It is the most common silent monitoring break after a ZooKeeper upgrade.

How Netdata helps

  • Netdata’s ZooKeeper collector issues the 4lw commands you configure. If mntr is not whitelisted, the per-second scrape fails visibly in the collector logs and the affected charts go to “no data” rather than to zero. Flatline-at-zero is the failure mode that hides outages; explicit “no data” is the one you want.
  • Per-second resolution means the gap between “whitelist changed” and “alert fires” is short. A daily scraper can lose a day of visibility before anyone notices; a per-second scraper cannot.
  • Correlating the missing mntr scrape against an ensemble-wide deployment event (rolling restart, image bump) makes the upgrade-induced regression obvious. The signature is “all zk_* metrics disappeared from this node at the same instant”, which is visually distinct from “one metric crossed a threshold”.
  • Netdata’s ML anomaly detection flags the joint pattern of “metric source went away while process uptime reset” as anomalous, even when no static threshold has been set on the affected charts.