ClickHouse Operations Guides on Netdata

ClickHouse active part count growing: reading MaxPartCountForPartition before it pages

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse active part count growing: reading MaxPartCountForPartition before it pages

Rising MaxPartCountForPartition is the leading indicator for the most common ClickHouse production failure: parts accumulating faster than background merges can consolidate them. A single partition crossing 500 active parts means you have hours, not days, before inserts delay and eventually fail with TOO_MANY_PARTS.

The thresholds are per-partition. A table with ten partitions at fifty parts each is healthy; one partition at 950 parts is approaching throttling. Projections create hidden parts inside the same table that count toward the same limits. Materialized views route inserts to separate target tables that can hit their own limits independently.

ClickHouse ALTER UPDATE/DELETE overuse: why mutations are not row updates

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse ALTER UPDATE/DELETE overuse: why mutations are not row updates

Your inserts are slowing down. system.merges shows long-running background tasks with is_mutation = 1, while the active part count climbs toward the parts_to_delay_insert threshold. Write latency rises and DelayedInserts increases even though memory, disk, and CPU are not exhausted. The culprit is usually an application treating ClickHouse as an OLTP store, issuing ALTER TABLE UPDATE or ALTER TABLE DELETE as routine operations.

ClickHouse async inserts: when async_insert fixes too-many-parts and when it hides it

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse async inserts: when async_insert fixes too-many-parts and when it hides it

Async inserts move batching responsibility from the client to the ClickHouse server. The server buffers rows and flushes them as larger blocks instead of persisting every INSERT as a separate part. When the root cause is an unbatchable client, this stops the small-inserts anti-pattern from flooding the merge pool. When the root cause is high-cardinality partitioning or sustained over-ingestion, async inserts relocate the crisis into an in-memory buffer that loses data on crash and hides backpressure from the application.

ClickHouse authentication failures: system.session_log, brute force, and credential drift

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse authentication failures: system.session_log, brute force, and credential drift

You notice a spike in failed connection attempts to ClickHouse: a security scanner flags repeated TCP 9000 probes, or an application logs connection timeouts after a secrets rotation. ClickHouse exposes authentication events through system.session_log, but only if the feature is enabled. Without it, fallback to server error logs and system.query_log exceptions.

The failures split into two patterns. Malicious: brute-force or credential-scanning campaigns against exposed TCP 9000 or HTTP 8123. Operational drift: a rotated password not updated in a client config, or a deployment shipping an old connection string. Distinguish them fast. Block an external attacker at the network layer; fix credential drift on the client.

ClickHouse background pool saturation: when merges and mutations starve

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse background pool saturation: when merges and mutations starve

Insert latency is climbing and DelayedInserts is ticking up. system.merges shows every slot occupied, yet system.parts keeps growing. When the background merge and mutation pool saturates, new merges queue instead of starting, parts accumulate, and the distance to insert rejections shrinks fast. Distinguish true thread starvation from I/O-bound stalls, identify when mutations are the culprit, and relieve pressure before inserts fail.

ClickHouse cannot connect to ZooKeeper/Keeper: diagnosing the coordination layer

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse cannot connect to ZooKeeper/Keeper: diagnosing the coordination layer

If SELECT * FROM system.zookeeper WHERE path = '/' fails, replicated tables flip to readonly, or ON CLUSTER DDL hangs, the coordination layer is broken. The server stays up and GET /ping returns Ok., so liveness checks miss the problem. Partial degradation can escalate to a write outage if replicated tables cannot re-establish sessions.

ClickHouse uses the coordination service for ReplicatedMergeTree leader election, replication log queues, insert deduplication, and distributed DDL. ClickHouse Keeper typically listens on port 9181; external ZooKeeper ensembles listen on 2181. The diagnostic path differs slightly, but the symptom is the same: ClickHouse cannot reliably complete coordination operations.

ClickHouse checksum mismatch and broken parts: detecting data corruption

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse checksum mismatch and broken parts: detecting data corruption

ClickHouse logs showing Checksum doesn't match, Broken part, or similar errors indicate data corruption. Affected parts move to system.detached_parts. Queries may throw exceptions or return partial results. On replicated clusters, a replica with corrupt parts may lag because it cannot validate fetched parts. Corruption does not self-resolve. You must quarantine the bad part, identify the root cause, and rebuild the data from a healthy source or backup.

ClickHouse client connections climbing: TCP 9000, HTTP 8123, and connection leaks

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse client connections climbing: TCP 9000, HTTP 8123, and connection leaks

TCPConnection or HTTPConnection climbing on a ClickHouse node means each active socket consumes a file descriptor. When growth is uncorrelated with query throughput, it is usually a connection leak or pool misconfiguration rather than healthy concurrency. If the count approaches max_connections, the server rejects new client connections. If the Linux nofile limit is reached first, queries and merges fail with “too many open files.”

ClickHouse DB::Exception: Too many parts - causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse DB::Exception: Too many parts - causes and fixes

When logs show DB::Exception: Too many parts (N). Merges are processing significantly slower than inserts (exception code 252), INSERTs that succeeded yesterday are now failing. Data backs up in ingestors, message queues, or client buffers, and every retry adds load to a system already drowning in small files.

At least one partition in a MergeTree table has exceeded the hard limit for active data parts, and background merges cannot consolidate them fast enough. Do not restart the server or raise the limit indefinitely. Reduce the part creation rate, remove blockers from the merge pipeline, and give background threads runway to catch up.

ClickHouse DelayedInserts climbing: the warning before too-many-parts

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse DelayedInserts climbing: the warning before too-many-parts

Insert latency is climbing and system.events.DelayedInserts is no longer flat. ClickHouse is sleeping during INSERT because at least one partition has crossed parts_to_delay_insert. The database still accepts writes, but injects a sleep before each insert commits. This is the warning window before hard failure. If the merge backlog is not resolved, DelayedInserts climbs until RejectedInserts starts ticking and clients receive DB::Exception: Too many parts.

ClickHouse detached parts piling up: reading system.detached_parts and reclaiming space

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse detached parts piling up: reading system.detached_parts and reclaiming space

Disk usage is climbing and system.disks shows unreserved_space shrinking, yet active part counts in system.parts look normal. Queries are not failing, merges appear healthy, and there is no insert rejection storm. The hidden consumer is often a growing pile of detached parts: directories that ClickHouse removed from the active dataset but left on the filesystem. Unlike active parts, detached parts do not participate in queries or merges, and ClickHouse never deletes them automatically. Standard monitoring that only watches system.parts with active = 1 misses them entirely. This article explains how to read system.detached_parts, interpret the reason column, decide whether to reattach or discard the data, and prevent silent recurrence.

ClickHouse disk space collapse: why merges need free space and how the spiral starts

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse disk space collapse: why merges need free space and how the spiral starts

Disk usage climbs from 75% to 85% over a week, then hits 90%. A few hours later ClickHouse rejects inserts, part counts spike, and the volume is at 98%. The system did not simply run out of space. It entered a self-reinforcing spiral where background merges, the mechanism that reclaims space, stalled because they needed free space to work.

ClickHouse disk space monitoring: free_space, unreserved_space, and the 80% target

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse disk space monitoring: free_space, unreserved_space, and the 80% target

ClickHouse disk space is not a simple capacity gauge. Merges write new parts before deleting old ones, so free space is an operational dependency of the storage engine. Insufficient disk does not just mean “running low”: background merges stall, parts accumulate, and the system enters a self-reinforcing death spiral.

This guide covers the system.disks metrics, why unreserved_space matters more than free_space, and operational targets that keep merges alive.

ClickHouse distributed DDL stuck: ON CLUSTER queries that never finish

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse distributed DDL stuck: ON CLUSTER queries that never finish

ON CLUSTER DDL is not atomic. The initiator writes a task into a shared queue in ZooKeeper or ClickHouse Keeper under /clickhouse/task_queue/ddl/. Each node pulls entries in order, executes the DDL locally, and reports status back. One slow, down, or misconfigured node can leave the entry unfinished forever, blocking all later DDL on that node and creating schema drift that is invisible until a replica rejects an insert or a subsequent ALTER fails.

ClickHouse distributed query amplification: one coordinator, many shard subqueries

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse distributed query amplification: one coordinator, many shard subqueries

A SELECT against a distributed table registers one entry in system.processes on the coordinator. That query fans out subqueries to every addressed shard, waits for intermediate results, then merges them locally. True CPU, memory, and network load is multiplied by the shard count, and the slowest participating shard dictates latency.

This amplification is invisible to cluster-average metrics. A dashboard averaging CPU or query latency across all nodes can look healthy while one shard is saturated, because the coordinator and healthy shards mask the straggler. Operators often misdiagnose the slowdown as a generalized cluster problem or a coordinator bottleneck, when the root cause is usually a single overloaded shard, a missing sharding key filter, or an expensive GLOBAL operator that broadcasts data across the network.

ClickHouse full table scan: partition pruning failures and the primary key

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse full table scan: partition pruning failures and the primary key

A query that returned in milliseconds yesterday now takes minutes and saturates CPU. Check whether it is reading every row before you add vCPUs or shards. A missing filter on the partition key or a mismatch with the primary key prefix forces ClickHouse to open every part and scan every granule. Symptoms are runaway read_rows counts, tail latency spikes, and sustained CPU decompression that hardware scaling cannot fix.

ClickHouse insert latency rising: the leading indicator of write-pipeline trouble

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse insert latency rising: the leading indicator of write-pipeline trouble

Your ClickHouse inserts are taking longer. A query that committed in 200 ms last week is now taking 5 seconds, then 15, then 30. In most databases this signals slow disks or lock contention. In ClickHouse, sustained insert latency is the earliest operational signal that the write pipeline is congesting. It precedes DelayedInserts, part-count alerts, and the hard stop of RejectedInserts by minutes to hours. Wait for the error and the merge debt is already severe.

ClickHouse Keeper latency high: the early warning before sessions expire

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Keeper latency high: the early warning before sessions expire

INSERTs to replicated tables slow down, ON CLUSTER DDL hangs, and the replication queue grows on followers. SELECT 1 and HTTP /ping stay healthy, and non-replicated tables are fine. The culprit is usually the coordination service, not ClickHouse itself.

Rising ZooKeeper or ClickHouse Keeper operation latency is a leading indicator. Replicated inserts, replication log updates, and distributed DDL all round-trip through Keeper. Because Keeper writes its transaction log synchronously, disk I/O on the Keeper node is the most common bottleneck. A degraded-but-connected coordination service is worse than a hard partition: it silently slows every replicated operation until sessions start expiring and replicas flip to readonly. This article explains how to read the early signals, isolate the cause, and fix it before sessions expire.

ClickHouse Keeper saturation spiral: too many tables, DDL storms, and cluster freeze

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Keeper saturation spiral: too many tables, DDL storms, and cluster freeze

INSERTs fail. Replicated tables flip read-only. ON CLUSTER DDL hangs. curl http://localhost:8123/ping still returns Ok. and SELECT 1 still works. This is Keeper saturation: the coordination layer is choking while liveness probes give false confidence.

The spiral starts when ZooKeeper or ClickHouse Keeper cannot keep up with metadata load. Every replicated table registers znodes and watches. Every DDL operation adds more. Coordination latency climbs until heartbeat traffic cannot complete within the negotiated session timeout; sessions expire, replicas become read-only, and writes fail. Reconnecting nodes then trigger a thundering herd.

ClickHouse killed by the OOM killer: RSS, max_server_memory_usage, and cgroup limits

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse killed by the OOM killer: RSS, max_server_memory_usage, and cgroup limits

You restart a pod and kubectl describe pod shows Reason: OOMKilled with exit code 137. Inside ClickHouse, MemoryTracking sits well below max_server_memory_usage, and system.text_log shows no warning. The process is gone, merges are dead, and replication queues are backing up.

The Linux OOM killer targets RSS, not ClickHouse’s internal MemoryTracking. Untracked allocations, jemalloc arena fragmentation, and cgroup accounting quirks create a persistent gap between what ClickHouse thinks it is using and what the kernel sees. In containerized environments, the cgroup OOM killer can evict the pod before ClickHouse ever triggers its own server-wide limit.

ClickHouse long-running queries: finding and killing the resource hog

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse long-running queries: finding and killing the resource hog

A query that should finish in seconds is still running after twenty minutes. Memory on the ClickHouse node is climbing, query latency has doubled, and you suspect a single query is holding resources it will never release. In ClickHouse, a long-running query can be a legitimate analytical job crunching terabytes, a Cartesian JOIN exploding in memory, or a GROUP BY that has spilled to disk and slowed to a crawl. Telling the difference determines whether you kill it or let it finish.

ClickHouse mark cache and uncompressed cache: reading low hit rates

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse mark cache and uncompressed cache: reading low hit rates

ClickHouse query latency spikes after a restart, or monitoring shows a sustained drop in mark cache hit rate. Before increasing mark_cache_size, determine whether you are seeing normal warmup or a cache that is too small for the working set.

The mark cache stores primary key index granule positions. The uncompressed cache stores decompressed column blocks. Low hit rates in each produce different symptoms. This guide covers how to read the metrics, distinguish warmup from real problems, and decide when tuning is warranted.

ClickHouse Memory limit (for query) exceeded: per-query limits and GROUP BY/JOIN blowups

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Memory limit (for query) exceeded: per-query limits and GROUP BY/JOIN blowups

Code: 241. DB::Exception: Memory limit (for query) exceeded means a single query’s allocations breached the max_memory_usage ceiling. ClickHouse tracks memory in a hierarchy: server-wide, per-user, and per-query. The server kills the query to protect the rest of the workload.

This differs from Memory limit (total) exceeded (server-level pressure) and Memory limit (for user) exceeded (profile-level pressure). A query-level breach usually stems from one of three patterns: a high-cardinality GROUP BY, an unbounded DISTINCT, or a JOIN that materializes more rows than expected.

ClickHouse Memory limit (total) exceeded - server-wide memory pressure and fixes

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Memory limit (total) exceeded - server-wide memory pressure and fixes

Code: 241. DB::Exception: Memory limit (total) exceeded: would use X bytes, current RSS Y, maximum Z. is the server-level cap, not a per-query limit. When ClickHouse’s MemoryTracking hits max_server_memory_usage (default 90% of physical RAM), the server kills the heaviest running queries to protect the process. New and existing queries fail until memory drops. Find the largest memory consumer and stop it before the OOM killer does.

ClickHouse memory pressure death spiral: runaway queries, retries, and OOM

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse memory pressure death spiral: runaway queries, retries, and OOM

MEMORY_LIMIT_EXCEEDED errors climb in the query log. Queries that normally finish in seconds now take minutes or are killed outright. The ClickHouse process is near its memory limit, but killing the heaviest query only frees capacity for a moment before another query is killed. If the application retries immediately, pressure never drops. With spill-to-disk enabled, the bottleneck shifts to disk I/O, starving background merges and slowing the whole system.

ClickHouse MemoryTracking vs MemoryResident: reading the memory gap correctly

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse MemoryTracking vs MemoryResident: reading the memory gap correctly

You finish a large batch query, open monitoring, and see ClickHouse MemoryTracking drop by 40 GB while MemoryResident barely moves. Or you watch RSS climb for hours after a restart while MemoryTracking tracks the rise steadily. Neither pattern indicates a leak. The gap between ClickHouse’s internal ledger and OS resident set size is normal: the server accounts for memory synchronously while jemalloc retains pages for reuse.

ClickHouse merge death spiral: when parts accumulate faster than merges consolidate

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse merge death spiral: when parts accumulate faster than merges consolidate

Insert latency climbs. Application logs show ClickHouse throttling writes. Eventually inserts fail with Too many parts. Disk usage rises even though ingestion volume is flat. The cluster is up but refusing writes.

This is the merge death spiral: a self-reinforcing loop where parts accumulate faster than background merges consolidate them. Every INSERT creates immutable on-disk parts. Background merge threads combine smaller parts into larger ones to keep query performance healthy and part counts low. When insert pressure exceeds merge throughput, the backlog grows, merge overhead increases, and the system chokes on its own structure.

ClickHouse merge duration climbing: the leading indicator of part explosion

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse merge duration climbing: the leading indicator of part explosion

system.merges shows elapsed times in hours. Your dashboards show P99 merge duration climbing over the past 48 hours. Rising merge duration is the earliest signal that your cluster is heading toward a part-count crisis, typically 1 to 3 days before inserts throttle or fail entirely.

ClickHouse merges not keeping up: diagnosing a stalled or starved merge pool

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse merges not keeping up: diagnosing a stalled or starved merge pool

When insert latency climbs and system.merges is empty while active parts grow, the background merge pool is likely stalled or starved. ClickHouse relies on background merges to consolidate immutable parts after each INSERT. Without merges, parts accumulate: query scans open more files, memory pressure shifts to the mark cache and file descriptor tables, and the system approaches the TOO_MANY_PARTS threshold.

ClickHouse monitoring checklist: the signals every production cluster needs

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse monitoring checklist: the signals every production cluster needs

ClickHouse failures usually begin as storage-structure debt: immutable parts accumulate faster than merges consolidate them, coordination sessions expire, or disk space drops below the threshold merges need to complete. Query latency degrades only after the crisis is hours old.

This checklist groups monitoring signals into four maturity levels. Level 1 is the minimum viable instrumentation to avoid data loss and unavailability. Each subsequent level adds leading indicators that catch part accumulation, replication divergence, and memory pressure while they are still reversible. The signals are drawn from ClickHouse system tables and OS-level metrics. They apply to single-node, sharded, and replicated setups; replicated tables add ZooKeeper/Keeper signals that belong in Level 2.

ClickHouse monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse monitoring maturity model: from survival to expert

Most production ClickHouse incidents are not mysterious. They are predictable storage-structure or coordination failures that better monitoring would have surfaced hours earlier. If you are running ClickHouse at scale, you need to know whether your observability is actually catching the failure modes that matter, or just proving that the process is running.

This maturity model is a diagnostic mirror, not a trophy case. Use it to audit your dashboards, tune alert noise, and decide what to instrument next. Each level builds on the last. Skipping levels leaves predictable gaps: teams with beautiful query-latency dashboards still get surprised by merge death spirals because they never instrumented part counts per partition.

ClickHouse mutation stuck: parts_to_do not decreasing and how to recover

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse mutation stuck: parts_to_do not decreasing and how to recover

Check system.mutations and find a mutation active for hours. parts_to_do has not moved in thirty minutes and parts are accumulating. Queries return, but insert latency climbs. On replicated tables, each replica processes mutations independently, so a stall on one node creates silent divergence while the rest of the cluster appears healthy.

Mutations rewrite data parts to apply ALTER UPDATE, ALTER DELETE, or projection changes. They run sequentially per table and share the background merge and mutation pool with regular merges. A stalled mutation blocks subsequent mutations for that table and consumes threads needed for merges. The result is merge starvation, insert delays, and eventually Too many parts rejections.

ClickHouse mutations silently blocking merges: the hidden cause of part growth

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse mutations silently blocking merges: the hidden cause of part growth

You are watching part counts climb on a ClickHouse node. system.merges shows active background tasks, health checks return Ok, and the log shows no mutation errors. Yet inserts are slowing and MaxPartCountForPartition is trending toward the delay threshold. The pool looks busy, so merges should be keeping up. They are not: some of those busy slots are mutations, and mutations starve merges silently.

ClickHouse No space left on device: emergency recovery when the data disk fills

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse No space left on device: emergency recovery when the data disk fills

When ClickHouse returns No space left on device to client inserts or the server log fills with write errors, the situation is past a simple capacity alert. ClickHouse does not degrade gradually on a full disk. Background merges halt immediately because they require temporary free space to write combined parts before removing source files. Once merges stop, small insert parts accumulate, metadata overhead grows, and disk usage accelerates. TTL-based expiration also stops because TTL cleanup is executed by merges. ClickHouse system tables such as system.query_log and system.part_log are MergeTree tables that can grow unbounded and consume the remaining space.

ClickHouse projections and hidden parts: the part count you can't see

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse projections and hidden parts: the part count you can’t see

Inserts fail with TOO_MANY_PARTS while system.parts on the base table looks comfortable. If you monitor only the source table, you are missing projection sub-parts inside base directories and independent parts in materialized view target tables. A single insert can spawn parts across projection subdirectories and multiple downstream tables. The number in system.parts is a floor, not a ceiling. The gap between visible and real part count is where merge crises begin.

ClickHouse query error rate high: reading exception codes 241, 252, and 999

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse query error rate high: reading exception codes 241, 252, and 999

What this means

ClickHouse logs failed queries to system.query_log with type = ‘ExceptionWhileProcessing’ and a numeric exception_code. The FailedQuery counter in system.events increments for each failure. Keep the FailedQuery / Query ratio below 1%. A sustained climb above that threshold signals a systemic issue, not a few bad queries.

The codes that dominate production incidents fall into five classes.

ClickHouse query latency P99 spikes: tail latency, hot shards, and cold data

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse query latency P99 spikes: tail latency, hot shards, and cold data

Your ClickHouse cluster shows healthy average query latency, but users report intermittent timeouts. P99 spikes while P50 stays flat. This divergence is tail latency: a subset of queries is dramatically slower. In distributed clusters, the culprit is often a single hot shard or a replica serving stale data. In tiered storage, a query touching cold S3 data can be one to two orders of magnitude slower.

ClickHouse Replica is lost: SYSTEM RESTORE REPLICA and recovering a diverged replica

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Replica is lost: SYSTEM RESTORE REPLICA and recovering a diverged replica

You run SELECT count() on two replicas and get different results. Your application returns inconsistent aggregations depending on which node answers. The replication dashboard looks green: ZooKeeper sessions are active, queue_size is zero, and no replica is readonly. The replica has permanently lost parts, but ZooKeeper does not know because the loss happened outside the replication log. Silent divergence hides behind healthy metrics until queries start returning wrong results.

ClickHouse replicas out of sync: when SELECT count() differs across replicas

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse replicas out of sync: when SELECT count() differs across replicas

You run the same SELECT count() against two replicas and get different numbers. The replica with the lower count still responds instantly, its HTTP ping returns Ok., its replication queue may even be empty, and its CPU and memory look fine. Standard liveness checks give false confidence while the replica serves stale or incomplete data.

ClickHouse replication is asynchronous and coordinated through ZooKeeper or ClickHouse Keeper. A replica can lose its session, miss fetches, or silently drop data, yet continue serving reads from local disk. Distributed queries hide the divergence unless you compare row counts or partition metadata across nodes.

ClickHouse ReplicatedDataLoss > 0: detecting and responding to lost parts

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse ReplicatedDataLoss > 0: detecting and responding to lost parts

ReplicatedDataLoss > 0 is a hard signal in ClickHouse. A nonzero value in system.events means the server has determined that a data part is missing and cannot be retrieved from any available replica. This is not replication lag, a transient fetch failure, or the normal ReplicatedPartFetchesOfMerged optimization.

Queries that touch the affected part can return incomplete results or errors. The immediate risk is silent divergence between replicas, where one replica serves stale or incomplete results without failing the query. Confirm the event, identify the scope, and determine whether a healthy peer still has the part.

ClickHouse replication lag: absolute_delay, queue_size, and catch-up diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse replication lag: absolute_delay, queue_size, and catch-up diagnosis

You notice absolute_delay climbing on one replica. SELECTs there return older rows than on peers, and failover to the lagging node risks losing recently inserted data. In ClickHouse, replication lag is not a single failure; it is a symptom with several distinct causes. A replica can fall behind because it cannot pull entries from the Keeper log, because it cannot fetch parts fast enough, because a mutation is blocking the queue, or because the source replica itself is too slow to serve data.

ClickHouse replication queue stuck: num_tries, last_exception, and dead entries

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse replication queue stuck: num_tries, last_exception, and dead entries

A replica can show low absolute_delay in system.replicas while system.replication_queue contains entries with num_tries in the hundreds and the same last_exception repeating for hours. These dead entries do not self-resolve. They block merges, fetches, or mutations, causing silent divergence and stale reads. This guide shows how to read the queue correctly, identify the failure mode from the entry type, and clear the blockage without making the replica diverge further.

ClickHouse slow queries: diagnosis from query_log to plan to fix

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse slow queries: diagnosis from query_log to plan to fix

P99 latency climbs before averages move. In ClickHouse, tail latency spikes usually mean a subset of queries is hitting cold data, scanning too many parts, or spilling to disk. The goal is to separate plan problems from resource problems fast, then confirm the fix before the next batch of queries arrives.

What this means

Slow queries show up as rising query_duration_ms in system.query_log for QueryFinish events, or rising elapsed in system.processes. ClickHouse latency is sensitive to how many parts a query opens, whether the mark cache is warm, and whether the pipeline stays in memory.

ClickHouse small inserts anti-pattern: why single-row inserts melt the merge pool

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse small inserts anti-pattern: why single-row inserts melt the merge pool

DB::Exception: Too many parts spikes, query latency climbs, and the background merge pool runs flat out while making no progress. Disk, CPU, and memory look healthy. The cause is usually single-row INSERTs from the application.

Every INSERT into a MergeTree table creates at least one immutable data part on disk. ClickHouse is optimized for batch inserts. When clients send single-row or micro-batch inserts, parts are created faster than background merges can consolidate them. This is the number one driver of part accumulation in production clusters.

ClickHouse system log tables eating disk: query_log, part_log, and TTL

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse system log tables eating disk: query_log, part_log, and TTL

You are investigating disk growth on a ClickHouse node. User tables look reasonable, but system.query_log or system.part_log are consuming tens or hundreds of gigabytes. These tables use MergeTree-family engines and are subject to the same part-count and merge pressure limits as production tables. Without a TTL rule, data accumulates forever. During incidents with high query error rates or retry storms, query_log can expand fast enough to threaten disk capacity and stall logging.

ClickHouse Table is in readonly mode: is_readonly on replicated tables and how to fix it

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Table is in readonly mode: is_readonly on replicated tables and how to fix it

Inserts fail with TABLE_IS_READ_ONLY. system.replicas shows is_readonly = 1 for affected tables and the replica rejects writes. For replicated tables, this is a coordination failure, not a disk or memory problem: the replica has lost its session with ClickHouse Keeper or ZooKeeper, or the ensemble is unreachable.

Reads from local parts may still succeed, which hides the failure from load balancers and monitoring probes that rely on SELECT 1 or the HTTP ping endpoint. Brief readonly states lasting seconds are normal during Keeper leader elections. Sustained readonly lasting minutes means the replica is diverging and inserts are being lost or routed elsewhere.

ClickHouse too many open files: file descriptors, part count, and nofile limits

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse too many open files: file descriptors, part count, and nofile limits

ClickHouse aborts queries with “Too many open files” or the server process dies. Logs show errors about failing to open column files or metadata. This is not a traditional leak; it is a capacity cliff. Every active MergeTree part keeps multiple files open, background merges temporarily spike that count, and the Linux nofile limit is usually the bottleneck. The default limit of 1024 is catastrophic for production ClickHouse.

ClickHouse too many partitions: why over-partitioning multiplies your part count

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse too many partitions: why over-partitioning multiplies your part count

You are watching MaxPartCountForPartition climb, or you have already seen DB::Exception: Too many parts. You check the table and see a modest total of active parts. The table looks healthy, but ClickHouse rejects inserts anyway.

The problem is not the table total. It is the partition boundary.

ClickHouse enforces part limits per partition, not per table. A high-cardinality PARTITION BY key like toYYYYMMDD(timestamp) or a business identifier creates thousands of isolated part pools. Parts never merge across those pools. One hundred partitions with thirty parts each produces 3,000 parts, multiplied merge-scheduling overhead, and higher file-descriptor pressure. One partition crossing the hard limit kills inserts for that partition while the rest of the table appears idle.

ClickHouse Too many simultaneous queries: max_concurrent_queries and query storms

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse Too many simultaneous queries: max_concurrent_queries and query storms

You run a query and ClickHouse returns “Too many simultaneous queries.” New connections either queue or fail outright. Queries that completed in seconds yesterday now time out. The server is not down, but it is not usable.

ClickHouse is optimized for fewer, heavier analytical queries. As concurrency rises, CPU, memory, and I/O contention increase non-linearly. A small spike can become a storm because each query is greedy.

ClickHouse TTL not deleting data: why expired rows survive and the disk fills

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse TTL not deleting data: why expired rows survive and the disk fills

A MergeTree table with a configured TTL can still accumulate disk usage when expired rows remain visible to SELECT. This is not a syntax or timezone error. ClickHouse enforces TTL only as a side effect of background merges. When merges stop or never start, TTL stops, and storage grows without warning.

What this means

ClickHouse evaluates TTL expressions only during merge operations. When the merge scheduler combines parts, it checks whether rows exceeded their TTL interval. Depending on table settings, it either drops the entire part if every row expired, or rewrites the part to exclude expired rows. Both paths require a merge. If the background merge pool is saturated, disk space is too low for temporary merge output, or a partition has no new inserts to trigger merge selection, expired rows remain indefinitely. The system does not log a warning for TTL skips; the only visible symptom is growing storage.

ClickHouse unauthorized DROP TABLE: auditing DDL and privilege anomalies

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse unauthorized DROP TABLE: auditing DDL and privilege anomalies

A production table disappears. An application returns “table does not exist.” In ClickHouse, DROP TABLE removes the table definition and MergeTree data parts from disk immediately. On replicated tables, a single ON CLUSTER command propagates through the coordination service and can erase the table everywhere before you intervene. There is no native undo.

This guide is for finding out what happened, determining blast radius, and closing the gaps.

ClickHouse ZooKeeper session has expired: causes, recovery, and tuning

Mon, 01 Jan 0001 00:00:00 +0000

ClickHouse ZooKeeper session has expired: causes, recovery, and tuning

When system.replicas shows is_session_expired = 1, the replica has lost its ZooKeeper session and stopped participating in replication. It rejects inserts, coordinated merges, and distributed DDL. Depending on quorum and load-balancer configuration, writes may shift silently to other replicas or halt for entire shards.

is_session_expired often appears alongside is_readonly = 1, but the two are distinct. Session expiration means the coordination session is dead. Read-only means the replica refuses writes. Session expiration is the most severe because it breaks the replica’s contract with the cluster.

How ClickHouse actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How ClickHouse actually works in production: a mental model for operators

ClickHouse is a column-oriented OLAP database built around the MergeTree family of table engines. It does not fail like a transactional database. Its failure modes are distinctive and almost always originate in the storage layer rather than the query layer. In production, most incidents begin as storage-structure debt: immutable data parts accumulate faster than background processes can consolidate them. By the time queries slow down or inserts fail, the underlying merge crisis has been developing for hours.