Kafka retention not deleting old segments: retention.ms, retention.bytes, and the active segment

You set retention.ms to 24 hours, but broker disk keeps climbing. A partition shows segment files older than the threshold, or the active segment has grown so large it consumes most of the volume. A topic with both retention.ms and retention.bytes may still appear to ignore them.

Kafka retention is not a continuous sweep. retention.ms and retention.bytes apply independently to closed segments, and the check interval adds latency. The active segment is never deleted by retention alone. On compacted topics, retention.bytes is ignored. Per-topic overrides shadow broker defaults.

This guide covers segment deletion mechanics and read-only checks to find the cause.

What this means

Kafka reclaims disk through a periodic retention check that runs every log.retention.check.interval.ms (default 5 minutes). During each run, the broker evaluates closed log segments against two independent policies:

  • Time-based: retention.ms (or log.retention.hours). A closed segment is eligible when the largest timestamp of its records exceeds the threshold.
  • Size-based: retention.bytes (or log.retention.bytes). A closed segment is eligible when the total size of all segments in the partition exceeds the threshold, from the oldest first.

These policies operate as an OR. Three rules exempt segments:

  1. The active segment – the segment currently receiving writes – is never deleted by retention. It rolls when segment.bytes, segment.ms, or the index file size limit is reached. Each offset-index entry covers roughly log.index.interval.bytes (default 4 KiB), so a 10 MiB index limits a segment to about 5 GiB. Once rolled, the segment becomes eligible in the next check cycle.
  2. After a segment is marked for deletion, file.delete.delay.ms (default 1 minute) must pass before the OS removes the file. Disk space is not freed instantly.
  3. For topics with cleanup.policy=compact (without delete), both retention.ms and retention.bytes are ignored. Compaction retains each unique key until a newer value arrives. Use max.compaction.lag.ms to bound how long an uncompacted record can persist, but that is not time retention.

Per-topic configs set via kafka-configs.sh take precedence over broker defaults in server.properties. A forgotten override means the broker default will not apply.

Because of the 5-minute check interval and the 1-minute deletion delay, retention.ms is a lower bound, not a real-time guarantee. A low-volume topic with a 10-minute retention can retain data for 20 minutes or more if the active segment has not rolled.

flowchart TD
    A[Disk growing despite retention] --> B{Topic compacted?}
    B -->|Yes| C{Policy includes delete?}
    C -->|No| D[retention.bytes ignored
check cleaner health]
    B -->|No| E{Closed segments past
retention.ms?}
    E -->|No| F[Active segment exempt
check segment roll settings]
    E -->|Yes| G{retention.bytes set?}
    G -->|Yes, not breached| H[Size limit not met
time policy still active]
    G -->|Breached / Not set| I[Wait for check interval
and file deletion delay]
    I --> J[Verify per-topic overrides
vs broker defaults]

Common causes

CauseWhat it looks likeFirst thing to check
Active segment is the only segmentPartition has one giant .log file and no closed segmentsSegment roll settings (segment.bytes, segment.ms) and write rate
Only one retention policy is breachedDisk exceeds size limit but segments are too young, or vice versaEffective config for both values
Topic is compacted without deletecleanup.policy=compact; retention.bytes ignored; disk grows with unique key countcleanup.policy and log cleaner dirty ratio
Per-topic override shadows broker defaultBroker default is 7 days but topic override is -1 or longerkafka-configs.sh --describe output
Segment rolls prematurely due to index sizeSegments roll at ~5 GiB because log.index.size.max.bytes (default 10 MiB) fills firstSegment sizes vs segment.bytes target
File deletion delay not elapsedSegments disappeared from Kafka metadata but df shows no freed spaceWait for file.delete.delay.ms; check open file handles

Quick checks

# Effective topic config
kafka-configs.sh --bootstrap-server localhost:9092 \
  --describe --entity-type topics --entity-name <topic>

# Broker defaults
grep -E 'log.retention|log.segment|cleanup.policy' /etc/kafka/server.properties

# Segment files in partition dir
ls -lth /var/lib/kafka-logs/<topic>-<partition>/

# Disk usage per log dir
kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe

# Cleaner errors in broker logs
grep -iE 'cleaner|compaction' /var/log/kafka/server.log

# Open file descriptors
ls /proc/$(pgrep -f kafka.Kafka)/fd | wc -l
cat /proc/$(pgrep -f kafka.Kafka)/limits | grep "Max open files"

How to diagnose it

  1. Confirm the effective configuration. Run kafka-configs.sh --describe for the topic. Compare retention.ms, retention.bytes, cleanup.policy, segment.ms, and segment.bytes to broker defaults in server.properties. Per-topic overrides shadow defaults completely.
  2. Identify the active segment. In the partition directory, the active segment is the highest-numbered .log file and is exempt from retention. If it is the only segment or contains data far older than retention.ms, it has not rolled. Check segment.bytes, segment.ms, and the index size limit. Low throughput keeps the segment open until segment.ms elapses.
  3. Verify closed segments are eligible. For closed segments, compare the largest timestamp in the segment against retention.ms; Kafka uses the time index, not filesystem modification time. Use kafka-dump-log.sh --files <segment>.timeindex if necessary. For size-based retention, sum the closed segment sizes and compare against retention.bytes. The policies apply independently: 10 GiB of closed segments will not delete anything if retention.bytes is 20 GiB.
  4. Check if compaction is the culprit. If cleanup.policy=compact (without delete), retention.bytes is ignored. Check the log cleaner dirty ratio via JMX (kafka.log:type=LogCleaner,name=max-dirty-percent) or broker logs. A dead cleaner lets compacted topics grow without bound.
  5. Account for check interval and deletion delay. Even eligible segments wait for log.retention.check.interval.ms and file.delete.delay.ms. Do not expect instant reclamation.
  6. Calculate runway. Multiply BytesInPerSec for the topic by retention.ms (in seconds) and the replication factor. If actual disk usage is much higher, retention or compaction is failing.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Disk space utilization on log.dirs volumesRetention failure leads directly to disk full and broker outage>75% TICKET; >90% or runway <4h PAGE
Log cleaner dirty ratio (max-dirty-percent)On compacted topics, a silent cleaner failure mimics a retention failure>50% sustained or climbing
BytesInPerSec per topicValidates whether disk growth aligns with throughput and retention mathRate exceeding retention reclaim capacity
Open file descriptor countEach segment consumes FDs; a segment explosion pressures this resource>75% of ulimit -n
Offline log directory countFinal consequence of unchecked disk growth; broker takes the directory offlineAny nonzero value PAGE
Partition count per brokerHigh partition counts amplify the active-segment headroom problem>4,000 partitions per broker

Fixes

WARNING: Altering topic configuration can delete data or cause OffsetOutOfRangeException for lagging consumers. Check consumer lag first and run during a maintenance window when possible.

Lower retention. Reduce retention.ms or retention.bytes with kafka-configs.sh --alter. This takes effect at the next check interval. Do not delete segment files manually; Kafka must remove them to keep indexes and metadata consistent.

Force faster segment rolling. Reduce segment.ms on the topic. The active segment rolls once its first record timestamp exceeds the new threshold, then becomes eligible for retention. Tradeoff: smaller segments increase file count and FD usage.

Enable size-based deletion on compacted topics. Change cleanup.policy to compact,delete. This restores retention.bytes enforcement. Tradeoff: older keys may be deleted entirely, changing semantics for consumers that expect indefinite retention.

Resolve per-topic override conflicts. Remove or lower unexpected topic overrides with kafka-configs.sh --alter. Verify with --describe.

Restart a broker with a dead cleaner. A controlled restart restarts the cleaner. Check broker logs for the underlying corrupt segment or error first, or the cleaner may die again. Tradeoff: restart evicts page cache and triggers ISR recovery.

Expand disk before adjusting retention. Add capacity to log.dirs first if you must preserve data. A full disk can take the directory offline.

Prevention

  • Treat retention.ms and retention.bytes as independent guardrails, not a combined policy. Set both and verify which triggers first for your throughput.
  • Monitor log cleaner dirty ratio and DeadThreadCount on every cluster with compacted topics, including __consumer_offsets.
  • Size segment.bytes and log.index.size.max.bytes together. If you raise segment.bytes above 5 GiB, raise log.index.size.max.bytes proportionally.
  • Keep 15-20% headroom per log.dirs volume to accommodate compaction overhead, reassignment, and the delay between eligibility and deletion.
  • Convert disk monitoring from percentage used to time-to-full by correlating BytesInPerSec with retention configuration.

How Netdata helps

Netdata exposes disk utilization per mount, Kafka JMX metrics such as BytesInPerSec per topic, and OS signals such as disk I/O latency and file-descriptor usage. Use these to calculate retention runway, detect a stuck log cleaner, and separate retention failures from producer backfills or reassignment traffic.