Docker Operations Guides on Netdata

Docker commands hang: docker ps, inspect, and exec freezes

Mon, 01 Jan 0001 00:00:00 +0000

Docker Commands Hang: docker ps, inspect & exec Freezes

docker ps, docker inspect, and docker exec hang while containers continue serving traffic. This is a Docker daemon hang: the management plane is dead while the data plane survives. Standard process monitors show dockerd as alive, yet you cannot manage, inspect, or evacuate workloads.

This guide covers how to distinguish a hang from a crash, identify whether the root cause is storage, a stuck shim, a plugin, or an internal deadlock, and recover without unnecessary host reboots or container kills.

Docker container cannot connect to another container

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container Cannot Connect To Another Container

Connection timeouts or “connection refused” between containers on the same host point to a failure in one of four layers: the bridge network, Docker’s embedded DNS, iptables, or the application bind address. Inter-container networking relies on Linux bridges, veth pairs, iptables rules, and an embedded DNS resolver at 127.0.0.11. This guide isolates the faulty layer and fixes it.

What This Means

Docker isolates each container in its own network namespace. Containers on the same custom bridge communicate through a Linux bridge and resolve each other by name via Docker’s embedded DNS. The default bridge network has no embedded DNS. If containers are on different networks, if the target application binds to 127.0.0.1, or if iptables rules were wiped by an external firewall reload, traffic stops. The symptom appears at the application layer, but the root cause can be Layer 2, Layer 3, or the application itself.

Docker container cannot connect to the internet: diagnosis and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Docker container cannot connect to the internet

The host may be online while Docker’s bridge, iptables rules, DNS proxy, or connection tracking table is in a broken state. Applications fail with connection timeouts, package managers stall, and external health checks return unhealthy. Isolate whether the failure is DNS, routing, packet filtering, or daemon state before fixing it.

What this means

Outbound connectivity from a container traverses the container’s network namespace, a veth pair attached to a bridge (docker0 or user-defined), iptables NAT and filter rules managed by Docker’s libnetwork, the host’s routing table, and the upstream physical interface. On user-defined networks, Docker’s embedded DNS resolver at 127.0.0.11 proxies queries to the host’s configured resolvers. On the default bridge network, there is no embedded DNS; containers inherit the host’s /etc/resolv.conf directly. A failure at any layer produces the same symptom: requests time out.

Docker container exits immediately: how to diagnose it

Mon, 01 Jan 0001 00:00:00 +0000

Docker container exits immediately: how to diagnose it

A container that exits immediately prints its ID during docker run, appears briefly in docker ps, then vanishes. docker ps -a lists it as Exited with an uptime measured in seconds. Unlike a restart loop, it stays stopped.

A Docker container is a Linux process wrapper, not a virtual machine. Its lifecycle is bound to PID 1 in the container’s namespace. When that process completes, crashes, or is killed, the container exits. An immediate exit means the process finished its work, failed to start, or was terminated during initialization. This guide gives you a diagnostic flow to find out why PID 1 terminated, interpret the exit code, and fix the root cause.

Docker container high CPU usage: causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Docker container high CPU usage: causes and fixes

A container alert for high CPU is easy to generate but hard to interpret. Docker reports CPU as a percentage of total host cores, so a value of 340% on an 8-core machine simply means the container is using 3.4 cores. That might be expected for a multi-threaded workload, or it might signal a runaway process, CFS throttling, or a cryptominer. This guide shows how to distinguish legitimate compute from pathological behavior, and how to fix the root cause without restarting the container blindly.

Docker container high memory usage: how to diagnose it

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container High Memory Usage: How To Diagnose It

Your container is sitting at 90% of its memory limit but has not been OOMKilled. Or it is being killed repeatedly and you cannot tell whether the limit is too low or the application is leaking. docker stats shows a single percentage, but that number mixes reclaimable page cache with anonymous memory that the kernel cannot reclaim. To diagnose this correctly, you need to decompose cgroup memory.stat, map it to your runtime’s actual allocations, and decide whether the problem is cache pressure, a runtime mismatch, or a true leak.

Docker container keeps restarting: causes, checks, and fixes

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container Keeps Restarting: Causes, Checks & Fixes

A container that restarts every few seconds is not self-healing. It is a crash loop that wastes CPU, floods logs, and masks the real failure. Docker’s restart policy can hide whether the application is OOM-killed, segfaulting, or waiting for a dependency that never arrives. This guide shows how to read the signals, map exit codes to causes, and stop the loop before it degrades the host.

Docker container memory leak: how to find one and prove it

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container Memory Leak: How To Find One And Prove It

Memory that only ever climbs is easy to spot. The harder problem is proving whether the growth is a leak, unbounded caching, or a limit set below the working set. During an incident, operators need to decide in minutes whether to page an on-call developer or bump a cgroup limit. This guide shows how to use cgroup memory.stat, process-level RSS, and container restart patterns to build a defensible diagnosis. You will be able to separate anonymous memory growth from reclaimable cache, identify whether the leak lives in application heap or runtime overhead, and present evidence that justifies either a code fix or a capacity change.

Docker container running but unhealthy: how to diagnose health check failures

Mon, 01 Jan 0001 00:00:00 +0000

Docker Container Running But Unhealthy: How To Diagnose Health Check Failures

The container is up. docker ps shows it running. But the status is (unhealthy) and your load balancer or orchestrator has stopped sending traffic. This means Docker’s health check command is returning a non-zero exit code, or timing out, while the container’s main process stays alive. The container is running, but Docker does not consider it ready.

An unhealthy state is not a crash. In Swarm, it triggers replacement. In Compose, depends_on with condition: service_healthy blocks downstream services. Even on a single host, an unhealthy mark often precedes a restart loop that buries the real error in noise. You need to distinguish between a broken application, a broken probe, and a broken runtime configuration.

Docker CPU throttling: the hidden cause of container latency

Mon, 01 Jan 0001 00:00:00 +0000

Docker CPU Throttling: The Hidden Cause Of Container Latency

Your application latency just spiked. p99 response times doubled or tripled. CPU dashboards show the container at 40% utilization. Memory is fine. Network is quiet. You restart the container, redeploy, or blame the code, but the pattern repeats.

The culprit is often CPU throttling. Docker uses Linux CFS bandwidth control to enforce CPU limits in discrete 100ms periods. A container can exhaust its quota in a burst, spend the rest of each period throttled by the kernel, and still report a modest average CPU over a longer window. This guide shows how to confirm throttling from cgroup metrics, calculate its severity, and fix it without guessing.

Docker daemon not responding: how to troubleshoot a hung dockerd

Mon, 01 Jan 0001 00:00:00 +0000

Docker Daemon Not Responding: How To Troubleshoot A Hung Dockerd

A hung Docker daemon is one of the more disorienting production failures you can face. The process is still running, your containers are still serving traffic, but every docker command hangs. You cannot inspect, stop, or create containers. Deployments stall. Automation times out.

This guide covers the diagnostic ladder from socket probe to storage driver investigation, explains when to wait versus when to restart, and describes what to avoid when you are not sure what is wrong.

Docker disk space full: how to troubleshoot /var/lib/docker

Mon, 01 Jan 0001 00:00:00 +0000

Docker Disk Space Full: How To Troubleshoot /var/lib/docker

You notice deployments failing with “no space left on device,” image pulls hanging, or the Docker daemon becoming sluggish. On a Docker host, everything lives under /var/lib/docker: image layers, container writable layers and logs, named volumes, and build cache. When this filesystem fills, the failure is cascading and abrupt. New containers cannot start, running containers may fail on writes, and daemon operations deadlock.

Docker DNS not working inside containers

Mon, 01 Jan 0001 00:00:00 +0000

Docker DNS not working inside containers

Your application logs show connection timeouts. Health checks against dependency names are failing. curl from inside a container returns “Could not resolve host” while the same name resolves fine on the host. DNS inside Docker is not a simple passthrough to the host resolver. It is a stack of namespace-specific forwarders, embedded resolvers, and inherited configuration that breaks in specific, repeatable ways.

This guide will help you determine whether the failure is a missing embedded DNS, a poisoned resolv.conf, an upstream forwarding issue, or a version regression. You will be able to distinguish between inter-container name resolution failures and external lookup failures, identify the root cause with safe read-only checks, and apply the correct fix without guessing.

Docker exit code 1: application errors and how to find them

Mon, 01 Jan 0001 00:00:00 +0000

Docker Exit Code 1: Application Errors And How To Find Them

A container exiting immediately with code 1 is a common production incident. Unlike exit code 137 (OOM killer) or code 125 (Docker daemon error), code 1 means the application inside the container called exit(1). Docker is only reporting what PID 1 did.

The challenge is that code 1 is a catch-all. It can mask an unhandled JavaScript exception, a Python import error, a missing configuration file, a shell script failing under set -e, or a Go binary that cannot reach its database. Time spent checking Docker daemon health or host memory is wasted when the real failure is an application-level error written to stdout or stderr.

Docker exit code 137: OOMKilled or SIGKILL?

Mon, 01 Jan 0001 00:00:00 +0000

Docker Exit Code 137: OOMKilled Or SIGKILL?

A container exits with code 137. Docker restarts it, or it stays down, and you need to know why. The number itself only tells you that the process received SIGKILL. What matters for your next step is whether the kernel’s cgroup OOM killer fired because the container exceeded its memory limit, or whether an external actor sent the signal. The remediation for an undersized memory limit is completely different from fixing a misconfigured stop timeout or an orchestrator sending a premature kill. This guide shows how to classify the cause in under a minute using only the Docker CLI and cgroup files.

Docker exit code 143: SIGTERM and graceful shutdown failures

Mon, 01 Jan 0001 00:00:00 +0000

Docker exit code 143: SIGTERM and graceful shutdown failures

You are reviewing container exit codes after a deployment or node drain and see 143. If your monitoring alerts on it, you might think something failed. Exit code 143 is not an error. It is 128 plus signal 15 (SIGTERM), and it means the container’s PID 1 process received SIGTERM and exited voluntarily. This is exactly what docker stop is designed to do.

Docker image cleanup: safe pruning strategies for production hosts

Mon, 01 Jan 0001 00:00:00 +0000

Docker image cleanup: safe pruning strategies for production hosts

When df -h /var/lib/docker shows 87% utilization, the urge to run docker system prune -a is strong. On production hosts, that is a mistake. Cleanup is not about finding the single command that reclaims the most space. It is about knowing exactly what each flag deletes, what it leaves behind, and which filters prevent a 3 a.m. image re-pull because a base layer was removed.

Docker image pull failures: registry, network, and auth diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

Docker Image Pull Failures: Registry, Network & Auth Diagnosis

What This Means

When you run docker pull, the daemon negotiates a TLS connection to the registry, authenticates if required, resolves the manifest for the requested tag and architecture, then downloads missing layers. If any step fails, the pull aborts.

Registry errors surface as HTTP 429 or 401/403 responses. Network errors appear as timeouts, connection resets, or TLS handshake failures. Local problems such as a full disk or a hung daemon can also abort a pull even when the registry is healthy. In orchestrated environments, a single node’s pull failure can trigger the scheduler to retry on other nodes, turning a localized auth error into a cluster-wide rate limit storm. Distinguishing these layers quickly is the difference between a five-minute fix and a prolonged outage.

Docker JVM memory tuning: heap, off-heap, and the cgroup mismatch

Mon, 01 Jan 0001 00:00:00 +0000

Docker JVM Memory Tuning: Heap, Off-Heap, And The cgroup Mismatch

Your Java container is OOMKilled at 02:00. Heap usage is 60%. Docker reports exit code 137 and OOMKilled: true. The JVM never threw an OutOfMemoryError.

In a container, the kernel enforces memory limits through cgroups, but the JVM heap is only one component of process RSS. Off-heap memory, metaspace, thread stacks, direct byte buffers, and GC overhead all count against the same cgroup limit. When total RSS crosses that limit, the kernel kills the container without a JVM-level error. In some JDK and kernel combinations, the JVM fails to detect the cgroup limit entirely and sizes the heap against host RAM, which guarantees an OOM kill.

Docker log rotation: preventing json-file logs from filling disk

Mon, 01 Jan 0001 00:00:00 +0000

Docker Log Rotation: Preventing json-file Logs From Filling Disk

Docker’s default json-file log driver appends every line of container stdout and stderr to a JSON file on the host under /var/lib/docker/containers/<id>/. Without size limits, that file grows monotonically. A single verbose container can consume tens of gigabytes, and because the driver is the default, this often happens silently until /var/lib/docker fills. At that point image pulls fail, container creates are rejected, and the daemon may hang on storage operations. This guide covers how to cap log files with daemon.json and per-container log-opt overrides, verify the caps are working, and choose a different driver when json-file is not appropriate.

Docker logs taking too much disk space: how to fix log growth

Mon, 01 Jan 0001 00:00:00 +0000

Docker Logs Taking Too Much Disk Space: How To Fix Log Growth

The default json-file logging driver captures every line a container writes to stdout and stderr and appends it to a file under /var/lib/docker/containers/. There is no size limit out of the box. A container with verbose logging, a stuck retry loop, or a crash spiral can grow its log file by gigabytes per day until the filesystem is full.

Docker memory limits: how to set them and what happens when they hit

Mon, 01 Jan 0001 00:00:00 +0000

Docker memory limits: how to set them and what happens when they hit

A container without a memory limit can consume all available RAM, force the kernel to reclaim page cache, push the system into swap, and eventually trigger a host-level OOM kill that takes down the Docker daemon or other critical processes. Setting --memory is not enough: limits can be silently ignored, misread by runtimes, or masked by swap behavior that turns a clean failure into a slow crawl.

Docker memory usage explained: anonymous, file, slab, and what counts

Mon, 01 Jan 0001 00:00:00 +0000

Docker memory usage explained: anonymous, file, slab, and what counts

You look at docker stats, see a container sitting at 1.2 GB of a 1.5 GB limit, and assume it is about to explode. It might be fine. That total includes page cache the kernel will reclaim the second another process asks for memory. Meanwhile, a different container reports 600 MB with no limit set, yet its anonymous memory grows 50 MB per hour and will force a host-level OOM kill before lunch.

Docker monitoring checklist: the signals every production host needs

Mon, 01 Jan 0001 00:00:00 +0000

Docker Monitoring Checklist: The Signals Every Production Host Needs

Production Docker incidents rarely look like Docker problems at first. They show up as application latency, deployment failures, or hosts that suddenly refuse to schedule containers. By the time you notice, the daemon may be hung, a log file has filled the disk, or a container has been silently throttled into unusable latency. This checklist groups the essential production signals into three priority tiers: must-have alerts that keep the host alive, should-have metrics that expose resource pressure before it becomes an outage, and nice-to-have security and internal signals for mature environments. Every signal includes where to read it from the raw cgroup filesystem or the Docker API so you can instrument hosts without guessing paths.

Docker OOMKilled: causes, detection, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

Docker OOMKilled: causes, detection, and prevention

A container exits with code 137 and restarts. The application loses in-memory state. Dependent services start failing. The restart loop begins. This is the OOMKilled pattern, and it is one of the most common and most misdiagnosed failure modes in Docker environments.

This article covers how to confirm an OOM kill, distinguish it from an external SIGKILL, understand why it happened, and prevent recurrence. It also covers the JVM-in-container memory mismatch, which is responsible for a large share of OOM kills in Java workloads.

Docker port binding: address already in use

Mon, 01 Jan 0001 00:00:00 +0000

Docker port binding: address already in use

A docker run -p 8080:80 or docker compose up fails with bind: address already in use. The error is clear, but the owner is not. ss may show a system service. docker ps may show nothing, yet Docker still refuses. The port can appear free while an orphaned DNAT rule or a split firewall backend blocks the bind.

Distinguish a genuine socket conflict from an orphaned DNAT rule, a rootless Docker regression, and a WSL2 iptables/nftables split brain. Then reclaim the port and prevent recurrence.

Docker published port not reachable: troubleshooting -p and EXPOSE

Mon, 01 Jan 0001 00:00:00 +0000

Docker published port not reachable: troubleshooting -p and EXPOSE

You mapped a port with -p 8080:80, but curl against the host IP returns connection refused. docker ps shows the mapping, the container is running, and the port still appears closed.

A published port depends on three layers: a runtime mapping rule (-p), a host forwarding path (iptables DNAT and FORWARD policy), and an application listener inside the container bound to an interface that receives the forwarded packet. EXPOSE in a Dockerfile is metadata. It does not publish ports, create firewall rules, or set bind addresses.

Docker socket security: why /var/run/docker.sock is root access

Mon, 01 Jan 0001 00:00:00 +0000

Docker socket security: why /var/run/docker.sock is root access

Mounting /var/run/docker.sock into a container grants that container a root-equivalent privilege boundary. This is not a Docker vulnerability. The daemon runs as root, listens on a filesystem socket, and trusts any client that can write to it.

Symptoms of abuse look like container escape or host compromise, but the container never escaped. It asked the root-owned daemon to perform privileged operations on its behalf. This guide explains the mechanism, how to audit exposure, and what constraints you can apply without rebuilding your pipeline.

Docker volume cleanup: finding and removing orphaned volumes

Mon, 01 Jan 0001 00:00:00 +0000

Docker volume cleanup: finding and removing orphaned volumes

Run docker system df and the Local Volumes line keeps growing. Run docker system prune and you reclaim images and build cache, but the volume count barely drops. A few weeks later the disk alert fires again.

Data persists in /var/lib/docker/volumes/ after its consumer is long gone. These volumes waste disk, complicate capacity planning, and can hide sensitive data in forgotten corners of the filesystem.

Reading docker system df: where Docker disk usage actually lives

Mon, 01 Jan 0001 00:00:00 +0000

Reading docker system df: Where Docker Disk Usage Actually Lives

You run docker system df but the numbers do not add up to what df -h reports. Maybe the Build Cache row is empty while /var/lib/docker/buildkit/ consumes tens of gigabytes. Maybe RECLAIMABLE is high but docker system prune barely frees space because overlay2 layer sharing masks the real unique cost. Or the daemon returns Error response from daemon when the disk is already full. This guide shows how to read docker system df precisely, what it hides, and how to triage the real consumers on the host.