$ guides / nvidia-gpu / nvidia-gpu-unauthorized-process-cryptomining ▌

Operations Guides

Unauthorized NVIDIA GPU processes: detecting cryptomining and rogue workloads

A GPU node that should be idle shows sustained utilization. Or a scheduler reports a job cannot land because memory is held, yet no known workload is running. nvidia-smi shows a process named python3 burning 99% SM utilization on a box that has no Python workload scheduled. The question to answer fast: is this an expected workload you have lost track of, or something that should not be there at all?

Cryptominers are the common case for genuinely unauthorized GPU use, but not the only one. Orphaned jobs from another team, zombie CUDA contexts holding memory after a crash, and compromised containers all look similar from the first nvidia-smi output. The diagnosis below separates them.

One caution up front: in shared multi-tenant clusters, “unauthorized” often just means “another team’s job you were not told about.” Confirm ownership before you kill anything or page anyone.

What this means

Every process using a GPU is visible to the NVIDIA driver. nvidia-smi --query-compute-apps and nvidia-smi pmon enumerate them with PID, name, memory footprint, and utilization. The detection problem is not visibility; it is attribution. You have a PID and a command name, and you need to decide whether that process belongs.

The practical workflow is a decision tree: is there a process on the GPU, does it match the expected workload set, and if not, what is it really?

flowchart TD
  A[GPU busy or memory held] --> B[Query compute apps and pmon]
  B --> C{Process listed?}
  C -->|No, but memory used| D[Zombie context: fuser /dev/nvidia*]
  C -->|Yes| E{Matches expected workload set?}
  E -->|Yes| F[Legitimate: track attribution]
  E -->|No| G[Inspect /proc/pid/exe and cmdline]
  G --> H{Real binary and owner known?}
  H -->|Another team| I[Confirm with owner, no page]
  H -->|Renamed or unknown| J[Compromised: contain node]

Two properties make GPU cryptomining distinctive. Miners want sustained compute, so they tend to pin SM utilization near 95-100% with moderate memory use. And they disguise themselves: process names are renamed to blend in (python3, java, system-looking names), so the command column in nvidia-smi is untrustworthy. Verification always means resolving the actual binary behind the PID.

Common causes

Cause	What it looks like	First thing to check
Cryptominer (compromised host or container)	Sustained 95%+ SM util, moderate memory, unknown or renamed process, unexpected user	`ls -la /proc/<pid>/exe` to resolve the real binary
Another team’s job in a shared cluster	Normal ML-looking process, high util and memory, runs under a valid user	Scheduler records and namespace ownership before acting
Zombie CUDA context	GPU memory held, but process list is empty or shows a dead PID	`fuser -v /dev/nvidia*` for stale device handles
Container escape or rogue pod in Kubernetes	Host-namespace PID with no pod assigned to that GPU	Map host PID to container via `/proc/<pid>/cgroup`
Orphaned job after scheduler failure	Job gone from scheduler but process still on GPU	Compare `--query-compute-apps` against scheduler state

Not every miner pins the GPU flat. Some mining malware monitors GPU activity and pauses when a legitimate user is active, so utilization may be intermittent rather than a clean sustained 95%. Intermittent, unexplained utilization spikes on a node that should be idle deserve the same investigation.

Quick checks

All of these are read-only and safe to run during an incident.

# 1. Which processes are on the GPUs right now
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory --format=csv

# 2. Per-process utilization snapshot (SM% and memory%)
nvidia-smi pmon -s um -c 1

# 3. Watch per-process utilization over time
nvidia-smi pmon -s um -d 1

The pmon columns that matter here are pid, sm (SM utilization %), mem (memory controller utilization %), fb (framebuffer MB used), and command. A process at 95%+ sm with modest fb is the classic miner shape: compute-hungry, memory-light.

# 4. Resolve the real binary behind a suspicious PID
ls -la /proc/<pid>/exe

# 5. See the full command line, not the truncated name
cat /proc/<pid>/cmdline | tr '\0' ' '

# 6. Who owns it and when it started
ps -o user=,group=,lstart= -p <pid>

# 7. If nvidia-smi shows no processes but memory is still used
fuser -v /dev/nvidia*

Step 4 is the check that catches renamed miners. The command field in pmon is trivially spoofed; /proc/<pid>/exe is a symlink to the actual executable on disk. If a process calling itself python3 resolves to a binary in /tmp, /dev/shm, or a hidden directory under a user’s home, that is your answer. Note that /proc/<pid>/exe reflects the binary the process was exec’d from; process-injection techniques can complicate this, so check cmdline, open files, and network connections too.

Step 7 covers the case where nvidia-smi reports memory in use but no processes. Processes that exit uncleanly can leave device file handles and CUDA contexts behind; fuser -v /dev/nvidia* finds PIDs still holding the device nodes.

How to diagnose it

Build the expected workload set. Before judging anything, list what should be on this node: scheduler job records, Kubernetes pod assignments, systemd units, known long-running services. Detection is comparison against an expected set, and without that set every process looks suspicious.
Enumerate actual GPU processes. Run --query-compute-apps and pmon -s um -c 1. Record PID, name, memory, and SM utilization per GPU.
Diff expected vs actual. Any process not in the expected set is a candidate. Any GPU with utilization or memory but no listed process goes down the zombie-context path (step 6).
Verify the real binary. For each candidate PID: ls -la /proc/<pid>/exe, cat /proc/<pid>/cmdline, and ps -o user= -p <pid>. A valid ML framework binary under a valid service account points to a workload-attribution problem. A renamed binary, a path in a temp directory, or an unexpected user points to compromise.
Map host PIDs to containers in Kubernetes. nvidia-smi reports host-namespace PIDs, so the number means nothing to kubectl directly. Read /proc/<pid>/cgroup: on containerized nodes the cgroup path contains the container runtime ID, which you can match against crictl ps output or the pod sandbox. The red flag is a process on a GPU that is not assigned to any current pod on that node, meaning something is using the device outside the device plugin’s allocation.
Handle the empty-list case. Memory used, no processes listed: run fuser -v /dev/nvidia* to find holders of the device files. If that also comes back empty, you are likely looking at a leaked context in the driver; cross-check dmesg for XID events and plan a GPU reset or node reboot if memory must be reclaimed.
Corroborate before concluding compromise. Check for the supporting evidence: unexpected outbound network connections from the PID (mining pools need egress), a cron entry or systemd unit that respawns it, and how long it has been running (lstart from quick check 6). A miner that has survived reboots has a persistence mechanism you must find, or killing the process accomplishes nothing.
In shared clusters, confirm ownership first. Before escalating, ask the other teams. A large fraction of “unauthorized GPU process” tickets in multi-tenant environments resolve to a job someone submitted through a side channel.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Per-process SM utilization (`pmon -s um`)	Miners pin compute; this is the primary detection signal	Sustained 95%+ from a process not in the expected set
Per-process framebuffer usage	Separates compute-hungry miners (moderate memory) from ML jobs (heavy memory)	High SM util with small, static memory footprint
`--query-compute-apps` process list vs scheduler state	Attribution: the diff between them IS the alert	Any process with no matching job or pod
GPU memory used with empty process list	Reveals zombie contexts and leaked memory after crashes	`memory.used` stays high after job exit
Device file holders (`fuser -v /dev/nvidia*`)	Finds processes evading the compute-apps list	PIDs holding `/dev/nvidia*` not visible to nvidia-smi
Outbound connections from GPU processes	Miners need pool egress; legitimate training usually talks to known endpoints	GPU process with connections to unknown external hosts

Fixes

Confirmed cryptominer

Kill the process, but treat that as step one of remediation, not the fix. kill <pid> and verify in pmon that utilization drops. Then find the persistence mechanism (cron, systemd unit, container entrypoint, compromised image) and the ingress path (exposed SSH, vulnerable service, poisoned image). If you cannot explain how it got in, assume it will come back and rebuild the node from a known-good image. Rotate any credentials the node could access; a host running a miner is a compromised host.

Tradeoff: rebuilding is disruptive but is the only response that closes the question. Killing alone is fast and keeps the node in service, at the cost of near-certain recurrence.

Another team’s job

Do not kill. Record the PID and owner, contact the team, and fix the process gap that let the job run untracked. In Kubernetes this usually means someone bypassed the scheduler or the device plugin allocation; in bare-metal clusters it means the expected-workload inventory is stale.

Zombie CUDA context holding memory

If a hung process still holds the context, kill -9 <pid> releases it. Be aware this unconditionally terminates that process, so confirm it is genuinely hung and not a slow-but-alive job first. If the process is already dead and only device handles remain, a GPU reset or node reboot may be required to reclaim the memory; this is disruptive to any other jobs on that GPU, so schedule it rather than doing it mid-shift.

Rogue pod in Kubernetes

Cordon the node, capture the evidence (PID mapping, container ID, image reference) before deleting anything, then delete the pod. Audit how it was scheduled: a pod using a GPU not assigned by the device plugin implies a privileged pod, a hostPath mount of /dev/nvidia*, or a bypassed admission policy.

Prevention

Maintain the expected workload set per node. Detection is a diff; without a current baseline of what should run, you have nothing to diff against. Scheduler state is the best source of truth.
Alert on the process-list diff, not just utilization. Sustained 95% utilization is noisy (legitimate jobs do this constantly). “Process on GPU not in the expected set” is quiet and precise.
Sample frequently enough. GPU state changes in seconds; process monitoring at minute resolution misses short-lived miners that pause when observed. Sub-10-second sampling catches the intermittent pattern.
Lock down /dev/nvidia* access in Kubernetes. Disallow privileged pods and hostPath mounts of device nodes through admission policy, so GPU access can only flow through the device plugin, which makes every allocation attributable to a pod.
Watch egress from GPU nodes. Mining requires outbound connectivity. Alerting on unexpected egress destinations from GPU workloads catches miners even when the process itself is well hidden.

How Netdata helps

Netdata collects per-GPU utilization, memory used and free, and per-process GPU stats from NVML, so the compute-apps list and utilization history are already on the dashboard when you need to ask “when did this start.”
Correlating per-process SM utilization against memory footprint on one screen surfaces the miner signature (high compute, moderate memory) without manual pmon sessions.
GPU memory staying high with an empty process list is visible as a divergence between the memory-used chart and the per-process chart, pointing straight at zombie contexts.
Sub-second sampling captures intermittent miners that pause during active use, which minute-resolution monitoring smooths into invisibility.
Because Netdata also collects host CPU, network, and process metrics, you can correlate an unknown GPU process with its egress connections and parent process in the same interface instead of jumping between tools.