NVIDIA persistence mode: why the GPU keeps re-initializing and P-state flaps

A training node accepts a job, and the first CUDA call takes two seconds before any kernel runs. The next job lands, and it happens again. Your dashboard shows the GPU bouncing between P0 and P8 all day, clocks collapsing to idle between launches, power draw dropping toward zero. It looks like throttling. It looks like the GPU is restarting constantly. It is neither: the driver is unloading GPU state every time the last CUDA context exits, and reinitializing it on the next one.

On a headless Linux GPU node without persistence mode, this is the default behavior. When no process holds the GPU open, the driver deinitializes the device. The next CUDA client pays the full initialization cost again, on the order of 1-3 seconds per GPU, largely due to ECC scrubbing behavior. Between launches the GPU drops to its idle performance state, which is what produces the P-state flapping in your dashboards.

The fix is one setting, but it has to survive reboots and driver reloads, and on modern driver branches the legacy way of setting it is deprecated in favor of a daemon. This guide covers how to confirm the diagnosis, which fix to apply, and how to monitor persistence mode as a configuration-compliance signal so it does not silently regress.

What this means

Persistence mode controls whether the NVIDIA kernel driver keeps GPU state loaded even when no compute contexts are active. It is a Linux-only concern, and it is per-device on multi-GPU systems.

Without persistence mode:

• Every new CUDA context pays init latency. The first CUDA call of each workload blocks for 1-3 seconds while the driver initializes the GPU. For batch schedulers running many short jobs, this is pure overhead on every job.
• The GPU drops to a low P-state between launches. P8/P12 at idle is normal power saving, but without persistence the GPU oscillates between P0 under load and deep idle states between every context. Dashboards show clocks and power collapsing repeatedly, which masquerades as thermal or power throttling.
• Monitoring has gaps. Metric collectors that sample via NVML can see brief unreachability or stale values while the driver unloads and reloads between jobs.
• Scheduler race conditions appear. Job launchers that probe the GPU before assigning work can race with driver deinitialization.

With persistence mode enabled, the driver holds the GPU initialized. The first CUDA call of a new process costs milliseconds instead of seconds, and NVML queries are consistently fast. Note that persistence mode does not pin the GPU at P0: an idle GPU still downclocks to P8 as normal power management. What changes is that the driver stays loaded, so the re-init cost and the unload/reload monitoring gaps disappear.

There are two mechanisms, and which one you should use depends on your driver branch:

1. Legacy persistence mode: nvidia-smi -pm 1. Simple, but NVIDIA has marked this solution as near end-of-life and deprecated it on some driver branches. On Tesla/datacenter drivers starting with version 535, settings applied this way are not preserved across restarts, and some driver builds log a kernel warning telling you to use the daemon instead.
2. The nvidia-persistenced daemon: a small userspace process that holds the GPU device files open, keeping the driver state loaded. This is the actively developed successor and the recommended path on current drivers.

Common causes

The last row matters. P-state flapping between P0 and P8 between jobs is the persistence symptom. Clock reduction during active compute with a throttle reason flag set is real throttling, and persistence mode will not fix it.

Quick checks

All of these are read-only unless noted.

# Is persistence mode on, per GPU?
nvidia-smi --query-gpu=persistence_mode --format=csv,noheader
# Expected in production: "Enabled" for every GPU

# Current P-state: is it dropping to P8/P12 between launches?
nvidia-smi --query-gpu=pstate --format=csv,noheader
# P8/P12 while a workload is mid-flight is a different problem (underperformance).
# Flapping between P0 and P8 across job boundaries is the persistence symptom.

# Time the first CUDA touch of a cold context.
# Without persistence this shows the 1-3s init; with persistence it is milliseconds.
# Simplest proxy without writing CUDA code: time a fresh nvidia-smi query.
time nvidia-smi --query-gpu=gpu_name --format=csv,noheader -i 0

# Is the modern daemon installed and running?
systemctl status nvidia-persistenced

# If the daemon runs, does its unit pass --persistence-mode?
systemctl cat nvidia-persistenced | grep -i ExecStart

# Look for the deprecation warning telling you to move off -pm:
dmesg -T | grep -i "NVRM.*persist"
# Typical line: "NVRM: Persistence mode is deprecated and will be removed
# in a future release. Please use nvidia-persistenced instead"

# Rule out real throttling as the cause of low clocks:
nvidia-smi --query-gpu=clocks_event_reasons.active --format=csv,noheader

One nuance on nvidia-smi -pm 1 as a check: running it to test whether the legacy path still works is a state change, not a read. On some driver branches it can fail with “Unknown Error”. That failure is itself diagnostic: use nvidia-persistenced on those branches.

How to diagnose it

Work through this in order. Most cases resolve at step 2.

1. Confirm the symptom pattern. Look at P-state, clock, and power history for the GPU. The persistence signature is oscillation tied to job boundaries: P0 and full clocks during a workload, P8 and near-zero power in the gaps, repeating all day. If clocks are depressed while utilization is high, stop here and investigate throttle reasons instead.

2. Check current persistence state. Run the persistence_mode query for every GPU. Disabled on a production compute node confirms the diagnosis.

3. Check whether something is supposed to be enabling it. Look for a systemd unit, init script, or config-management rule that runs nvidia-smi -pm 1 or starts nvidia-persistenced. A very common finding: someone ran nvidia-smi -pm 1 interactively months ago, it worked, and the setting silently disappeared at the next reboot or driver reload. Persistence mode does not persist across reboots on its own.

4. Determine which mechanism your driver branch supports. Check dmesg for the NVRM: Persistence mode is deprecated warning. If present, or if nvidia-smi -pm 1 returns “Unknown Error”, the legacy path is gone or broken on this branch and you need nvidia-persistenced. On datacenter drivers from version 535 onward, assume the daemon is the correct path and that legacy settings will not survive restarts.

5. If the daemon is installed but the GPU still deinitializes, inspect the unit’s ExecStart line. Some distro packages start nvidia-persistenced with --no-persistence-mode by default, which means the daemon runs but does not actually keep the GPU initialized. The fix is adding --persistence-mode to the unit.

6. Verify the fix end to end. Enable persistence, then watch P-state and first-call latency across several job launches. The first CUDA call of each new process should drop from seconds to milliseconds, and monitoring gaps between jobs should disappear. The GPU will still sit at P8 during genuine idle time; that is normal and not the regression you fixed.

flowchart TD
    A[P-state flaps, 1-3s first CUDA call] --> B{persistence_mode enabled?}
    B -- No --> C{Driver branch still supports -pm 1?}
    B -- Yes --> D[Symptom is not persistence. Check clocks_event_reasons for real throttling]
    C -- Yes --> E[Enable, but persist via boot config]
    C -- No, deprecated or Unknown Error --> F[Use nvidia-persistenced with --persistence-mode]
    E --> G[Survives reboot? Add systemd unit or config mgmt]
    F --> G
    G --> H[Monitor persistence_mode as config-compliance signal]

Metrics and signals to monitor

Treat persistence_mode as part of a broader control-plane configuration-drift check. The same query snapshot that reads persistence mode can also read compute_mode, ecc.mode.current, mig.mode.current, and power limits, all of which drift for the same reasons: reboots, driver reloads, and incomplete automation. See the monitoring checklist for the full configuration baseline.

Fixes

Enable persistence mode the right way for your driver

Modern path (recommended): nvidia-persistenced. The daemon ships with the driver at /usr/bin/nvidia-persistenced on most distributions, with a systemd unit included in the driver package:

# Enable and start the daemon
systemctl enable --now nvidia-persistenced

# Verify the unit starts it in persistence mode
systemctl cat nvidia-persistenced | grep ExecStart
# You want: /usr/bin/nvidia-persistenced --persistence-mode ...
# If the unit lacks the flag, add a drop-in:
systemctl edit nvidia-persistenced
# [Service]
# ExecStart=
# ExecStart=/usr/bin/nvidia-persistenced --persistence-mode

The empty ExecStart= line before the replacement is required when overriding a unit’s command via a drop-in. This is a safe change: the daemon only holds device files open.

Legacy path (older branches only): nvidia-smi -pm 1. This works on branches where it is not yet deprecated, but it is a runtime setting. It is lost on reboot and on driver reload, so it must be reapplied from a boot-time unit or config management. Do not rely on a one-time interactive invocation. If your fleet spans driver branches, standardize on the daemon and skip the legacy path entirely.

Tradeoff: a GPU held initialized draws slightly more idle power than one allowed to fully deinitialize. For production compute nodes this is almost always the right trade; for power-sensitive idle capacity, weigh it deliberately.

VM passthrough caveat

If the same host also passes GPUs through to VMs, do not leave nvidia-persistenced holding a device file open for a GPU you are about to assign to a VM. Operators have reported host crashes in that combination. Stop the daemon before VM start and restart it after the GPU is released; a libvirt hook is the standard pattern. Bare-metal and container-only GPU nodes are unaffected.

Prevention

• Manage it as code. Persistence enablement belongs in the same config-management path that installs the driver, not in a runbook step someone runs by hand. If it is not in automation, it will regress at the next reboot.
• Standardize on the daemon. The legacy -pm path is near end-of-life per NVIDIA’s own documentation, and its behavior already differs across branches (deprecation warnings, “Unknown Error” failures, settings not surviving restart on 535+ datacenter drivers). One mechanism, fleet-wide, removes a class of per-node variance.
• Alert on configuration drift, not symptoms. Poll persistence_mode per GPU and ticket when it reads Disabled on a production node. This catches the regression at reboot time instead of after users notice slow job starts.
• Separate the lookalikes in your dashboards. Annotate or gate P-state and clock panels so that idle-state drops between jobs are visually distinct from throttle-reason-driven clock reduction during compute. The former is a config issue; the latter is thermal or power. For the broader signal taxonomy, see how an NVIDIA GPU actually works in production and the monitoring maturity model.

How Netdata helps

• Per-second P-state, clock, and power collection makes the flap pattern obvious: minute-resolution monitoring smears the P0/P8 oscillation into an average that looks like mild underperformance.
• Persistence mode state per GPU, collected alongside the rest of the NVML configuration snapshot, turns this into a compliance check instead of a post-incident discovery.
• Throttle-reason correlation lets you rule out hw_thermal_slowdown and sw_power_cap in the same view where you see the P-state behavior, so you do not chase a cooling problem that does not exist.
• Driver-health latency signals (NVML query response time) surface the per-context init cost and any monitoring gaps caused by driver unload between jobs.
• Reboot and driver-reload correlation ties a persistence_mode flip back to the exact event that caused it, which is usually a reboot or driver update, not a human change.

Related guides

• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA GPU monitoring checklist: the signals every production GPU fleet needs
• NVIDIA GPU monitoring maturity model: from survival to expert