NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver

You run nvidia-smi and get:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

This error means the userspace side of the driver (NVML, the library nvidia-smi talks to) cannot reach the kernel side. Either the nvidia kernel module is not loaded, it is loaded but built for a different kernel than the one running, the kernel refused to load it, or the GPU itself is gone from the PCIe bus. nvidia-smi exits non-zero and prints the message to stderr; the error string is the reliable signal, not the exact exit code.

The two most common triggers in production are boring: a kernel update landed via apt or dnf and the NVIDIA module was never rebuilt for the new kernel, or Secure Boot blocked an unsigned module. The scary variant is rarer: the GPU fell off the bus (Xid 79 in dmesg), which is a hardware event and no amount of driver reinstalling will fix it.

What this means

nvidia-smi is a thin client over NVML. NVML opens the /dev/nvidia* device nodes and issues ioctls to the nvidia.ko kernel module, which in turn talks to the GPU over PCIe. When you see this error, the chain is broken at one of three points:

1. The kernel module is missing or not loaded. Nothing created the device nodes, so NVML has nothing to talk to. Typical after a kernel upgrade when DKMS did not rebuild the module, or after a driver install that never built a module at all.
2. The module exists but the kernel refused to load it. Secure Boot without an enrolled machine owner key (MOK) is the usual cause on modern distros. Loading the wrong module variant for the GPU (the open nvidia-open module on a Maxwell or Pascal card) also lands here: the module loads but refuses to drive the GPU.
3. The module is loaded but the GPU is gone. The PCIe link failed catastrophically and the device fell off the bus. dmesg shows NVRM: Xid ... 79. This is unrecoverable without a reboot, and often indicates failing hardware.

A related but different error is Failed to initialize NVML: Driver/library version mismatch. That means the kernel module and userspace library are different versions (usually a half-applied driver upgrade) and needs a different fix. Do not conflate the two.

Common causes

Quick checks

Run these in order. All are read-only and safe.

# 1. Confirm the error and note the exit code
nvidia-smi; echo "exit=$?"

# 2. Is the kernel module loaded?
lsmod | grep nvidia

# 3. Do the device nodes exist?
ls -la /dev/nvidia*

# 4. Which kernel is running, and what did DKMS build?
uname -r
dkms status

# 5. What did the NVIDIA driver (NVRM) say at boot or load time?
dmesg -T | grep -i nvrm

# 6. Any Xid errors, especially 79 (GPU fallen off the bus)?
dmesg -T | grep -i "NVRM: Xid"

# 7. Is the GPU still on the PCI bus?
lspci -d 10de:

# 8. Is Secure Boot on?
mokutil --sb-state

Interpretation shortcuts:

• lsmod empty and dkms status shows the module built for a different kernel than uname -r: classic post-upgrade breakage. Rebuild or reboot into the kernel the module was built for.
• lsmod empty and dkms status is empty: no module was ever built. The driver package may not include a kernel module at all (some headless/server metapackages are userspace-only), or kernel headers are missing so DKMS never ran.
• lsmod empty, DKMS shows the module built for the running kernel, but Secure Boot is enabled: the kernel is refusing the unsigned module. Look for signature or lockdown messages in dmesg, for example with dmesg -T | grep -iE "nvidia|lockdown|signature|module"; the exact wording varies by kernel and distribution.
• Module loaded but dmesg shows the probe failing with a message about the GPU not being supported by the open kernel module (no GSP firmware path): you installed nvidia-open on a pre-Turing GPU.
• lspci -d 10de: returns nothing, or dmesg shows NVRM: Xid ... 79: hardware-level failure. Skip to that fix section; do not reinstall anything.

How to diagnose it

Work through this decision path:

flowchart TD
  A[nvidia-smi fails: couldn't communicate] --> B{lsmod shows nvidia module?}
  B -->|No| C{dkms status: built for running kernel?}
  C -->|No or empty| D[Install headers, rebuild via DKMS or reinstall driver]
  C -->|Yes, but not loaded| E{Secure Boot enabled?}
  E -->|Yes| F[Enroll MOK or sign module]
  E -->|No| G[modprobe nvidia, check dmesg for refusal]
  B -->|Yes| H{dmesg NVRM errors?}
  H -->|Xid 79| I[GPU fell off bus: reboot, then hardware diagnosis]
  H -->|Unsupported by open module| J[Switch to proprietary module variant]
  H -->|Other Xid or hang| K[Driver crash: check Xid codes, plan reload or reboot]
  H -->|Nothing| L[Check device nodes and permissions]

1. Reproduce and capture. Run nvidia-smi and record the exit code with echo $?. Non-zero is the signal. NVIDIA documents nvidia-smi return code 9 as “NVIDIA driver is not loaded” and return code 15 as “The GPU has fallen off the bus or has otherwise become inaccessible”; record the code with the driver version rather than treating it as a general contract for every failure mode.
2. Check module state. lsmod | grep nvidia. If empty, the problem is load-side (steps 3-4). If present, the problem is probe-side or hardware-side (step 5).
3. Compare DKMS state to the running kernel. dkms status against uname -r. A mismatch after a kernel update is the single most common cause. Also confirm linux-headers-$(uname -r) (Debian/Ubuntu) or kernel-devel-$(uname -r) (RHEL/Fedora) is installed; DKMS silently does nothing without headers.
4. Check Secure Boot. mokutil --sb-state. If enabled and the module was never signed or the MOK enrollment was skipped at install time, the kernel will refuse the module. The nvidia-smi error is identical to the missing-module case, which is why this gets misdiagnosed.
5. Read NVRM messages. dmesg -T | grep -i nvrm and dmesg -T | grep -i "NVRM: Xid". This is where the driver tells you the truth: probe failures, unsupported GPU for the module variant, and Xid error codes. Xid 79 (“GPU has fallen off the bus”) is definitive for the hardware case. Other Xids (48, 95, 74) indicate different fault classes and different responses; see the Xid classification in the GPU operations material before acting on them.
6. Confirm the GPU is still on the bus. lspci -d 10de:. If the device vanished from PCIe entirely, no driver action will bring it back.
7. Rule out the near-miss errors. If you instead see Failed to initialize NVML: Driver/library version mismatch, the running kernel module and the installed userspace are different versions. That is fixed by unloading/reloading the module (usually requires stopping all GPU processes) or rebooting, not by DKMS rebuilds.

Metrics and signals to monitor

This failure is binary when it happens, but it rarely arrives without warning. These are the signals worth having before it does.

Fixes

Kernel upgraded, module not rebuilt

On Debian/Ubuntu, ensure headers for the running kernel exist, then rebuild:

sudo apt install linux-headers-$(uname -r)
sudo dkms autoinstall
sudo modprobe nvidia

On RHEL/Fedora with akmods, the module builds at boot; if it failed, check akmods output and kernel-devel for the running kernel, then force a rebuild. If a rebuild is not possible right now, booting back into the previous kernel (from the bootloader menu) restores the old module pairing immediately. Tradeoff: you stay on the old kernel until the build is sorted.

No kernel module in the installed packages

Some driver metapackages (headless/server variants) install userspace only and do not pull in DKMS or a kernel module. dkms status will be empty and no package provides nvidia.ko. Fix: install the matching DKMS/kernel-module package for your driver branch (for example the nvidia-dkms-<branch> package on Ubuntu), then dkms autoinstall and modprobe nvidia.

Secure Boot blocking the module

Two options, pick per policy:

• Enroll a MOK. Reinstall or reconfigure the driver package so it generates and signs the module, then enroll the key at the blue MOK manager screen on the next boot. This keeps Secure Boot on.
• Disable Secure Boot in UEFI. Faster, but you are lowering the boot-integrity posture of the node. On shared or compliance-bound machines this is usually not yours to decide unilaterally.

Wrong module variant for the GPU

Since the R560 driver branch, NVIDIA defaults to the open-source kernel modules, and distro packages increasingly default to nvidia-open. The open module only supports Turing and newer architectures (RTX 20xx, 30xx, 40xx, and datacenter equivalents). Maxwell, Pascal, and Volta cards require the proprietary module. If dmesg shows the open module refusing your GPU, remove the -open package variant and install the proprietary one for the same branch, then reboot.

Driver crash or wedged GPU

If the module is loaded, the GPU is on the bus, but NVRM shows a crash or the management path is wedged, first check whether any processes still hold the GPU: nvidia-smi --query-compute-apps=pid,process_name --format=csv or fuser -v /dev/nvidia*. If you can drain the node, a driver reload (modprobe -r nvidia after stopping all GPU consumers, then modprobe nvidia) may recover it. If processes are stuck in uninterruptible sleep or the module will not unload, only a reboot clears it. Do not force this during a running job; draining first is the tradeoff.

GPU fallen off the bus (Xid 79)

This is hardware. The PCIe link failed and the GPU is unresponsive; no software fix exists. The response is:

1. Confirm: dmesg -T | grep -i "NVRM: Xid" shows Xid 79; lspci -d 10de: may no longer list the device.
2. Drain workloads from the node and reboot it.
3. After reboot, check whether the GPU re-enumerates (lspci -d 10de:, nvidia-smi -L).
4. If it does not come back, the card, riser, or slot is dead: RMA or reseat/replace.
5. If it does come back, treat it as suspect. A GPU that falls off the bus once tends to repeat, usually under load. On cloud instances, stop/start (not reboot) to move to different underlying hardware.

Prevention

• Gate kernel updates on GPU nodes. On nodes with DKMS-built drivers, hold kernel upgrades or verify dkms status shows a build for the new kernel before rebooting into it. A one-line check in your reboot automation prevents the most common occurrence of this error.
• Install kernel headers with the kernel. Ensure linux-headers / kernel-devel packages track the kernel package so DKMS can build at update time.
• Decide on a Secure Boot policy before deployment. Either enroll MOK keys as part of the driver install runbook, or standardize on Secure Boot disabled for GPU nodes. The worst outcome is discovering the policy at 3 a.m.
• Pin the module variant to the GPU generation in your inventory. If your fleet includes Pascal or older cards, automation that installs nvidia-open by default will break them.
• Enable persistence mode in production (nvidia-smi -pm 1, via a systemd unit so it survives reboot). This removes the load/unload churn that produces transient reachability failures.
• Alert on Xid events, not just on nvidia-smi failure. Xid 79 and friends appear in the kernel log the moment the fault occurs; waiting for the next monitoring scrape to discover unreachability costs you minutes.

How Netdata helps

• GPU reachability as a monitored signal. Netdata collects NVML-based GPU metrics per device; when a GPU stops answering, the collection gap plus the prior metric history tells you exactly when communication broke and what the GPU was doing right before.
• Xid correlation. Netdata’s kernel log monitoring can alert on NVRM: Xid entries as they appear, so a fallen-off-the-bus event pages on the Xid itself, not on the next failed nvidia-smi scrape.
• Pre-failure context. Management-path slowness, ECC error deltas, and PCIe errors often precede a full communication failure. Per-second history for these, on the same dashboard, answers “hardware or driver?” quickly.
• Fleet-wide drift detection. Comparing driver version, persistence mode, and GPU count across nodes makes it obvious when one node diverged after a package update, which is the usual preamble to this error on the next reboot.

Related guides

• NVIDIA GPU operations guides (hub): GPU signal taxonomy, Xid classification, and failure patterns such as thermal cascades, ECC degradation, and PCIe link degradation.