NVIDIA driver and CUDA version mismatch: subtle failures and crashes

The loud version of this problem is easy: a job dies at startup with CUDA driver version is insufficient for CUDA runtime version, and the fix is a driver upgrade. The version that ruins your week is quieter. A kernel update goes out fleet-wide, the NVIDIA module rebuilds on most nodes but not all, and now a subset of training jobs crash with kernel-launch failures or, worse, run to completion with subtly wrong results. Nothing in your dashboards looks broken: GPUs up, memory fine, temperatures normal.

Driver and CUDA version skew can produce all four outcomes at once: hard startup failures, mid-run crashes, performance regressions, and silent incorrect results. The fix depends less on the GPU than on your change-management discipline, because the mismatch almost always arrives through an update: a kernel bump, a driver branch change, a container toolkit upgrade, or a new framework image that expects a newer driver than the host has.

This guide covers how the compatibility model actually works, how to tell the loud failure from the subtle ones, and how to keep version skew from reaching production.

What this means

There are two version axes operators routinely conflate:

• The kernel driver (the kernel module plus the userspace driver library, libcuda.so). This is what nvidia-smi reports as “Driver Version”.
• The CUDA runtime/toolkit the application was built against, either installed on the host or shipped inside a container image. This is what nvcc --version or torch.version.cuda reports.

The “CUDA Version” shown in the nvidia-smi header is neither of these. It is the driver API compatibility version: the maximum CUDA version the installed driver can support. It tells you nothing about which toolkit is installed, and misreading it is the single most common source of confusion in this failure mode.

The compatibility rules, as of mid-2026:

• CUDA 13.x requires a minimum driver of 580.
• CUDA 12.x requires a minimum driver of 525, and minor-version compatibility is capped below 580.
• CUDA 11.x requires a minimum driver of 450, capped below 525.

A newer driver runs applications built against older CUDA versions. The failure direction is a toolkit that is newer than the driver. When that gap is large, you get the loud error. When the gap is small or the skew is elsewhere (kernel module vs userspace library, host driver vs container libraries), you get the subtle failures: CUDA error: unspecified launch failure, XID 13/31/32 events, numerically wrong results, or a training run that is 15% slower for no visible reason.

flowchart TD
  A[CUDA job fails or misbehaves] --> B{Exact error string?}
  B -->|"driver version is insufficient"| C[Driver too old for toolkit
upgrade driver or pin toolkit]
  B -->|"unspecified launch failure / illegal access"| D[Check dmesg for XID 13/31/32]
  B -->|"no error: wrong results or slow"| E[Suspect driver bug or silent corruption]
  D --> F{Module vs userspace skew?}
  F -->|yes| G[Reload or reboot after update]
  F -->|no| H{Container in play?}
  H -->|yes| I[Check container toolkit
package version matching]
  H -->|no| J[Check driver release notes
for known kernel-pattern bugs]
  E --> J

Common causes

Two of these deserve extra detail because they are easy to miss.

Architecture deprecation is a version mismatch too. CUDA 13.0 removed support for Maxwell, Pascal, and Volta GPUs, and the R580 branch is the last driver branch to support them. A fleet with mixed generations can have a “correct” driver on paper and still strand older cards when the toolkit moves forward.

Driver bugs that only surface under specific kernel patterns are real. One reported example: RTX 5090 cards on the open R595 driver corrupt the GPU front end (XID 13, XID 32) within seconds of bursty small-kernel launches in PyTorch, while other workloads on the same driver look fine. Similarly, CUDA kernels using tensor-core sparsity via mma.sp PTX on Hopper can intermittently produce incorrect results on certain R535 releases. The operational signature is identical in both cases: the failure follows the workload change, not the infrastructure change, and the root cause is the driver.

Quick checks

All read-only and safe to run during an incident.

# Driver and firmware versions on this node
nvidia-smi --query-gpu=driver_version,vbios_version --format=csv,noheader

# Kernel module actually loaded (compare against driver_version above)
cat /proc/driver/nvidia/version
lsmod | grep nvidia

# Device nodes present and driver talking to the GPU
nvidia-smi -L
ls -la /dev/nvidia*

# Toolkit version the app was built against (host install, if present)
nvcc --version

# What the framework sees (PyTorch example)
python -c "import torch; print(torch.__version__, torch.version.cuda)"

# XID history: version skew commonly surfaces as 13, 31, 32
dmesg -T | grep -i "NVRM: Xid"

# Container toolkit packages: must match exactly
dpkg -l | grep -E "nvidia-container|libnvidia-container"

How to diagnose it

1. Capture the exact error string before doing anything else. “CUDA driver version is insufficient for CUDA runtime version” is a compatibility-table problem. “Unspecified launch failure” and “illegal memory access” point at XID correlation first. No error at all, with wrong results, points at driver-bug or silent-corruption territory. Each path is different; do not skip to upgrading the driver.

2. For the loud error, check the compatibility table. Compare the installed driver (nvidia-smi) against the minimum for the toolkit the application was built with (nvcc --version, the framework’s reported CUDA version, or the container image’s tag). Remember that the “CUDA Version” in the nvidia-smi header is the driver’s ceiling, not the toolkit.

3. For post-update failures, check module vs userspace skew. After a driver package update without a reboot, the loaded kernel module can be from the old version while userspace libraries are from the new one, or the reverse after a partial rollback. Compare cat /proc/driver/nvidia/version against the nvidia-smi driver version. On nodes that just rebooted into a new kernel, verify the NVIDIA module actually rebuilt for that kernel; a failed DKMS rebuild leaves the node with no module at all, which presents as a driver-communication failure rather than a version error. See NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver for that path.

4. Correlate XID events with the version story. XID 32 (invalid or corrupted push buffer) is commonly triggered by toolkit/driver mismatch; XID 13 and XID 31 can also surface from version skew, though they more often indicate application bugs. The distinguishing test: did these XIDs start appearing on nodes immediately after a driver, kernel, toolkit, or container-image change, with no application code change? If yes, suspect skew before suspecting the code.

5. For containers, check the toolkit layer separately. Since container toolkit v1.18.0, the libnvidia-container* library versions must exactly match the nvidia-container-toolkit* package versions, and mismatched package versions violate the toolkit’s supported package matrix. Mounting CUDA compat libraries from the container was disabled by default in v1.17.4 and re-exposed via a dedicated compat hook in v1.17.5; if your images rely on forward compatibility, a container toolkit upgrade can silently remove the mechanism that made them work.

6. For “runs but wrong” or “runs but slow,” treat the driver as a suspect, not a bystander. Pin the job to the previous known-good driver on one node and compare outputs and step time. Check the toolkit and driver release notes for known silent-corruption issues in your version range; for example, NVIDIA documented a cuBLAS issue, introduced in cuBLAS 12.6 Update 2, that could produce incorrect results or invalid memory access in cublasLtMatmul() when (data-type size × matrix leading dimension) exceeded a signed 32-bit integer; it affected algorithm configuration ID 66 on compute capabilities 9.0, 10.x, and 11.0 and was fixed in CUDA 13.0 Update 3. Subtle numerical differences between driver versions can also break training reproducibility even when nothing is “broken.”

Metrics and signals to monitor

Fixes

Driver too old for the toolkit: upgrade the driver. This is the correct default fix. Choose the branch deliberately: Production Branch for general fleet use, Long-Term Support Branch when you need a stable target for years, New Feature Branch only when you need something it carries and accept its short lifecycle. As of mid-2026, R580 LTSB is supported through June 2028 and is the last branch supporting Maxwell/Pascal/Volta; R595 Production Branch runs through March 2027; R535 LTSB reached end of life in June 2026, so any fleet still on it is on borrowed time.

Driver cannot move: pin the toolkit down. If the host driver is frozen (managed environment, certification constraints), build or select container images whose CUDA version fits within the driver’s supported range. For CUDA versions newer than the driver on an LTSB branch, NVIDIA’s forward compatibility package can bridge the gap where the branch supports it; NVIDIA’s current matrix supports 13.x compatibility packages on 535+, 570+, 580+, 590+, and 595+, while NFBs are not supported compatibility targets.

Kernel update broke the module: rebuild and reload. Reinstall or rebuild the kernel module for the running kernel (DKMS normally handles this; check its status when it does not). Warning: loading a freshly built module on a node with active GPU workloads requires draining jobs first, and in many cases a reboot is the clean path. Treat this as a disruptive action, not a hot fix.

Module/userspace skew: finish the update. A driver update that updated userspace but left the old module loaded (or vice versa) is resolved by completing the transition: reload the module or reboot the node after draining. Do not leave nodes in the half-updated state; that state produces exactly the intermittent crash pattern that is hardest to debug.

Container toolkit mismatch: align the packages. Bring libnvidia-container* and nvidia-container-toolkit* to exactly matching versions. If your images depend on CUDA forward compatibility inside containers, verify the compat hook is enabled after the upgrade rather than assuming the old behavior survived.

Driver bug under your kernel pattern: roll back and pin. If XIDs and failures track a specific driver release and a specific workload shape, pin the fleet to the previous known-good release, file the issue with NVIDIA with the XID pattern and a repro, and gate the new release behind a canary before trying again.

Prevention

• Inventory versions as fleet state, not node trivia. Track driver, VBIOS, InfoROM, toolkit, and container toolkit package versions across every node. Version drift between nodes in the same pool is a defect, not an observation.
• Stage driver rollouts with a canary and an XID watch. Roll new driver branches to a small subset first, run representative workloads including your most kernel-pattern-intensive jobs, and correlate XID 13/31/32 patterns against the candidate release before widening. Some driver bugs only appear under specific kernel patterns, so “it booted and ran nvidia-smi” is not validation.
• Make kernel and driver updates one atomic change. A kernel update that reboots nodes without confirming the NVIDIA module rebuilt is how you get a fleet where 5% of nodes fail jobs for days. Gate node re-entry to service on the module loading cleanly and a smoke CUDA context succeeding.
• Keep a reproducibility baseline per driver release. A short, deterministic training or inference check whose outputs you compare across driver versions catches the “subtle numerical differences” class of problem before your researchers do.
• Track branch EOL dates in the same place you track capacity. A driver branch going end of life is a scheduled event. Discovering it during an incident means the calendar failed, not the driver.
• Watch security fixes as a driver-version input. Driver CVEs occasionally force a version floor. That is easier to absorb when your rollout pipeline already exists.

How Netdata helps

• Per-node driver version visibility turns fleet version drift into something you can query, so “which nodes diverged from the baseline” is a dashboard question instead of an SSH loop.
• Kernel log and XID correlation lets you line up XID 13/31/32 events with the moment a driver, kernel, or image rollout landed, which is the fastest way to separate version skew from application bugs.
• GPU signal context around the crash: when a launch failure or crash hits, utilization, memory, ECC, and throttle state at sub-minute resolution around the event tell you whether the GPU itself was implicated or the failure was purely in the software stack.
• Post-update regression detection: comparing step time, utilization shape, and power draw before and after a driver change surfaces the quiet performance-regression class of mismatch.
• Cross-node comparison makes the canary pattern operational: one node on a new driver behaving differently from its peers is visible immediately rather than after the next failed training run.

Related guides

• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• CUDA out of memory with free memory available: GPU memory fragmentation
• DCGM, nvidia-smi, and NVML: which NVIDIA GPU telemetry source to use
• DCGM nv-hostengine down or hung: GPU telemetry goes dark or stale
• NVIDIA DCGM exporter on Kubernetes: mapping GPU metrics to pods
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate
• NVIDIA Fabric Manager not running: NVSwitch GPUs lose NVLink