NVIDIA Fabric Manager not running: NVSwitch GPUs lose NVLink

Your multi-GPU training job hangs during NCCL initialization, or fails with cudaErrorSystemNotReady (error 802) the moment a process touches CUDA. Each GPU shows up in nvidia-smi with normal temperature, memory, and utilization. You burn hours on NCCL debug logs, InfiniBand checks, and application-level bisection before someone runs systemctl status nvidia-fabricmanager and finds the service failed two days ago after a reboot.

On DGX and HGX systems, the NVSwitch fabric that gives GPUs their all-to-all NVLink connectivity is not self-configuring. A userspace daemon, nv-fabricmanager (the nvidia-fabricmanager systemd unit), programs the switch routing tables and health-checks the fabric. If that daemon crashes, never started, or aborted on a version mismatch, the GPUs are individually healthy and collectively deaf.

What this means

On PCIe-attached GPUs, NVLink (where present) is a point-to-point connection between GPU pairs and needs no external coordination. On NVSwitch systems, the baseboard carries NVSwitch ASICs and every GPU-to-GPU path goes through them. The switches must be configured before any NVLink traffic flows. That configuration is FM’s job.

The failure is asymmetric, which is what makes it deceptive:

• Per-GPU health signals (temperature, ECC, utilization, memory) look normal because the GPUs themselves are fine.
• The failure only appears when software tries to use the fabric: CUDA init fails with cudaErrorSystemNotReady, or NCCL hangs after channel/tree setup but before P2P communication starts.
• In a multi-node job, one node with FM down stalls the entire NCCL communication ring. The symptom appears on every node, so operators routinely chase the network on healthy nodes while the fault sits on one.

flowchart TD
  FM["nvidia-fabricmanager down"] --> SW["NVSwitch fabric unconfigured"]
  SW --> NL["GPU-to-GPU NVLink paths dead"]
  NL --> CUDA["CUDA init: cudaErrorSystemNotReady"]
  NL --> NCCL["NCCL hang after channel setup"]
  NCCL --> RING["Whole multi-node job stalls"]
  FM -.masks.-> SMI["Per-GPU nvidia-smi still looks healthy"]

One more trap: a green systemctl status is not proof the fabric is healthy. FM can run in a stay-resident mode (FM_STAY_RESIDENT_ON_FAILURES) where the daemon reports active (running) but the fabric was never initialized. Verify fabric state, not just process state.

Common causes

The most common root cause is the first row. FM is lockstep-versioned with the NVIDIA kernel driver: during initialization it checks the loaded driver stack version and aborts if they do not match. A driver update without a matching nvidia-fabricmanager (and libnvidia-nscq) update leaves a daemon that refuses to start. Unattended-upgrades produces exactly this: the driver package and the FM package update independently, and repository lag can make a matching FM version unavailable even when you ask for it. There are confirmed distribution bugs of this shape (for example, FM 535.161.08 shipped against driver 535.171.04 in Ubuntu repos).

Quick checks

All read-only. Run them on the suspect node before restarting anything.

# 1. Is the service up, and for how long?
systemctl status nvidia-fabricmanager

# 2. Why did it fail or abort?
journalctl -u nvidia-fabricmanager --since "24 hours ago" --no-pager | tail -50

# 3. FM's own log (if enabled in its config) often has the clearest error
tail -50 /var/log/fabricmanager.log

# 4. Driver vs FM package version match
nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1
dpkg -l | grep -E "nvidia-fabricmanager|libnvidia-nscq"    # Debian/Ubuntu
# rpm -qa | grep -E "nvidia-fabricmanager|libnvidia-nscq"  # RHEL-family

# 5. What does the driver think of the fabric?
nvidia-smi -q | grep -A 4 -i fabric

The fifth check matters most. On a healthy system the Fabric section reports state Completed and status Success. In Progress, an error state, or a missing Fabric section on an NVSwitch system means the fabric never formed, regardless of what systemd says about the process.

Two checks that prevent embarrassing false alarms:

# Confirm this node actually has NVSwitch hardware
nvidia-smi -q | grep -i -A 2 "Product Name"
ls /dev/nvidia-nvswitch* 2>/dev/null

If the node has no NVSwitch ASICs, FM has nothing to do and exits with NV_WARN_NOTHING_TO_DO. That is not a fault; it is the expected outcome of starting FM on the wrong platform (see the diagnosis section for which platforms this hits).

How to diagnose it

1. Confirm the symptom class. CUDA applications fail with cudaErrorSystemNotReady (802) at init, or NCCL hangs after channel/tree setup completes but before P2P traffic starts. If individual-GPU jobs run fine and only multi-GPU jobs fail, the fabric is the prime suspect.

2. Check FM process state. systemctl status nvidia-fabricmanager. Three distinct outcomes: inactive (dead) (never started), failed (crashed or aborted), and active (running) (process alive, fabric state still unproven).

3. Check fabric state via the driver. nvidia-smi -q | grep -A 4 -i fabric. State Completed / status Success means the fabric formed. Anything else on an NVSwitch system means it did not. This catches the FM_STAY_RESIDENT_ON_FAILURES=1 case where systemd is green but the fabric is dead.

4. Read the FM logs. journalctl -u nvidia-fabricmanager and /var/log/fabricmanager.log. The most valuable line is the version-compatibility abort: FM prints the driver version it found and the version it requires. If you see that, skip to the version-mismatch fix.

5. Rule out the no-NVSwitch platform case. If the log shows NV_WARN_NOTHING_TO_DO, the node has no NVSwitch devices. This hits two common situations: systems with NVLink bridges but no switch (for example, bridge-connected A40 pairs), and GB200/GB300 NVL72 compute nodes, where the NVSwitch ASICs live in dedicated switch trays, not on the compute nodes. On those nodes FM should not be installed or running at all; the correct action is to stop and disable the unit, not fix it. Some GPU health tooling explicitly treats this state as healthy on GH200/GB200 nodes.

6. In a multi-node failure, bisect by node. If one node’s FM is down, the whole NCCL ring stalls and every node’s logs look similar. Run the nvidia-smi -q fabric check on every node in the job and fix the node whose fabric is not Completed, not the node whose logs you happened to read first.

7. For multi-node NVLink (NVL72-class), check the cross-node layer too. On GB200/GB300 NVL72 systems, FM manages the per-switch-tray fabric while a separate IMEX service orchestrates cross-node memory export/import. If intra-node NVLink works but inter-node does not, FM is fine and IMEX is the suspect.

Metrics and signals to monitor

Alerting guidance for this pattern: page only on NVSwitch systems, after an uptime gate (to exclude cold start and planned restarts), when FM has been down long enough to matter (over a minute), and NVSwitch-dependent jobs are failing or unable to initialize NCCL. On non-NVSwitch systems, or when no affected jobs exist, it is a ticket.

Fixes

Version mismatch: align FM to the driver

FM, libnvidia-nscq, and the kernel driver must be the same version.

# Identify the running driver branch
nvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1

# Install the FM package matching that exact version, then restart
# Debian/Ubuntu example; package name varies by distro and branch:
apt-get install nvidia-fabricmanager-<branch>=<exact-driver-version>
systemctl restart nvidia-fabricmanager

Two cautions. First, distribution repositories can lag NVIDIA’s releases, so an exact-match FM package may not exist in your configured repos; pulling the driver and FM together from NVIDIA’s own repo avoids the split-brain. Second, if the driver was updated but the node has not rebooted, the running kernel module may differ from the on-disk driver, and FM matches against the running module. Check what is actually loaded before installing anything. After the restart, verify with the nvidia-smi -q fabric check, not just systemd.

Service down after reboot

# Start and enable so it survives the next reboot
systemctl enable --now nvidia-fabricmanager

# Verify the fabric formed
nvidia-smi -q | grep -A 4 -i fabric

If the unit starts but the fabric state never reaches Completed, go back to the logs: on a genuine NVSwitch system that almost always points at version mismatch or a switch-side fault.

Fabric dead with FM running (stay-resident mode)

If FM_STAY_RESIDENT_ON_FAILURES=1 left a green-but-dead daemon, restarting the unit clears the resident state and forces a fresh fabric initialization attempt. Treat the restart as a diagnostic: if the fabric comes up Completed, the transient fault cleared. If it fails again, the journal from the new attempt has the real error. Note that on A100, FM exit behavior toward running CUDA jobs is governed by ABORT_CUDA_JOBS_ON_FM_EXIT; on H100 and later that option is ignored and running jobs continue after FM exits, which can mask the failure even longer.

Platform has no NVSwitch

Do not fix FM; remove it from the equation. Stop, disable, and uninstall the package on bridge-connected NVLink systems and on GB200/GB300 compute nodes. Keeping the unit enabled there guarantees a boot-time failure and alert noise forever.

After any recovery

Restart affected jobs from scratch. CUDA contexts that initialized (or hung) against a dead fabric do not recover in place when the fabric comes back. For multi-node jobs, re-run the fabric check on every participating node before relaunching.

Prevention

• Pin the driver/FM pair. Upgrade the driver and nvidia-fabricmanager (plus libnvidia-nscq) as one atomic change, from one repository, in one maintenance window. Never let them drift.
• Disable unattended-upgrades for GPU packages. Unattended upgrades touching the driver or FM independently is a documented cause of version-mismatch outages. Blacklist the NVIDIA packages or disable the service on NVSwitch nodes, and move upgrades to a controlled process.
• Gate reboots on FM health. After any reboot of a DGX/HGX node, verify the fabric state before marking the node schedulable. A node that rejoins the pool with FM down poisons the next multi-node job that lands on it.
• Alert on fabric state, not just process state. Because stay-resident mode makes systemd lie, the monitored signal must be the driver’s view of the fabric (Completed/Success), with the uptime and active-job guards from the alerting guidance above.
• Know your platforms. Keep a per-node-class record of which systems have NVSwitch ASICs and which layer owns cross-node NVLink (IMEX on NVL72). This turns the “is FM even supposed to run here?” question into a lookup instead of a debugging session.

How Netdata helps

• Netdata tracks the nvidia-fabricmanager service state and process liveness as a first-class metric on NVSwitch nodes, so a post-reboot FM failure pages before a job discovers it.
• Per-GPU health (temperature, ECC, utilization) alongside fabric and NVLink signals makes the “GPUs healthy, fabric dead” asymmetry visible in one view, which is the exact correlation this failure hides behind.
• Driver version exposure across the fleet lets you spot driver/FM version drift before the next restart turns it into an outage.
• NCCL hang and cudaErrorSystemNotReady symptoms in job logs correlated against FM state and uptime turn a multi-hour bisection into a single-node identification.
• Cross-node comparison of step time and per-GPU signals surfaces the straggler pattern a single dead fabric creates in a multi-node job.

Related guides

• How an NVIDIA GPU actually works in production: a mental model for operators
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• CUDA out of memory with free memory available: GPU memory fragmentation
• NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate
• NVIDIA GPU ECC disabled: the silent data-corruption risk
• NVIDIA GPU HBM (memory) temperature: the thermal limit most teams miss
• NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU
• NVIDIA GPU HW Power Brake Slowdown: the chassis is cutting GPU power
• NVIDIA GPU fan at 0%: fan failure on air-cooled cards
• NVIDIA GPU memory leak: framebuffer usage climbing without a plateau