Monitoring MIG-partitioned NVIDIA GPUs: per-instance metrics

On a MIG-enabled GPU (A100, A30, H100 and later), the physical card is partitioned into isolated GPU instances, each with its own SMs, memory partition, and failure domain. Every monitoring habit built around whole-GPU metrics breaks here: aggregate utilization, aggregate memory, and even some health counters describe the card, not the tenant. An instance can be OOM while nvidia-smi shows healthy free memory on the card.

The second surprise: nvidia-smi and NVML do not attribute utilization to MIG devices at all. Per-instance utilization reads as N/A by design. The supported path for per-instance metrics is DCGM, queried with MIG-aware entity types, and not every DCGM field is available at instance granularity.

This guide covers what MIG changes about the metric surface, which DCGM entities and fields work per instance, how to enable MIG-aware collection, and the version-dependent behavior you need to know before trusting the numbers.

Why per-GPU metrics mislead on MIG

MIG partitions compute and memory but not everything:

flowchart TD
    GPU["Physical GPU (A100 / H100)"]
    GPU --> GI1["GPU instance 0
SMs + dedicated HBM partition"]
    GPU --> GI2["GPU instance 1
SMs + dedicated HBM partition"]
    GPU --> GI3["GPU instance 2
SMs + dedicated HBM partition"]
    GI1 --> CI1["Compute instance(s)"]
    GI2 --> CI2["Compute instance(s)"]
    GPU -.-> SHARED["Shared: thermal domain, power budget,
ECC sources, PCIe/NVLink path, driver"]

Isolated per instance:

• SM allocation (compute capacity)
• HBM framebuffer partition (with hardware memory isolation between tenants)
• L2 cache slices and memory controllers assigned to the partition

Shared across all instances on the card:

• Thermal domain: one instance running hot heats the whole die
• Power budget: all instances draw from one enforced power limit
• The PCIe/NVLink path and the driver itself

This split dictates the monitoring model. Utilization and framebuffer usage must be read per instance, because each tenant only sees their partition. Thermal, power, and several reliability signals remain card-level, because the hardware domain is shared. You need both views: per-instance for capacity and workload health, aggregate for thermal/power and hardware degradation. For the underlying failure-model background, see How an NVIDIA GPU actually works in production.

The DCGM entity model

DCGM exposes the same metrics at different entity granularities. The entity type prefix determines what you are reading:

The profiling field IDs are the same across entity types. What changes is whether the value is the card aggregate or the instance slice. A useful consequence: GPU-level graphics-engine activity is scaled by the instance’s SM allocation ratio. With two instances each holding 42 of 98 SMs, the card-level activity attributed to one instance is roughly (42/98) times the instance-level reading. Do not compare instance-level and card-level activity numbers directly without that scaling in mind.

One enumeration gotcha: MIG-enabled GPUs enumerate differently depending on DCGM configuration. The parent GPU may be invisible and only MIG instances appear, or vice versa. If your DCGM GPU count does not match nvidia-smi -L, check MIG mode and group configuration before assuming a hardware fault.

Which fields work per MIG instance

Field availability at MIG granularity depends on GPU generation and DCGM version. Treat this table as a starting point and verify against your DCGM version’s field documentation.

Two field-ID cautions. First, field IDs and names have shifted across DCGM releases; confirm names against dcgmi dmon -l or your version’s field reference rather than hardcoding IDs from documentation written for another release. Second, requesting an unsupported field typically returns a blank or zero value without an error, which is worse than an explicit failure: your dashboard shows a plausible-looking flatline.

Enabling MIG-aware collection

Prerequisites:

• MIG mode enabled and instances created (nvidia-smi mig -lgi lists GPU instances, nvidia-smi mig -lci lists compute instances).
• DCGM installed. The MIG user guide recommends DCGM v2.0.13 or later for GPU utilization metrics and v3 or later for MIG device monitoring; current releases are 4.x.
• nv-hostengine running as root. Profiling metrics require administrator privileges; an unprivileged host engine will silently lack the DCGM_FI_PROF_* fields.
• On Ampere and earlier, profiling metrics require the datacenter-gpu-manager-4-proprietary package. It is bundled in the dcgm-exporter container but needs separate installation for host deployments.

Manual verification with dcgmi dmon, using entity-aware queries against a group containing your MIG devices:

# Confirm MIG instances exist and note their IDs
nvidia-smi mig -lgi
nvidia-smi mig -lci

# Watch profiling fields per MIG entity (group must include the MIG devices)
dcgmi dmon -e 1001,1002,1003,1004 -g <mig-group>

For dcgm-exporter (the Prometheus path), the default default-counters.csv already enables the core profiling fields listed in the table above. If you need SM_ACTIVE or SM_OCCUPANCY, uncomment them in your counters file and restart the exporter. XID error and clock-event cumulative counters are also opt-in and commented out by default.

One tool to avoid for this job: dcgmi stats does not support profiling metrics for MIG devices. SM utilization and memory utilization come back as N/A or Not Found for MIG instances. This is a confirmed limitation (NVIDIA DCGM GitHub issue #58). Use dcgmi dmon with entity prefixes instead.

Version and generation differences

These differences bite during fleet upgrades and mixed-generation clusters:

• MIG mode persistence. On Ampere (A100, A30), MIG mode persists across reboots via InfoROM. On Hopper and later (H100, H200, B200), MIG mode is not persistent: it resets when the driver reloads. After any driver reload or reboot on Hopper+, instances are gone until you re-enable MIG and recreate them. Most teams handle this with a systemd unit that reapplies the MIG configuration at boot. This also means your per-instance time series will have gaps and re-created instance identities after every reload; do not alert on the gap as a node failure.
• GPU reset requirement. Enabling MIG mode requires a GPU reset on Ampere, which is disruptive: drain all workloads from the GPU first. On Hopper+, no reset is required.
• Minimum driver versions for MIG (per the NVIDIA MIG user guide): A100/A30 need R525 (>=525.53), H100/H200 need R450 (>=450.80.02), B200 needs R570 (>=570.133.20).
• On DGX systems, stop nvsm and DCGM services before enabling MIG mode, or the mode change fails with “In use by another client” errors.

Known gotchas

• N/A utilization is by design, not a fault. NVML does not support utilization attribution to MIG instances. Any pipeline scraping nvidia-smi --query-gpu=utilization.gpu for a MIG device will produce N/A forever. Migrate the collection to DCGM entities.
• DCGM_FI_PROF_SM_ACTIVE above 100% on MIG devices has been reported on NVIDIA forums (February 2024) and was not documented as fixed in DCGM 4.x at review time. Treat >100% readings as a known anomaly, not proof of a broken collector.
• Utilization not reaching 100% under burn tests on partitioned A100s has been reported against dcgm-exporter (issue #639). If your instance shows a plateau below 100% under a known-saturating workload, check the issue before distrusting the workload.
• Instance metrics lie by omission across tenants. One instance can be thermally throttled because a neighbor instance saturated the shared power budget. Per-instance metrics alone will not explain this; you need card-level power draw and throttle reasons alongside.
• Placement failures are invisible in utilization data. A MIG profile that cannot be placed (wrong granularity, fragmented free slices) produces no metric at all, just a failed creation command. Alert on configuration drift from the expected instance set, not on the absence of metrics.

Signals to watch in production

How Netdata helps

• Netdata’s NVIDIA GPU collector surfaces per-device utilization, framebuffer, temperature, power, and throttle reasons at per-second resolution, which is what you need to catch the cross-tenant thermal and power contention that per-instance tools cannot see.
• On MIG systems, the card-level view and the per-instance view answer different questions; correlating a tenant’s throughput drop against card-level sw_power_cap or thermal throttle events is usually the fastest path to “my instance is fine, the card is contended.”
• Framebuffer tracking per device catches the slow-growth leaks that precede instance OOMs, and pairs naturally with the fragmentation failure mode covered in CUDA out of memory with free memory available.
• ECC counter and XID-aware alerting at the card level catches the shared-hardware degradation that affects every tenant on the GPU regardless of partition boundaries.
• Anomaly detection on utilization and DRAM activity per device helps flag tenants behaving outside their historical envelope, which on multi-tenant MIG nodes is often the first sign of a runaway job.

Related guides

• How an NVIDIA GPU actually works in production: a mental model for operators
• CUDA out of memory: diagnosing NVIDIA GPU framebuffer exhaustion
• CUDA out of memory with free memory available: GPU memory fragmentation
• NVIDIA BAR1 memory exhaustion: mapping failures with free framebuffer
• NVIDIA GPU ECC errors: corrected, uncorrected, volatile, and aggregate
• NVIDIA GPU ECC disabled: the silent data-corruption risk
• NVIDIA GPU HBM (memory) temperature: the thermal limit most teams miss
• NVIDIA GPU HBM progressive failure: from single-bit errors to a dead GPU
• NVIDIA GPU HW Power Brake Slowdown: the chassis is cutting GPU power
• NVIDIA GPU fan at 0%: fan failure on air-cooled cards
• NVIDIA Fabric Manager not running: NVSwitch GPUs lose NVLink
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver