$ guides / smartctl-disk-monitoring / smartctl-sas-phy-errors ▌

Operations Guides

SAS PHY error counters: transport health for SAS drives

You are running SAS drives and something is off. I/O latency has spiked on a specific drive, or the kernel log is showing SAS link resets and error recovery messages. You reach for smartctl to check SMART health, but there is no UDMA_CRC_Error_Count attribute. SAS drives do not use ATA SMART attributes. They use SCSI log pages, and their transport health lives in a different set of counters: the SAS PHY error counters.

These counters are the SAS equivalent of SATA’s UDMA CRC error count, but with more granular detail about what is failing at the physical layer. Four counters track different categories of transport-layer corruption between the HBA, expanders, and drives. Growth in any of them means the physical link is unreliable, and the cause is almost never the drive’s media. It is the cable, backplane, HBA port, or expander.

What the counters measure

SAS drives report health through SCSI log pages. Transport health comes from the SAS PHY log page (Protocol Specific log page, page 0x18), accessed with smartctl -l sasphy.

The four SAS PHY error counters:

Counter	What it measures
Invalid DWORD count	Corrupted data words received on the PHY. The link received bits that do not form a valid data word.
Running disparity error count	8b/10b encoding violations. Errors mean signal integrity is degraded.
Loss of DWORD synchronization count	The PHY lost lock on the incoming data stream and had to re-synchronize. Each event causes a link interruption.
Phy reset problem count	The PHY encountered a problem during the reset sequence. Frequent resets indicate an unstable link.

All four counters are cumulative. They never decrease on their own. A static, low count from past events is acceptable. What matters is growth: any increase means the transport layer is actively experiencing errors.

The counters are per-PHY, not per-device. A SAS wide port (for example, a 4-lane link between an HBA and an expander) has 4 sets of counters, one per lane. Errors on a single PHY within a wide port can cause degraded throughput without complete link loss, making the problem easy to miss if you only check aggregate throughput.

flowchart LR
    HBA[Host HBA] -->|"PHY 0-3"| CBL[SAS cable / backplane]
    CBL --> EXP[Expander]
    EXP --> DRV1[Drive A]
    EXP --> DRV2[Drive B]

Invalid DWORD and running disparity errors typically point to cable or backplane signal integrity issues between the HBA and the expander. Loss of sync and PHY reset problems can indicate HBA-side PHY degradation or an expander firmware problem.

Common causes

Cause	What it looks like	First thing to check
Bad or loose SAS cable	Invalid DWORD count and running disparity errors climbing together on a specific PHY	Reseat both ends of the cable, then check if counters stop growing
Failing backplane port	Errors on one PHY only, regardless of which drive is in that bay	Move the drive to a different bay and see if the errors follow the drive or stay on the port
HBA PHY degradation	PHY reset problems and loss of sync on one HBA port across all drives behind it	Check HBA firmware and compare error rates across HBA ports
Marginal expander	Errors spread across multiple PHYs connected to the same expander	Check expander firmware, temperature, and event logs
Reboot transient	All PHY counters spike once during a server reboot, then stop growing	Compare snapshots before and after reboot to confirm no ongoing growth

Quick checks

# Read SAS PHY error counters for a specific drive
smartctl -l sasphy /dev/sdX

# Explicit SCSI device type (more reliable behind some HBAs)
smartctl -d scsi -l sasphy /dev/sgN

# Read counters from the kernel sysfs interface (covers HBA and expander PHYs too)
cat /sys/class/sas_phy/*/invalid_dword_count
cat /sys/class/sas_phy/*/running_disparity_error_count
cat /sys/class/sas_phy/*/loss_of_dword_sync_count
cat /sys/class/sas_phy/*/phy_reset_problem_count

# Check kernel logs for SAS transport events
journalctl -k | grep -iE "sas|link reset|phy|device offline" | tail -30

# List SAS topology to identify PHY-to-device mapping
ls -la /sys/class/sas_phy/
ls -la /sys/class/sas_device/

Destructive command. Resetting counters destroys your historical baseline. Only do this after you have recorded the current values and are confident the physical fix is in place:

# Reset PHY error counters on a specific drive (erases all historical counts)
smartctl -l sasphy,reset /dev/sdX

How to diagnose it

Snapshot the counters twice. Read the counters, wait 10 to 30 minutes (or through a representative workload), then read them again. The goal is to confirm active growth, not just a historical count. A single non-zero reading from months ago is not actionable.
Identify which PHY is affected. The counters are per-PHY. Walk /sys/class/sas_phy/ to see each PHY independently. The PHY number in smartctl output does not directly map to the SCSI target ID shown by lsscsi. Mapping a PHY to a block device requires walking the sysfs topology; start at the PHY entry and follow the device symlinks through the port and end device nodes.
Check both sides of the link. SAS is a point-to-point protocol. Errors can be on the HBA-side PHY, the expander-side PHY, or the drive-side PHY. If you only check the drive, you may miss HBA-side PHY degradation. Check the HBA’s sysfs PHY counters and the expander’s counters if accessible.
Check for correlated media errors. Run smartctl -a /dev/sdX to check the SCSI grown defect list and background scan results. If PHY errors are growing but media errors are absent, the problem is purely transport. If both are growing, the transport errors may be causing phantom read failures that the drive logs as media problems. Fix the transport issue first, then re-evaluate.
Rule out reboot transients. SAS expanders log invalid DWORD, running disparity, and loss of dword sync events during server reboots. If the counters jumped right after a reboot and have not moved since, the spike is expected. Always correlate counter jumps with known events (maintenance, power cycles) before escalating.
Check if the adapter is disabling PHYs automatically. Some storage controllers automatically disable SAS PHYs when invalid dword errors exceed a threshold. If a PHY has disappeared entirely, check the HBA or controller logs for automatic PHY-disable events.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Invalid DWORD count (per-PHY)	Corrupted words on the link indicate signal integrity problems	Any growth from baseline
Running disparity error count	Encoding violations point to cable or backplane issues	Growth correlated with invalid DWORD count
Loss of DWORD sync count	Link interruptions causing I/O stalls	Growth accompanied by latency spikes or kernel EH messages
PHY reset problem count	Unstable link requiring resets	Growth correlated with device disappear/reappear events
Kernel SAS events	Host-visible transport errors and recovery actions	“link reset”, “phy down”, “device offline” messages
I/O latency (iostat await)	Transport retries manifest as latency before error counters move	await spiking on one device without CPU or queue depth explanation
SCSI grown defect list	Confirms whether transport errors are causing media-level issues	Growth alongside PHY error growth

Fixes

Bad or loose SAS cable

The most common cause. Reseat both ends of the cable. If counters continue to grow after reseating, swap the cable with a known-good one. SFF-8087, SFF-8643, and mini-SAS HD connectors can appear seated but have marginal pin contact. If the cable runs through a chassis backplane, the backplane trace itself may be the problem.

Failing backplane port

Move the drive to a different bay in the same enclosure. If the errors follow the drive, the drive-side PHY or internal cable may be failing. If the errors stay on the original bay, the backplane port or its trace is the problem. In hot-swap enclosures, backplane connector fatigue is a known failure mode.

HBA PHY degradation

If errors appear on all drives behind a specific HBA port, the HBA PHY itself is likely failing. Check the HBA firmware version against vendor advisories. If firmware is current, the HBA may need replacement. Moving drives to a different HBA port confirms the diagnosis.

Expander issues

If errors are spread across multiple PHYs that all connect through the same expander, the expander is the suspect. Check expander firmware, temperature, and any expander-level event logs. Expander firmware bugs are a known source of broad PHY error outbreaks across all attached drives.

Prevention

Baseline at deployment. Capture SAS PHY counter values when a drive or enclosure is first installed. Without a baseline, you cannot distinguish historical errors from active growth.
Monitor rate of change, not absolute values. A static count of 50 from two years ago is stable. A count that went from 0 to 5 this week is active. Alerting logic must track growth, not thresholds.
Exclude reboot windows from alerting. Loss of dword sync and invalid DWORD counts spike during reboots. If your monitoring does not account for this, you will get false alarms after every maintenance cycle.
Track per-PHY, not per-device. Monitoring only the drive-side counters misses HBA-side and expander-side PHY problems. Use the kernel sysfs interface at /sys/class/sas_phy/ for complete PHY coverage.
Do not apply ATA CRC logic to SAS. UDMA_CRC_Error_Count (ATA ID 199) does not exist on SAS drives. If your monitoring framework uses ATA attribute templates for SAS devices, it is silently missing transport health data.

How Netdata helps

SAS PHY error counters are cumulative and grow slowly. The diagnostic value is in the rate of change, not the absolute count. Per-second metric collection detects a counter increment within seconds and correlates it with I/O latency spikes on the same device.
The four PHY error counters should be read alongside kernel SAS events (link resets, phy-down messages) and per-device I/O latency. If all three move together, you have confirmed transport-layer degradation and can skip straight to physical inspection.
sysfs-based per-PHY counters at /sys/class/sas_phy/ provide broader coverage than drive-level smartctl queries because they include HBA-side and expander-side PHYs that drive-level commands cannot reach.