$ guides / lvm / lvm-raid-mismatch-count ▌

Operations Guides

LVM RAID mismatch count: data integrity after a scrub

A non-zero raid_mismatch_count means the scrub found blocks where data on one leg disagrees with data on the other leg(s). This surfaces as the m flag in position 9 of lv_attr and in the raid_mismatch_count field of lvs. It demands investigation, though some mismatches turn out to be benign.

The operational question is not “do we have mismatches” but “is this real corruption or expected noise.” RAID1 and RAID10 arrays can report non-zero mismatch counts from documented edge cases in the kernel write path that produce alignment differences in transient data areas. Mismatches appearing after an unclean shutdown are expected until the scrub completes and should not trigger paging on initial boot.

This article covers how to interpret a non-zero mismatch count, how to distinguish benign from real corruption, and how to safely repair without making things worse.

What this means

When you run lvchange --syncaction check on a RAID LV, the kernel reads every block from every leg and compares them. Any block where the legs disagree increments the mismatch counter. The check action is read-only: it detects and reports but does not modify data.

After the scrub completes, raid_mismatch_count holds the total number of mismatched blocks found. The m character in position 9 of lv_attr (for example, Rwi-a-r-m-) indicates mismatches exist. The count persists until a subsequent scrub runs to completion with zero discrepancies.

A non-zero count means one of three things:

Real data corruption. One or more blocks on a leg are genuinely wrong, typically from media degradation or a firmware bug.
Benign alignment differences. RAID1 and RAID10 can produce these in transient data areas due to write-path edge cases. They do not represent corruption, but the counter cannot distinguish them from real problems.
Transient post-crash state. After an unclean shutdown, dirty regions may show mismatches until the scrub reconciles them.

You cannot distinguish these three cases from the mismatch count alone. The count is a number, not a verdict. Correlating it with SMART data, kernel logs, and the trend across multiple scrubs is what separates a non-event from an incident.

Common causes

Cause	What it looks like	First thing to check
Benign RAID1/RAID10 alignment	Small, stable mismatch count on a healthy array with no dmesg errors and no degraded legs	Re-run the scrub; if the count is stable or drops, likely benign
Post-crash dirty regions	`m` flag appears immediately after unclean shutdown, before scrub completes	Wait for the post-boot scrub to finish, then re-check
Real media corruption	Mismatch count grows over repeated scrubs, dmesg shows medium errors on a PV	Check SMART data and dmesg for the affected device
dm-integrity counter stuck	Mismatches on LVs with `--raidintegrity y`, count never resets after check	Check lvm2 version for the May 2026 fix (commit 49c11b41)
Unallocated space mismatches	Count is non-zero but the filesystem on the LV reports no errors	Write to all free space to force synchronization, then re-scrub

Quick checks

# Check health status of all RAID LVs — position 9 of lv_attr
lvs -o lv_name,vg_name,lv_attr,raid_mismatch_count,raid_sync_action

# Filter for unhealthy LVs only
lvs -o lv_name,vg_name,lv_attr | awk 'substr($3,9,1) != "-" && NR>1'

# Detailed RAID health per device from device-mapper (no LVM lock needed)
dmsetup status <vg>-<lv>
# Health chars: A = alive and in-sync, a = alive but not in-sync, D = dead/failed

# Monitor an in-progress scrub
lvs -a -o name,raid_sync_action,sync_percent,copy_percent

# Verify no PVs are missing or degraded
pvs -o pv_name,vg_name,pv_attr,pv_size,pv_free
# Missing PVs may not appear in pvs output at all; check vgs for partial flag
<!-- TODO: verify exact pvs/vgs output format for missing PVs across lvm2 versions -->

# Check for kernel I/O errors on underlying devices
dmesg | grep -i 'error\|medium\|fail' | tail -50

# Check VG metadata consistency
vgck -v <vgname>

How to diagnose it

Confirm the scrub has completed. A mismatch count is only meaningful after raid_sync_action shows the check is done and sync_percent is 100. If the scrub is still running, wait. Mismatches during an in-progress scrub are not final.
Rule out the post-crash transient. If the system recently rebooted after an unclean shutdown, the m flag may appear until the post-boot scrub reconciles dirty regions. Wait for the scrub to complete, then re-check the count.
Check whether any legs are degraded. Use dmsetup status <vg>-<lv> and look at the health characters. If any device shows D (dead), the array is degraded and you have a bigger problem than the mismatch count. Address the dead leg first. See LVM RAID or mirror degraded.
Determine the RAID level. RAID1 and RAID10 can produce benign mismatches from write-path edge cases. These exist in transient data areas and do not represent corruption, but the counter cannot distinguish them from real problems.
Run a second scrub and compare counts. After the first scrub reports a mismatch count, run lvchange --syncaction check again. If the count drops or disappears, it was likely transient or benign. If the count is identical, it could be persistent corruption or a stable benign alignment difference. If the count grows, investigate hardware immediately.
Check for unallocated space mismatches. Mismatches can reside in unallocated space on the filesystem. The RAID layer tracks all blocks, including ones the filesystem has never written to. Writing to all free space can force synchronization and reduce the count. This is a known cause of confusing-but-benign mismatch reports.
Check SMART and dmesg for hardware errors. If you suspect real corruption, correlate the mismatch with SMART error logs and kernel I/O error messages for the underlying physical devices. Medium errors, read retries, and reallocated sectors all point to media degradation.
Check whether dm-integrity is enabled. If the LV was created with --raidintegrity y, there was a known bug where syncaction check would detect mismatches but the counter would never reset to zero, and syncaction repair was refused with the message “Use syncaction check to detect and correct integrity checksum mismatches.” This was fixed in upstream commit 49c11b41 (May 2026), which allows syncaction check and syncaction repair directly on integrity-enabled LVs.

flowchart TD
    A["raid_mismatch_count > 0"] --> B{"Scrub completed?"}
    B -- No --> C["Wait for completion
Do not alert yet"]
    B -- Yes --> D{"Recent unclean
shutdown?"}
    D -- Yes --> E["Wait for post-boot scrub
Re-check count"]
    D -- No --> F{"Any leg shows D
in dmsetup status?"}
    F -- Yes --> G["Address dead leg first
Higher priority than mismatches"]
    F -- No --> H{"RAID1 or RAID10?"}
    H -- Yes --> I["May be benign
Re-scrub and compare"]
    H -- No --> J["More likely real
Check SMART and dmesg"]
    I --> K{"Count stable or
dropping?"}
    K -- Yes --> L["Likely benign
Monitor trend"]
    K -- No --> M["Investigate hardware
Consider repair"]
    J --> M

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`raid_mismatch_count`	Direct indicator of data divergence between legs	Any non-zero value after a completed scrub
`lv_attr` position 9 (`m`)	Health flag that surfaces mismatches without querying the count	`m` appears where `-` was previously
`raid_sync_action`	Tells you whether a scrub is in progress and what type	Stuck in `check` or `repair` for hours with no progress
`copy_percent`	Rebuild or resync progress for the array	Stuck at the same value for over an hour
`dmsetup status` health chars	Per-device RAID health (A/a/D)	Any `D` means a dead leg, which takes priority over mismatch investigation
PV accessibility	Missing PV means degraded array, not just mismatches	PV absent from `pvs` output, VG showing partial flag in `vgs`
Kernel I/O errors in dmesg	Hardware-level signal of media degradation	`medium error`, `I/O error`, or read retries on PV devices
SMART error logs	Physical disk health independent of the LVM layer	Reallocated sectors, pending sectors, UDMA CRC errors

Fixes

Run a check scrub

# Read-only detection — does not modify data
lvchange --syncaction check <vg>/<lv>

This is always the first step. It populates raid_mismatch_count without writing anything. Monitor progress with lvs -a -o name,raid_sync_action,sync_percent. Do not act on a count until the scrub completes. This action is safe to run during production operation, though it generates additional read I/O.

Run a repair scrub

# WARNING: repair makes data consistent but may pick the wrong copy.
# It cannot determine which leg has the correct data.
lvchange --syncaction repair <vg>/<lv>

The repair action detects mismatches and writes corrected blocks to make the legs consistent. The critical limitation: repair does not know which copy is correct. The man page explicitly warns that repair may result in consistent but incorrect data. If you have application-level checksums or filesystem integrity verification, run them after repair.

For RAID1 where you know one PV is failing or corrupt, prefer lvchange --rebuild <PV> <vg>/<lv> over repair. The --rebuild flag rebuilds from the known-good leg to the specified device, which is safer when you have high confidence about which leg holds the correct data.

The dm-integrity workaround (older lvm2)

If your LV uses --raidintegrity y and you are running lvm2 from before the May 2026 fix, syncaction repair may be refused. The workaround is to temporarily disable integrity:

# WARNING: temporarily removes the integrity layer.
# This is disruptive and bypasses integrity checking until re-enabled.
lvconvert --raidintegrity n <vg>/<lv>
lvchange --syncaction repair <vg>/<lv>
lvconvert --raidintegrity y <vg>/<lv>

Newer lvm2 (with commit 49c11b41) allows syncaction check and syncaction repair directly on integrity-enabled LVs without this workaround.

Force synchronization of unallocated space

If mismatches persist primarily in unallocated space and you have confirmed the filesystem itself is healthy:

# WARNING: this fills the filesystem completely until ENOSPC.
# Applications writing to the same filesystem will fail while this runs.
# Run only during a maintenance window or on a quiesced filesystem.
dd if=/dev/zero of=/mountpoint/fillfile bs=1M status=progress
rm /mountpoint/fillfile
lvchange --syncaction check <vg>/<lv>

This writes zeros to all free space, forcing the RAID layer to synchronize blocks that were never written. Schedule it during a maintenance window. After the fill file is removed, re-scrub to verify the mismatch count has dropped.

Prevention

Schedule regular scrubs. Run lvchange --syncaction check on RAID LVs periodically (weekly or monthly depending on data criticality). Regular scrubs catch corruption early and keep the mismatch count meaningful as a trend signal rather than a one-time surprise.
Do not page on post-boot mismatches. After an unclean shutdown, mismatches may appear transiently until the scrub completes. Gate alerts on system uptime greater than 600 seconds and require the scrub to have completed before alerting.
Verify your lvm2 version if using dm-integrity. The May 2026 fix (commit 49c11b41) changes how syncaction check and syncaction repair interact with integrity-enabled LVs. Older versions have a known bug where the mismatch counter never resets and repair is refused.
Track mismatch count trends, not just current values. A stable count of 12 that has been the same for months is different from a count that went from 0 to 12 in the last scrub. Historical data tells you whether the situation is changing, which is the single most useful signal for distinguishing benign from active corruption.
Correlate with SMART data. Media errors are the most common cause of real (non-benign) mismatches. Proactive SMART monitoring catches degrading disks before they produce RAID-level corruption.
Document which LVs are RAID1/RAID10. These levels can produce benign mismatches. Having this context in your runbook prevents unnecessary escalation when a stable non-zero count appears on a known-healthy RAID1 volume.

How Netdata helps

Per-second collection on raid_mismatch_count and lv_attr health flags catches the transition from - to m immediately, rather than waiting for the next cron-based lvs poll that may be tens of seconds behind.
ML-based anomaly detection on mismatch count trends distinguishes a stable, known-benign count from a sudden jump that indicates active corruption, reducing false escalations on RAID1 volumes with expected alignment differences.
Correlation with disk I/O latency and error rates from /proc/diskstats and SMART data connects RAID-level mismatches to the specific underlying physical device that is degrading.
Kernel log collection surfaces dmesg I/O errors, medium errors, and device reset messages alongside the mismatch flag, providing the corroboration needed to separate real corruption from benign alignment differences.
Historical trending of mismatch counts across scrub cycles shows whether the count is stable, growing, or shrinking over time, which is the single most useful signal for distinguishing benign from active corruption.