SQL Server HADR_SYNC_COMMIT waits: a synchronous secondary throttling primary commits

HADR_SYNC_COMMIT at the top of your wait statistics is a counterintuitive failure. The primary replica looks idle: CPU is low, local I/O is fast, batch requests look normal. Yet every write transaction stalls. The cause is not on the primary but on the synchronous secondary, the network between replicas, or the AG transport itself.

In synchronous-commit mode, the primary cannot acknowledge a transaction commit until the secondary hardens the log record to disk. Every write transaction pays that round-trip tax. When the secondary or the path to it degrades, that tax becomes seconds of latency per commit. Under sustained write load, the delays lengthen lock hold times on the primary and cascade into blocking, worker thread growth, and eventually THREADPOOL waits. The application experiences this as a general slowdown, but the root cause is a single replica that cannot keep up.

What this means

HADR_SYNC_COMMIT is the time the primary spends waiting for all synchronous-commit secondaries to acknowledge that a log block has been hardened (written to the log file on the secondary) at or beyond the commit LSN. The local log flush and the remote log block copy are initiated in parallel. The primary thread waits on the remote harden acknowledgement and then on the local flush if it has not completed yet. If the local flush is slow, you also see WRITELOG waits. If only the remote side is slow, you see HADR_SYNC_COMMIT.

The mechanism matters because it explains the failure signature:

  • Primary CPU and local disk I/O can be completely healthy. The bottleneck is somewhere else.
  • The wait does not identify which secondary is slow when you have more than one synchronous replica. You need to correlate per-replica.
  • The wait does not by itself tell you whether the delay is in the network transport, secondary log file I/O, secondary CPU, or the AG flow-control gates. You need additional signals.
  • Every transaction that waits on HADR_SYNC_COMMIT is also holding its locks. As wait time grows, lock hold times grow. The natural endpoint is blocking chains and worker thread exhaustion under high write concurrency.

The extended event hadr_db_commit_mgr_harden_still_waiting fires when a committed LSN has not been acknowledged by all synchronous-commit secondaries for more than two seconds. That gives you a precise trigger for active investigation rather than relying on cumulative wait stats.

Common causes

CauseWhat it looks likeFirst thing to check
Secondary log file I/O stallHADR_SYNC_COMMIT rising on primary, secondary redo queue low, secondary log write latency above 5mssys.dm_io_virtual_file_stats on the secondary, focused on LOG files
Network path degradationHADR_SYNC_COMMIT rising, secondary log I/O looks fine, send queue stable but log send rate reducedRound-trip latency between replica endpoints; AG transport counters
Readable secondary workload competing with redoHADR_SYNC_COMMIT rising on primary, redo queue growing on secondary, secondary CPU busy with read queriesActive requests on the secondary; redo-related waits
AG flow-control gates throttlingHADR_SYNC_COMMIT rising, network and disk look fine, Flow Control/sec nonzero on primarySQLServer:Availability Replica counters for Flow Control/sec and Flow Control Time (ms/sec)
Primary write burst overwhelming pipelineHADR_SYNC_COMMIT spikes during ETL or index maintenance, send queue grows temporarilyLog generation rate on primary during the window; index rebuild MAXDOP

Quick checks

Run these read-only checks on the primary first, then on the secondary. All are safe during an incident.

-- Confirm HADR_SYNC_COMMIT is currently the dominant wait (delta sampling)
SELECT TOP 10
    wait_type,
    waiting_tasks_count,
    wait_time_ms,
    CAST(100.0 * wait_time_ms / SUM(wait_time_ms) OVER() AS DECIMAL(5,2)) AS pct
FROM sys.dm_os_wait_stats
WHERE wait_type NOT IN (
    'SLEEP_TASK', 'BROKER_TO_FLUSH', 'BROKER_TASK_STOP',
    'CLR_AUTO_EVENT', 'CLR_MANUAL_EVENT', 'LAZYWRITER_SLEEP',
    'SQLTRACE_BUFFER_FLUSH', 'WAITFOR', 'XE_TIMER_EVENT',
    'HADR_WORK_QUEUE', 'LOGMGR_QUEUE', 'CHECKPOINT_QUEUE',
    'REQUEST_FOR_DEADLOCK_SEARCH', 'XE_DISPATCHER_WAIT')
AND waiting_tasks_count > 0
ORDER BY wait_time_ms DESC;

-- Per-replica send and redo queue from the primary
SELECT
    ag.name AS ag_name,
    ar.replica_server_name,
    DB_NAME(drs.database_id) AS database_name,
    drs.synchronization_state_desc,
    drs.log_send_queue_size / 1024.0 AS send_queue_mb,
    drs.log_send_rate / 1024.0 AS send_rate_mbps,
    drs.redo_queue_size / 1024.0 AS redo_queue_mb,
    drs.redo_rate / 1024.0 AS redo_rate_mbps
FROM sys.dm_hadr_database_replica_states drs
JOIN sys.availability_replicas ar ON drs.replica_id = ar.replica_id
JOIN sys.availability_groups ag ON ar.group_id = ag.group_id
ORDER BY drs.log_send_queue_size DESC;

-- Active HADR_SYNC_COMMIT waits with resource details
SELECT
    r.session_id,
    r.wait_type,
    r.wait_time / 1000.0 AS wait_seconds,
    r.resource_description
FROM sys.dm_exec_requests r
WHERE r.wait_type = 'HADR_SYNC_COMMIT'
ORDER BY r.wait_time DESC;

-- AG replica connectivity and synchronization health
SELECT
    ag.name AS ag_name,
    ar.replica_server_name,
    ars.role_desc,
    ars.connected_state_desc,
    ars.synchronization_health_desc,
    ars.last_connect_error_description
FROM sys.dm_hadr_availability_replica_states ars
JOIN sys.availability_replicas ar ON ars.replica_id = ar.replica_id
JOIN sys.availability_groups ag ON ar.group_id = ag.group_id;

-- Secondary log write latency (run on the secondary)
SELECT
    DB_NAME(vfs.database_id) AS database_name,
    mf.name AS file_name,
    mf.type_desc,
    vfs.io_stall_write_ms,
    vfs.num_of_writes,
    CASE WHEN vfs.num_of_writes > 0
         THEN vfs.io_stall_write_ms * 1.0 / vfs.num_of_writes
         ELSE 0 END AS avg_write_latency_ms
FROM sys.dm_io_virtual_file_stats(NULL, NULL) vfs
JOIN sys.master_files mf
    ON vfs.database_id = mf.database_id AND vfs.file_id = mf.file_id
WHERE mf.type_desc = 'LOG'
ORDER BY avg_write_latency_ms DESC;

How to diagnose it

Diagnosis has to peel apart the possible locations of the bottleneck: secondary log file I/O, the network transport, the secondary redo path (readable workload interference), and the AG flow-control gates. The fixes are different for each.

flowchart TD
    A[HADR_SYNC_COMMIT dominant on primary] --> B{Send queue growing?}
    B -- Yes --> C[Network or send pipeline]
    B -- No --> D{Secondary log write latency high?}
    D -- Yes --> E[Secondary storage]
    D -- No --> F{Redo queue growing on secondary?}
    F -- Yes --> G[Secondary redo blocked or CPU-bound]
    F -- No --> H{Flow Control per sec nonzero?}
    H -- Yes --> I[AG transport throttling]
    H -- No --> J[Check readable workload + write burst]

  1. Confirm the wait is sustained and not background. Snapshot sys.dm_os_wait_stats 60 seconds apart, compute deltas, and verify that HADR_SYNC_COMMIT is in the top waits for the delta window. A single cumulative read is meaningless.

  2. Inspect each synchronous replica individually. The primary cannot tell you which replica is slow when more than one is in synchronous commit. Run the send-queue query and look for the outlier.

  3. On the slow secondary, measure log file write latency directly. Join sys.dm_io_virtual_file_stats to sys.master_files and look at write latency on the LOG files for the AG databases. Sustained write latency above 5ms is degraded; above 15ms is severe and will dominate HADR_SYNC_COMMIT.

  4. On the slow secondary, check the redo queue. A growing redo queue means the secondary is keeping up with harden but not with apply. That does not directly cause HADR_SYNC_COMMIT (the wait is on harden, not redo), but it is the same overload pattern and points to secondary CPU or redo-thread contention.

  5. On the primary, check the AG flow-control counters. SQL Server has two flow-control gates: a transport-level gate per availability replica and a database-level gate per availability database. When either gate is hit, the primary pauses sending log blocks until the secondary drains. This shows up as Flow Control/sec and Flow Control Time (ms/sec) being nonzero on SQLServer:Availability Replica, and it produces HADR_SYNC_COMMIT growth that is invisible to network bandwidth and disk latency metrics.

  6. Check whether readable secondary queries are competing with redo. Readable secondaries serve read queries that consume CPU, I/O, and memory resources needed by redo threads. Under load, redo falls behind processing the log stream, which can delay harden acknowledgement indirectly. Look at active requests on the secondary and at redo-related waits such as HADR_REDO_WAIT.

  7. Measure network round-trip time at the SQL layer, not just with ping. AG log transport uses TCP and the database mirroring endpoints. The most reliable operator check is to capture the hadr_db_commit_mgr_harden_still_waiting extended event, which fires precisely when the primary has been waiting beyond two seconds for a harden acknowledgement.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
HADR_SYNC_COMMIT as percentage of total waits (delta)Primary indicator that synchronous replication is the bottleneckSustained above 20-30% of waits during write windows
AG send queue per replicaWhether the secondary is keeping up with log shipmentGrowing send queue on a synchronous replica
AG redo queue per replicaDistinguishes harden problems from redo problemsGrowing redo queue, even when send queue is empty
Secondary log file write latencyThe most common root cause of HADR_SYNC_COMMITSustained above 5ms on the LOG file
Flow Control/sec on Availability ReplicaReveals AG transport throttling invisible to network and disk metricsAny sustained nonzero value
Required synchronized secondaries to commitDetermines whether a degraded replica can be safely moved to asyncA value greater than zero means primary writes will block on a degraded secondary
Write transaction commit latency (app or Query Store)The user-visible symptomSudden increase correlated with HADR_SYNC_COMMIT
THREADPOOL waits on primaryLate-stage cascade from sustained commit blockingAny sustained THREADPOOL indicates imminent worker exhaustion

Fixes

Emergency relief: switch the offending replica to asynchronous commit

When HADR_SYNC_COMMIT is causing user-visible impact and you have identified which replica is the source, the fastest relief is to switch that replica’s availability mode to asynchronous commit.

Warning: This accepts data-loss exposure on failover to that replica until the underlying cause is fixed and the replica is moved back to synchronous. Confirm the RPO impact with stakeholders before running it.

-- Destructive: changes AG availability mode, affecting RPO
-- Verify required_synchronized_secondaries_to_commit before running
ALTER AVAILABILITY GROUP [YourAGName]
MODIFY REPLICA ON N'SecondaryServer'
WITH (AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT);

This is a configuration change, not a failover. The replica stays connected and continues to receive log, just without the primary waiting for harden acknowledgements. If required_synchronized_secondaries_to_commit is greater than zero, you may need to lower it before writes will flow without blocking.

Switch back to synchronous only after you have fixed the storage, network, or workload cause. Otherwise the problem returns immediately.

Secondary storage fixes

If secondary log write latency is the cause, the fix is at the storage layer:

  • Move the secondary log files to faster storage. Log file latency is dominated by write IOPS and write latency, not throughput.
  • Confirm secondary log files are on dedicated spindles or disk tiers, not sharing with data files or other workloads.
  • Verify autogrowth is not triggering during the incident. Log file autogrow cannot use Instant File Initialization and stalls all log writes for the duration. Pre-size log files to peak workload.
  • On cloud VMs, verify the disk tier supports the log write IOPS you need. Cloud disk throttling is sudden and severe once the provisioned limit or burst credits are exhausted.

Network path fixes

If send queue is growing but log send rate is dropping, the network path is the suspect:

  • Confirm the AG endpoints are using dedicated network interfaces where possible.
  • Check for MTU mismatches causing fragmentation. AG log transport is sensitive to retransmissions.
  • Look for routing changes or congestion on the inter-replica path that does not show up in basic ping.
  • Verify the SQL Server endpoints are healthy and the certificates, if used, are not expired. Expired endpoint certificates cause disconnects that look like network problems.

Readable secondary interference

If redo queue is growing and the secondary is serving read queries:

  • Move reporting workloads off the secondary during peak write windows.
  • Investigate sessions on the secondary competing with redo for CPU, I/O, and memory. Check for redo-related waits.
  • Consider whether the secondary should be a read-intent-only target rather than a general read scale-out replica.

Flow-control gate tuning

When Flow Control/sec is nonzero and you are on SQL Server 2022 or later, the increased database-level flow-control gate limits are already in effect. On older supported builds, trace flag 12310 (startup only, requires restart) backports the higher gate limits.

Confirm that 12310 is appropriate for your workload and version before applying. It is a startup trace flag and cannot be enabled with DBCC TRACEON. Test in a non-production environment first.

For synchronous replicas, log stream compression is disabled by default starting in SQL Server 2016 to avoid adding latency. Do not enable trace flag 9592 (which enables compression on synchronous replicas) without measuring the latency impact carefully. Compression trades CPU for bytes, and on a synchronous path the extra CPU latency can worsen HADR_SYNC_COMMIT rather than help it.

Primary write burst management

If HADR_SYNC_COMMIT spikes during index maintenance or bulk loads, the primary is overwhelming the send pipeline:

  • Reduce MAXDOP for index rebuilds and reorganizations.
  • Batch large modifications instead of running them as single large transactions.
  • Consider temporarily switching affected replicas to async during maintenance windows, with explicit communication about RPO.

Prevention

  • Treat synchronous secondaries as production tier. They need the same storage class, network path, and CPU headroom as the primary. A secondary on cheaper storage silently throttles every primary commit.
  • Monitor HADR_SYNC_COMMIT as a time series with delta sampling, not as a single cumulative read. The DMV is cumulative since startup; without deltas you cannot see current impact.
  • Monitor AG flow-control counters on every primary. Nonzero values are an early warning that the AG transport is throttling before HADR_SYNC_COMMIT grows.
  • Monitor redo queue in time-to-catch-up units, not just bytes. Divide redo queue size by redo rate to estimate failover RTO.
  • Track secondary log write latency independently. Most teams monitor primary I/O stall but forget the secondary, which is where AG latency originates.
  • Schedule maintenance with AG impact in mind. Index rebuilds and bulk loads generate enormous log volume that strains the AG pipeline even when the secondary is healthy.
  • Validate failover readiness regularly. A synchronous replica that reports HEALTHY may still have grown redo queue that compromises RTO on failover.
  • Review required_synchronized_secondaries_to_commit against your actual RPO and RTO requirements. A higher value provides stronger consistency but means more replicas can block primary writes when one degrades.

How Netdata helps

  • Per-second wait statistics sampling with deltas surfaces HADR_SYNC_COMMIT as a current signal rather than a cumulative lifetime total. A 60-second snapshot is the smallest meaningful window for triage.
  • AG send queue, redo queue, log send rate, and redo rate per replica are collected alongside primary wait statistics, so you can correlate HADR_SYNC_COMMIT growth on the primary with the offending replica’s queue depth.
  • I/O stall per database file is captured continuously on both primary and secondary, so secondary log write latency can be trended and alerted on without setting up a separate collection job.
  • Flow-control counters on the Availability Replica object are exposed as time series, making AG transport throttling visible before it manifests as HADR_SYNC_COMMIT.
  • Anomaly detection on commit latency and transaction throughput surfaces the user-visible impact early, before THREADPOOL waits develop.

Netdata’s Microsoft SQL Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.