Oracle 'Thread N cannot allocate new log': the archive hang that masquerades as up

The alert log shows Thread 1 cannot allocate new log, sequence 12345. The instance is OPEN. lsnrctl status returns services. Your basic availability check, the one that runs SELECT 1 FROM DUAL, passes. The dashboard is green.

Meanwhile, every session that needs to commit is frozen on log file switch (archiving needed). TPS is at zero. Application connection pools are hung on their next write. New non-SYSDBA logins get ORA-00257; new SYSDBA logins may succeed but immediately block on the first redo-generating statement. This is the archive hang: instance up, commits dead.

The root cause is binary. The archiver cannot write filled online redo logs to the archive destination, so LGWR cannot reuse those logs, so the redo log buffer cannot be flushed, so every COMMIT in the database blocks. The instance stays OPEN the entire time. Recovery is conceptually simple (give ARCn somewhere to write), but diagnosis during the panic window is hard if you have not rehearsed it.

This article walks the failure mechanism, the safe read-only checks that confirm an archive hang in under a minute, the live fixes (space, FRA size, RMAN delete), and the monitoring signals that catch the cliff before sessions freeze.

What this means

When Oracle runs in ARCHIVELOG mode, ARCn copies each filled online redo log to the archive destination before LGWR can reuse it. The database has a finite number of redo log groups (typically 3-5), so this copy must keep up with redo generation or the cycle stalls.

The hang cascade is mechanical:

1. ARCn cannot write. Destination full, NFS unreachable, ASM disk group full, permission changed, or FRA exhausted.
2. Online redo logs fill and cannot be marked reusable.
3. LGWR cycles through all redo log groups and finds none available.
4. Foreground sessions that need to write redo (every COMMIT, every DML, even some SELECTs that touch undo) block on log file switch (archiving needed).
5. The alert log emits Thread N cannot allocate new log, sequence N. By the time you see this message, the hang is already in progress.

This defeats naive monitoring because instance status in V$INSTANCE stays OPEN and ACTIVE, the listener stays responsive, and a read-only health probe that does not generate redo can complete against an existing session. The database is technically alive and completely unusable.

flowchart TD
    A["ARCn cannot write
dest full / NFS down / FRA full"] --> B["Online redo logs fill
cannot be marked reusable"]
    B --> C["LGWR cycles through groups
finds none available"]
    C --> D["Alert log:
cannot allocate new log"]
    C --> E["Foreground COMMITs block on
log file switch archiving needed"]
    E --> F["TPS drops to zero
instance stays OPEN"]
    F --> G["Basic health checks pass
listener responds"]
    G --> H["Monitoring says UP
application says DOWN"]

Common causes

Quick checks

Run these read-only checks in order. They are safe during the incident and will confirm an archive hang inside a minute.

-- 1. Confirm the wait event that defines an archive hang
SELECT EVENT, COUNT(*) AS waiters
FROM V$SESSION
WHERE STATE = 'WAITING' AND WAIT_CLASS != 'Idle'
GROUP BY EVENT ORDER BY waiters DESC;
-- A dominant 'log file switch (archiving needed)' count, with TPS near zero, confirms the hang.

-- 2. Archive destination status and any error string
SELECT DEST_ID, STATUS, DESTINATION, ERROR
FROM V$ARCHIVE_DEST_STATUS WHERE STATUS != 'INACTIVE';
-- STATUS must be VALID. ERROR or DEFERRED is the smoking gun.

-- 3. FRA utilization if archives live in the FRA
SELECT NAME,
       SPACE_LIMIT/1073741824  AS limit_gb,
       SPACE_USED/1073741824   AS used_gb,
       SPACE_RECLAIMABLE/1073741824 AS reclaimable_gb
FROM V$RECOVERY_FILE_DEST;

-- 4. Archiver process health
SELECT PROCESS, STATUS, STATE, SEQUENCE#
FROM V$ARCHIVE_PROCESSES;
-- STATE should be IDLE or BUSY, not STOPPED.

-- 5. Online redo log group status (which groups are archived and reusable)
SELECT GROUP#, THREAD#, SEQUENCE#, BYTES/1048576 AS mb,
       ARCHIVED, STATUS, FIRST_TIME
FROM V$LOG ORDER BY GROUP#;
-- STATUS = ACTIVE or CURRENT with ARCHIVED = NO means the log cannot be reused.

# 6. OS-level free space on the archive destination
df -h /u01/app/oracle/fast_recovery_area  # or wherever log_archive_dest_N points

# 7. Tail the alert log for the cannot allocate new log and any destination errors
adrci exec="show alert -tail 50"

-- 8. Confirm TPS has collapsed
SELECT VALUE FROM V$SYSSTAT WHERE NAME = 'user commits';
-- Sample twice, 30 seconds apart. A flat counter is the functional definition of a hang.

How to diagnose it

1. Confirm it is an archive hang, not a lock cascade or a checkpoint stall. The wait event distribution from Quick check 1 is decisive. log file switch (archiving needed) across most active sessions, with TPS near zero, means archive hang. log file switch (checkpoint incomplete) means DBWn cannot flush dirty buffers; the cause and the fix are different. enq: TX - row lock contention with one or few blockers means a lock cascade, not an archive problem.
2. Locate the destination that failed. V$ARCHIVE_DEST_STATUS lists every configured destination. Look for STATUS = ERROR and read the ERROR column. If you use the FRA, V$RECOVERY_FILE_DEST tells you whether FRA exhaustion is the cause. The two views will usually agree.
3. Verify with OS-level evidence. Cross-check df, asmcmd lsdg, or NFS mount state. The database view and the OS view should agree. If they disagree, suspect a stale mount, an exhausted inode table on a small filesystem, or a permissions change on the destination directory.
4. Trace the wait chain if the picture is not clear. Walk V$SESSION.BLOCKING_SESSION from blocked foreground sessions up to LGWR and ARCn. With a Diagnostics Pack license, ASH-based wait chain reports give 1-second sampling granularity and longer history than the live V$SESSION view.
5. Capture a hanganalyze if normal SYSDBA login is blocked. When even sqlplus / as sysdba cannot log in (most commonly because a logon trigger performs DML that itself blocks on the redo path), connect with sqlplus -prelim / as sysdba and run oradebug hanganalyze 3. The prelim connection does not initialize a full session and bypasses logon triggers.

Metrics and signals to monitor

Fixes

Choose the fix by cause. The pattern is always the same: give ARCn somewhere to write, then let it drain the backlog.

Free space on the archive destination

If df shows the filesystem full, the fastest recovery is to remove archive logs that have already been backed up to tertiary storage. Use RMAN, not rm:

# Remove archive logs already backed up at least once to SBT (tape) or disk.
# Adjust DEVICE TYPE to match where your backups actually land (SBT for tape,
# DISK for disk-based backups). The wrong device type will delete nothing.
rman target / <<EOF
DELETE ARCHIVELOG ALL BACKED UP 1 TIMES TO DEVICE TYPE SBT;
EOF

Do not manually rm archive logs from the filesystem. Oracle does not know they are gone, the control file still references them, and you create recovery gaps. Always go through RMAN so the repository stays consistent.

Increase FRA size

If the FRA is the constraint, raise DB_RECOVERY_FILE_DEST_SIZE:

ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE = <larger_value> SCOPE = BOTH;

This is online and safe. Pair it with RMAN DELETE OBSOLETE to remove backups and archive logs that fall outside the configured retention policy. Otherwise the FRA fills again.

Recover an unreachable NFS or ASM destination

If the destination is NFS, restore the mount with the HARD option and direct I/O that Oracle requires. If ASM, add disks to the disk group or rebalance. After the destination is reachable again, ARCn will start draining the backlog automatically. No restart is required.

Add online redo log groups as a buffer

Adding redo log groups does not fix the root cause, but it buys time during a transient destination outage or while ARCn catches up:

ALTER DATABASE ADD LOGFILE GROUP <n> ('<member_path>') SIZE <current_size>;

This is a temporary runway extender, not a permanent fix. If redo consistently outpaces archiving, you need more archive throughput, not more redo log groups.

Data Guard transport stall

If a MANDATORY standby destination is unreachable and primary redo stalls, either restore connectivity or, as a last resort, defer the destination with ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_N = DEFER. Deferring a MANDATORY destination sacrifices data protection for primary availability. Reverse it the moment the standby is reachable again.

Prevention

• Monitor archive destination free space as a first-class signal, not an afterthought. This is the single highest-value monitoring you can add to an Oracle deployment. Alert at 85%.
• Compute archive runway daily. time_to_hang = unarchived_redo_space_remaining / (redo_rate - archive_throughput). If the denominator is positive, you are on a clock whether you know it or not.
• Validate that archive throughput sustains at least 2x the normal redo generation rate. Headroom absorbs spikes; matching redo rate does not.
• Keep at least 24 hours of archive logs at peak redo rate at the destination. Anything less invites a cascading failure during a backup delay.
• Align RMAN retention with FRA size. If retention requires more files than the FRA can hold, Oracle’s auto-deletion policy stops working and the FRA fills.
• Run a DML-based health check. A read-only SELECT 1 FROM DUAL does not exercise LGWR. A health check that inserts, commits, then deletes into a small heartbeat table exercises the full write path and catches the archive hang directly.
• Distinguish the red herrings. Occasional Private strand flush not complete lines in the alert log without cannot allocate new log are documented as expected behavior. A handful of Checkpoint not complete per day on a busy system is not an emergency. The signature of the real hang is cannot allocate new log combined with log file switch (archiving needed) waiters.
• Rehearse the prelim connection. The first time you try sqlplus -prelim / as sysdba should not be during an outage.

How Netdata helps

• Per-second TPS deviation from baseline. A near-zero commit rate during business hours is the functional signature of the archive hang. Per-second resolution and rolling baselines surface this within seconds, not the 5-minute window most polling-based tools need.
• Correlated alert log, destination, and TPS signals. Seeing cannot allocate new log lines coincide with a free-space drop on the archive mount and a TPS collapse shortens diagnosis from 30 to 60 minutes of confused triage down to under a minute.
• Archive destination and FRA utilization trends. Tracking the slope of free-space decline days before the cliff lets you plan cleanup rather than react to a hang.
• Wait event distribution over time. A shift in the dominant wait class from User I/O or Commit to Configuration (where the log file switch events live) is the leading indicator that the redo path is stalled.
• ML anomaly detection on redo generation rate. Sudden spikes in redo from bulk loads, supplemental logging being turned on, or a runaway batch compress archive runway. Anomaly alerts give hours of warning instead of seconds.

Netdata’s Oracle Database monitoring brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• How Oracle Database actually works in production: a mental model for operators
• Oracle ‘Checkpoint not complete’: redo log sizing, DBWn, and log-switch stalls
• Oracle ’log file sync’ waits: slow commits, LGWR, and the redo path
• Oracle Database monitoring checklist: the signals every production instance needs
• Oracle Database monitoring maturity model: from survival to expert
• Oracle redo generation rate: capacity planning for archiving and Data Guard
• Oracle redo log switch frequency: undersized logs and checkpoint pressure
• Oracle slow commit cascade: when redo storage degrades and every transaction waits