Oracle RMAN backup failures: V$RMAN_BACKUP_JOB_DETAILS and silent RPO loss

RMAN backups can fail for days without visible application impact. The database stays OPEN, queries succeed, transactions commit, and the alert log may show nothing actionable. The only authoritative record is in V$RMAN_BACKUP_JOB_DETAILS, a view that many shops either do not query or query incorrectly by filtering on STATUS = ‘FAILED’ and missing the more common partial-failure states.

The operational consequence is silent RPO loss. If the last successful backup is 9 days old and you discover this only when you need to restore, your effective recovery point objective is 9 days, regardless of what your runbook says. RMAN does not raise a pager when backups stop working. The scheduler runs, the script exits 0, and the database continues serving traffic with no safety net.

What this means

Each row in V$RMAN_BACKUP_JOB_DETAILS represents one backup job invocation, identified by SESSION_KEY, with START_TIME, END_TIME, INPUT_TYPE (DB FULL, DB INCR, ARCHIVELOG, and so on), and STATUS.

STATUS is a VARCHAR2 column with seven documented values:

The two states that bite operators are COMPLETED WITH WARNINGS and COMPLETED WITH ERRORS. A check that treats the job as healthy when STATUS != ‘FAILED’ will report green on jobs that skipped archived logs, dropped datafiles from the backup set, or hit channel errors that RMAN tolerated.

A second failure class is even quieter: the job never ran at all. OS cron stopped, the DBMS_SCHEDULER job was disabled, TNS connectivity broke, or the backup host crashed. V$RMAN_BACKUP_JOB_DETAILS gets no new row for that window. A STATUS-based query returns nothing, looks healthy, and silently extends RPO by another day.

Severity guidance: default backup status failures to TICKET (no successful backup inside the policy window, typically 24 hours), and escalate to PAGE only when RMAN is the sole recovery mechanism and the gap violates a documented RPO. Defaulting to PAGE on every backup blip trains operators to mute the channel.

Common causes

Quick checks

These queries are read-only and safe to run during production hours.

-- Last 10 backup jobs with status and elapsed time
SELECT SESSION_KEY, INPUT_TYPE, STATUS,
       TO_CHAR(START_TIME, 'YYYY-MM-DD HH24:MI') AS start_time,
       TO_CHAR(END_TIME,   'YYYY-MM-DD HH24:MI') AS end_time,
       ROUND((END_TIME - START_TIME) * 24 * 3600) AS elapsed_sec
FROM V$RMAN_BACKUP_JOB_DETAILS
ORDER BY START_TIME DESC
FETCH FIRST 10 ROWS ONLY;  -- 12c+. Pre-12c: wrap in a subquery with ROWNUM.

-- Any non-COMPLETED jobs in the last 7 days
SELECT SESSION_KEY, INPUT_TYPE, STATUS, START_TIME, END_TIME
FROM V$RMAN_BACKUP_JOB_DETAILS
WHERE STATUS <> 'COMPLETED'
  AND START_TIME > SYSDATE - 7
ORDER BY START_TIME DESC;

-- Last successful backup per input type
SELECT INPUT_TYPE, MAX(END_TIME) AS last_successful
FROM V$RMAN_BACKUP_JOB_DETAILS
WHERE STATUS = 'COMPLETED'
GROUP BY INPUT_TYPE
ORDER BY INPUT_TYPE;

-- Error and warning text for a specific job
-- Replace &session_key with a SESSION_KEY from the query above.
-- Join RMAN output by the job session RECID and STAMP; do not substitute SESSION_KEY for SESSION_RECID.
SELECT ro.OUTPUT
FROM V$RMAN_OUTPUT ro
JOIN V$RMAN_BACKUP_JOB_DETAILS jd
  ON ro.SESSION_RECID = jd.SESSION_RECID
 AND ro.SESSION_STAMP = jd.SESSION_STAMP
WHERE jd.SESSION_KEY = &session_key
ORDER BY ro.RECID;

-- Verify the scheduler actually ran the backup job
SELECT JOB_NAME, STATUS, ACTUAL_START_DATE, RUN_DURATION, ADDITIONAL_INFO
FROM DBA_SCHEDULER_JOB_RUN_DETAILS
WHERE JOB_NAME LIKE '%BACKUP%'
  AND ACTUAL_START_DATE > SYSDATE - 7
ORDER BY ACTUAL_START_DATE DESC;

How to diagnose it

1. Pull the last 14 days of jobs from V$RMAN_BACKUP_JOB_DETAILS. Anything other than COMPLETED is a candidate failure. Treat COMPLETED WITH WARNINGS and COMPLETED WITH ERRORS as failures for triage purposes until proven otherwise.

2. For each non-COMPLETED row, join the job to V$RMAN_OUTPUT on SESSION_RECID and SESSION_STAMP, then read the full text. The actionable error (RMAN-06059, ORA-19504, ORA-27072, sbt errors, “skipped” notices) is almost always in the last 30 lines of the job’s output.

3. Confirm the absence-of-backup case. If there is no row for a window where one was expected, the failure is upstream of RMAN. Check DBA_SCHEDULER_JOB_RUN_DETAILS for DBMS_SCHEDULER-driven backups, or OS cron and the backup host for externally scheduled jobs. The RMAN views only see jobs that actually reached the database.

4. Compare INPUT_TYPE coverage against policy. A common drift: ARCHIVELOG backups still succeed while DB FULL or DB INCR jobs fail. The database looks protected because “backups are running,” but the last full is weeks old and the recoverability window has shrunk to whatever the archived logs can cover.

5. Validate retention on the source. Old rows age out of the control file when CONTROL_FILE_RECORD_KEEP_TIME is shorter than your audit horizon. If you use a recovery catalog, query RC_RMAN_BACKUP_JOB_DETAILS for longer history; its 19c catalog view documentation lists the same running/ completed/failed status family and identifies SESSION_KEY, SESSION_RECID, and SESSION_STAMP as the join keys for catalog output.

flowchart TD
    A[Backup window opens] --> B{Row in V$RMAN_BACKUP_JOB_DETAILS?}
    B -- No --> C[Job never reached RMAN]
    C --> D[Check scheduler, cron, TNS, backup host]
    B -- Yes --> E{STATUS value?}
    E -- COMPLETED --> F[Healthy. Update last-success metric.]
    E -- COMPLETED WITH WARNINGS --> G[Partial gap. Read V$RMAN_OUTPUT.]
    E -- COMPLETED WITH ERRORS --> G
    E -- FAILED --> G
    G --> H{Skipped logs or datafiles?}
    H -- Yes --> I[RPO silently extended. Treat as failed.]
    H -- No --> J[Recoverable. Document and fix root cause.]

Metrics and signals to monitor

Fixes

Treat each subsection as a separate workstream.

Missing archived logs (RMAN-06059)

RMAN-06059 fires when RMAN expects an archived log that is no longer on disk. The backup completes with warnings, but the resulting backup set has a redo gap. Resolution options, in order of preference:

• Restore the missing archived log from a secondary destination (Data Guard standby, archive log mirror, or backup piece) and rerun the backup.
• If the log is genuinely lost and the window is already exposed, run CROSSCHECK ARCHIVELOG ALL followed by DELETE EXPIRED ARCHIVELOG from the RMAN prompt, then take a fresh incremental backup to reset the recoverability point. These commands only remove catalog entries for files that no longer exist on disk. Always review the CROSSCHECK output before DELETE.
• Prevent recurrence by reviewing archive deletion policy. DELETE OBSOLETE should not remove archived logs that are still needed by the most recent backup set.

Overlapping RMAN sessions

Two RMAN jobs targeting the same database simultaneously will collide on channel allocation or snapshot control file locks. The first job succeeds, the second ends with COMPLETED WITH ERRORS. Fixes:

• Serialize backup jobs through the scheduler. DBMS_SCHEDULER job chains or a single master script that calls each backup type in sequence prevents overlap.
• If concurrency is intentional (for example, ARCHIVELOG backups running alongside a level-0 incremental), separate them by window or use separate channels and snapshot control file locations.

Channel, media manager, or sbt errors

COMPLETED WITH ERRORS with ORA-195xx or sbt errors typically points to the media management layer (NetBackup, Tivoli, Data Domain, OSB) or the underlying storage. The RMAN side is healthy. Coordinate with the backup media team and check the media manager’s own logs, which V$RMAN_OUTPUT only echoes in summary form.

Job never started

No row in V$RMAN_BACKUP_JOB_DETAILS for the window means the failure is upstream. Common causes: OS cron stopped or its environment is missing ORACLE_HOME, the DBMS_SCHEDULER job was disabled or its credential object expired, TNS connectivity broke, or the backup host is down. Each upstream layer needs its own health check. The RMAN view cannot detect them.

Skipped tablespaces

COMPLETED WITH WARNINGS with “skipped” lines for offline or read-only tablespaces is benign only if the skip is intentional and the tablespace is recoverable by other means (for example, it is read-only and was backed up after being set read-only). If the skip was unexpected, the tablespace is silently outside the recovery set. Take a fresh backup and update the backup script’s inclusion list.

History truncation in the control file

V$RMAN_BACKUP_JOB_DETAILS is populated from the control file. Rows age out based on CONTROL_FILE_RECORD_KEEP_TIME. If your audit requirement is longer than the configured retention, raise the parameter, use a recovery catalog, or export the view to a historical table on a schedule.

Prevention

The reliable prevention pattern is a last-success metric that does not depend on STATUS at all. Track MAX(END_TIME) WHERE STATUS = ‘COMPLETED’ per INPUT_TYPE, and alert when that timestamp crosses the policy window. This single check catches both the failed-job and the job-never-ran cases, because both produce no new successful END_TIME.

Layered on top, monitor for any row in the last 24 hours with STATUS in (COMPLETED WITH WARNINGS, COMPLETED WITH ERRORS, FAILED). Treat COMPLETED WITH WARNINGS as a TICKET and COMPLETED WITH ERRORS as a TICKET or PAGE depending on whether the error represents a recoverability gap.

Quarterly, run a restore test on a non-production system. Backups that complete successfully are not the same as backups that restore. Track the last successful restore test date as a TICKET-grade signal when it exceeds 90 days.

How Netdata helps

• Per-second collection of Oracle metrics, including backup job status from V$RMAN_BACKUP_JOB_DETAILS, surfaces silent failures within minutes rather than after the next operator query.
• The natural unit of alerting is “no successful backup in N hours,” which catches both the failed-job and the job-never-ran failure modes that STATUS-only checks miss.
• Correlating RMAN job status with FRA utilization, archive destination free space, and alert log ORA- errors shortens root cause analysis: a COMPLETED WITH ERRORS row paired with a spike in V$RECOVERY_FILE_DEST usage points directly at an FRA pressure cascade.
• ML anomaly detection on backup duration and INPUT_TYPE cadence flags drift (jobs taking 3x longer than baseline, or ARCHIVELOG backups succeeding while DB INCR silently stops) before they cross a hard threshold.
• The same dashboard that shows instance health, redo generation, and archive destination state also shows the last-successful-backup timestamp, so RPO risk is visible alongside the signals that predict the next outage.

Netdata’s Oracle Database monitoring brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• How Oracle Database actually works in production: a mental model for operators
• Oracle ‘Checkpoint not complete’: redo log sizing, DBWn, and log-switch stalls
• Oracle ’log file sync’ waits: slow commits, LGWR, and the redo path
• Oracle Database monitoring checklist: the signals every production instance needs
• Oracle Database monitoring maturity model: from survival to expert
• ORA-00257: archiver error, connect internal only until freed
• Oracle redo generation rate: capacity planning for archiving and Data Guard
• Oracle redo log switch frequency: undersized logs and checkpoint pressure
• Oracle slow commit cascade: when redo storage degrades and every transaction waits