RMAN backups can fail for days without visible application impact. The database stays OPEN, queries succeed, transactions commit, and the alert log may show nothing actionable. The only authoritative record is in V$RMAN_BACKUP_JOB_DETAILS, a view that many shops either do not query or query incorrectly by filtering on STATUS = ‘FAILED’ and missing the more common partial-failure states.

The operational consequence is silent RPO loss. If the last successful backup is 9 days old and you discover this only when you need to restore, your effective recovery point objective is 9 days, regardless of what your runbook says. RMAN does not raise a pager when backups stop working. The scheduler runs, the script exits 0, and the database continues serving traffic with no safety net.

What this means

Each row in V$RMAN_BACKUP_JOB_DETAILS represents one backup job invocation, identified by SESSION_KEY, with START_TIME, END_TIME, INPUT_TYPE (DB FULL, DB INCR, ARCHIVELOG, and so on), and STATUS.

STATUS is a VARCHAR2 column with six possible values:

STATUSMeaning
RUNNING WITH WARNINGSJob in progress; warnings emitted
RUNNING WITH ERRORSJob in progress; errors emitted
COMPLETEDJob finished cleanly
COMPLETED WITH WARNINGSJob finished; some operations warned
COMPLETED WITH ERRORSJob finished; some operations failed
FAILEDJob aborted; backup set incomplete or unusable

The two states that bite operators are COMPLETED WITH WARNINGS and COMPLETED WITH ERRORS. A check that treats the job as healthy when STATUS != ‘FAILED’ will report green on jobs that skipped archived logs, dropped datafiles from the backup set, or hit channel errors that RMAN tolerated.

A second failure class is even quieter: the job never ran at all. OS cron stopped, the DBMS_SCHEDULER job was disabled, TNS connectivity broke, or the backup host crashed. V$RMAN_BACKUP_JOB_DETAILS gets no new row for that window. A STATUS-based query returns nothing, looks healthy, and silently extends RPO by another day.

Severity guidance: default backup status failures to TICKET (no successful backup inside the policy window, typically 24 hours), and escalate to PAGE only when RMAN is the sole recovery mechanism and the gap violates a documented RPO. Defaulting to PAGE on every backup blip trains operators to mute the channel.

Common causes

CauseWhat it looks likeFirst thing to check
Missing archived logs (RMAN-06059)COMPLETED WITH WARNINGS; archived log gap in V$RMAN_OUTPUTJoin V$RMAN_BACKUP_JOB_DETAILS.SESSION_KEY to V$RMAN_OUTPUT.SESSION_RECID
Overlapping RMAN sessionsCOMPLETED WITH ERRORS on the second job; channel allocation failures in outputLook for concurrent rows with overlapping START_TIME/END_TIME
Channel or media manager errorsCOMPLETED WITH ERRORS or FAILED; ORA-195xx or sbt errors in outputInspect V$RMAN_OUTPUT for the failing SESSION_KEY
Job never startedNo new row for the windowCheck DBA_SCHEDULER_JOB_RUN_DETAILS or OS cron logs
Skipped offline or read-only tablespaceCOMPLETED WITH WARNINGS; “skipped” lines in outputCross-reference skipped tablespaces against recovery requirements
History truncated by control file age-outOld SESSION_KEY rows disappearVerify CONTROL_FILE_RECORD_KEEP_TIME or use a recovery catalog

Quick checks

These queries are read-only and safe to run during production hours.

-- Last 10 backup jobs with status and elapsed time
SELECT SESSION_KEY, INPUT_TYPE, STATUS,
       TO_CHAR(START_TIME, 'YYYY-MM-DD HH24:MI') AS start_time,
       TO_CHAR(END_TIME,   'YYYY-MM-DD HH24:MI') AS end_time,
       ROUND((END_TIME - START_TIME) * 24 * 3600) AS elapsed_sec
FROM V$RMAN_BACKUP_JOB_DETAILS
ORDER BY START_TIME DESC
FETCH FIRST 10 ROWS ONLY;  -- 12c+. Pre-12c: wrap in a subquery with ROWNUM.

-- Any non-COMPLETED jobs in the last 7 days
SELECT SESSION_KEY, INPUT_TYPE, STATUS, START_TIME, END_TIME
FROM V$RMAN_BACKUP_JOB_DETAILS
WHERE STATUS <> 'COMPLETED'
  AND START_TIME > SYSDATE - 7
ORDER BY START_TIME DESC;

-- Last successful backup per input type
SELECT INPUT_TYPE, MAX(END_TIME) AS last_successful
FROM V$RMAN_BACKUP_JOB_DETAILS
WHERE STATUS = 'COMPLETED'
GROUP BY INPUT_TYPE
ORDER BY INPUT_TYPE;

-- Error and warning text for a specific job
-- Replace &session_key with a SESSION_KEY from the query above.
-- V$RMAN_OUTPUT uses SESSION_RECID, which maps to SESSION_KEY in V$RMAN_BACKUP_JOB_DETAILS.
SELECT OUTPUT
FROM V$RMAN_OUTPUT
WHERE SESSION_RECID = &session_key
ORDER BY RECID;

-- Verify the scheduler actually ran the backup job
SELECT JOB_NAME, STATUS, ACTUAL_START_DATE, RUN_DURATION, ADDITIONAL_INFO
FROM DBA_SCHEDULER_JOB_RUN_DETAILS
WHERE JOB_NAME LIKE '%BACKUP%'
  AND ACTUAL_START_DATE > SYSDATE - 7
ORDER BY ACTUAL_START_DATE DESC;

How to diagnose it

  1. Pull the last 14 days of jobs from V$RMAN_BACKUP_JOB_DETAILS. Anything other than COMPLETED is a candidate failure. Treat COMPLETED WITH WARNINGS and COMPLETED WITH ERRORS as failures for triage purposes until proven otherwise.

  2. For each non-COMPLETED row, join SESSION_KEY to V$RMAN_OUTPUT.SESSION_RECID and read the full text. The actionable error (RMAN-06059, ORA-19504, ORA-27072, sbt errors, “skipped” notices) is almost always in the last 30 lines of the job’s output.

  3. Confirm the absence-of-backup case. If there is no row for a window where one was expected, the failure is upstream of RMAN. Check DBA_SCHEDULER_JOB_RUN_DETAILS for DBMS_SCHEDULER-driven backups, or OS cron and the backup host for externally scheduled jobs. The RMAN views only see jobs that actually reached the database.

  4. Compare INPUT_TYPE coverage against policy. A common drift: ARCHIVELOG backups still succeed while DB FULL or DB INCR jobs fail. The database looks protected because “backups are running,” but the last full is weeks old and the recoverability window has shrunk to whatever the archived logs can cover.

  5. Validate retention on the source. Old rows age out of the control file when CONTROL_FILE_RECORD_KEEP_TIME is shorter than your audit horizon. If you use a recovery catalog, query the catalog’s equivalent view for longer history.

flowchart TD
    A[Backup window opens] --> B{Row in V$RMAN_BACKUP_JOB_DETAILS?}
    B -- No --> C[Job never reached RMAN]
    C --> D[Check scheduler, cron, TNS, backup host]
    B -- Yes --> E{STATUS value?}
    E -- COMPLETED --> F[Healthy. Update last-success metric.]
    E -- COMPLETED WITH WARNINGS --> G[Partial gap. Read V$RMAN_OUTPUT.]
    E -- COMPLETED WITH ERRORS --> G
    E -- FAILED --> G
    G --> H{Skipped logs or datafiles?}
    H -- Yes --> I[RPO silently extended. Treat as failed.]
    H -- No --> J[Recoverable. Document and fix root cause.]

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Last successful END_TIME per INPUT_TYPEDirectly defines recoverability windowAny INPUT_TYPE older than its policy window
STATUS distributionDistinguishes hard failures from partial successesAny COMPLETED WITH WARNINGS or COMPLETED WITH ERRORS
SESSION_KEY presence per expected windowCatches the no-row failure modeZero new rows in the expected window
V$RMAN_OUTPUT warning patternsSurfaces RMAN-06059 and similar before they extend RPORMAN-06059, ORA-195xx, “skipped” notices
CONTROL_FILE_RECORD_KEEP_TIMESets how long V$RMAN_BACKUP_JOB_DETAILS history survivesValue shorter than your backup audit requirement
DBA_SCHEDULER_JOB_RUN_DETAILS.STATUSCatches scheduler-side failures upstream of RMANFAILED rows or no rows for expected backup jobs
V$RECOVERY_FILE_DEST space utilizationFRA full can cascade into RMAN failures and archive hangsSPACE_USED / SPACE_LIMIT above 0.85

Fixes

Treat each subsection as a separate workstream.

Missing archived logs (RMAN-06059)

RMAN-06059 fires when RMAN expects an archived log that is no longer on disk. The backup completes with warnings, but the resulting backup set has a redo gap. Resolution options, in order of preference:

  • Restore the missing archived log from a secondary destination (Data Guard standby, archive log mirror, or backup piece) and rerun the backup.
  • If the log is genuinely lost and the window is already exposed, run CROSSCHECK ARCHIVELOG ALL followed by DELETE EXPIRED ARCHIVELOG from the RMAN prompt, then take a fresh incremental backup to reset the recoverability point. These commands only remove catalog entries for files that no longer exist on disk. Always review the CROSSCHECK output before DELETE.
  • Prevent recurrence by reviewing archive deletion policy. DELETE OBSOLETE should not remove archived logs that are still needed by the most recent backup set.

Overlapping RMAN sessions

Two RMAN jobs targeting the same database simultaneously will collide on channel allocation or snapshot control file locks. The first job succeeds, the second ends with COMPLETED WITH ERRORS. Fixes:

  • Serialize backup jobs through the scheduler. DBMS_SCHEDULER job chains or a single master script that calls each backup type in sequence prevents overlap.
  • If concurrency is intentional (for example, ARCHIVELOG backups running alongside a level-0 incremental), separate them by window or use separate channels and snapshot control file locations.

Channel, media manager, or sbt errors

COMPLETED WITH ERRORS with ORA-195xx or sbt errors typically points to the media management layer (NetBackup, Tivoli, Data Domain, OSB) or the underlying storage. The RMAN side is healthy. Coordinate with the backup media team and check the media manager’s own logs, which V$RMAN_OUTPUT only echoes in summary form.

Job never started

No row in V$RMAN_BACKUP_JOB_DETAILS for the window means the failure is upstream. Common causes: OS cron stopped or its environment is missing ORACLE_HOME, the DBMS_SCHEDULER job was disabled or its credential object expired, TNS connectivity broke, or the backup host is down. Each upstream layer needs its own health check. The RMAN view cannot detect them.

Skipped tablespaces

COMPLETED WITH WARNINGS with “skipped” lines for offline or read-only tablespaces is benign only if the skip is intentional and the tablespace is recoverable by other means (for example, it is read-only and was backed up after being set read-only). If the skip was unexpected, the tablespace is silently outside the recovery set. Take a fresh backup and update the backup script’s inclusion list.

History truncation in the control file

V$RMAN_BACKUP_JOB_DETAILS is populated from the control file. Rows age out based on CONTROL_FILE_RECORD_KEEP_TIME. If your audit requirement is longer than the configured retention, raise the parameter, use a recovery catalog, or export the view to a historical table on a schedule.

Prevention

The reliable prevention pattern is a last-success metric that does not depend on STATUS at all. Track MAX(END_TIME) WHERE STATUS = ‘COMPLETED’ per INPUT_TYPE, and alert when that timestamp crosses the policy window. This single check catches both the failed-job and the job-never-ran cases, because both produce no new successful END_TIME.

Layered on top, monitor for any row in the last 24 hours with STATUS in (COMPLETED WITH WARNINGS, COMPLETED WITH ERRORS, FAILED). Treat COMPLETED WITH WARNINGS as a TICKET and COMPLETED WITH ERRORS as a TICKET or PAGE depending on whether the error represents a recoverability gap.

Quarterly, run a restore test on a non-production system. Backups that complete successfully are not the same as backups that restore. Track the last successful restore test date as a TICKET-grade signal when it exceeds 90 days.

How Netdata helps

  • Per-second collection of Oracle metrics, including backup job status from V$RMAN_BACKUP_JOB_DETAILS, surfaces silent failures within minutes rather than after the next operator query.
  • The natural unit of alerting is “no successful backup in N hours,” which catches both the failed-job and the job-never-ran failure modes that STATUS-only checks miss.
  • Correlating RMAN job status with FRA utilization, archive destination free space, and alert log ORA- errors shortens root cause analysis: a COMPLETED WITH ERRORS row paired with a spike in V$RECOVERY_FILE_DEST usage points directly at an FRA pressure cascade.
  • ML anomaly detection on backup duration and INPUT_TYPE cadence flags drift (jobs taking 3x longer than baseline, or ARCHIVELOG backups succeeding while DB INCR silently stops) before they cross a hard threshold.
  • The same dashboard that shows instance health, redo generation, and archive destination state also shows the last-successful-backup timestamp, so RPO risk is visible alongside the signals that predict the next outage.

Netdata’s Oracle Database monitoring brings these signals together with per-second metrics and ML anomaly detection.