Oracle Database Operations Guides on Netdata

How Oracle Database actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

Oracle is dense, but four abstractions make most production incidents legible: the shared memory region (SGA), per-process private memory (PGA), the background processes that move data to disk, and the wait-event model that tells you where time is going. When you see log file sync climbing, or free buffer waits appearing, or the database hanging while basic health checks still pass, you should know immediately which subsystem is involved and what it competes for.

ORA-00020: maximum number of processes exceeded

Mon, 01 Jan 0001 00:00:00 +0000

ORA-00020 is a connection-admission cliff-edge. At 99% of the configured PROCESSES limit, everything works. At 100%, new connections are refused and the error cascades into every monitoring tool, DBA session, and application pool. Existing sessions keep running, which is why this is often discovered late.

The symptom: applications report connection failures, the listener answers TCP but the database rejects the actual connect with ORA-00020, and your usual SYSDBA session may also be refused. The alert log fills with the error. If your connection pool retries in a tight loop, the situation worsens before it improves, because each retry is another failed admission attempt.

ORA-00060: deadlock detected while waiting for resource

Mon, 01 Jan 0001 00:00:00 +0000

ORA-00060 fires when Oracle’s background deadlock detector finds a wait cycle among sessions competing for enqueues. By the time you see it, Oracle has already resolved it: one session was picked as the victim, its current statement was rolled back, and the other sessions in the cycle continued. The victim session is still connected and its transaction is still open. The application must commit or roll it back.

The work is finding which sessions deadlocked, what SQL each was running, and why their lock ordering conflicted. Oracle’s trace file states this explicitly: ORA-00060 is a user application design issue, not an Oracle error. The fix lives in the application or the schema, not in a database parameter.

ORA-00257: archiver error, connect internal only until freed

Mon, 01 Jan 0001 00:00:00 +0000

Mid-incident: users report the application is hung. New connections from application servers fail with ORA-00257. Existing sessions are stuck and return no error. The instance reports OPEN and ACTIVE. The listener responds to TCP probes. A read-only health check says the database is up. The database is not up.

ORA-00257 is the connect-time symptom of an Archive Hang. The archiver process (ARCn) cannot copy filled online redo logs to the archive destination. Online redo logs fill, the log writer (LGWR) cannot switch to a new log group, and every session that needs to generate redo freezes. Only non-SYSDBA sessions attempting to connect see an error. Everyone else just waits.

ORA-00600: internal error code, arguments - triage and what to capture

Mon, 01 Jan 0001 00:00:00 +0000

ORA-00600 is Oracle’s catch-all internal error code. A server process hit an unexpected condition inside the kernel and failed an internal assertion. The database may stay up, the failing call rolls back, and the system may look normal for minutes or hours, but the engine has reported a condition you cannot fix from SQL.

The message format is ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], []. The first bracketed token, here [kdsgrp1], is the internal message number and the single most important input for Oracle Support. Multiple distinct bugs can assert in the same internal function, so the argument narrows the search but does not uniquely identify the bug. The full argument list plus the exact database version, down to patch level, is what Support needs to find the matching known bug.

ORA-01555: snapshot too old, rollback segment too small

Mon, 01 Jan 0001 00:00:00 +0000

ORA-01555 is an Oracle read-path failure. A long-running query needs an old undo version of a block that has already been overwritten, so the query errors out:

ORA-01555: snapshot too old (rollback segment too small)

The “rollback segment” wording is a legacy artifact. Manual rollback segments were deprecated when Automatic Undo Management (AUM) was introduced in 9i. The actual cause is undo pressure inside an AUM undo tablespace. Treating this as a “rollback segment” problem leads down the wrong path.

ORA-01578: ORACLE data block corrupted (file #, block #)

Mon, 01 Jan 0001 00:00:00 +0000

ORA-01578 is raised when Oracle reads a data block whose contents fail internal validation. The block is marked corrupt and the read fails. The instance keeps running because corruption rarely crashes it, but every query touching that block errors until the block is repaired or bypassed.

Two risks follow. First, the data in that block is inaccessible or lost. Second, backups may be compromised: RMAN detects corrupt blocks during backup by default (MAXCORRUPT=0), but without proactive VALIDATE you may not know when corruption started or which backup pieces are clean.

ORA-01652: unable to extend temp segment in tablespace

Mon, 01 Jan 0001 00:00:00 +0000

ORA-01652 fires when Oracle cannot allocate another extent for a temporary segment in a tablespace. In production the tablespace is almost always the default TEMP, and the consumers are operations that have spilled out of the PGA: large sorts, hash joins, global temporary tables, and temporary LOBs. The error is a hard stop for the failing statement, while every other session that needs temp behind it queues on direct path write temp.

ORA-01653: unable to extend table in tablespace

Mon, 01 Jan 0001 00:00:00 +0000

ORA-01653 is the hard stop. A DML statement needs to extend a table segment, Oracle cannot allocate the next extent in the tablespace, and the statement fails immediately. There is no graceful degradation: the tablespace was performing normally a moment ago, and now any write that needs new space is rejected. Existing committed data is intact, but inserts, updates that grow row size, index maintenance behind constraints, and any operation that requires new extents all error out.

ORA-01654: unable to extend index in tablespace

Mon, 01 Jan 0001 00:00:00 +0000

When ORA-01654 fires, an index segment tried to allocate a new extent and could not. The statement fails. Depending on the index, this can stop a bulk load, an ALTER INDEX ... REBUILD, or any DML that modifies the index. It is the index-specific sibling of ORA-01653: same mechanism (segment extent allocation failure), same fix surface (tablespace capacity), different failing object.

Two operator surprises are common. First, the failing index often lives in a different tablespace than its base table. Default schemas create indexes alongside tables, but production layouts commonly split indexes into a dedicated INDX tablespace. When the error fires, investigate the index’s tablespace, not the table’s.

ORA-04031: unable to allocate bytes of shared memory (shared pool)

Mon, 01 Jan 0001 00:00:00 +0000

ORA-04031 fires when a session needs a chunk of shared pool memory and Oracle cannot find a single contiguous free region large enough. The full message lists four values: bytes requested, pool name, allocation type, and heap name. A typical first sighting: ORA-04031: unable to allocate 4032 bytes of shared memory ("shared pool","unknown object","sga heap","row cache buffers"). It can also fire against the large pool, java pool, or streams pool.

ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT

Mon, 01 Jan 0001 00:00:00 +0000

ORA-04036 fires when instance-wide PGA consumption crosses PGA_AGGREGATE_LIMIT, the hard cap introduced in 12c. This is not a tuning warning. The database is defending itself by killing or interrupting work.

PGA_AGGREGATE_TARGET is a soft target Oracle tries to honor. PGA_AGGREGATE_LIMIT is an enforced ceiling. Sessions can exceed the target legitimately. Crossing the limit triggers Oracle to abort the call of, and then terminate, the sessions holding the most untunable PGA. SYS and most background processes are not eligible for termination, which produces a distinct and more dangerous failure mode described below.

ORA-07445: exception encountered, core dump — a process crash in Oracle code

Mon, 01 Jan 0001 00:00:00 +0000

ORA-07445 in the alert log means an Oracle server process received a fatal operating system signal and dumped core. A foreground or background process crashed inside Oracle kernel code, PMON cleaned up the session, and Oracle wrote an incident to the Automatic Diagnostic Repository (ADR). Unlike a normal ORA- error returned to a client, ORA-07445 is the kernel telling you a process died underneath the database.

This error is always critical. ORA-07445 in a production instance warrants immediate investigation regardless of frequency, and the standard escalation is to capture the trace and engage Oracle Support. It is not a tuning signal, not a capacity warning, and not something you can ignore because the database appears to keep running. A single ORA-07445 can be an isolated bug hit; a cluster of them can mean corruption, a bad patch, or failing hardware.

ORA-30036: unable to extend segment in undo tablespace

Mon, 01 Jan 0001 00:00:00 +0000

A session executing DML fails with ORA-30036: unable to extend segment by N in undo tablespace. The transaction rolls back and any other session needing undo for the same tablespace also fails. Unlike ORA-01555 (“snapshot too old”), which is a read-path failure, ORA-30036 is a write-path failure: Oracle cannot find space to record the before-image of a data change.

The error is cliff-edge. The undo tablespace is the shared resource every DML writes to. When it is exhausted by ACTIVE undo (undo records belonging to uncommitted transactions that Oracle cannot overwrite), no new DML can record its undo and the operation fails immediately.

Oracle 'buffer busy waits': hot blocks, sequence headers, and index leaf splits

Mon, 01 Jan 0001 00:00:00 +0000

buffer busy waits fires when a session needs a buffer that another session currently has pinned in the buffer cache. It is a contention signal, not an I/O signal. The waiting session is blocked by a session holding the buffer, not by storage latency.

Since Oracle 10.1 this event has been distinct from read by other session, which fires when a session waits for another session to finish reading a block from disk into cache. Before 10.1 both conditions collapsed into buffer busy waits. Modern Oracle (19c, 23ai) keeps the four-event split: buffer busy waits, read by other session, gc buffer busy acquire, and gc buffer busy release. The last two are RAC-only.

Oracle 'Checkpoint not complete': redo log sizing, DBWn, and log-switch stalls

Mon, 01 Jan 0001 00:00:00 +0000

The alert log says Checkpoint not complete followed by Current log# N seq# N mem# N: <path>. Foreground sessions stall on log file switch (checkpoint incomplete) at every log switch. Commits hesitate for tens of milliseconds to seconds, and the stall repeats each time LGWR wraps to the next redo log group. The wrong fix (enlarging redo logs when the real bottleneck is DBWn I/O) only buys minutes.

This guide assumes you understand the Oracle write path from how Oracle Database works in production.

Oracle 'cursor: pin S wait on X': mutex contention on hot cursors

Mon, 01 Jan 0001 00:00:00 +0000

Sessions accumulating on cursor: pin S wait on X want a shared (S) mutex pin on a cached cursor while another session holds an exclusive (X) pin on the same cursor object. The X holder is usually hard parsing, invalidating the cursor, or doing library cache maintenance. Waiters queue behind a single mutex.

This event sits in the library cache mutex family alongside library cache: mutex X, cursor: mutex S, and cursor: mutex X. All of them reflect contention on the shared SQL area. cursor: pin S wait on X specifically points at cursor-level pin contention, almost always driven by a small number of hot SQL IDs that are parsed, invalidated, or executed frequently, or that have accumulated an unreasonable number of child cursors.

Oracle 'db file scattered read': multiblock reads, full scans, and plan regressions

Mon, 01 Jan 0001 00:00:00 +0000

A sudden spike in db file scattered read wait time on an OLTP database is one of the most reliable signals of an execution plan regression. The wait event itself is benign on analytics and warehouse workloads, where multiblock full scans are the expected access path. On a transactional system it usually means a query that used to do a handful of index reads is now scanning whole tables.

This guide covers the failure pattern, the path from wait event to offending SQL_ID, and the tradeoffs in the common fixes. It is scoped to single-instance and RAC databases doing buffered multiblock reads. From 11g onward, many large serial scans bypass the cache via direct path read instead; that event is covered where it affects diagnosis.

Oracle 'db file sequential read': single-block index reads and buffer cache misses

Mon, 01 Jan 0001 00:00:00 +0000

db file sequential read is the dominant single-block I/O wait event in Oracle OLTP. Each time a foreground process needs one block, finds it missing from the buffer cache, and waits for the read from a datafile into the SGA, the session accrues time on this event. The name is historical: “sequential” refers to the single-block read into a specific buffer cache slot, not to a sequential scan pattern. Multiblock scans are tracked separately as db file scattered read.

Oracle 'enq: TM - contention': unindexed foreign keys and table-level locks

Mon, 01 Jan 0001 00:00:00 +0000

enq: TM - contention is the wait event a session emits when it wants a DML (table) enqueue on an object and another session already holds a conflicting mode. Unlike enq: TX - row lock contention, which is row-level and usually about uncommitted transactions, TM contention is almost always structural: an unindexed foreign key that forces Oracle to take a full table lock where it would otherwise take a row lock.

Oracle 'enq: TX - row lock contention': blocking sessions and uncommitted DML

Mon, 01 Jan 0001 00:00:00 +0000

The classic symptom is partial slowness. Some transactions succeed. Some hang indefinitely. Application logs show threads stuck inside database calls. There are no ORA- errors in the alert log, CPU is low, I/O latency is normal, the instance is OPEN, and the listener responds.

When you query V$SESSION for the waiters, they are all parked on the same event: enq: TX - row lock contention. When you walk BLOCKING_SESSION up the chain, you usually land on a session whose STATUS is INACTIVE and whose wait is SQL*Net message from client. That session is sitting idle with an open, uncommitted transaction. One stuck holder, many waiters behind it.

Oracle 'free buffer waits': when DBWn can't clean buffers fast enough

Mon, 01 Jan 0001 00:00:00 +0000

Sessions waiting on free buffer waits cannot find a clean buffer in the cache to read a new block into. The buffer cache is full of dirty buffers that DBWn has not yet flushed, so foreground processes that need to read a new block have nowhere to put it.

This is a write-path bottleneck, not a read-path problem. The root cause is DBWn falling behind on writes: either the storage cannot absorb the write rate, or DBWn itself is CPU-starved or process-limited. The fix is on the write side. Throwing a bigger buffer cache at the problem does not help and frequently makes it worse.

Oracle 'library cache: mutex X' waits: parsing pressure and cursor contention

Mon, 01 Jan 0001 00:00:00 +0000

When library cache: mutex X shows up as a dominant wait event, the database is burning CPU on parsing rather than on query execution. Sessions serialize behind exclusive mutexes that protect the shared SQL area, throughput erodes, and the instance still reports OPEN/ACTIVE on every basic availability check. The shared SQL cache cannot keep up with the rate of new SQL text the application is sending.

This pattern almost always appears alongside elevated parse count (hard), declining shared pool free memory, and the sibling waits cursor: pin S wait on X and latch: shared pool. If the rate is high enough and sustained, shared pool fragmentation follows and ORA-04031 becomes a real risk.

Oracle 'log file sync' waits: slow commits, LGWR, and the redo path

Mon, 01 Jan 0001 00:00:00 +0000

Every COMMIT in Oracle is a synchronous handshake with the Log Writer (LGWR). The foreground session hands off its redo, waits for LGWR to flush that redo to the online redo logs, and only then returns control to the application. The wait event that covers this round trip is log file sync. When LGWR is slow, every committing transaction is slow. This is the most common “everything is uniformly slow” pattern in Oracle and the single most important latency signal to monitor.

Oracle 'Thread N cannot allocate new log': the archive hang that masquerades as up

Mon, 01 Jan 0001 00:00:00 +0000

The alert log shows Thread 1 cannot allocate new log, sequence 12345. The instance is OPEN. lsnrctl status returns services. Your basic availability check, the one that runs SELECT 1 FROM DUAL, passes. The dashboard is green.

Meanwhile, every session that needs to commit is frozen on log file switch (archiving needed). TPS is at zero. Application connection pools are hung on their next write. New non-SYSDBA logins get ORA-00257; new SYSDBA logins may succeed but immediately block on the first redo-generating statement. This is the archive hang: instance up, commits dead.

Oracle archive log destination full: V$ARCHIVE_DEST_STATUS, the ERROR state, and space

Mon, 01 Jan 0001 00:00:00 +0000

The most dangerous Oracle outage masquerades as “database up.” The instance shows OPEN and ACTIVE in V$INSTANCE, the listener answers TCP probes, existing sessions stay connected, and basic availability checks pass. But the database is frozen because ARCn cannot write archived redo logs and LGWR cannot switch online redo log groups. Every session that needs to generate redo hangs on log file switch (archiving needed), and new non-SYSDBA connections receive ORA-00257.

Oracle autoextend hit MAXSIZE: the space gotcha with a half-empty filesystem

Mon, 01 Jan 0001 00:00:00 +0000

ORA-01653 (table) or ORA-01654 (index) fires on a production instance. Application writes fail. You log in, check df on the datafile filesystem, and see hundreds of gigabytes free. ASM disk group shows plenty of headroom. There is no obvious space problem at the storage layer. Yet Oracle insists the tablespace cannot extend.

This is the autoextend ceiling gotcha. The datafile has AUTOEXTEND ON, but it has reached its MAXSIZE. Oracle refuses to grow the file further even though the disk beneath has abundant space. Disk-only monitoring never sees this coming because the disk is not the constraint. The per-datafile MAXSIZE is.

Oracle blocking sessions: finding the blocker at the head of the chain

Mon, 01 Jan 0001 00:00:00 +0000

Blocking sessions are the most common cause of “the database is slow but everything looks healthy.” One session holds an uncommitted transaction on a row; every other session that wants to touch that row queues behind it on enq: TX - row lock contention. The instance is OPEN, the listener responds, CPU is often low, and TPS quietly decays. From the outside it looks like a performance regression rather than a lock incident.

Oracle buffer cache hit ratio: the most misused metric in Oracle monitoring

Mon, 01 Jan 0001 00:00:00 +0000

The buffer cache hit ratio appears in nearly every legacy monitoring template, executive dashboard, and database health report. It is also one of the least useful signals for diagnosing real production problems.

The formula is simple: 1 - (physical reads / (db block gets + consistent gets)), computed from cumulative counters in V$SYSSTAT. A high percentage looks reassuring. A low percentage looks alarming. Neither reaction is reliably correct. A system doing nothing but SELECT * FROM dual in a loop has a 99.99% hit ratio and zero useful work. A system running large parallel analytics might sit at 70% and be performing exactly as designed.

Oracle connection and session exhaustion: PROCESSES, SESSIONS, and pool sizing

Mon, 01 Jan 0001 00:00:00 +0000

New Oracle connections start failing with ORA-00020 (maximum number of processes exceeded) or ORA-00018 (maximum number of sessions exceeded). The listener still answers TCP on 1521 but refuses new connections with TNS-12516 or TNS-12519 because the instance has no free handler to hand over. Existing sessions often keep working, masking the problem until an application tier’s pool needs to grow or refresh and fails.

This is a cliff-edge failure. The database works normally at 99% of the PROCESSES limit and hard-fails at 100%. There is no graceful degradation and no backpressure. The first you hear of it is frequently a user-facing outage when an app tier’s pool churns.

Oracle Data Guard transport and apply lag: RPO, RTO, and gap detection

Mon, 01 Jan 0001 00:00:00 +0000

Data Guard lag is two measurements operators often treat as one. Transport lag is how far behind the standby is in receiving redo: your real-time Recovery Point Objective (RPO) exposure, the data you would lose if the primary failed right now. Apply lag is how far behind the standby is in applying that redo: your real-time Recovery Time Objective (RTO) exposure, the extra time the standby needs to finish catching up before it can open after a failover.

Oracle Database monitoring checklist: the signals every production instance needs

Mon, 01 Jan 0001 00:00:00 +0000

Oracle production instances have a wide surface area: CPU, memory (SGA plus PGA), storage I/O, network, process slots, locks, file descriptors. They also have multiple single points of failure (LGWR, the archiver, the listener) and several failure modes that masquerade as “database up” while the application is frozen.

The four-level framework below is drawn from the Signal Catalog and Maturity Levels in the Oracle Database playbook. Each level is additive: Level 2 includes everything in Level 1, and so on. Signals are grouped by the question they answer. Use this checklist to audit your own monitoring against what actually pages you at 3 a.m.

Oracle Database monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

Oracle monitoring is a stack of progressively deeper signals, each layer catching failure modes the layer below cannot see. Teams that jump from “is the instance up” directly to “ASH analysis” usually miss the middle, where the highest-frequency production incidents live.

This model maps four levels of monitoring maturity. It is a prioritization framework, not a tooling roadmap: which signals earn their place at each level, which failure modes they expose, and what blind spots remain if you stop there. Most production Oracle estates sit between Level 1 and Level 2, with a few critical Level 3 signals missing.

Oracle Fast Recovery Area full: db_recovery_file_dest_size, reclaimable space, and DELETE OBSOLETE

Mon, 01 Jan 0001 00:00:00 +0000

The Fast Recovery Area (FRA) is Oracle’s self-managing location for recovery-related files: archived redo logs, RMAN backups, flashback logs, and control file autobackups. Its size is bounded by the DB_RECOVERY_FILE_DEST_SIZE parameter, a hard quota Oracle enforces internally on the total bytes these files can occupy.

When the FRA fills, every consumer that writes to it stalls at once. ARCn cannot write archived redo logs. RMAN backups fail. Flashback log creation fails. The most dangerous consequence is the archive stall: online redo logs cannot be reused until they are archived, so LGWR eventually cannot switch to a new log group, and every session that needs to generate redo freezes.

Oracle hard parse storm: literal SQL, bind variables, and shared pool churn

Mon, 01 Jan 0001 00:00:00 +0000

CPU is pegged. The top non-idle wait event is library cache: mutex X or cursor: pin S wait on X. Active sessions are climbing, but logical reads are flat or falling: sessions are burning CPU on parsing, not execution. Transactions per second is unstable or declining. If the storm runs long enough, ORA-04031 starts appearing in the alert log.

This is the parse storm pattern. It is almost never a database bug. It is an application pattern meeting Oracle’s shared SQL model.

Oracle HugePages not configured: page-table overhead and wasted SGA memory

Mon, 01 Jan 0001 00:00:00 +0000

When Oracle’s SGA runs on regular 4KB pages, the Linux kernel maintains a page table entry for every page the SGA touches. A 100GB SGA spans roughly 26 million 4KB pages. Every Oracle process that maps the SGA (dedicated servers, background processes, parallel slaves) carries page table structures tracking those mappings. This per-process page table overhead is invisible to Oracle’s own memory views: V$SGA, V$PGASTAT, and V$SGASTAT report the SGA as configured, not as the OS actually consumes it. You only see the overhead at the OS level in /proc/<oracle_pid>/status (the VmPTE field) or system-wide in /proc/meminfo (PageTables).

Oracle instance status: OPEN, MOUNTED, RESTRICTED, and detecting a real outage

Mon, 01 Jan 0001 00:00:00 +0000

An Oracle instance that reports STATUS = OPEN and DATABASE_STATUS = ACTIVE is not necessarily serving production traffic. Several intermediate and degraded states look healthy to a naive health check while blocking user work: restricted mode left on after maintenance, a quiesce in progress, a resumable operation suspended on a space error, or a shutdown pending behind stuck sessions. Equally common is the opposite failure: a physical standby correctly reports READ ONLY WITH APPLY and gets paged as “down” because the check assumed READ WRITE.

Oracle listener errors TNS-12516 / TNS-12519: no available handler and connection refusals

Mon, 01 Jan 0001 00:00:00 +0000

Applications start failing with TNS-12516 or TNS-12519. Your TCP health probe to port 1521 still returns green. Existing database sessions keep running and serving queries, but every new connection attempt from the application pool is refused. The listener is up, the instance is up, and basic health checks pass, while new connections cannot be established.

TNS-12516 (“listener could not find available handler with matching protocol stack”) and TNS-12519 (“no appropriate service handler found”) mean the listener accepted the TCP socket but could not route the client to a database service handler. The listener accepts the TCP connection, checks its registered services and handlers, and only then spawns or hands off to a dedicated server process. When no handler is available, the client gets a TNS error after the TCP layer already succeeded.

Oracle lock contention cascade: one idle session that stalls the whole application

Mon, 01 Jan 0001 00:00:00 +0000

An application opens a transaction, updates a few rows on a hot table, and never commits. A developer’s SQL tool is waiting for input, a connection pool returned a dirty connection, or a batch job is mid-update. From the database’s perspective the session is INACTIVE and waiting on SQL*Net message from client. From the application’s perspective, every other transaction touching those rows is stuck.

What follows is the lock contention cascade. Waiters queue on enq: TX - row lock contention. Application connection pools, seeing requests time out, spin up more sessions. Those new sessions hit the same rows and queue too. Process and session counts climb toward the PROCESSES and SESSIONS limits. Eventually ORA-00020 starts refusing new connections, the pool is wedged, and a single stuck session has become an application-wide outage.

Oracle logical reads spiking: the system-wide symptom of a bad plan

Mon, 01 Jan 0001 00:00:00 +0000

A sudden spike in session logical reads from V$SYSSTAT is usually the system-wide fingerprint of a plan regression. One query switches from an index scan doing 10 buffer gets per execution to a full table scan doing a million, and at 100 executions per second the database is suddenly doing 100 million additional buffer gets per second. CPU saturates, response times climb, and the whole system feels slow even though no individual component has failed.

Oracle out of memory: the Linux OOM killer, SGA, PGA, and random session deaths

Mon, 01 Jan 0001 00:00:00 +0000

Oracle sessions are dying at random. Users report sudden disconnects with no application-side explanation. There is no ORA- error returned to the client, no blocking session, no lock chain. The sessions simply vanish. In dmesg or journalctl -k you find the evidence: Out of memory: Kill process <pid> (oracle...).

The root cause: Oracle’s combined memory footprint exceeds physical RAM. The SGA (ideally pinned in hugepages), aggregate PGA across all dedicated server processes, and OS overhead push the system past available memory. The Linux OOM killer does not understand Oracle’s internal memory model. It ranks processes by memory consumption and kills the highest-scoring victim. Oracle server processes score high because they map the SGA and carry their own PGA, so they die first.

Oracle PGA memory pressure: over-allocation, temp spills, and PGA_AGGREGATE_TARGET

Mon, 01 Jan 0001 00:00:00 +0000

Program Global Area (PGA) is the private memory each dedicated server process uses for sorting, hashing, bitmap operations, and session state. It is allocated outside the SGA, one chunk per process, and managed as an aggregate pool with a soft target (PGA_AGGREGATE_TARGET) and, from Oracle 12c onward, a hard ceiling (PGA_AGGREGATE_LIMIT). PGA pressure has two opposite failure signatures, and the operator’s job is to tell them apart quickly.

The first signature is “too little PGA”: work areas cannot fit, the optimizer still picks hash joins and sorts, and Oracle silently spills those work areas to temp tablespace. The wait events direct path read temp and direct path write temp show up, temp fills faster, and query latency rises without a clean error. The second signature is “too much PGA”: total allocation grows past the soft target toward the hard limit, and either sessions hit ORA-04036 or, if PGA plus SGA exceeds the box, the Linux OOM killer targets Oracle server processes.

Oracle physical I/O latency: per-datafile hotspots and AVGIOTIM

Mon, 01 Jan 0001 00:00:00 +0000

When a datafile’s read or write latency rises, every session touching that file slows. Wait events like db file sequential read and db file scattered read dominate the top timed events, and throughput drops. Oracle reports I/O statistics across several views with different units, coverage, and granularity. Querying V$FILESTAT without understanding that AVGIOTIM is in centiseconds, not milliseconds, leads to misdiagnosis by 10x.

The views that matter

Three views cover physical I/O statistics, each with different scope and granularity.

Oracle RAC global cache waits: gc buffer busy, interconnect health, and workload affinity

Mon, 01 Jan 0001 00:00:00 +0000

In Oracle RAC, the global cache (gc) wait events describe time spent moving or coordinating access to data blocks across instances. When these waits dominate the top wait list, the cluster is paying for cross-instance coordination instead of serving work. The signal is clear in V$SYSTEM_EVENT, but the cause is not.

gc buffer busy waits are the most misunderstood of these events. They are not transfer waits. They are contention waits. The same hot blocks are being requested from multiple instances, and sessions queue behind an in-flight transfer instead of getting their own block right away. The fix is almost always workload affinity, not interconnect tuning.

Oracle redo generation rate: capacity planning for archiving and Data Guard

Mon, 01 Jan 0001 00:00:00 +0000

Redo generation rate is the single most important capacity signal on the Oracle write path. It measures, in bytes per second, how fast change vectors are produced, and therefore how fast every downstream consumer must drain them: online redo logs, the archiver, Data Guard redo transport, and archive log storage.

Unlike log file sync wait time, which tells you commits are already slow, redo generation rate is a planning signal. A database generating 50 MB/s of redo needs redo log groups sized for that rate, an archiver that can sustain it, a Data Guard network link that can carry it, and archive storage that can absorb it. If any consumer falls behind the rate, redo backs up until the database hangs.

Oracle redo log switch frequency: undersized logs and checkpoint pressure

Mon, 01 Jan 0001 00:00:00 +0000

Redo log switch frequency is a direct proxy for redo throughput pressure. When the database cycles through online redo log groups faster than DBWn can flush dirty buffers or ARCn can archive filled logs, every committing session starts waiting. The database does not crash, but transactions stall.

This article covers what switch frequency measures, the thresholds that separate normal operation from pressure, the cascade from undersized logs into checkpoint not complete waits, and the operational levers for fixing it. See the Oracle Database mental model if you need background on Oracle’s redo mechanism and background processes.

Oracle RMAN backup failures: V$RMAN_BACKUP_JOB_DETAILS and silent RPO loss

Mon, 01 Jan 0001 00:00:00 +0000

RMAN backups can fail for days without visible application impact. The database stays OPEN, queries succeed, transactions commit, and the alert log may show nothing actionable. The only authoritative record is in V$RMAN_BACKUP_JOB_DETAILS, a view that many shops either do not query or query incorrectly by filtering on STATUS = ‘FAILED’ and missing the more common partial-failure states.

The operational consequence is silent RPO loss. If the last successful backup is 9 days old and you discover this only when you need to restore, your effective recovery point objective is 9 days, regardless of what your runbook says. RMAN does not raise a pager when backups stop working. The scheduler runs, the script exits 0, and the database continues serving traffic with no safety net.

Oracle slow commit cascade: when redo storage degrades and every transaction waits

Mon, 01 Jan 0001 00:00:00 +0000

Every write transaction is slow. TPS is down, p99 commit latency is up, and the slowdown hits writes uniformly across transaction types. No single SQL_ID is the culprit. CPU is underutilized while sessions pile up on a wait event. This is the Oracle slow commit cascade: the storage behind your online redo logs can no longer service LGWR’s commit flushes fast enough, and every COMMIT in the database pays the price.

Oracle slow query diagnosis: top SQL by buffer gets and elapsed time

Mon, 01 Jan 0001 00:00:00 +0000

The application is slow and you suspect one or a few SQL statements, but you do not know which. Oracle answers this precisely: rank statements in V$SQL by BUFFER_GETS (logical I/O) and ELAPSED_TIME (database time), convert cumulative counters into per-execution rates, and compare against the wait-event profile.

This is a triage article for the case where the instance is OPEN and ACTIVE, the listener responds, and the symptom is degraded query or transaction latency rather than a total hang. If every session is frozen on a single wait event, start with the blocking sessions guide instead.

Oracle SQL plan regression: when a good query suddenly starts doing full scans

Mon, 01 Jan 0001 00:00:00 +0000

A query that returned in 10 milliseconds for months now takes 10 seconds. The application is timing out. CPU on the database host is pinned. The instance is OPEN, the listener responds, existing connections work, but every page load crawls. There is no error in the alert log. Nothing crashed.

This is SQL plan regression. Oracle’s cost-based optimizer chose a new execution plan for a statement that used to perform well. The old plan walked an index and touched a handful of blocks. The new plan does a full table scan and touches millions. The SQL_ID is the same, but the PLAN_HASH_VALUE changed, and BUFFER_GETS per execution jumped by one to three orders of magnitude.

Oracle tablespace full: monitoring used percent against max capacity

Mon, 01 Jan 0001 00:00:00 +0000

A tablespace full event in Oracle is a cliff-edge failure. Operations that worked seconds ago return ORA-01653 (table) or ORA-01654 (index). The instance is OPEN, the listener responds, existing SELECTs may still succeed, but any INSERT, UPDATE, or index maintenance that needs to allocate a new extent fails. There is no graceful degradation.

Oracle allocates space on demand, extent by extent. The moment a segment cannot extend, the operation errors out. If the failing tablespace is UNDO, the failure cascades to every transaction in the database. If it is SYSTEM or SYSAUX, internal operations can stall.

Oracle temp tablespace full: sorts, hash joins, and PGA spill

Mon, 01 Jan 0001 00:00:00 +0000

ORA-01652 hits your alert log and a batch job, an ETL run, or an analytical query dies mid-execution. Sessions report ORA-01652: unable to extend temp segment by N in tablespace TEMP. Throughput to the rest of the database may be fine, but anything that needs work space on disk is now blocked.

The temp tablespace is the overflow for PGA. Sorts, hash joins, bitmap operations, global temporary tables (GTTs), and temporary LOBs all allocate here when work does not fit in per-process memory. Unlike a permanent tablespace full event, temp full does not usually mean the database is down. It means a specific class of query cannot run.

Oracle undo pressure spiral: long transactions, long queries, and ORA-01555/30036

Mon, 01 Jan 0001 00:00:00 +0000

ORA-01555 (“snapshot too old”) and ORA-30036 (“unable to extend undo segment”) are two faces of the same pressure: undo extents are being consumed faster than they can be reclaimed or retained. In production they usually arrive together, in a spiral where one large uncommitted transaction starves a fleet of long-running queries, then starves new writes.

The classic trigger is a batch job that updates or deletes millions of rows in a single transaction with no intermediate commits. It generates enormous undo. At the same time, reports and ETL reads need older undo blocks for read consistency. The undo tablespace fills with ACTIVE extents that cannot be reclaimed. Under the default RETENTION NOGUARANTEE, Oracle starts stealing UNEXPIRED undo to make room, and the long reads start failing with ORA-01555. If ACTIVE undo fills everything, new DML fails outright with ORA-30036.

Oracle undo tablespace usage: ACTIVE vs UNEXPIRED vs EXPIRED and UNDO_RETENTION

Mon, 01 Jan 0001 00:00:00 +0000

Oracle’s undo tablespace holds the before-images of changed blocks for transaction rollback and read consistency. A session can undo its own work, PMON can roll back a dead session, and a long-running query can see the database as it existed at the query’s start SCN. Those uses compete for the same finite space, and the three extent states in DBA_UNDO_EXTENTS (ACTIVE, UNEXPIRED, EXPIRED) track which extents can be reused and which cannot.