In Oracle RAC, the global cache (gc) wait events describe time spent moving or coordinating access to data blocks across instances. When these waits dominate the top wait list, the cluster is paying for cross-instance coordination instead of serving work. The signal is clear in V$SYSTEM_EVENT, but the cause is not.

gc buffer busy waits are the most misunderstood of these events. They are not transfer waits. They are contention waits. The same hot blocks are being requested from multiple instances, and sessions queue behind an in-flight transfer instead of getting their own block right away. The fix is almost always workload affinity, not interconnect tuning.

The interconnect is the other half. Block transfers between instances happen over the cluster interconnect, and latency there dominates transfer waits. On a healthy dedicated interconnect, average gc transfer waits sit under 1ms. Above 3ms indicates congestion. Above 10ms indicates interconnect failure or silent failover to the public network.

What this means

The gc% events in V$SYSTEM_EVENT split into two families. Confusing them wastes diagnosis time.

Transfer waits describe the block actually moving over the interconnect:

  • gc cr grant 2-way, gc cr block 2-way, gc cr block 3-way for consistent read transfers
  • gc current grant 2-way, gc current block 2-way, gc current block 3-way for current mode transfers

Average latency under 1ms is normal on a dedicated high-speed interconnect (InfiniBand, 10GbE or better). Above 3ms indicates congestion or an undersized link. Above 10ms indicates interconnect failure or failover to the public network.

Contention waits describe a session waiting for a block that someone else is already transferring:

  • gc buffer busy acquire: another session on the same local instance already has an open global cache lock request for that block, so your session waits behind it instead of issuing a redundant request
  • gc buffer busy release: a remote instance holds the global cache lock for the block and has not released it yet, so your session waits for the remote transfer to complete

These are not interconnect problems. They are workload placement problems. The same hot block is being touched from multiple instances, and the cluster is serializing on the lock instead of serving concurrent work.

flowchart TD
  A["gc% waits in top events"] --> B{"Transfer or contention?"}
  B -->|"gc cr/current block"| C["Transfer wait: tune interconnect"]
  B -->|"gc buffer busy acquire/release"| D["Contention: fix workload affinity"]
  C --> E{"avg latency"}
  E -->|"< 1ms"| F["Normal cross-instance traffic"]
  E -->|"> 3ms"| G["Congestion: NIC, UDP, MTU"]
  E -->|"> 10ms"| H["Verify V$CLUSTER_INTERCONNECTS IS_PUBLIC"]

Common causes

CauseWhat it looks likeFirst thing to check
Workload not partitioned across instancesgc buffer busy acquire and release dominate; same SQL_IDs active on multiple instancesV$ACTIVE_SERVICES and where sessions actually land vs. service placement
Interconnect failover to public networkAverage gc transfer waits jump 10x to 100x; private interface traffic near zeroV$CLUSTER_INTERCONNECTS.IS_PUBLIC on the active interface
Interconnect congestion or packet lossgc waits >3ms average; gc blocks lost non-zero; NIC errorsOS-level NIC stats and UDP socket overflows
Hot blocks from monotonic keysgc buffer busy concentrated on specific segments; local buffer busy waits tooV$INSTANCE_CACHE_TRANSFER and V$SEGMENT_STATISTICS
Instance rebalance after node failuregc traffic spikes during recovery; transfers between surviving nodesCluster node status and recent instance restarts

Quick checks

-- gc wait events by total time waited
SELECT EVENT, TOTAL_WAITS, TIME_WAITED_MICRO,
       ROUND(TIME_WAITED_MICRO/NULLIF(TOTAL_WAITS,0)/1000, 2) AS avg_ms
FROM V$SYSTEM_EVENT
WHERE EVENT LIKE 'gc%'
ORDER BY TIME_WAITED_MICRO DESC;

-- verify Oracle is using the private interconnect, not the public network
SELECT NAME, IP_ADDRESS, IS_PUBLIC, CON_ID
FROM V$CLUSTER_INTERCONNECTS;
-- IS_PUBLIC should be NO for the interface Oracle is actually using

-- per-instance block transfer and congestion stats
<!-- TODO: verify INSTANCE column name; may be INST_ID on some versions -->
SELECT INST_ID, CLASS, CR_BLOCK, CR_BUSY, CR_CONGESTED,
       CURRENT_BLOCK, CURRENT_BUSY, CURRENT_CONGESTED
FROM V$INSTANCE_CACHE_TRANSFER
ORDER BY CR_BLOCK + CURRENT_BLOCK DESC;
-- CR_BUSY / CURRENT_BUSY non-zero = contention. CR_CONGESTED = interconnect saturation.

-- compare with all configured interfaces to catch misconfiguration
SELECT * FROM V$CONFIGURED_INTERCONNECTS;

# verify interconnect interface and MTU at the OS level
ip link show <interconnect_interface>
# look for mtu 9000 if jumbo frames are configured

# verify OCR-registered interface configuration (read-only)
oifcfg getif
# confirm the private subnet is registered as cluster_interconnect, not public

-- services defined in the cluster (use srvctl or DBA_SERVICES for preferred-instance placement)
SELECT NAME, NETWORK_NAME FROM V$ACTIVE_SERVICES;

How to diagnose it

  1. Confirm gc waits are a meaningful slice of DB time. A handful of milliseconds per wait on a busy cluster is normal. Look at gc wait time as a percentage of total DB time. Above 15% sustained warrants investigation.

  2. Separate transfer waits from contention waits. If gc cr block 2-way, gc cr block 3-way, gc current block 2-way, or gc current block 3-way dominate with high average latency, you have an interconnect problem. If gc buffer busy acquire or gc buffer busy release dominate, you have a workload placement problem.

  3. For transfer waits, verify the interconnect interface. Query V$CLUSTER_INTERCONNECTS. The IS_PUBLIC column should be NO. If it is YES, Oracle is using the public network for cache fusion traffic, and latency will be 10x to 100x higher than expected. This is a critical misconfiguration.

  4. For transfer waits on a correctly private interconnect, check OS-level signals. Look at NIC error counters, UDP socket buffer overflows, switch port errors, and MTU mismatches. A jumbo frame configured on one node and standard 1500 on another silently fragments packets.

  5. For contention waits, identify the hot blocks. V$INSTANCE_CACHE_TRANSFER shows which instance pairs exchange the most blocks. CR_BUSY and CURRENT_BUSY indicate contention rather than clean transfers. Drill into V$SEGMENT_STATISTICS for buffer busy waits per segment to find the specific objects.

  6. For contention waits, look at where the workload is running. If the same SQL_IDs are active on multiple instances against the same hot tables, that is the cause. RAC cache fusion tolerates cross-instance access but does not make parallel writers on the same block cheap.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
gc wait average latencyDirect measure of interconnect performance>3ms sustained, >10ms is page-worthy
gc waits as % of DB timeShows whether gc is the bottleneck or just noise>15% sustained
V$CLUSTER_INTERCONNECTS.IS_PUBLICCatches silent failover to public networkIS_PUBLIC = YES on the active interface
V$INSTANCE_CACHE_TRANSFER.CR_BUSY / CURRENT_BUSYDistinguishes contention from clean transfersNon-trivial values sustained
gc blocks lostIndicates dropped interconnect packetsAny non-zero value
NIC error counters and UDP socket overflowsLower-layer causes of high gc latencyNon-zero errors or overflows

Fixes

Interconnect failover to public network

This is the highest-impact RAC performance problem and the easiest to verify. If V$CLUSTER_INTERCONNECTS shows the public interface in use, Oracle fell back from the private interconnect. Causes include bond failure, switch failure, or a misconfigured interface registration in the OCR.

The fix is to correct the OCR-registered interface with oifcfg and restart the affected instance or instances. The private subnet must be registered as cluster_interconnect, and the public subnet as public. This is disruptive: cache fusion traffic re-routes only after instance restart. Plan a window. Do not attempt this on a live production cluster without testing the change in a non-production environment first.

Interconnect congestion and packet loss

If the private interconnect is in use but gc latency is high, the link is saturated or losing packets. Common causes:

  • UDP receive buffer sizes too small, causing socket overflows and silent packet loss. Check OS-level UDP stats.
  • Switch port speed fixed instead of auto-negotiate, or duplex mismatches.
  • MTU mismatch between nodes, with jumbo frames on one and standard 1500 on another, causing fragmentation.
  • Single interconnect link where a bonded pair should be in use.

Verify jumbo frames are configured end-to-end on the private network. Check ip link show on all nodes for MTU 9000. Check switch configuration for the private VLAN.

Workload without service affinity

This is the most common RAC performance mistake. Running the same workload across all instances without service-based partitioning causes gc buffer busy waits because every instance fights over the same hot blocks.

The fix is service-based workload partitioning. Define application-specific services with preferred and available instances, and route each application tier to its service. A service that serves the OLTP workload should prefer one instance and only fail over to another when the preferred instance is down. A batch service should prefer a different instance. Under normal operation, each hot block has one writer instance, and cache fusion transfers happen only on failover, not during steady state.

This is a design change, not a parameter change. It requires application connection string changes, service definitions, and coordination with whoever owns the application deployment.

Hot blocks from monotonic keys

If gc buffer busy waits concentrate on specific segments, look at the workload pattern. Monotonically increasing sequences or right-hand index inserts on a single table create hot leaf blocks. When sessions on multiple instances insert into the same hot index leaf, every insert triggers a cross-instance lock transfer.

Local fixes include reverse key indexes (which spread inserts across the index but defeat range scans), hash partitioning of the index, or larger sequence cache sizes to reduce enq: SQ - contention on the sequence itself. The RAC-specific fix is still workload affinity: keep the inserting workload on one instance so the hot block never crosses the interconnect.

Prevention

  • Service-based workload partitioning. Every application tier should map to a service with preferred instances. Without this, RAC scales worse than single instance under write contention.
  • Interconnect monitoring, not just database monitoring. Track gc wait latency, V$CLUSTER_INTERCONNECTS.IS_PUBLIC, NIC error counters, and gc blocks lost as first-class signals.
  • Jumbo frames on the private network from day one. Retrofitting MTU 9000 on a live cluster is error-prone.
  • UDP buffer sizing on all nodes. Default OS UDP receive buffer sizes are often too small for RAC cache fusion traffic under load.
  • Service placement review after every node failure. When a node restarts, services may rebalance. Verify the workload is back on its preferred instances before declaring the incident over.

How Netdata helps

  • Per-second gc wait latency. The average wait time per gc event is the single most useful RAC performance signal. A spike from 1ms to 5ms shows up immediately, before users notice.
  • IS_PUBLIC correlation. When gc latency jumps, the first question is whether Oracle fell back to the public network. Correlating gc waits with V$CLUSTER_INTERCONNECTS output answers that question in one view.
  • gc waits vs. local buffer busy waits. Distinguishing local hot block contention (single instance) from cross-instance contention (RAC) tells you whether the fix is index design or service placement.
  • Transfer waits vs. contention waits. Splitting gc cr/current block waits from gc buffer busy waits tells you whether to tune the interconnect or tune the workload.
  • Per-instance load balance. If one instance carries 80% of the active sessions, service affinity is not working. Correlating active session count per instance with gc wait time shows this clearly.

Netdata’s Oracle Database monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.