Split Brain in Oracle RAC: What Every DBA Must Know

   If you have worked with Oracle RAC long enough, you know the cluster is only as strong as its communication layer. Most days, RAC behaves beautifully ; instances coordinate, cache fusion works seamlessly, and life is good. But when inter-node communication breaks down, things can get ugly very quickly.

One of the most dangerous scenarios in a RAC environment is a split brain condition. It is not just a theoretical concept., it is something that can bring your cluster to its knees if not handled correctly. I have personally seen environments where a poorly configured interconnect or a flaky switch triggered node evictions during peak hours

In this article, I will break down what split brain really means in practical terms, how Oracle RAC prevents it, and what you as a DBA should monitor proactively. We will also go through essential commands, real-world scenarios, and operational insights that are rarely discussed in documentation.

What Split Brain Actually Means in RAC
At its core, split brain is a communication failure between RAC nodes.
When nodes stop seeing each other over the private interconnect, each node may assume it is the only surviving member of the cluster. Now imagine both nodes continuing to process transactions independently against the same storage—that is where data corruption risk comes in.

In a healthy RAC setup:

• The Nodes exchange heartbeat signals over the private interconnect
• Clusterware keeps track of node membership & All nodes agree on cluster state

In a split scenario:

• Heartbeat is lost &  Nodes cannot validate each other
• Each node believes the other is down

Oracle does not allow both to continue. It forces a decision.

Why Split Brain Happens in Production
From experience, split brain is rarely caused by a single obvious failure. It is often a combination of infrastructure and configuration issues.
Common triggers I have seen:

• Interconnect failure 
• NIC issues, bonding misconfiguration, or cable problems

• Network switch problems
• Especially when interconnect traffic shares switches with other workloads

• High CPU or OS hang
• Node becomes unresponsive and misses heartbeat intervals

• Improper interconnect configuration
• Wrong interface priority or MTU mismatch

Split brain is not always a "network down" scenario. Even latency spikes or packet drops can trigger it.

How Oracle RAC Protects Your Data
Oracle RAC is designed with a strong principle: Data integrity is more important than availability.

When a split condition is detected, Oracle Clusterware steps in immediately.

1. Node Eviction 
Oracle resolves the conflict by evicting one node from the cluster.

• The node that loses quorum or heartbeat validation is removed
• Its database instance is terminated & Access to shared storage is blocked

This ensures only one side continues processing.

2. Voting Disk Mechanism
Voting disks act as arbiters.

• Each node votes to confirm cluster membership
• Majority wins & the losing node is evicted

Hence, Always use an odd number of voting disks (3 or 5)

3. Heartbeat Monitoring
Cluster Synchronization Services (CSS):

• Tracks node liveness and Uses heartbeat signals over interconnect
• Missed heartbeats beyond threshold trigger eviction

4. Fencing
Once evicted, a Node is completely isolated from shared storag and Prevents accidental writes

Essential Commands Every RAC DBA Should Know
During an incident, speed matters. You do not have time to google commands.
Cluster Health Checks : 

crsctl check cluster -all
crsctl stat res -t

Node Information olsnodes -n

Voting Disk Status crsctl query css votedisk

CRS Logs (Eviction Analysis) cd $GRID_HOME/log/<node_name>/crsd/

When Node Eviction Becomes a Problem

While node eviction protects data, frequent evictions are a red flag.
You might face some Application disruptions / Session drops / Performance instability

If evictions happen often, it is not a RAC problem-- it is an infrastructure problem.
Focus on the areas such as Interconnect latency / Network packet drops / CPU starvation 

Mini Case Study: The Invisible Network Issue
In one production setup, we observed random node evictions during peak batch processing.
Initial suspicion was Database load & High I/O

But the real issue was a misconfigured network switch intermittently dropping jumbo frames.

Symptoms we noticed : 

• No full network outage and

• Just enough packet loss to miss heartbeats

Fix:

• Standardized MTU across nodes and switches

• Dedicated interconnect VLAN

Lesson learnt : Split brain is often triggered by subtle network inconsistencies, not total failures.

DBA Perspective

• Eviction is not failure- it is protection
• Many teams panic when they see node eviction. It is actually RAC doing its job.

• Interconnect deserves dedicated design
• Never treat it as "just another network interface

• Monitoring must go beyond database

• such as Network latency  OS load &  NIC errors

• CRS logs are gold during incidents
• Do not rely only on alert logs

• Virtual environments increase risk
• Resource contention can mimic node failure

• Timeout parameters are not tuning knobs for beginners
• Changing misscount or disktimeout without understanding can make things worse

Quick Takeaways

• Split brain occurs when RAC nodes lose communication
• Oracle resolves it using node eviction
• Voting disks ensure majority-based decisions
• Interconnect stability is critical
• Frequent evictions indicate deeper infrastructure issues
• Monitoring CRS logs is essential for diagnosis
• Always prioritize data integrity over uptime

Conclusion

Split brain is one of those topics that sounds theoretical until you experience it in production. And when you do, it quickly becomes clear why Oracle designed RAC the way it did.

The cluster does not try to be overly smart -, it simply ensures that only one consistent version of truth exists. That is why node eviction happens so aggressively. It may feel disruptive, but it is preventing something far worse: data corruption.

As a DBA, your responsibility is not just to understand split brain, but to prevent the conditions that lead to it. That means investing time in network design, validating interconnect configurations, and monitoring system health beyond the database layer.
If you are running RAC today, take a moment to review your setup:

• Are voting disks properly configured?

• Is your interconnect isolated and reliable?

• Do you actively monitor cluster health?

Because in RAC, stability is not accidental—it is engineered.

FAQs
1. Does split brain always lead to data corruption?
No. Oracle prevents corruption using node eviction and fencing mechanisms.

2. Can I disable node eviction in RAC?
No. It is a core safety feature and cannot be disabled.

3. What is the ideal number of voting disks?
Always use an odd number -. typically 3 or 5.

4. How do I identify which node was evicted?
Check CRS logs under $GRID_HOME/log/<node>/crsd/ and alert logs.

5. Can high CPU cause split brain?
Yes. If a node cannot send heartbeat signals due to CPU starvation, it may be evicted.

What is your experience with RAC node evictions in production?
Have you ever traced a split brain issue back to something unexpected like network latency or OS load?
Drop your insights in comments below., these are the scenarios every DBA learns from.