Oracle Instance Startup Failure | Instance terminated by GEN0

  

SCOPE

Recently a server running both Oracle databases with version 11.1.0.2  had to be forced powered cycled due to a NFS issue. Since it wasn’t possible to login to the O/S to shutdown the DBs an instance recovery was required on both the 11.2 databases.

The database startup failed with errors in alert log as below.

EXCERPT

ALTER DATABASE MOUNT /* db agent *//* {0:9:118} */
Dumping diagnostic data in directory=[cdmp_20210729060244], requested by (instance=1,
osid=51180 (PMON)), summary=[incident=168017].
GEN0 (ospid: 51196): terminating the instance due to error 472
System state dump requested by (instance=1, osid=51196 (GEN0)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/admin/TTRADEDB/diag/rdbms/TTRADEDB/TTRADEDB/trace/DEUBRU9B_diag_51200_20190714060245.trc
Dumping diagnostic data in directory=[cdmp_20190714060245], requested by (instance=1, osid=51196 (GEN0)), summary=[abnormal instance termination].
Instance terminated by GEN0, pid = 51196

The GEN0 is usually associated with ASM due to bugs or issues in the past.

Cause 

The issue is caused by internal, unpublished Bug 13483672 "ORA-7445 [strlen()+16] creating database dependencies for large number of disk groups" which causes a buffer overflow if the diskgroup dependencies of the database resource exceed a certain size.

The Bug 13483672 has been fixed in 11.2.0.3 PSU 3 and windows 11.2.0.3.7 patch bundle. Interim patch has also been provided for 11.2.0.2 on certain platforms.

workaround

Set hidden parameter "_notify_crs" to false, which will prevent the database instance from notifying the CRS daemon process when diskgroups are being mounted:

SQL> startup nomount
....
SQL> alter system set "_notify_crs"=FALSE scope=spfile;

System altered.

SQL> alter database mount;

Apply Grid Infrastructure 11.2.0.3 PSU 3 (or windows 11.2.0.3 patch bundle 7) or higher which contains the fix for bug 13483672

Please note that the GI PSU needs to be applied to both the Grid Infrastructure as well as the Database home! 

Do post your comments which are highly appreciated.

-- Nikhil