Saturday, 19 September 2026

ORA-12547 from SRVCTL While SQL*Plus Startup Works – Tracing It Back to a Post-Patching Relink Issue

I recently worked on an Oracle database that could be started normally from SQL*Plus, but failed every time Clusterware tried to start it through srvctl. The error returned by Clusterware was ORA-12547: TNS:lost contact.

The final fix was to relink the database Oracle Home, but I did not reach that conclusion from the error message alone. The useful clues came from comparing SQL*Plus and SRVCTL startup behaviour, noticing what was happening to the Oracle executable during the SRVCTL attempt, and then going back through the last database startup and patching history.

The SRVCTL Startup Failure

The database resource was registered with Clusterware, but the startup failed immediately:

$ srvctl start database -d BPL1VODA

PRCR-1079 : Failed to start resource ora.bpl1voda.db
CRS-5017: The resource action "ora.bpl1voda.db start"
encountered the following error:

ORA-12547: TNS:lost contact

CRS-2674: Start of 'ora.bpl1voda.db' failed

ORA-12547 is not very specific. Oracle describes it as the partner process unexpectedly disappearing, usually during process startup. In a local database startup like this, I would not immediately treat the word TNS as evidence of a listener or network problem. The process being launched by Clusterware may simply be terminating before the startup can complete.

SQL*Plus Startup Worked

Before spending more time inside CRS, I wanted to know whether the database itself had a startup problem. I set the database environment and tried a direct startup as the Oracle software owner.

$ export ORACLE_SID=BPL1VODA
$ export ORACLE_HOME=/u01/app/oracle/product/19.0.0.0/BPL
$ export PATH=$ORACLE_HOME/bin:$PATH

$ sqlplus / as sysdba

SQL> startup;

The database started successfully.

That was an important separation point. If there had been a basic problem with the SPFILE, control files, database files, ASM disk groups or recovery, I would normally expect the manual startup to expose it as well.

Instead, the behaviour was:

  • SQL*Plus startup worked.
  • SRVCTL startup consistently failed with ORA-12547.

At that stage it looked as though something was different in the Clusterware-managed startup path. I also verified the Oracle Home registered with the database resource rather than assuming that SRVCTL was using the same environment as my shell.

$ echo $ORACLE_HOME
$ ORACLE_HOME=/u01/app/oracle/product/19.0.0.0/BPL

$ which oracle

$ srvctl config database -d BLL1VODA
$ srvctl status database -d BPL1VODA

The configured database home was the expected one, so I continued looking at what happened when Clusterware actually attempted the startup.

The Oracle Executable Kept Changing Back

This was the first behaviour that made the Oracle Home itself suspicious.

While troubleshooting, I checked $ORACLE_HOME/bin/oracle. Most of the executables in the same database home were owned by oracle:oinstall, but the main Oracle executable was showing this state:

-rwxr-s--x. 1 grid asmadmin 464944488 Sep 11 17:27 oracle

I corrected the ownership and permissions of the executable and verified them before trying another startup

The surprising part was that the correction did not stay in place. The moment I ran:

$ srvctl start database -d BLP1VODA

the executable returned to:

grid:asmadmin
-rwxr-s--x

and the database startup failed again with the same ORA-12547

At first this looked like a straightforward executable permission problem. But repeatedly correcting the file manually was clearly not solving anything. Clusterware was touching the binary again during startup, so there was another reason behind the behaviour.

The s in the group execute position also showed that the set-GID bit was being used. ASM environments have specific handling around the group and set-GID state of the RDBMS oracle executable, including Oracle's setasmgidwrap processing. For that reason, I did not use the permission change alone as proof of the root cause.

What was more relevant was the complete behaviour: SQL*Plus could start the database, SRVCTL could not, and the state of the main executable was being changed again as soon as Clusterware tried to launch the database.

Going Back to the Last Time the Database Was Running

Instead of continuing to retry the same startup, I went back through the timeline. I wanted to know when this database had last been running normally and, more importantly, why it had been stopped.

That brought the recent patching activity into the investigation.

I correlated the instance startup information with the SQL patch history and the Oracle Home inventory:

SQL> SELECT instance_name,
       host_name, SQL > 
       startup_time,
       status
FROM v$instance;


SQL> SELECT patch_id,
action,
status,
action_time,
description FROM dba_registry_sqlpatch ORDER BY action_time DESC;

$ORACLE_HOME/OPatch/opatch lsinventory

The database had been stopped as part of a patching activity. The patch information confirmed that patching had taken place, and when I correlated the timestamps with the database and Clusterware history, I could not find a successful normal SRVCTL-managed startup after that activity.

One detail is worth being precise about. V$INSTANCE does not tell us whether an instance was started through SQL*Plus or SRVCTL, and DBA_REGISTRY_SQLPATCH does not record the startup method. I used these timestamps together with the patch and Clusterware history to reconstruct the sequence.

What I now had was a much more useful timeline:

  • The database had previously been running normally.
  • It was stopped for patching.
  • The database Oracle Home had gone through patching.
  • There was no confirmed successful SRVCTL-managed startup afterwards.
  • A direct SQL*Plus startup still worked.
  • An SRVCTL startup failed with ORA-12547.
  • Correcting the Oracle executable manually did not persist; the SRVCTL attempt changed it back again.

Why Relinking Became the Next Logical Test

At this point there was still no single log line saying, "the Oracle Home must be relinked." The direction came from the combination of evidence.

The database engine itself could start, the problem appeared only through the Clusterware-managed path, the Oracle executable was being modified during that path, and the last significant change to the Oracle Home had been patching.

Rather than continuing to change file permissions manually, I decided to rebuild the executables in the database Oracle Home.

With the database stopped, I ran:

$ cd $ORACLE_HOME/bin

$ relink all

The relink completed successfully. I then returned to the same operation that had been failing:

$ srvctl start database -d BPL1VODA

$ srvctl status database -d BPL1VODA

This time the database started successfully through SRVCTL.

That result was much more useful than simply seeing SQL*Plus work. The exact Clusterware-managed startup that had been failing before the relink was now working without manually changing the executable.

What I Consider the Most Likely Root Cause

I would not document this incident as "ORA-12547 is fixed by relink." ORA-12547 has several possible causes, and relinking should not be used as a generic response to that error.

For this case, the strongest explanation is that the database Oracle Home had been left in an inconsistent executable or linking state following the patching activity.

The reasoning is based on the sequence:

  • The Oracle Home had recently been patched.
  • The problem appeared on the subsequent Clusterware-managed startup.
  • SQL*Plus could still start the instance.
  • SRVCTL repeatedly failed with ORA-12547.
  • The main Oracle executable was being returned to the same unexpected ownership/group state during the failed SRVCTL attempt.
  • Relinking the database Oracle Home corrected the condition.
  • The same SRVCTL startup succeeded immediately afterwards.

Unless an older patching or relink log contains a definite linker error, I would still avoid claiming that a previous relink failure was conclusively proven. What the evidence does show is that rebuilding the Oracle executables corrected the post-patching state that was preventing Clusterware from starting the database.

Where to Look for Evidence of a Relink Problem

After recovering the database, it is worth going back through the logs rather than treating the successful startup as the end of the investigation.

Oracle writes relink output below $ORACLE_HOME/install. I would look for the logs created during the original patching window as well as the new successful relink log.

# Relink history
$ ls -ltr $ORACLE_HOME/install/relinkActions*

$ grep -Ein 
'error|failed|fatal|undefined|cannot find|not found|ld:|collect2|make.*error' 
$ORACLE_HOME/install/relinkActions*.log

# OPatch history

$ ls -ltr $ORACLE_HOME/cfgtoollogs/opatch/

$ grep -RniE 'relink|linking|make|error|failed' ORACLE_HOME/cfgtoollogs/opatch/ 2>/dev/null

# Clusterware agent trace

$ grep -iE 'ORA-12547||failed|error' ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/crsd_oraagent_oracle.trc

An older relinkActions log from the patching window is particularly useful. Messages such as undefined reference, cannot find, linker errors, make: *** Error or permission failures would provide direct evidence that a previous linking operation did not finish cleanly.

The CRS agent trace may not contain an explicit relink error. Clusterware may only know that the process it launched terminated and therefore report ORA-12547. This is why the patching timeline and Oracle Home history can be more useful than searching for one specific error string.

What the SQL*Plus Test Actually Proved

The successful SQL*Plus startup was initially one reason to suspect SRVCTL itself, but it is important not to take that test further than it proves.

It proved that the database instance could be started directly using that Oracle Home. It did not prove that the entire Oracle Home and the Clusterware/ASM-managed startup path were in the expected state.

That difference was exactly where this issue was hiding.

One Check I Would Add After Patching

For a database managed by Oracle Clusterware, seeing an open database after patching is not enough for me to consider the startup validation complete.

I also want to know that the database can be managed through the same path that will be used during the next restart or failover.

$ srvctl status database -d BPL1VODA

$ srvctl stop database -d BPL1VODA

$ srvctl start database -d BPL1VODA

$ srvctl status database -d BPL1VODA

In this case, a manual startup would have given a false sense that everything was fine. The problem only became visible when Clusterware attempted to manage the database through SRVCTL.

The error itself was only one part of the investigation. The useful path was to compare the two startup methods, observe what Clusterware was doing to the executable, trace the database back to the patching activity, and then test the Oracle Home rather than continuing to modify permissions manually.

Oracle References



Labels: , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)