Overcoming OPatch Issues

 

It is every DBA’s nightmare to be given the task of patching an Oracle environment during a tight scheduled downtime window, to be rewarded with problems, resulting in failure. Maybe even having to resort to an Oracle Home backup with no new patches applied. The following paragraphs aim to explain and provide solutions to some of the “gotchas” I have experienced when patching Oracle 11.1.0.7.

The Patch Set Update

Installing Interim Patches

Patch Set Updates (PSUs) are proactive cumulative patches containing recommended bug fixes that are released on a regular and predictable schedule. PSUs include recommended Patch Bundles and Critical Patch Updates (CPUs). The 5th digit in the Oracle version is incremented for each PSU released against the patch set version.

When installing interim patches on top of 11.1.0.7.1 PSU, it is possible to encounter an Opatch error code 73 as seen in the example below.

[oracle@g###01a patches]$ unzip p7006588_111071_Linux-x86-64.zip
Archive:  p7006588_111071_Linux-x86-64.zip
   creating: 7006588/
   creating: 7006588/files/
   creating: 7006588/files/lib/
   creating: 7006588/files/lib/libserver11.a/
  inflating: 7006588/files/lib/libserver11.a/kfn.o
  inflating: 7006588/files/lib/libserver11.a/kfnc.o
  inflating: 7006588/files/lib/libserver11.a/ksmp.o
  inflating: 7006588/files/lib/libserver11.a/kspt.o
   creating: 7006588/etc/
   creating: 7006588/etc/config/
  inflating: 7006588/etc/config/inventory.xml
  inflating: 7006588/etc/config/actions.xml
   creating: 7006588/etc/xml/
  inflating: 7006588/etc/xml/GenericActions.xml
  inflating: 7006588/etc/xml/ShiphomeDirectoryStructure.xml
  inflating: 7006588/README.txt
[oracle@g###01a patches]$ cd 7006588/
[oracle@g###01a 7006588]$ opatch apply
Invoking OPatch 11.1.0.6.2
Oracle Interim Patch Installer version 11.1.0.6.2
Copyright   2007, Oracle Corporation.  All rights reserved.
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.2
OUI version       : 11.1.0.7.0
OUI location      : /u01/app/oracle/product/11.1.0/asm/oui
Log file location : /u01/app/oracle/product/11.1.0/asm/cfgtoollogs/opatch/opatch2009-12-08_15-07-33PM.log
ApplySession failed: Patch ID is null.
System intact, OPatch will not attempt to restore the system
OPatch failed with error code 73

Solution

Upgrade OPatch to 11.1.0.6.8 or higher will resolve the issue.

Oracle recommends that all customers be on the latest version of OPatch.
Oracle Doc ID: 224346.1 provides the instructions to update OPatch to the latest version.

Installing CRS Patch Bundle #1

The CRS Bundle Patch has been renamed as CRS PSU. CRS PSU and RDBMS PSU are two separate patches.

For 11.1.0.7, the CRS PSU is patch 8287931 and the RDBMS PSU is patch 8833297.

Despite what the README.txt file says in patch 8287931, using opatch napply to install and invoking [pre/post]rootpatch.sh may result in numerous errors, even causing corruption to your CRS Oracle Home.

The Opatch Auto option

As stated in Oracle Documentation (Oracle® Universal Installer and OPatch User's Guide), “The opatch auto command automates all of these tasks for patching the CRS home and all other applicable RDBMS homes”. The auto option has been available since Oracle 10gR2. Although a great utility, the auto option is not without problems.

<CRSHome>/Opatch/opatch auto <patch_directory>

Opatch Auto Fails verifying resources

ERROR verify failure: exit code 34560 for patch_crs

 

Solution

Opatch is bundled with Oracle Configuration Manager. The OCM path must include an actual ocm.rsp response file and not just path.
E.g.

<CRSHome>/OPatch/ocm/bin/ocm.rsp


instead of:

<CRSHome>/OPatch/ocm/bin

To generate a OCM response file, execute the following commands as root user

cd <CRSHome>/OPatch/ocm/bin

<CRSHome>/OPatch/ocm/bin/emocmrsp

 

ERROR verify failure: exit code 32512 for cat

[root@d###01a PSUpatches]$ <CRSHome>/OPatch/opatch auto /u02/PSUpatches/8287931

 

auto_patch.pl: log file is /u02/PSUpatches/opatchauto_2010-05-27_15:12:45.log

Discovering environment to patch

 

Checking if Clusterware is up

 

Looking for configured cluster nodes

 

. .

 

Oracle version for Oracle Home /u01/app/oracle/product/11.1.0/asm/ is 11.1.0.7.0

The patch is applicable for this Oracle Home /u01/app/oracle/product/11.1.0/asm/

verify failure: exit code 32512 for cat

Action failed, do you want to retry it? (y/n/abort/N/N1-N2/help):

n
ignored at user request, continuing

Solution

 

This error can be ignored as confirmed in Doc ID: 844983.1

 

ERROR verify failure: exit code 18688 for patch_crs

You cannot continue from this error, Opatch will exit. To discover why I had hit this particular error I ran the following command manually from the opatch auto screen output:

 

Executing /u01/app/oracle/product/11.1.0/crs/OPatch/opatch napply -local -silent -ocmrf /u01/app/oracle/product/11.1.0/crs/OPatch/ocm/bin/ocm.rsp -oh /u01/app/oracle/product/11.1.0/crs -id 8287931 as oracle on d****01a

 

verify failure: exit code 18688 for patch_crs

 

Action failed, do you want to retry it? (y/n/abort/N/N1-N2/help):

 

Re-executing command manually as oracle user:

 

[oracle@D****01A 111070]$ /u01/app/oracle/product/11.1.0/crs/OPatch/opatch napply -local -silent -ocmrf /u01/app/oracle/product/11.1.0/crs/OPatch/ocm/bin/ocm.rsp -oh /u01/app/oracle/product/11.1.0/crs -id 8287931

Invoking OPatch 11.1.0.6.9

 

Oracle Interim Patch Installer version 11.1.0.6.9

Copyright (c) 2009, Oracle Corporation.  All rights reserved.

 

UTIL session

 

Oracle Home       : /u01/app/oracle/product/11.1.0/crs

Central Inventory : /u01/app/oraInventory

   from           : /etc/oraInst.loc

OPatch version    : 11.1.0.6.9

OUI version       : 11.1.0.7.0

OUI location      : /u01/app/oracle/product/11.1.0/crs/oui

Log file location : /u01/app/oracle/product/11.1.0/crs/cfgtoollogs/opatch/opatch2010-05-27_15-16-23PM.log

 

Patch history file: /u01/app/oracle/product/11.1.0/crs/cfgtoollogs/opatch/opatch_history.txt

 

Invoking utility "napply"

Checking conflict among patches...

Checking if Oracle Home has components required by patches...

Checking conflicts against Oracle Home...

OPatch continues with these patches:   8287931

 

Do you want to proceed? [y|n]

Y (auto-answered by -silent)

User Responded with: Y

 

Running prerequisite checks...

Prerequisite check "CheckApplicable" failed.

The details are:

 

Patch 8287931:

Copy Action: Destination File "/u01/app/oracle/product/11.1.0/crs/install/onsconfig" is not writeable.

'oracle.crs, 11.1.0.7.0': Cannot copy file from 'onsconfig' to '/u01/app/oracle/product/11.1.0/crs/install/onsconfig'

 

UtilSession failed: Prerequisite check "CheckApplicable" failed.

 

OPatch failed with error code 73

 

Solution

 

From the above output we see that file: /u01/app/oracle/product/11.1.0/crs/install/onsconfig is not writeable.

 

To fix the permission problem, execute the following command as root user:

 

chmod 754 /u01/app/oracle/product/11.1.0/crs/install/onsconfig

 

Then startup CRS stack and re-execute opatch auto.

 

ERROR verify failure: exit code 65280 for cat

You cannot continue from this error, Opatch will fail and exit. This error relates to a problem with ssh between the nodes.

 

The Opatch log file shows the error which occurs after the perl script auto_patch.pl invokes the “cat” command:

 

Wed Jun  9 08:50:17 2010:  Oracle version for Oracle Home /u01/app/oracle/product/11.1.0/asm is 11.1.0.7.0

Wed Jun  9 08:53:39 2010:  remote: invoked with -v cat oracle d****01a n /bin/cat /u01/app/oracle/product/11.1.0/crs/a.txt

Wed Jun  9 08:53:39 2010:  run: invoked with -v cat oracle n /usr/bin/ssh d****01a /u01/app/oracle/product/11.1.0/crs/OPatch/crs/auto_patch.pl -x -v 0 -oh /u01/app/oracle/product/11.1.0/crs -och /u01/app/oracle/product/11.1.0/crs -patchn 8287931 -patchdir /patchdir /bin/cat /u01/app/oracle/product/11.1.0/crs/a.txt

Wed Jun  9 08:53:39 2010:  run: exit code 65280 after /usr/bin/ssh d****01a /u01/app/oracle/product/11.1.0/crs/OPatch/crs/auto_patch.pl -x -v 0 -oh /u01/app/oracle/product/11.1.0/crs -och /u01/app/oracle/product/11.1.0/crs -patchn 8287931 -patchdir /patchdir /bin/cat /u01/app/oracle/product/11.1.0/crs/a.txt

Wed Jun  9 08:53:39 2010:  verify: invoke with cat /patchdir/opatchauto_2010-06-09_08:39:42.2.tmp 65280

Wed Jun  9 08:53:39 2010:  osimulate is

 

Wed Jun  9 08:53:39 2010:  verify failure: exit code 65280 for cat

 

Wed Jun  9 08:53:39 2010:  Action failed, do you want to retry it? (y/n/abort/N/N1-N2/help):

 

Interestingly the tmp file provides the real error:

 

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

The RSA host key for d****01a has changed,

and the key for the according IP address 196.6.204.10

is unknown. This could either mean that

DNS SPOOFING is happening or the IP address for the host

and its host key have changed at the same time.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that the RSA host key has just been changed.

The fingerprint for the RSA key sent by the remote host is

50:94:28:e6:b1:18:7b:ac:d2:ee:d0:33:4c:b8:6c:1f.

Please contact your system administrator.

Add correct host key in /home/oracle/.ssh/known_hosts to get rid of this message.

Offending key in /home/oracle/.ssh/known_hosts:4

RSA host key for ****01a has changed and you have requested strict checking.

Host key verification failed.

 

Solution

 

From the above output we see there is an issue with RSA key fingerprint #4 causing problems with ssh to remote node. The solution is:

 

  1. Edit the /home/oracle/.ssh/known_hosts file and remove the offending record.
  2. Perform user equivalence check for the oracle user
    su - oracle
    ssh <node1> date
    ssh <node2> date

  3. The first time you execute the above commands the OS will prompt for a password. Once the correct password is entered, the RSA host key will be reset.
  4. Perform user equivalence check again to verify
  5. startup CRS stack and re-execute opatch auto

_______________________________________________________________________________

Did you find the article useful?

Please provide your feedback by voting now.

If you have a comment or question, please complete and submit the form below.

_______________________________________________________________________________