Monday 24 July 2017

Step by Step process of Applying PSU patch

How to apply PSU patch and what are the pre-requisite parameter we have to take care before going for a patch.

Important- $ORACLE_HOME -> Before going for PSU patch we have to check $ORACLE_HOME must have minimum 2GB of space. 
Here- prdrms is the server name and prdrms is the database name as well.


Ø  Login to the server prdrms server and sudo to oracle, validate the instance name  and set the profile as below :-
         prdrms:/home1/mukhers3> sudo su - oracle
            prdrms:/apps/oracle> ps -aef|grep pmon
prdrms:/apps/oracle> . /prdrms/admin/profile/oracle.profile
  oracle 21889166        1   0   Feb 26      -  0:20 ora_pmon_PRDRMS

Ø  Check the Operating System version from the below command

prdrms /prdrms/admin/profile=>function aixversion {
  OSLEVEL=$(oslevel -s)
 AIXVERSION=$(echo "scale=1; $(echo $OSLEVEL | cut -d'-' -f1)/1000" | bc)
  AIXTL=$(echo $OSLEVEL | cut -d'-' -f2 | bc)
 AIXSP=$(echo $OSLEVEL | cut -d'-' -f3 | bc)
   echo "AIX ${AIXVERSION} - Technology Level ${AIXTL} - Service Pack ${AIXSP}"
 }
aixversion
prdrms /prdrms/admin/profile=>aixversion
AIX 6.1 - Technology Level 8 - Service Pack 3
prdrms /prdrms/admin/profile=>

Check the operating system bit from the below command.

prdrms /prdrms/admin/profile=>getconf KERNEL_BITMODE
64

Ø  Check the database version as below :-

SQL> select * from v$version ;

BANNER
----------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
PL/SQL Release 11.2.0.3.0 - Production
CORE    11.2.0.3.0      Production
TNS for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production
NLSRTL Version 11.2.0.3.0 – Production

Ø  Go to MOS (mysupport.oracle.com-Patches and Upgrades section) and download the relevant patch set selecting the appropriate Operating System as well as Database version by searching it with patch number. Download the exact PSU patchset from Supoort.Oracle.com.In this case the patchset is p19769496_112030_AIX64-5L.zip.

Ø  Psftp/winscp the patchset to the prdrms server under /prdrms/oracle/temp.

Ø  Unzip the patchset prdrms /prdrms/oracle/temp=>unzip p19769496_112030_AIX64-5L.zip

Ø  After unzipping it will create a folder similar to below.

drwxrwxr-x   15 oracle   oradba         4096 Dec 10 20:57 19769496

Ø  Export PATH as below :-

export PATH=$ORACLE_HOME/OPatch:$PATH

Ø  Check perl version and check that whether the minimum perl required for the patch is installed
by using perl –v

Ø  Check that the proper Oracle home is getting pointed and binaries are getting invoked from proper location

prdrsm /prdrms/oracle/temp=>which opatch
/prdrms/oracle/product/11.2.0/OPatch/opatch

prdrms /apps/oracle=>which unzip
/prdrms/oracle/product/11.2.0/bin/unzip

Ø  Check that whether minimum opatch utility is installed as per Patch Read me document.

prdrms /prdrms/oracle/temp=>opatch version
OPatch Version: 11.2.0.3.6
OPatch succeeded.
Oracle recommends that you use the latest released OPatch version for 11.2, which is available for download from My Oracle Support patch 6880880by selecting the 11.2.0.0.0 release.

Ø  Check what are the patches installed in the database from operating system level:-

prdrms /prdrms/oracle/temp=>opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.


Oracle Home       : /prdrms/oracle/product/11.2.0
Central Inventory : /prdrms/oracle/oraInventory
   from           : /prdrms/oracle/product/11.2.0/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.3.0
Log file location : /prdrms/oracle/product/11.2.0/cfgtoollogs/opatch/opatch2015-02-26_16-57-23PM_1.log

Lsinventory Output file location : /prdrms/oracle/product/11.2.0/cfgtoollogs/opatch/lsinv/lsinventory2015-02-26_16-57-23PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.3.0
There are 1 product(s) installed in this Oracle Home.


Interim patches (1) :

Patch  17540582     : applied on Mon Jan 20 10:55:24 PST 2014
Unique Patch ID:  16985513
Patch description:  "Database Patch Set Update : 11.2.0.3.9 (17540582)"
   Created on 7 Jan 2014, 04:32:31 hrs PST8PDT
Sub-patch  16902043; "Database Patch Set Update : 11.2.0.3.8 (16902043)"
Sub-patch  16619892; "Database Patch Set Update : 11.2.0.3.7 (16619892)"
Sub-patch  16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)"
Sub-patch  14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
Sub-patch  14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"
   Bugs fixed:
 
--------------------------------------------------------------------------------

OPatch succeeded.

One off Patch Conflict Detection and Resolution

Ø  Determine whether any currently installed one-off patches conflict with the PSU patch as follows:

prdrms /prdrms/oracle/temp=>opatch prereq CheckConflictAmongPatchesWithDetail -phBaseDir ./19769496 -invPtrLOc $ORACLE_HOME/oraInst.loc
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /prdrms/oracle/product/11.2.0
Central Inventory : /prdrms/oracle/oraInventory
   from           : /prdrms/oracle/product/11.2.0/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.3.0
Log file location : /prdrms/oracle/product/11.2.0/cfgtoollogs/opatch/opatch2015-02-19_12-08-36PM_1.log

Invoking prereq "checkconflictamongpatcheswithdetail"

Prereq "checkConflictAmongPatchesWithDetail" passed.

OPatch succeeded.

Execute the below check conflict command from outside of the patch directory if you want to use the central inventory.

prdrms /prdrms/oracle/temp=>opatch prereq CheckConflictAmongPatchesWithDetail -phBaseDir ./19769496
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /prdrms/oracle/product/11.2.0
Central Inventory : /prdrms/oracle/oraInventory
   from           : /prdrms/oracle/product/11.2.0/oraInst.loc
OPatch version    : 11.2.0.3.6
OUI version       : 11.2.0.3.0
Log file location : /prdrms/oracle/product/11.2.0/cfgtoollogs/opatch/opatch2015-02-19_12-08-45PM_1.log

Invoking prereq "checkconflictamongpatcheswithdetail"

Prereq "checkConflictAmongPatchesWithDetail" passed.

OPatch succeeded.

Ø  Check if there is sufficient space in the oracle home directory as PSU patches normally consume 2 GB of space in the oracle home.

prdrms /prdrms/oracle/temp=>df -g $ORACLE_HOME
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/prdrms_oraclelv     30.00     16.87   44%    48963     2% /prdrms/oracle

Ø  Shutdown the listener and database.

prdrms /apps/oracle=>lsnrctl stop LISTENER_PRDRMS

SQL> shu immediate;

Ø  Ensure that no oracle process is running from O/s end and no jobs is running like below :-
ps –aef|grep –i oracle
$ jobs
If any jobs/processes arerunning, pleasekill those processes using kill -9 process id and stop the ongoing jobs. If any applications jobs are running, coordinate with ITAS DBA team.

Ø  Comment all the crontab entries relevant to the database.

Ø  Take a backup of Oracle Home directory and oracle inventory file in a mount point where there is sufficient amount of space.

          prdrms /prdrms/oracle/product/11.2.0=>cd /prdrms/oracle/product
prdrms /prdrms/oracle/product=>ls -ltr
total 8
drwxr-xr-x   80 oracle   oradba         4096 Feb 12 03:04 11.2.0
prdrms /prdrms/oracle/product=>tar -cvf /prdrms/tmp002/11.2.0_PRDRMS.tar 11.2.0
prdrms /prdrms/oracle/product=>cat $ORACLE_HOME/oraInst.loc
inventory_loc=/prdrms/oracle/oraInventory
inst_group=oradba
prdrms /prdrms/oracle/product=>cd /prdrms/oracle
prdrms /prdrms/oracle=>tar -cvf /prdrms/tmp002/oraInventory_PRDRMS.tar oraInventory

Ø  Go to the patch directory and apply the patch after confirm no database process is running from the Oracle Home

prdrms /prdrms/oracle/temp/19769496=>pwd
/prdrms/oracle/temp/19769496

prdrms /prdrms/oracle/temp/19769496=> opatch apply

OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
                             (Oracle Home = '/prdrms/oracle/product/11.2.0')
            [Feb 26, 2015 5:51:07 PM]    User Responded with: Y
 Press Enter when asked email and select ‘YES’ about security updates when asked

Ø  It was noticed that opatch failed with the error message “Copy failed from '/prdrms/oracle/temp/19769496/18031683/files/lib/libnnz11.so' to '/prdrms/oracle/product/11.2.0/lib/libnnz11.so'...” .Do you want to proceed? [y|n]
Start to wait for user-input at Thu Feb 26 17:51:38 PST 2015
Finish waiting for user-input at Thu Feb 26 17:51:44 PST 2015
User Responded with: N

OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: Copy failed from '/prdrms/oracle/temp/19769496/18031683/files/lib/libnnz11.so' to '/prdrms/oracle/product/11.2.0/lib/libnnz11.so'...
     ‘Restoring "/prdrms/oracle/product/11.2.0" to the state prior to running NApply...
[Feb 26, 2015 5:51:49 PM]    OPatch failed to restore the files from backup area. Not running "make".
[Feb 26, 2015 5:51:49 PM]    OUI-67124:
                             NApply was not able to restore the home.  Please invoke the following scripts:
                               - restore.[sh,bat]
                               - make.txt (Unix only)
                             to restore the ORACLE_HOME.  They are located under
                             "/prdrms/oracle/product/11.2.0/.patch_storage/NApply/2015-02-26_17-49-26PM"

Ø  Oracle failed to restore Oracle home, hence instructed us to run restore.sh as above.

Ø  Restore .sh was executed as below :-

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2015.02.26 17:53:19 =~=~=~=~=~=~=~=~=~=~=~=
./restore.sh
This script is going to restore the Oracle Home to the previous state.
It does not perform any of the following:
- Running init/pre/post scripts
- Oracle binary re-link
- Customized steps performed manually by user
Please use this script with supervision from Oracle Technical Support.
About to modify Oracle Home( /prdrms/oracle/product/11.2.0 )
Do you want to proceed? [Y/N]
Y
User responded with : Y
Restore script completed.

Ø  After running restore.sh,Oracle home has been relinked as below :-

 prdrms /prdrms/oracle/product/11.2.0/.patch_storage/NApply/2015-02-26_17-49-26PM=>$ORACLE _HOME/bin/relink all;

writing relink log to: /prdrms/oracle/product/11.2.0/install/relink.log

Ø  Once relinking is completed , opatch is applied again and it is completed successfully  as below :-

                 prdrms /prdrms/oracle/temp/19769496=> opatch apply

          [Feb 26, 2015 6:02:32 PM]    --------------------------------------------------------------------------------
[Feb 26, 2015 6:02:32 PM]    OUI-67008:OPatch Session completed with warnings.
We can ignore the warning as it is specific to some AIX environments.

Ø  Once the patch apply has finished perform the following post patching activities.
Bring the database up.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> @utlrp.sql

Ø  Bounce the database again and open it, start the listener.

Ø  Check the alert log for any error/warning messages after database is started.

Post Patching Steps

1) Check that no object owner by “SYS” is in invalid state as below:-

SQL> select OWNER,OBJECT_NAME,OBJECT_TYPE from dba_objects where OWNER = 'SYS' and status <>'VALID';

If any object is invalid, compile them.

2) Validate the PSU level from O/S level and database level by below query:-

set linesize 120
col instance_name format a15 heading "Instance|Name"
col host_name     format a15 heading "Hostname"
col version       format a10 heading "Version"
col comments      format a20 heading "Patch|Applied"
col id                       heading "Patch|Number"
col to_char(r.action_time,'DD-Mon-YYYY@HH24:mi:ss')  format a20 heading "Date"
rem ******************

prompt MISC: APPLIED CPU HISTORY BY DATE

select distinct i.instance_name, i.host_name, i.version, r.comments, r.id, to_char(r.action_time,'DD-Mon-YYYY@HH24:mi:ss')
from v$instance i, sys.registry$history r
where r.comments like 'CPU%' or r.comments like 'PSU%'
order by to_char(r.action_time,'DD-Mon-YYYY@HH24:mi:ss');
/

!$ORACLE_HOME/OPatch/opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc |grep "Patch" | grep "applied on"