How to perform Grid patch and Database patch in Oracle RAC Manual Method (557934.1)( 1486109.1)

 

How to perform Grid patch and Database patch in Oracle RAC Manual Method (557934.1)( 1486109.1)

Note :  if you are following this document then always use full path. In place of $GRID_HOME , use full path GRID HOME and  for $ORACLE_HOME use full path of ORACLE HOME. This document is created only for reference .

Usually,  oracle release quarterly patch. Tentative dates are

1.      Q1➡January 17

2.      Q2➡ April 17

3.      Q3➡July 17

4.      Q4➡October 17

 

1.      Patches are available in different format

2.      Patch sets

A.     Patch sets update (PSU) / BP (PSU patches in Exadata is called Bundel patch (888828.1) ➡ Release time January, April, July, October. Oracle always recommend  going with PSU.

B.     RDBMS PSU

C.     GI PSU

D.     OJVM PSU (They will apply in only Database Home)

E.     Interim patches / Oneoff ➡ if you bundle one or two patches then they called interim patches or merge patch

 

How to download  the Latest Patch

1.      Log in to My Oracle Support. (https://support.oracle.com/)

2.      On the main My Oracle Support page, click Patches and Updates tab.

3.      In the Patch Search group, select Product or Family (Advanced).

4.      In the Product field, select Oracle Database.

5.      In the Release field select the release number. For example, Oracle 19.0.0.0

6.      Click Search.

7.      Any available patch updates are displayed in the Patch Search page.

8.      Select the patch number and click ReadMe. The README page is displayed and contains information about the patch set and how to apply the patches to your installation.

9.      Return to the Patch Search page, click Download, and save the file on your system.

10.  Use the unzip utility provided with Oracle Database 19c to uncompress the

11.  Oracle patch updates that you downloaded from My Oracle Support.

12.  The unzip utility is located in the $ORACLE_HOME\bin directory

 

 

 

Impotent point before starting the patching

1.      It is highly recommended to take a backup of the Oracle home binaries, the Grid home binaries, and Central Inventory prior to applying patches. For further information, refer to My Oracle Support document 565017.1

2.      You must use Latest OPatch utility to apply the patch For further information, refer to My Oracle Support document 293369.1

3.      Please refer 6880880 for downloading latest OPatch utility

4.      You can use below command to check the latest Opatch version

$$ORACLE_HOME/opatch version

5.      If your OPatch utility is not in latest version, then you must upgrade it accordingly.

6.      For upgrading  the OPatch utility ,just remove or rename existing OPatch directory and copy new OPatch utility which you have downloaded and again check the version of OPatch utility and also grant required permission (Permission is most important point during the patching)

$$ORACLE_HOME/OPatch/opatch version

$cd $ORACLE_HOME

$chmod -R oragrid:oinstall Opatch

$chmod -R 755 Opatch

7.      Before beginning patch application, check the consistency of inventory information for Grid home and each Oracle home to be patched. Run this command as the respective ORACLE_HOME user to check the consistency:

$ <ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>

8.       If this command succeeds, it lists the Oracle components that are installed in the home. Save the output so that you have the status prior to the patch application.If this command fails, contact Oracle Support for assistance.

9.       For more information, please check read me file

10.    Now you can start the patching process

11.    Use below command to check the GRID HOME issue before patching

cluvfy stage -pre patch

12.    Use below command to check the GRID HOME issue after patching

cluvfy stage -post patch

13.    The CLUVFY command line for patching ensures that the GRID_HOME can receive the new patch and also ensures that the patch application process completed successfully leaving the home in the correct state. For more information, please check 30839369

14.    There are two ways of patching 

A. Rolling ➡Node by Node patching (This is a default mode)

B. Non-Rolling ➡Patching all Nodes at same time

 

Steps To Involve inGRID Patching

1.      Opatch Location of GRID HOME : $ORACLE_HOME/OPatch

2.      Stop GI Home as root user

3.      Unlock the GI Home as root user

4.      grant permission patch directory as root user

5.      As grid user do pre-checks

6.      Do OPatch version and current lspatches checks

A.     Conflict pre-checks

B.     System space pre-checks

7.      Apply Patch:

8.      Post verification

9.      Lock and start GI

 

Steps Involve in Oracle Home Patching

1.      Opatch Location for ORACLE HOME : $ORACLE_HOME/OPatch

2.      Stop DB Home as root user

3.      grant permission on patch directory as root user

4.      As oracle user do pre-checks

5.      Do OPatch version and current lspatches checks

A.     Conflict pre checks:

B.     System space pre-checks:

6.      Apply Patch:

7.      Post verification

8.      Start ORACLE_HOME as root user

 

Enable/Disable CRON (crontab -e)

To Disable: s/^/##PSU_OCT_21_PRE##/g

To Enable: s/##PSU_OCT_21_PRE##//g

 

Check CRS Services

As per my experience , we should take restart of server and CRS before starting the patching. Take all the pre-checks whatever you can take

$crsctl check crs

$crsctl check cluster -all

 

First, We Will Check Conflict

As a GRID HOME user because below is the GRID HOME patch

 

$GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35643107

$GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35655527

$GRID _HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35652062

$GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35553096

$GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/33575402

 

Or we can use above commands with ORACLE_HOME (Here you need to mention the location of path )option as below

 

$GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35643107 -oh $GRID_HOME

$GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35655527 -oh $GRID_HOME

$GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35652062 -oh $GRID_HOME

$GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35553096 -oh $GRID_HOME

$GRID_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/33575402 -oh $GRID_HOME

 

 

As anORACLE HOME user because below is the ORACLE HOME patch

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35643107

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35655527

 

You can use above commands with ORACLE HOME option as below

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35643107 -oh $ORACLE_HOME

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/35642822/35655527 -oh $ORACLE_HOME

 

Now Run Opatch System Space Check

As a GRID HOME user because below is the GRID HOME patch

$GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile <UNZIPPED_PATCH_LOCATION>/35642822/35643107

$GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile <UNZIPPED_PATCH_LOCATION>/35642822/35655527

$GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile <UNZIPPED_PATCH_LOCATION>/35642822/35652062

$GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile <UNZIPPED_PATCH_LOCATION>/35642822/35553096

$GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile <UNZIPPED_PATCH_LOCATION>/35642822/33575402

As an ORACLE HOME user because below is the ORACLE HOME patch

$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile <UNZIPPED_PATCH_LOCATION>/35642822/35643107

$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile <UNZIPPED_PATCH_LOCATION>/35642822/35655527

 

Or we can also use above commands with ORACLE_HOME (Here you need to mention the location of path)

Example:

 

$GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile <UNZIPPED_PATCH_LOCATION>/35642822/35643107 -oh $GRID_HOME

$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile <UNZIPPED_PATCH_LOCATION>/35642822/35643107 - oh $ORACLE_HOME

 

You can also take these pre checks by creating text file.Create a temporary text file /tmp/patch_list_gihome.txt with the following content

$cat /tmp/patch_list_gihome.txt

<UNZIPPED_PATCH_LOCATION>/35642822/35643107

<UNZIPPED_PATCH_LOCATION>/35642822/35655527

<UNZIPPED_PATCH_LOCATION>/35642822/35652062

<UNZIPPED_PATCH_LOCATION>/35642822/35553096

<UNZIPPED_PATCH_LOCATION>/35642822/33575402

 

 $ GRID_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt

 

$ cat /tmp/patch_list_dbhome.txt

<UNZIPPED_PATCH_LOCATION>/35642822/35643107

<UNZIPPED_PATCH_LOCATION>/35642822/35655527

 

 $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt

 

Password Less Authentication

Once conflict and system space check done you also need to check password less authentication between nodes across the cluster along with GRID ,ORACLE and ROOT user. If it is not enabled, please enable it. Usually there is a separate procedure to enable it in the project for which you are working. Let’s try to enable it through oracle recommended method.

We can use sshUserSetup.sh file located on $ORACLE_HOME/deinstall to setup the password less authentication.

Syntex :

$./sshUserSetup.sh -user oracle -hosts "dbhost1 dbhost2" -noPromptPassphrase -confirm -advanced

$./sshUserSetup.sh -user grid -hosts "dbhost1 dbhost2" -noPromptPassphrase -confirm -advanced

 

 

Stopping the database and ORACLE_HOME

Now need to stop the all-database home and database on server for which you are going to apply the patch as below

$ORACLE_HOME/bin/srvctl stop database -d database_name

$ORACLE_HOME/bin/srvctl stop home -o $ORACLE_HOME -s /home/oracle/rac1_JAN21_PSU_03April.txt -n rac1

 

Grant the permission

Grant required permission to oui-patch.xml and other directories , if you miss then your patching will fail. You can find oraInst.loc inside the $GRID_HOME location. Default location of oraInst.loc is inside /etc/ folder.

chown oragrid:oinstall oui-patch.xml

chmod 660 oui-patch.xml

 

Unlock the GI Stack on Node one

For unlocking the GRID_HOME we will use rootcrs.sh file located on $GRID_HOME/crs/install as root user.

 

 

$GRID_HOME/crs/install/rootcrs.sh -prepatch -nonrolling   ➡for nonrolling manner    (Now a days no one is use everyone perform patching via only rolling manner)

$GRID_HOME/crs/install/rootcrs.sh -prepatch➡for rolling manner

 

Prepatch will unlock your GRID_HOME and it will change the permission some of the files owned by root user and give the permission as grid owner. So, you can patch your GRID HOME without permission issue. It will also shut down TFA and restart your GRID HOME and change the state from NORMAL to ROLLING PATCH.

 

Now Login Via GRID User and Apply Patch One by One

Now login via GRID user and set environment

 

$export ORACLE_HOME=/u01/app/19.0.0.0/grid

$export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH

$export TNS_ADMIN=$ORACLE_HOME/network/admin

$which opatch

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local -silent  <UNZIPPED_PATCH_LOCATION>/35642822/35643107

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local -silent  <UNZIPPED_PATCH_LOCATION>/35642822/35655527

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local -silent  <UNZIPPED_PATCH_LOCATION>/35642822/35652062

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local -silent  <UNZIPPED_PATCH_LOCATION>/35642822/35553096

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local -silent  <UNZIPPED_PATCH_LOCATION>/35642822/33575402

 

You can check the logs in below location

$GRID_HOME/cfgtoollogs/opatch

 

Check Patches Are Applied or Not in Grid Home

$GRID_HOME/OPatch/opatch lspatches|grep -i applied

$GRID_HOME/OPatch/opatch lspatches|sort -nr

$GRID_HOME/OPatch/opatch lsinv|grep -i applied

 

Now GRID PATCHING gets completed via manual method  however some steps are pending but we will perform once we will complete Database patching . Whatever you have done in first node you need to perform same steps in other nodes as well.

 

Let’s start Database ORACLE HOME patching

Now Login Via Oracle User and Patches One by One

First, we need to prepatch on Database ORACLE HOME      (This is  not mandatory )

<UNZIPPED_PATCH_LOCATION>/35642822/35643107/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME

 

$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local -silent  <UNZIPPED_PATCH_LOCATION>/35642822/35643107

$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE _HOME -local -silent  <UNZIPPED_PATCH_LOCATION>/35642822/35655527

 

First, we need to post patch on Database ORACLE HOME (This is  not mandatory )

<UNZIPPED_PATCH_LOCATION>/35642822/35655527/custom/scripts/postpatch.sh-dbhome $ORACLE_HOME

 

Check Patches Are Applied or Not in Oracle Home

$ORACLE _HOME/OPatch/opatch lspatches|grep -i applied

$ORACLE _HOME/OPatch/opatch lspatches|sort -nr

$ORACLE _HOME/OPatch/opatch lsinv|grep -i applied

 

Login via Grid User and run post patch scripts

$GRID_HOME/rdbms/install/rootadd_rdbms.sh

 

 

Now lock the GI Stack on Node one

For locking the GRID_HOME we will use rootcrs.sh file located on $GRID_HOME/crs/install as root user.

 

 

$GRID_HOME/crs/install/rootcrs.sh -postpatch

 

This one will start your Clusterware and revert the changes which has performed during prepatch and also change the status of Clusterware from ROLLING PATCH to NORMAL.

 

Start the database and ORACLE_HOME

Login via ORACLE user and start the all-database home and database on server

$ORACLE_HOME/bin/srvctl start home -o $ORACLE_HOME -s /home/oracle/rac1_JAN21_PSU_03April.txt -n rac1

$ORACLE_HOME/bin/srvctl stop database -d database_name

 

Rollback Conflict Patches

$OH/OPatch/opatch rollback -id 32933487 -silent

$OH/OPatch/opatch rollback -id 29511771 -silent

$OH/OPatch/opatch rollback -id 20713578 -invPtrLoc $ORACLE_HOME/oraInst.loc

 

GRID And DATABASE Patching Through opatchauto Method

First, We Need to Take thePre-Checks Via opatchauto command. Login via root user and run below command to perform the precheck on  all the nodes

 

$export ORACLE_HOME=/u01/app/19.0.0.0/grid

$export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH

$export TNS_ADMIN=$ORACLE_HOME/network/admin

$which opatch

$GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/35642822 --analyze

Above opatchauto command will take the pre checks of GRID HOME as well as ORACLE HOME. This utility you cannot run parallel on nodes within the cluster. You need to run this command one by one on all the nodes. Once above command gets completed then take required action according to output.

 

Now apply the patch via opatchauto

Below command will apply  the patch on both GRID HOME and ORACLE HOME on the node where you are applying the patch.

$GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/35642822

 

Once patching gets completed then bring up the database and apply datapatch on all the databases.

 

Issue which I have faced during patching

Issue 1 -

Grid Infrastructure gives error CRS-6706: Oracle Clusterware Release Patch Level ('748994161') Does Not Match Software Patch Level (Doc ID 2348013.1)

After applying the patch when I have executed rootcrs.sh -postpatch it was failing

2017-11-19 16:29:27: Oracle CRS stack has been shut down
2017-11-19 16:29:27: The stack was already down before stopping it
2017-11-19 16:29:27: Starting CRS without resources...
2017-11-19 16:29:27: OHASD needs to be up for disabling CRS resource
2017-11-19 16:29:27: Executing cmd: /u01/app/12.2.0.1/grid/bin/crsctl start crs -noautostart
2017-11-19 16:29:27: Command output:
> CRS-6706: Oracle Clusterware Release patch level ('748994161') does not match Software patch level ('0'). Oracle Clusterware cannot be started.
> CRS-4000: Command Start failed or completed with errors.

 

Solution :

Run the following command as the root user to complete the patching set up behind the scenes:

$GRID_HOME/bin/clscfg -localpatch

$GRID_HOME/crs/install/rootcrs.sh -lock

$GRID_HOME/bin/crsctl start crs

 

Issue 2 –

This issue I have faced when I was applying the grid patch from OCI console. On first node patching gets completed successfully but on second node it gets failed because it was not able o shut down the databases. So we have created a SR with oracle support then as discussed with oracle support this was a unpublished bug  36200983 and Bug 36118349.

 

AgentErrorWrMsg:DBAAS-60225:[FATAL] [DBAAS-60225] Job 'stop_db_instances_on_node-copc3prnvm04' has failed on node(s): copc3prnvm04.
AgentErrorWrMsg:DBAAS-70430:[FATAL] [DBAAS-70430] Could not stop the instances of following databases from node 'copc3prnvm04': [ java.lang.NullPointerException].

 

Oracle has recommended to run below command from root user and again resume the job.

 

dbaascli grid patch --targetVersion 19.21.0.0.0 --nodeList copc3prnvm04 --continueWithDbDowntime --pilot_additional_args stop_db_instances=false