Oracle Database 12c and OEM Cloud Control 12c Installation on Ubuntu

Oracle Database and Enterprise Manager Cloud Control 12c Installation on Ubuntu

Pre-Requisites

1.     Perform pre-requisites as listed in Linux install of Oracle Database and Oracle Enterprise Manager for 12c version, this would include:

• Linux Packages required 
  
  apt-get -y install elfutils libaio1 libaio-dev libstdc++* numactl pdksh sysstat unixodbc-dev unixodbc build-essential gawk ksh libmotif* libXtst* alien libtool glibc-* binutils cpp-* debhelper g++* gcc* gcc-*-base gettext html2text lib32z1 lib32ncurses5 intltool-debian lib32z1-dev libc6 libc6-dev libc6-dev-i386 libelf-dev libelf1 libltdl-dev libltdl7 libodbcinstq4-1 libodbcinstq4-1:i386 libqt*-core libqt*-gui libsqlite3-0 lsb lsb-core make odbcinst pax po-debconf rpm rpm-common unzip lib*crypt xorg* glibc-devel.i386 setarch rng-utils
  
• Kernel Configuration 
  
  echo "#" | sudo tee -a /etc/sysctl.conf
  echo "# Oracle 12C entries" | sudo tee -a /etc/sysctl.conf
  echo "fs.aio-max-nr=1048576" | sudo tee -a /etc/sysctl.conf
  echo "fs.file-max=6815744" | sudo tee -a /etc/sysctl.conf
  echo "kernel.shmall=2097152" | sudo tee -a /etc/sysctl.conf
  echo "kernel.shmmni=4096" | sudo tee -a /etc/sysctl.conf
  echo "kernel.sem=250 32000 100 128" | sudo tee -a /etc/sysctl.conf
  echo "net.ipv4.ip_local_port_range=9000 65500" | sudo tee -a /etc/sysctl.conf
  echo "net.core.rmem_default=262144" | sudo tee -a /etc/sysctl.conf
  echo "net.core.rmem_max=4194304" | sudo tee -a /etc/sysctl.conf
  echo "net.core.wmem_default=262144" | sudo tee -a /etc/sysctl.conf
  echo "net.core.wmem_max=1048586" | sudo tee -a /etc/sysctl.conf
  echo "kernel.shmmax=1073741824" | sudo tee -a /etc/sysctl.conf 
  
  Run command to make changes effective:
  sysctl -p
  
• Security Limits
  
  cp /etc/security/limits.conf /etc/security/limits.conf.original
  echo "#Oracle 12C shell limits:" | sudo tee -a /etc/security/limits.conf
  echo "oracle soft nproc 2048" | sudo tee -a /etc/security/limits.conf
  echo "oracle hard nproc 16384"| sudo tee -a /etc/security/limits.conf
  echo "oracle soft nofile 1024" | sudo tee -a /etc/security/limits.conf
  echo "oracle hard nofile 65536" | sudo tee -a /etc/security/limits.conf
• It is required to create softlinks to below lib files to avoid installation failure when Oracle looks up for them
  
  mkdir /usr/lib64
  ln -s /usr/lib /usr/lib/lib64
  ln -s /lib/x86_64-linux-gnu/libgcc_s.so.1 /lib64/
  ln -s /usr/bin/awk /bin/awk
  ln -s /usr/bin/basename /bin/basename
  ln -s /usr/bin/rpm /bin/rpm
  ln -s /usr/lib/x86_64-linux-gnu/libc_nonshared.a /usr/lib64/
  ln -s /usr/lib/x86_64-linux-gnu/libpthread_nonshared.a /usr/lib64/
  ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /lib64/
  ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/lib64/
  ln /usr/lib/x86_64-linux-gnu/libpthread_nonshared.a /usr/lib64/libpthread_nonshared.a
  
  In case if you get missing reference for below library, make soft link and try relinking: (you may verify using ldd command for references of library that fails linking)
  ln -s <ORACLE_HOME>/lib/libclntshcore.so.12.1 /usr/lib/

As Ubuntu is not a certified Linux for Oracle, it would require to make additional changes in ins_rdbms.mk and env_rdbms.mk files for installation to complete (not failing at relink step).  All credits for these changes goes to a post that I found on internet (can't remember the name though), without their post I could not have get past this hurdle.
  
To make it short I have summarized the changes below, you would require to add "-Wl,--no-as-needed" for modules that fail in compiling.

<ORACLE_HOME>/rdbms/lib$ cat env_rdbms.mk | grep "-Wl,--no-as-needed"
ORACLE_LINKLINE=$(ORACLE_LINKER) -Wl,--no-as-needed $(PL_FLAGS) $(ORAMAI) $(SSORED) $(TTCSOI) \
EXTPROC_LINKLINE=$(LINK) $(OPT) -Wl,--no-as-needed $(EXTPMAI) $(PROD_EXTPROC_OPT) \
HSDEPXA_LINKLINE=$(LINK) $(OPT) -Wl,--no-as-needed $(HSDEPMAI) $(HSALIB_SUBSET1) \
HSOTS_LINKLINE=$(LINK) $(OPT) -Wl,--no-as-needed $(HSOTSMAI) $(HSALIB_SUBSET1) \
KFED_LINKLINE=$(LINK) $(S0MAIN) -Wl,--no-as-needed $(SSKFEDED) $(SKFEDPT) \
KFOD_LINKLINE=$(LINK) $(S0MAIN) -Wl,--no-as-needed $(SSKFODED) $(SKFODPT) $(KFODOBJ) \
KFNDG_LINKLINE=$(LINK) $(S0MAIN) -Wl,--no-as-needed $(SSKFNDGED) $(SKFNDGPT) $(KFNDGOBJ) \
AMDU_LINKLINE=$(LINK) $(S0MAIN) -Wl,--no-as-needed $(SSKFMUED) $(SKFMUPT) $(LLIBCLNTSH) \




Specify library file names for below modules in ins_rdbms.mk, and the ones that may fail, if any, during linking.

<ORACLE_HOME>/rdbms/lib$ cat ins_rdbms.mk | grep lnnz12
        $(ORAPWD_LINKLINE) -lnnz12
        $(TG4PWD_LINKLINE) -lnnz12
<ORACLE_HOME>/rdbms/lib$ cat ins_rdbms.mk | grep lons
        $(PLSHPROF_LINKLINE) -lons
        $(RMAN_LINKLINE) -lons
<ORACLE_HOME>/rdbms/lib$ cat ins_rdbms.mk | grep lagtsh
        $(EXTPROC_LINKLINE) -lagtsh
        $(HSOTS_LINKLINE) -lagtsh

<ORACLE_HOME>/network/lib$ cat ins_net_server.mk | grep "\-l"
        $(TNSLSNR_LINKLINE) -lnnz12 -lons

Installation

1.    Downloaded softwares

Database -> /u01/stage/DB12c
OEM -> /u01/stage/OMS

2.    Add user “oracle” and groups “dba” and “oinstall” and prepare directories

# groupadd -g 202 dba
# groupadd -g 203 oinstall
# useradd -m -d /home/oracle -u 203 -g oinstall -G dba -s /bin/bash -p <pass> oracle
# mkdir -p /u01/{oem12c,db12c,admin}
# mkdir -p /u01/oem12c/{middleware,agent12c}
# chown -R oracle:oinstall /u01/*
# chmod 755 -R /u01/*

3.    Install Oracle Database Software and create database EMREP to be used as OEM repository
 
a. Unzip software as “oracle”

$ cd /u01/stage/DB12c
$ unzip linuxamd64_12102_database_1of2.zip
$ unzip linuxamd64_12102_database_2of2.zip

b. After unzipping prepare response file

$ cd /u01/stage/DB12c/database/response
$ vi oemdb.rsp

db_install.rsp is sample file provided by Oracle that can be used as a template after modification, the parameters that I used are:
oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v12.1.0
oracle.install.option=INSTALL_DB_SWONLY
ORACLE_HOSTNAME=symgrNCS01
UNIX_GROUP_NAME=oinstall
SELECTED_LANGUAGES=en
ORACLE_HOME=/u01/db12c
ORACLE_BASE=/u01/admin
oracle.install.db.InstallEdition=EE
oracle.install.db.DBA_GROUP=dba
oracle.install.db.OPER_GROUP=dba
oracle.install.db.BACKUPDBA_GROUP=dba
oracle.install.db.DGDBA_GROUP=dba
oracle.install.db.KMDBA_GROUP=dba
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false
DECLINE_SECURITY_UPDATES=true

c. Invoke Silent Installation

$ cd /u01/stage/DB12c/database
$  ./runInstaller -ignoreSysPrereqs -silent -responseFile /u01/stage/DB12c/database/response/oemdb.rsp

d. Prepare pfile and create spfile for instance
Although DBUA can be used to create database, I preferred to go with manual method.

*.compatible='12.1.0.2'
*.control_files='/u02/oradb/control01.ctl'
*.db_block_size=8192
*.db_create_file_dest='/u02/oradb/'
*.db_create_online_log_dest_1='/u02/oradb/redologs/'
*.db_create_online_log_dest_2='/u02/oradb/mirrlogs/'
*.db_name='EMREP'
*.db_securefile='PERMITTED'
*.diagnostic_dest='/u02/data/admin/'
*.job_queue_processes=50
*.log_archive_dest='/u02/oradb/arch/'
*.log_archive_format='log%t_%s_%r.arc'
*.memory_target=6G
*.open_cursors=350
*.parallel_max_servers=8
*.processes=300
*.undo_tablespace='UNDOTBS1'

e. Startup nomount instance and run create database script

$ sqlplus / as sysdba
SQL> CREATE DATABASE EMREP ARCHIVELOG
DATAFILE SIZE 500M AUTOEXTEND ON
SYSAUX DATAFILE SIZE 300M AUTOEXTEND ON
UNDO TABLESPACE UNDOTBS1 DATAFILE SIZE 200M AUTOEXTEND ON
DEFAULT TEMPORARY TABLESPACE TEMP1 TEMPFILE SIZE 200M AUTOEXTEND ON
LOGFILE
GROUP 1 SIZE 500M,
GROUP 2 SIZE 500M,
GROUP 3 SIZE 500M
CHARACTER SET AL32UTF8;

Run catalog.sql, catproc.sql and pupbld.sql respectively.

f. Create password file and listener for database

$ cd $ORACLE_HOME/dbs
$ orapwd file=orapwEMREP password=<pass> ignorecase=y

$ vi $ORACLE_HOME/network/admin/listener.ora
EMREP =
 (DESCRIPTION=
   (ADDRESS=(PROTOCOL=TCP)(HOST=srv01)(PORT=1521)
    (CONNECT_DATA=(SID=EMREP))
   )
 )

SID_LIST_EMREP=
 (SID_LIST=
  (SID_DESC=
   (ORACLE_HOME=/u01/db12c)
   (SID_NAME=EMREP)
  )
 )

$ lsnrctl start EMREP

4.    Install Oracle Enterprise Manager

a. Unzip software and prepare response file

$ cd /u01/stage/OMS/
$ unzip em12105_linux64_disk1.zip
$ unzip em12105_linux64_disk2.zip
$ unzip em12105_linux64_disk3.zip

$ cd /u01/stage/OMS/em12105_linux/response
$ vi oms_new_install.rsp




new_install.rsp is sample file provided by Oracle that can be used as a template after modification, the parameters that I used are:

RESPONSEFILE_VERSION=2.2.1.0.0
UNIX_GROUP_NAME="dba"
INVENTORY_LOCATION="/u01/oraInventory"
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false
DECLINE_SECURITY_UPDATES=true
INSTALL_UPDATES_SELECTION="skip"
ORACLE_MIDDLEWARE_HOME_LOCATION=/u01/oem12c/middleware
ORACLE_HOSTNAME=srv01
AGENT_BASE_DIR=/u01/oem12c/agent12c
WLS_ADMIN_SERVER_USERNAME="weblogic"
WLS_ADMIN_SERVER_PASSWORD="oracle12"
WLS_ADMIN_SERVER_CONFIRM_PASSWORD="oracle12"
NODE_MANAGER_PASSWORD="oracle12"
NODE_MANAGER_CONFIRM_PASSWORD="oracle12"
ORACLE_INSTANCE_HOME_LOCATION=/u01/oem12c/middleware/gc_inst
CONFIGURE_ORACLE_SOFTWARE_LIBRARY=false
DATABASE_HOSTNAME="srv01"
LISTENER_PORT="1521"
SERVICENAME_OR_SID="EMREP"
SYS_PASSWORD="oracle12"
SYSMAN_PASSWORD="oracle12"
SYSMAN_CONFIRM_PASSWORD="oracle12"
DEPLOYMENT_SIZE="SMALL"
MANAGEMENT_TABLESPACE_LOCATION="/media/itadmin/oradb/EMREP/datafile/mgmt01.dbf"
CONFIGURATION_DATA_TABLESPACE_LOCATION="/media/itadmin/oradb/EMREP/datafile/mgmt_emc_depot01.dbf"
JVM_DIAGNOSTICS_TABLESPACE_LOCATION="/media/itadmin/oradb/EMREP/datafile/mgmt_deepdive01.dbf"
AGENT_REGISTRATION_PASSWORD="oracle12"
AGENT_REGISTRATION_CONFIRM_PASSWORD="oracle12"
FROM_LOCATION="../oms/Disk1/stage/products.xml"
DEINSTALL_LIST={"oracle.sysman.top.oms","12.1.0.5.0"}
b_upgrade=false
EM_INSTALL_TYPE="NOSEED"
CONFIGURATION_TYPE="ADVANCED"
TOPLEVEL_COMPONENT={"oracle.sysman.top.oms","12.1.0.5.0"}

b. Invoke OEM installation

$ cd /u01/stage/OMS/em12105_linux/
$ ./runInstaller -silent -responseFile /u01/stage/OMS/em12105_linux/response/oms_new_install.rsp

c. Obtain Oracle Management Agent for AIX

Since this OEM is a Linux install and all monitored servers are running AIX, we would require OEM to be equipped with software of Oracle Management Agent 12c for AIX to monitor them, this can be obtained using ONLINE method or OFFLINE method.
ONLINE method: https://docs.oracle.com/cd/E24628_01/install.121/e39876/downloading_agent_sw.htm#EMGSG111
OFFLINE method: https://docs.oracle.com/cd/E24628_01/install.121/e22624/install_agent.htm#EMBSC294

d. Add Targets for Monitoring

After downloading agent software using any of above two methods, proceed with deployment using “Add Targets” menu

5.    Configure Alerts and Notification Methods.
We are all done with installation, you may proceed with configuring alerts and notifications!

Enjoy.
-Anurag

Applying July-2016 PSU on 2-Node RAC with Standby (Rolling)

Hey everyone!, I recently experimented on opatchauto tool for complete automation of patching. The environment I used comprised of a 2-node RAC with a 2-node Standby database in place.

Environment Details:

Nodes to be Patched:
Primary
Node 1 - RAC01
Node 2 - RAC02
 Standby
Node 3 - SBY01
Node 4 - SBY02

Database Version 12.1.0.2 64bit
Grid Version (Active/Release/Software) 12.1.0.2 64bit
Operating System RHEL v7.2 (Mapio) on all nodes.

Kindly note that there were no interim patches applied to either grid infrastructure or database.

Patches to be applied:
1. Patch 6880880: OPatch patch of version 12.2.0.1.7 for Oracle software releases 12.1.0.x (installer)
and 12.2.0.x (AUG 2016)
Since opatch version in my environment is 12.1.0.1.3, it needs to be upgraded to 12.2.0.1.7

2. Patch 23273629: GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.2.160719 (JUL2016)

Note: Patch  23273629  includes database PSU and GI PSU, it does not includes OJVM PSU July-2016.

Upgrade OPATCH 

OPatch needs to be upgraded on GRID_HOME and RDBMS_HOME of all nodes

1. Download OPatch version 12.2.0.1.7

Downloaded patch on RAC01, RAC02, SBY01 and SBY02 as:
/u01/stage/opatch/p6880880_121010_Linux-x86-64.zip

2. Check current OPatch version
Run below commands against GRID_HOME or RDBMS_HOME as  grid owner user and oracle owner user respectively
$ export ORACLE_HOME=<GRID_HOME or RDBMS_HOME>
$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ opatch version

This would show 12.1.0.1.3

3. Upgrade OPatch on Database software and verify
On all nodes as oracle user
$ export ORACLE_HOME=/oracle/app/product/12.1.0.2/db_1
$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_bkp_121013
$ unzip /u01/stage/opatch/p6880880_121010_Linux-x86-64.zip -d $ORACLE_HOME
$ opatch version

4. Upgrade OPatch on Grid software and verify
On all nodes as grid user
$ export ORACLE_HOME=/u01/app/product/12.1.0/grid
$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_bkp_121013
$ unzip /u01/stage/opatch/p6880880_121010_Linux-x86-64.zip -d $ORACLE_HOME
$ opatch version

The output of above commands will now show 12.2.0.1.7 opatch version.

Validate Oracle Inventory

It is important to ensure that Oracle Inventory is consistent on each GRID_HOME and RDBMS_HOME that are to be patched, follow below steps to confirm:

For Oracle Grid Infrastructure: as grid user on all nodes
$ export ORACLE_HOME=/u01/app/product/12.1.0/grid
$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME
The successful execution of command will list Oracle components and patches, if any for Oracle Home.

For Oracle RDBMS: as oracle user on all nodes
$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
$ $ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME

Apply PSU

1. Download Patch 23273629

Downloaded patch on RAC01, RAC02, SBY01 and SBY02 as:
/u01/stage/gipsu_jul16/p23273629_121020_Linux-x86-64.zip

The installation of patch will be carried-out as root user, however it is must that owner of the zip file is grid user. As root perform below steps to ensure this:

# cd /u01/stage/gipsu_jul16/
# unzip p23273629_121020_Linux-x86-64.zip
# chown grid:oinstall –R /u01/stage/gipsu_jul16/23273629

Failing to give correct permissions would fail opatchauto with below error:

==Following patches FAILED in apply:

Patch: /u01/stage/gipsu_july16/23273629/23054327
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-14-39PM_1.log 
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException:
Prerequisite check "CheckApplicable" failed.


2. Check for any conflicts
Although no interim patches are applied thus check would pass. As grid user run below command to check for conflicts in both  GRID_HOME and RDBMS_HOME on node RAC01 and SBY01,  run as root :

# export ORACLE_HOME=/u01/app/product/12.1.0/grid
# export PATH=$ORACLE_HOME/OPatch:$PATH
# opatchauto apply /u01/stage/gipsu_jul16/23273629 -analyze

3. Stop logshipping from PRIMARY and stop Managed Recovery Process on STANDBY
Defer respective log_archive_dest_n parameter and cancel managed recovery.

4. Apply patch on STANDBY Nodes
Logon to SBY01 and SBY02 as root user, and follow below steps on one node at a time

Source environment for GRID HOME and Patch GRID_HOME on Node 1 and Node 2
NODE 1
# export ORACLE_HOME=/u01/app/product/12.1.0/grid
# export PATH=$ORACLE_HOME/OPatch:$PATH
[root@SBY01 23273629]# opatchauto apply /u01/stage/gipsu_jul16/23273629 -oh /u01/app/product/12.1.0/grid

System initialization log file is /u01/app/product/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2016-08-31_09-54-25PM.log.

Session log file is /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2016-08-31_09-54-31PM.log
The id for this session is HW39
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/product/12.1.0/grid

Executing OPatch prereq operations to verify patch applicability on CRS Home........

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/product/12.1.0/grid successfully
[init:init] Executing GIRACPrereqAction action on home /u01/app/product/12.1.0/grid

Executing prereq operations before applying on CRS Home........

[init:init] GIRACPrereqAction action completed on home /u01/app/product/12.1.0/grid successfully
[shutdown:shutdown] Executing GIShutDownAction action on home /u01/app/product/12.1.0/grid

Performing prepatch operations on CRS Home........

Prepatch operation log file location: /u01/app/product/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_SBY01_2016-08-31_09-54-58PM.log

[shutdown:shutdown] GIShutDownAction action completed on home /u01/app/product/12.1.0/grid successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/product/12.1.0/grid

Start applying binary patches on CRS Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/product/12.1.0/grid successfully
[startup:startup] Executing GIStartupAction action on home /u01/app/product/12.1.0/grid

Performing postpatch operations on CRS Home........

Postpatch operation log file location: /u01/app/product/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_SBY01_2016-08-31_10-02-32PM.log

[startup:startup] GIStartupAction action completed on home /u01/app/product/12.1.0/grid successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/product/12.1.0/grid

Verifying patches applied on CRS Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/product/12.1.0/grid successfully
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:sby01
CRS Home:/u01/app/product/12.1.0/grid
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/gipsu_jul16/23273629/21436941
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_21-57-34PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054246
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_21-57-34PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054327
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_21-57-34PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054341
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_21-57-34PM_1.log

NODE 2
# export ORACLE_HOME=/u01/app/product/12.1.0/grid
# export PATH=$ORACLE_HOME/OPatch:$PATH
[root@SBY02 23273629]# opatchauto apply /u01/stage/gipsu_july16/23273629 -oh /u01/app/product/12.1.0/grid

System initialization log file is /u01/app/product/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2016-08-31_10-13-29PM.log.

Session log file is /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2016-08-31_10-13-37PM.log
The id for this session is 5AIS
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/product/12.1.0/grid

Executing OPatch prereq operations to verify patch applicability on CRS Home........

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/product/12.1.0/grid successfully
[init:init] Executing GIRACPrereqAction action on home /u01/app/product/12.1.0/grid

Executing prereq operations before applying on CRS Home........

[init:init] GIRACPrereqAction action completed on home /u01/app/product/12.1.0/grid successfully
[shutdown:shutdown] Executing GIShutDownAction action on home /u01/app/product/12.1.0/grid

Performing prepatch operations on CRS Home........

Prepatch operation log file location: /u01/app/product/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_SBY02_2016-08-31_10-14-04PM.log

[shutdown:shutdown] GIShutDownAction action completed on home /u01/app/product/12.1.0/grid successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/product/12.1.0/grid

Start applying binary patches on CRS Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/product/12.1.0/grid successfully
[startup:startup] Executing GIStartupAction action on home /u01/app/product/12.1.0/grid

Performing postpatch operations on CRS Home........

Postpatch operation log file location: /u01/app/product/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_SBY02_2016-08-31_10-21-21PM.log

[startup:startup] GIStartupAction action completed on home /u01/app/product/12.1.0/grid successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/product/12.1.0/grid

Verifying patches applied on CRS Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/product/12.1.0/grid successfully
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:SBY02
CRS Home:/u01/app/product/12.1.0/grid
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/gipsu_jul16/23273629/21436941
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_22-16-28PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054246
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_22-16-28PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054327
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_22-16-28PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054341
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_22-16-28PM_1.log


Source environment for RDBMS HOME and Patch RDBMS_HOME on both nodes
I would recommend to shut down standby database before patching RDBMS_HOME, this is because opatchauto will attempt to load modified SQLs in database by running datapatch utility.

As oracle user stop database:
$ srvctl stop database -d prod

NODE 1
# export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
# export PATH=$ORACLE_HOME/OPatch:$PATH
[root@SBY01 23273629]# opatchauto apply /u01/stage/gipsu_jul16/23273629 -oh /u01/app/oracle/product/12.1.0/dbhome_1

System initialization log file is /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2016-08-31_10-51-22PM.log.

Session log file is /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2016-08-31_10-51-38PM.log
The id for this session is R9CQ
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Executing OPatch prereq operations to verify patch applicability on RAC Home........

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[init:init] Executing GIRACPrereqAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Executing prereq operations before applying on RAC Home........

[init:init] GIRACPrereqAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[init:init] Executing RACDBPrereqAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Executing prereq operations before applying on RAC Home........

[init:init] RACDBPrereqAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[shutdown:prepare-shutdown] Executing RACPrepareShutDownAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Preparing RAC Home to bring down database service........

[shutdown:prepare-shutdown] RACPrepareShutDownAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[shutdown:shutdown] Executing RACShutDownAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Stopping the database service on RAC Home for patching........

[shutdown:shutdown] RACShutDownAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:prepare] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Performing prepatch operation on RAC Home........

[offline:prepare] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Start applying binary patches on RAC Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:finalize] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Performing postpatch operation on RAC Home........

[offline:finalize] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[startup:startup] Executing RACStartupAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Starting the database service on RAC Home........

[startup:startup] RACStartupAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[startup:finalize] Executing RACFinalizeStartAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

No step execution required.........

[startup:finalize] RACFinalizeStartAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[online:product-patching] Executing RACOnlineAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Trying to apply SQL patches on RAC Home.

[WARNING] The database instance 'prod1' from '/u01/app/oracle/product/12.1.0/dbhome_1', in host'SBY01' is not running. SQL changes, if any,  will not be applied.
To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).
Refer to the readme to get the correct steps for applying the sql changes.

[online:product-patching] RACOnlineAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Verifying patches applied on RAC Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:SBY01
RAC Home:/u01/app/oracle/product/12.1.0/dbhome_1
Summary:

==Following patches were SKIPPED:

Patch: /u01/stage/gipsu_jul16/23273629/21436941
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/stage/gipsu_jul16/23273629/23054341
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/gipsu_jul16/23273629/23054246
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_22-52-06PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054327
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_22-52-06PM_1.log

NODE 2
# export ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1
# export PATH=$ORACLE_HOME/OPatch:$PATH
[root@SBY02 23273629]# opatchauto apply /u01/stage/gipsu_jul16/23273629 -oh /u01/app/oracle/product/12.1.0/dbhome_1


System initialization log file is /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2016-08-31_10-56-08PM.log.

Session log file is /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2016-08-31_10-56-15PM.log
The id for this session is YI37
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Executing OPatch prereq operations to verify patch applicability on RAC Home........

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[init:init] Executing GIRACPrereqAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Executing prereq operations before applying on RAC Home........

[init:init] GIRACPrereqAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[init:init] Executing RACDBPrereqAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Executing prereq operations before applying on RAC Home........

[init:init] RACDBPrereqAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[shutdown:prepare-shutdown] Executing RACPrepareShutDownAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Preparing RAC Home to bring down database service........

[shutdown:prepare-shutdown] RACPrepareShutDownAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[shutdown:shutdown] Executing RACShutDownAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Stopping the database service on RAC Home for patching........

[shutdown:shutdown] RACShutDownAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:prepare] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Performing prepatch operation on RAC Home........

[offline:prepare] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Start applying binary patches on RAC Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:finalize] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Performing postpatch operation on RAC Home........

[offline:finalize] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[startup:startup] Executing RACStartupAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Starting the database service on RAC Home........

[startup:startup] RACStartupAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[startup:finalize] Executing RACFinalizeStartAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

No step execution required.........

[startup:finalize] RACFinalizeStartAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[online:product-patching] Executing RACOnlineAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Trying to apply SQL patches on RAC Home.

[WARNING] The database instance 'prod2' from '/u01/app/oracle/product/12.1.0/dbhome_1', in host'SBY02' is not running. SQL changes, if any,  will not be applied.
To apply. the SQL changes, bring up the database instance and run the command manually from any one node (run as oracle).
Refer to the readme to get the correct steps for applying the sql changes.

[online:product-patching] RACOnlineAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Verifying patches applied on RAC Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:SBY02
RAC Home:/u01/app/oracle/product/12.1.0/dbhome_1
Summary:

==Following patches were SKIPPED:

Patch: /u01/stage/gipsu_jul16/23273629/21436941
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/stage/gipsu_jul16/23273629/23054341
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/gipsu_jul16/23273629/23054246
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_22-56-43PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054327
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_22-56-43PM_1.log

You may now start the standby database as oracle user:
$ srvctl start database -d prod -o mount


5. Apply Patch on PRIMARY Nodes
Apply on one node at a time:

NODE 1
# export ORACLE_HOME=/u01/app/product/12.1.0/grid
# export PATH=$ORACLE_HOME/OPatch:$PATH
[root@RAC01 23273629]# opatchauto apply /u01/stage/gipsu_jul16/23273629

System initialization log file is /u01/app/product/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2016-08-31_11-07-14PM.log.

Session log file is /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2016-08-31_11-07-25PM.log
The id for this session is 8UDP
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/product/12.1.0/grid
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Executing OPatch prereq operations to verify patch applicability on CRS Home........


Executing OPatch prereq operations to verify patch applicability on RAC Home........

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/product/12.1.0/grid successfully
[init:init] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[init:init] Executing GIRACPrereqAction action on home /u01/app/oracle/product/12.1.0/dbhome_1
[init:init] Executing GIRACPrereqAction action on home /u01/app/product/12.1.0/grid

Executing prereq operations before applying on CRS Home........


Executing prereq operations before applying on RAC Home........

[init:init] GIRACPrereqAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[init:init] GIRACPrereqAction action completed on home /u01/app/product/12.1.0/grid successfully
[init:init] Executing RACDBPrereqAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Executing prereq operations before applying on RAC Home........

[init:init] RACDBPrereqAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[shutdown:prepare-shutdown] Executing RACPrepareShutDownAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Preparing RAC Home to bring down database service........

[shutdown:prepare-shutdown] RACPrepareShutDownAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[shutdown:shutdown] Executing GIShutDownAction action on home /u01/app/product/12.1.0/grid

Performing prepatch operations on CRS Home........

Prepatch operation log file location: /u01/app/product/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_RAC01_2016-08-31_11-10-11PM.log

[shutdown:shutdown] GIShutDownAction action completed on home /u01/app/product/12.1.0/grid successfully
[offline:prepare] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Performing prepatch operation on RAC Home........

[offline:prepare] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Start applying binary patches on RAC Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:finalize] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Performing postpatch operation on RAC Home........

[offline:finalize] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/product/12.1.0/grid

Start applying binary patches on CRS Home........

[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/product/12.1.0/grid

Start applying binary patches on CRS Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/product/12.1.0/grid successfully
[startup:startup] Executing GIStartupAction action on home /u01/app/product/12.1.0/grid

Performing postpatch operations on CRS Home........

Postpatch operation log file location: /u01/app/product/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_RAC01_2016-08-31_11-26-30PM.log

[startup:startup] GIStartupAction action completed on home /u01/app/product/12.1.0/grid successfully
[startup:finalize] Executing RACFinalizeStartAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

No step execution required.........

[startup:finalize] RACFinalizeStartAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[online:product-patching] Executing RACOnlineAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Trying to apply SQL patches on RAC Home.

[online:product-patching] RACOnlineAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/product/12.1.0/grid

Verifying patches applied on CRS Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/product/12.1.0/grid successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Verifying patches applied on RAC Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:RAC01
RAC Home:/u01/app/oracle/product/12.1.0/dbhome_1
Summary:

==Following patches were SKIPPED:

Patch: /u01/stage/gipsu_jul16/23273629/21436941
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/stage/gipsu_jul16/23273629/23054341
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/gipsu_jul16/23273629/23054246
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-12-54PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054327
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-12-54PM_1.log


Host:RAC01
CRS Home:/u01/app/product/12.1.0/grid
Summary:

==Following patches were SKIPPED:

Patch: /u01/stage/gipsu_jul16/23273629/21436941
Reason: This patch is already been applied, so not going to apply again.

Patch: /u01/stage/gipsu_jul16/23273629/23054246
Reason: This patch is already been applied, so not going to apply again.


==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/gipsu_jul16/23273629/23054327
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-23-59PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054341
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-23-59PM_1.log

NODE 2
[root@RAC02 23273629]# opatchauto apply /u01/stage/gipsu_jul16/23273629

System initialization log file is /u01/app/product/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2016-08-31_11-40-46PM.log.

Session log file is /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2016-08-31_11-40-59PM.log
The id for this session is VYJ3
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/dbhome_1
[init:init] Executing OPatchAutoBinaryAction action on home /u01/app/product/12.1.0/grid

Executing OPatch prereq operations to verify patch applicability on CRS Home........


Executing OPatch prereq operations to verify patch applicability on RAC Home........

[init:init] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[init:init] OPatchAutoBinaryAction action completed on home /u01/app/product/12.1.0/grid successfully
[init:init] Executing GIRACPrereqAction action on home /u01/app/oracle/product/12.1.0/dbhome_1
[init:init] Executing GIRACPrereqAction action on home /u01/app/product/12.1.0/grid

Executing prereq operations before applying on RAC Home........


Executing prereq operations before applying on CRS Home........

[init:init] GIRACPrereqAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[init:init] GIRACPrereqAction action completed on home /u01/app/product/12.1.0/grid successfully
[init:init] Executing RACDBPrereqAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Executing prereq operations before applying on RAC Home........

[init:init] RACDBPrereqAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[shutdown:prepare-shutdown] Executing RACPrepareShutDownAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Preparing RAC Home to bring down database service........

[shutdown:prepare-shutdown] RACPrepareShutDownAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[shutdown:shutdown] Executing GIShutDownAction action on home /u01/app/product/12.1.0/grid

Performing prepatch operations on CRS Home........

Prepatch operation log file location: /u01/app/product/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_RAC02_2016-08-31_11-44-51PM.log

[shutdown:shutdown] GIShutDownAction action completed on home /u01/app/product/12.1.0/grid successfully
[offline:prepare] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Performing prepatch operation on RAC Home........

[offline:prepare] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Start applying binary patches on RAC Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:finalize] Executing RACPatchingAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Performing postpatch operation on RAC Home........

[offline:finalize] RACPatchingAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[offline:binary-patching] Executing OPatchAutoBinaryAction action on home /u01/app/product/12.1.0/grid

Start applying binary patches on CRS Home........

[offline:binary-patching] OPatchAutoBinaryAction action completed on home /u01/app/product/12.1.0/grid successfully
[startup:startup] Executing GIStartupAction action on home /u01/app/product/12.1.0/grid

Performing postpatch operations on CRS Home........

Postpatch operation log file location: /u01/app/product/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_RAC02_2016-08-31_11-51-21PM.log

[startup:startup] GIStartupAction action completed on home /u01/app/product/12.1.0/grid successfully
[startup:finalize] Executing RACFinalizeStartAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

No step execution required.........

[startup:finalize] RACFinalizeStartAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[online:product-patching] Executing RACOnlineAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Trying to apply SQL patches on RAC Home.

[online:product-patching] RACOnlineAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/product/12.1.0/grid

Verifying patches applied on CRS Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/product/12.1.0/grid successfully
[finalize:finalize] Executing OracleHomeLSInventoryGrepAction action on home /u01/app/oracle/product/12.1.0/dbhome_1

Verifying patches applied on RAC Home.

[finalize:finalize] OracleHomeLSInventoryGrepAction action completed on home /u01/app/oracle/product/12.1.0/dbhome_1 successfully
OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:RAC02
RAC Home:/u01/app/oracle/product/12.1.0/dbhome_1
Summary:

==Following patches were SKIPPED:

Patch: /u01/stage/gipsu_jul16/23273629/21436941
Reason: This patch is not applicable to this specified target type - "rac_database"

Patch: /u01/stage/gipsu_jul16/23273629/23054341
Reason: This patch is not applicable to this specified target type - "rac_database"


==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/gipsu_jul16/23273629/23054246
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-45-45PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054327
Log: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-45-45PM_1.log


Host:RAC02
CRS Home:/u01/app/product/12.1.0/grid
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /u01/stage/gipsu_jul16/23273629/21436941
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-47-23PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054246
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-47-23PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054327
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-47-23PM_1.log

Patch: /u01/stage/gipsu_jul16/23273629/23054341
Log: /u01/app/product/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-08-31_23-47-23PM_1.log


6. Start Logshipping on PRIMARY and Managed Recovery Process on STANDBY
Enabled log_archive_dest_n parameter on primary and start MRP on standby.

Verify Patch

PROD
[oracle@RAC01 ~]$ opatch lspatches
23054327;OCW Patch Set Update : 12.1.0.2.160719 (23054327)
23054246;Database Patch Set Update : 12.1.0.2.160719 (23054246)

[grid@RAC01 ~]$ opatch lspatches
23054341;ACFS Patch Set Update : 12.1.0.2.160719 (23054341)
23054327;OCW Patch Set Update : 12.1.0.2.160719 (23054327)
23054246;Database Patch Set Update : 12.1.0.2.160719 (23054246)
21436941;WLM Patch Set Update: 12.1.0.2.5 (21436941)

[oracle@RAC02 ~]$ opatch lspatches
23054327;OCW Patch Set Update : 12.1.0.2.160719 (23054327)
23054246;Database Patch Set Update : 12.1.0.2.160719 (23054246)

[grid@RAC02 ~]$ opatch lspatches
23054341;ACFS Patch Set Update : 12.1.0.2.160719 (23054341)
23054327;OCW Patch Set Update : 12.1.0.2.160719 (23054327)
23054246;Database Patch Set Update : 12.1.0.2.160719 (23054246)
21436941;WLM Patch Set Update: 12.1.0.2.5 (21436941)


STANDBY
[oracle@SBY01 ~]$ opatch lspatches
23054327;OCW Patch Set Update : 12.1.0.2.160719 (23054327)
23054246;Database Patch Set Update : 12.1.0.2.160719 (23054246)

[grid@SBY01 ~]$ opatch lspatches
23054341;ACFS Patch Set Update : 12.1.0.2.160719 (23054341)
23054327;OCW Patch Set Update : 12.1.0.2.160719 (23054327)
23054246;Database Patch Set Update : 12.1.0.2.160719 (23054246)
21436941;WLM Patch Set Update: 12.1.0.2.5 (21436941)


[oracle@SBY02 ~]$ opatch lspatches
23054327;OCW Patch Set Update : 12.1.0.2.160719 (23054327)
23054246;Database Patch Set Update : 12.1.0.2.160719 (23054246)

[grid@SBY02 ~]$ opatch lspatches
23054341;ACFS Patch Set Update : 12.1.0.2.160719 (23054341)
23054327;OCW Patch Set Update : 12.1.0.2.160719 (23054327)
23054246;Database Patch Set Update : 12.1.0.2.160719 (23054246)
21436941;WLM Patch Set Update: 12.1.0.2.5 (21436941)

While logged-on to any of PRIMARY node as oracle user
SQL> col ACTION_TIME for a30
SQL> col DESCRIPTION for a60
SQL> set lines 150
SQL> select PATCH_ID, ACTION_TIME, DESCRIPTION from dba_registry_sqlpatch;

  PATCH_ID ACTION_TIME                    DESCRIPTION
---------- ------------------------------ ------------------------------------------------------------
  23054246 31-AUG-16 11.55.27.704930 PM   Database Patch Set Update : 12.1.0.2.160719 (23054246)


All done!