opatch auto 의 이해

ORACLE_HOME, GRID_HOME 에 최신 OPatch 파일을 풀어 주고..

 

테스트에 사용된 Patchset LInux x86_64 11.2.0.3.7 GI Patch set 이다.

 

패치과정을 살펴보고 어떤식으로 진행되는지 Review 해보도록 한다.

 여기서 혼동하지 말아야할 것은 1번, 2번 서버가 있다고 하면..

opatch auto 를 실행한 서버만 auto 로 Patch 가 적용된다고 하는것이다. 절대 2번 서버도 같이 패치되지 않는다. 

2번도 별도로 진행해 주어야 한다.

순서

1. 1번 node down

2. 패치적용

3. 1번 node up

4. 2번 node down

5. 패치적용

6. 2번 node up

 

[root:/oragrid/product/11.2.0/OPatch]#./opatch auto /oragrid -oh /oragrid/product/11.2.0 -ocmrf /oragrid/product/11.2.0/ocm.rsp
Executing /oragrid/product/11.2.0/perl/bin/perl ./crs/patch11203.pl -patchdir / -patchn oragrid -oh /oragrid/product/11.2.0 -ocmrf /oragrid/product/11.2.0/ocm.rsp -paramfile /oragrid/product/11.2.0/crs/install/crsconfig_params
/oragrid/product/11.2.0/crs/install/crsconfig_params
/oragrid/product/11.2.0/crs/install/s_crsconfig_defs

This is the main log file: /oragrid/product/11.2.0/cfgtoollogs/opatchauto2013-12-04_23-29-49.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /oragrid/product/11.2.0/cfgtoollogs/opatchauto2013-12-04_23-29-49.report.log

2013-12-04 23:29:49: Starting Clusterware Patch Setup
Using configuration parameter file: /oragrid/product/11.2.0/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.rac.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.rac.db' on 'rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /oragrid/product/11.2.0
patch //oragrid/16619898  apply successful for home  /oragrid/product/11.2.0
patch //oragrid/16619892  apply failed  for home  /oragrid/product/11.2.0
CRS-4123: Oracle High Availability Services has been started.

 

패치 Step을 살펴보자..

cat /oragrid/product/11.2.0/cfgtoollogs/opatchauto2013-12-04_23-29-49.report.log

***********  Configuration Data  ***********
* It shows only those targets that will be patched in this session *

crs_home=/oragrid/product/11.2.0      owner=oracle      opatch_ver=11.2.0.3.5

*********** Steps to be executed as owner unless specified as root ***********

1: /oragrid/product/11.2.0/OPatch/opatch prereq CheckComponents -ph //oragrid/16619898 -oh /oragrid/product/11.2.0

2: /oragrid/product/11.2.0/OPatch/opatch prereq CheckComponents -ph //oragrid/16619892 -oh /oragrid/product/11.2.0

3: /oragrid/product/11.2.0/OPatch/opatch prereq CheckConflictAgainstOH -ph //oragrid/16619898 -oh /oragrid/product/11.2.0

4: /oragrid/product/11.2.0/OPatch/opatch prereq CheckConflictAgainstOH -ph //oragrid/16619892 -oh /oragrid/product/11.2.0

5: /oragrid/product/11.2.0/bin/emctl stop dbconsole

6: /oragrid/product/11.2.0/crs/install/rootcrs.pl -unlock : run as root

7: /sbin/fuser -k /oragrid/product/11.2.0/bin/crsctl.bin : run as root

8: /oragrid/product/11.2.0/OPatch/opatch prereq CheckApplicable -ph //oragrid/16619898 -oh /oragrid/product/11.2.0

9: /oragrid/product/11.2.0/OPatch/opatch prereq CheckApplicable -ph //oragrid/16619892 -oh /oragrid/product/11.2.0

10: /oragrid/product/11.2.0/OPatch/opatch napply //oragrid/16619898 -local -silent -ocmrf /oragrid/product/11.2.0/ocm.rsp -oh /oragrid/product/11.2.0 -invPtrLoc /oragrid/product/11.2.0/oraInst.loc

11: /oragrid/product/11.2.0/OPatch/opatch napply //oragrid/16619892 -local -silent -ocmrf /oragrid/product/11.2.0/ocm.rsp -oh /oragrid/product/11.2.0 -invPtrLoc /oragrid/product/11.2.0/oraInst.loc

12: /oragrid/product/11.2.0/bin/emctl start dbconsole

 

자세하게 패치 로그를 살펴보자..

[root:/root]#vi /oragrid/product/11.2.0/cfgtoollogs/opatchauto2013-12-04_23-29-49.log

2013-12-04 23:29:49: Using Oracle CRS home /oragrid/product/11.2.0
2013-12-04 23:29:49: Checking parameters from paramfile /oragrid/product/11.2.0/crs/install/crsconfig_params to validate installer variables
2013-12-04 23:29:49: The configuration parameter file /oragrid/product/11.2.0/crs/install/crsconfig_params is valid
2013-12-04 23:29:49: ### Printing the configuration values from files:
2013-12-04 23:29:49:    /oragrid/product/11.2.0/crs/install/crsconfig_params
2013-12-04 23:29:49:    /oragrid/product/11.2.0/crs/install/s_crsconfig_defs
2013-12-04 23:29:49: ASM_AU_SIZE=1
2013-12-04 23:29:49: ASM_DISCOVERY_STRING=
2013-12-04 23:29:49: ASM_DISKS=/dev/raw/raw1
2013-12-04 23:29:49: ASM_DISK_GROUP=DATA
2013-12-04 23:29:49: ASM_REDUNDANCY=EXTERNAL
2013-12-04 23:29:49: ASM_SPFILE=
2013-12-04 23:29:49: ASM_UPGRADE=false
2013-12-04 23:29:49: CLSCFG_MISSCOUNT=
2013-12-04 23:29:49: CLUSTER_GUID=
2013-12-04 23:29:49: CLUSTER_NAME=rac-cluster
2013-12-04 23:29:49: CRFHOME="/oragrid/product/11.2.0"
2013-12-04 23:29:49: CRS_LIMIT_CORE=unlimited
2013-12-04 23:29:49: CRS_LIMIT_MEMLOCK=unlimited
2013-12-04 23:29:49: CRS_LIMIT_OPENFILE=65536
2013-12-04 23:29:49: CRS_LIMIT_STACK=2048
2013-12-04 23:29:49: CRS_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
2013-12-04 23:29:49: CRS_STORAGE_OPTION=1
2013-12-04 23:29:49: CSS_LEASEDURATION=400
2013-12-04 23:29:49: DIRPREFIX=
2013-12-04 23:29:49: DISABLE_OPROCD=0
2013-12-04 23:29:49: EXTERNAL_ORACLE=/opt/oracle
2013-12-04 23:29:49: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
2013-12-04 23:29:49: GNS_ADDR_LIST=
2013-12-04 23:29:49: GNS_ALLOW_NET_LIST=
2013-12-04 23:29:49: GNS_CONF=false
2013-12-04 23:29:49: GNS_DENY_ITF_LIST=
2013-12-04 23:29:49: GNS_DENY_NET_LIST=
2013-12-04 23:29:49: GNS_DOMAIN_LIST=
2013-12-04 23:29:49: GPNPCONFIGDIR=/oragrid/product/11.2.0
2013-12-04 23:29:49: GPNPGCONFIGDIR=/oragrid/product/11.2.0
2013-12-04 23:29:49: GPNP_PA=
2013-12-04 23:29:49: HOST_NAME_LIST=rac1,rac2
2013-12-04 23:29:49: ID=/etc/init.d
2013-12-04 23:29:49: INIT=/sbin/init
2013-12-04 23:29:49: INITCTL=/sbin/initctl
2013-12-04 23:29:49: ISROLLING=true
2013-12-04 23:29:49: IT=/etc/inittab
2013-12-04 23:29:49: JLIBDIR=/oragrid/product/11.2.0/jlib
2013-12-04 23:29:49: JREDIR=/oragrid/product/11.2.0/jdk/jre/
2013-12-04 23:29:49: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
2013-12-04 23:29:49: MSGFILE=/var/adm/messages
2013-12-04 23:29:49: NETWORKS="eth0"/192.168.56.0:public,"eth1"/10.10.10.0:cluster_interconnect
2013-12-04 23:29:49: NEW_HOST_NAME_LIST=
2013-12-04 23:29:49: NEW_NODEVIPS='rac1-vip/255.255.255.0/eth0,rac2-vip/255.255.255.0/eth0'
2013-12-04 23:29:49: NEW_NODE_NAME_LIST=
2013-12-04 23:29:49: NEW_PRIVATE_NAME_LIST=
2013-12-04 23:29:49: NODELIST=rac1,rac2
2013-12-04 23:29:49: NODE_NAME_LIST=rac1,rac2
2013-12-04 23:29:49: OCFS_CONFIG=
2013-12-04 23:29:49: OCRCONFIG=/etc/oracle/ocr.loc
2013-12-04 23:29:49: OCRCONFIGDIR=/etc/oracle
2013-12-04 23:29:49: OCRID=
2013-12-04 23:29:49: OCRLOC=ocr.loc
2013-12-04 23:29:49: OCR_LOCATIONS=NO_VAL
2013-12-04 23:29:49: OLASTGASPDIR=/etc/oracle/lastgasp
2013-12-04 23:29:49: OLD_CRS_HOME=
2013-12-04 23:29:49: OLRCONFIG=/etc/oracle/olr.loc
2013-12-04 23:29:49: OLRCONFIGDIR=/etc/oracle
2013-12-04 23:29:49: OLRLOC=olr.loc
2013-12-04 23:29:49: OPROCDCHECKDIR=/etc/oracle/oprocd/check
2013-12-04 23:29:49: OPROCDDIR=/etc/oracle/oprocd
2013-12-04 23:29:49: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
2013-12-04 23:29:49: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
2013-12-04 23:29:49: ORACLE_BASE=/oracle
2013-12-04 23:29:49: ORACLE_HOME=/oragrid/product/11.2.0
2013-12-04 23:29:49: ORACLE_OWNER=oracle
2013-12-04 23:29:49: ORA_ASM_GROUP=dba
2013-12-04 23:29:49: ORA_DBA_GROUP=dba
2013-12-04 23:29:49: PRIVATE_NAME_LIST=
2013-12-04 23:29:49: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
2013-12-04 23:29:49: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
2013-12-04 23:29:49: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
2013-12-04 23:29:49: RC_KILL=K15
2013-12-04 23:29:49: RC_KILL_OLD=K96
2013-12-04 23:29:49: RC_KILL_OLD2=K19
2013-12-04 23:29:49: RC_START=S96
2013-12-04 23:29:49: REUSEDG=false
2013-12-04 23:29:49: SCAN_NAME=rac-scan
2013-12-04 23:29:49: SCAN_PORT=1521
2013-12-04 23:29:49: SCRBASE=/etc/oracle/scls_scr
2013-12-04 23:29:49: SILENT=false
2013-12-04 23:29:49: SO_EXT=so
2013-12-04 23:29:49: SRVCFGLOC=srvConfig.loc
2013-12-04 23:29:49: SRVCONFIG=/var/opt/oracle/srvConfig.loc
2013-12-04 23:29:49: SRVCONFIGDIR=/var/opt/oracle
2013-12-04 23:29:49: TZ=Asia/Seoul
2013-12-04 23:29:49: UPSTART_INIT_DIR=/etc/init
2013-12-04 23:29:49: USER_IGNORED_PREREQ=true
2013-12-04 23:29:49: VNDR_CLUSTER=false
2013-12-04 23:29:49: VOTING_DISKS=NO_VAL
2013-12-04 23:29:49: ### Printing other configuration values ###
2013-12-04 23:29:49: CLSCFG_EXTRA_PARMS=
2013-12-04 23:29:49: HAS_GROUP=dba
2013-12-04 23:29:49: HAS_USER=root
2013-12-04 23:29:49: HOST=rac1
2013-12-04 23:29:49: OLR_DIRECTORY=/oragrid/product/11.2.0/cdata
2013-12-04 23:29:49: OLR_LOCATION=/oragrid/product/11.2.0/cdata/rac1.olr
2013-12-04 23:29:49: ORA_CRS_HOME=/oragrid/product/11.2.0
2013-12-04 23:29:49: SUPERUSER=root
2013-12-04 23:29:49: VF_DISCOVERY_STRING=
2013-12-04 23:29:49: crscfg_trace=1
2013-12-04 23:29:49: crscfg_trace_file=/oragrid/product/11.2.0/cfgtoollogs/opatchauto2013-12-04_23-29-49.log
2013-12-04 23:29:49: hosts=
2013-12-04 23:29:49: osdfile=/oragrid/product/11.2.0/crs/install/s_crsconfig_defs
2013-12-04 23:29:49: parameters_valid=1
2013-12-04 23:29:49: paramfile=/oragrid/product/11.2.0/crs/install/crsconfig_params
2013-12-04 23:29:49: platform_family=unix
2013-12-04 23:29:49: srvctl_trc_suff=0
2013-12-04 23:29:49: user_is_superuser=1
2013-12-04 23:29:49: ### Printing of configuration values complete ###
2013-12-04 23:29:49: INC is /oragrid/16619898/files/crs/install /oragrid/product/11.2.0/perl/lib/5.10.0/x86_64-linux-thread-multi /oragrid/product/11.2.0/perl/lib/5.10.0 /oragrid/product/11.2.0/perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /oragrid/product/11.2.0/perl/lib/site_perl/5.10.0 /oragrid/product/11.2.0/perl/lib/5.10.0/x86_64-linux-thread-multi /oragrid/product/11.2.0/perl/lib/5.10.0/x86_64-linux-thread-multi /oragrid/product/11.2.0/perl/lib/5.10.0 /oragrid/product/11.2.0/perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /oragrid/product/11.2.0/perl/lib/site_perl/5.10.0 /oragrid/product/11.2.0/perl/lib/site_perl .

2013-12-04 23:29:49: No -patchfile specified, assuming the patch is already uncompressed
2013-12-04 23:29:49: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:29:49: The patch ids are 16619898 16619892
2013-12-04 23:29:49: The patch ids are 16619898 16619892
2013-12-04 23:29:49: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch query -get_patch_type //oragrid/16619898 -oh /oragrid/product/11.2.0
2013-12-04 23:29:49: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch query -get_patch_type //oragrid/16619898 -oh /oragrid/product/11.2.0 '
2013-12-04 23:29:52: Removing file /tmp/fileV1NRL2
2013-12-04 23:29:52: Successfully removed file: /tmp/fileV1NRL2
2013-12-04 23:29:52: /bin/su successfully executed

2013-12-04 23:29:52: output is  This patch is a "legacy_bundle_top" patch.

2013-12-04 23:29:52: Patch type is "legacy_bundle_top"
2013-12-04 23:29:52: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch query -get_patch_type //oragrid/16619892 -oh /oragrid/product/11.2.0
2013-12-04 23:29:52: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch query -get_patch_type //oragrid/16619892 -oh /oragrid/product/11.2.0 '
2013-12-04 23:29:53: Removing file /tmp/fileKFA5PW
2013-12-04 23:29:53: Successfully removed file: /tmp/fileKFA5PW
2013-12-04 23:29:53: /bin/su exited with rc=75

2013-12-04 23:29:53: output is
2013-12-04 23:29:53: Patch type is
2013-12-04 23:29:53: GI patches are //oragrid/16619898 //oragrid/16619892
2013-12-04 23:29:53: DB patches are //oragrid/16619898/custom/server/16619898 //oragrid/16619892
2013-12-04 23:29:53: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:29:53: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch util saveconfigurationsnapshot -configFile /oragrid/product/11.2.0/cfgtoollogs/opatchauto2013-12-04_23-29-49.cfg.log
2013-12-04 23:29:53: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch util saveconfigurationsnapshot -configFile /oragrid/product/11.2.0/cfgtoollogs/opatchauto2013-12-04_23-29-49.cfg.log '
2013-12-04 23:29:53: Removing file /tmp/fileeYSuiU
2013-12-04 23:29:53: Successfully removed file: /tmp/fileeYSuiU
2013-12-04 23:29:53: /bin/su exited with rc=1

2013-12-04 23:29:53: The Oracle Home /product/11.2.0 is not valid directory. Please give proper Oracle Home.
 OPatch returns with error code = 1

2013-12-04 23:29:53: Opening file /etc/oracle/ocr.loc
2013-12-04 23:29:53: Value (FALSE) is set for key=local_only
2013-12-04 23:29:53: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:29:53: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch version -oh /oragrid/product/11.2.0
2013-12-04 23:29:53: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch version -oh /oragrid/product/11.2.0 '
2013-12-04 23:29:54: Removing file /tmp/file55ZPYR
2013-12-04 23:29:54: Successfully removed file: /tmp/file55ZPYR
2013-12-04 23:29:54: /bin/su successfully executed

2013-12-04 23:29:54: opatch version in oracle home /oragrid/product/11.2.0  is 11.2.0.3.5

2013-12-04 23:29:54: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:29:54: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:29:54: silent mode option is -silent -ocmrf /oragrid/product/11.2.0/ocm.rsp
2013-12-04 23:29:54: Running /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:29:54: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:29:54: Processing oracle home /oragrid/product/11.2.0
2013-12-04 23:29:54: Opening file /etc/oracle/ocr.loc
2013-12-04 23:29:54: Value (FALSE) is set for key=local_only
2013-12-04 23:29:54: Home type of /oragrid/product/11.2.0 is CRS
2013-12-04 23:29:54: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:29:54: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:29:54: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch version -oh /oragrid/product/11.2.0
2013-12-04 23:29:54: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch version -oh /oragrid/product/11.2.0 '
2013-12-04 23:29:54: Removing file /tmp/fileCveUDR
2013-12-04 23:29:54: Successfully removed file: /tmp/fileCveUDR
2013-12-04 23:29:54: /bin/su successfully executed

2013-12-04 23:29:54: opatch version in oracle home /oragrid/product/11.2.0  is 11.2.0.3.5

2013-12-04 23:29:54: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch util checkMinimumOPatchVersion -ph //oragrid/16619898 -version 11.2.0.3.5 -oh /oragrid/product/11.2.0
2013-12-04 23:29:54: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch util checkMinimumOPatchVersion -ph //oragrid/16619898 -version 11.2.0.3.5 -oh /oragrid/product/11.2.0 '
2013-12-04 23:29:57: Removing file /tmp/fileTO5UVS
2013-12-04 23:29:57: Successfully removed file: /tmp/fileTO5UVS
2013-12-04 23:29:57: /bin/su successfully executed

2013-12-04 23:29:57: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch util checkMinimumOPatchVersion -ph //oragrid/16619892 -version 11.2.0.3.5 -oh /oragrid/product/11.2.0
2013-12-04 23:29:57: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch util checkMinimumOPatchVersion -ph //oragrid/16619892 -version 11.2.0.3.5 -oh /oragrid/product/11.2.0 '
2013-12-04 23:30:00: Removing file /tmp/fileGxUG83
2013-12-04 23:30:00: Successfully removed file: /tmp/fileGxUG83
2013-12-04 23:30:00: /bin/su successfully executed

2013-12-04 23:30:00: Status of opatch version check  for /oragrid/product/11.2.0 is 1
2013-12-04 23:30:00: Opatch version check passed for oracle home  /oragrid/product/11.2.0
2013-12-04 23:30:00: Opatch version check passed  for all oracle homes
2013-12-04 23:30:00: Opening file /etc/oracle/ocr.loc
2013-12-04 23:30:00: Value (FALSE) is set for key=local_only
2013-12-04 23:30:00: The cluster nodes are rac1 rac2
2013-12-04 23:30:00: checking if path /oragrid/product/11.2.0/crs/install is shared
2013-12-04 23:30:00: Running as user oracle: /oragrid/product/11.2.0/bin/cluvfy comp ssa -t software -s /oragrid/product/11.2.0/crs/install -n rac1,rac2 -display_status
2013-12-04 23:30:00: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/bin/cluvfy comp ssa -t software -s /oragrid/product/11.2.0/crs/install -n rac1,rac2 -display_status '
2013-12-04 23:30:20: Removing file /tmp/filePBo6vp
2013-12-04 23:30:20: Successfully removed file: /tmp/filePBo6vp
2013-12-04 23:30:20: /bin/su exited with rc=1

2013-12-04 23:30:20: return code for shared check is 256
2013-12-04 23:30:20: output of sharedness check is
 Verifying shared storage accessibility

 Checking shared storage accessibility...

 "/oragrid/product/11.2.0/crs/install" is not shared

 Shared storage check failed on nodes "rac2,rac1"

 Verification of shared storage accessibility was unsuccessful on all the specified nodes.
 NODE_STATUS::rac2:VFAIL
 NODE_STATUS::rac1:VFAIL
 OVERALL_STATUS::VFAIL

2013-12-04 23:30:20: the ishared value is 0
2013-12-04 23:30:20: The oracle home /oragrid/product/11.2.0 is not shared
2013-12-04 23:30:20: Processing oracle home /oragrid/product/11.2.0
2013-12-04 23:30:20: Opening file /etc/oracle/ocr.loc
2013-12-04 23:30:20: Value (FALSE) is set for key=local_only
2013-12-04 23:30:20: Home type of /oragrid/product/11.2.0 is CRS
2013-12-04 23:30:20: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:30:20: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch prereq CheckComponents -ph //oragrid/16619898 -oh /oragrid/product/11.2.0
2013-12-04 23:30:20: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch prereq CheckComponents -ph //oragrid/16619898 -oh /oragrid/product/11.2.0 '
2013-12-04 23:30:23: Removing file /tmp/fileU3icUM
2013-12-04 23:30:23: Successfully removed file: /tmp/fileU3icUM
2013-12-04 23:30:23: /bin/su successfully executed

2013-12-04 23:30:23: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch prereq CheckComponents -ph //oragrid/16619892 -oh /oragrid/product/11.2.0
2013-12-04 23:30:23: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch prereq CheckComponents -ph //oragrid/16619892 -oh /oragrid/product/11.2.0 '
2013-12-04 23:30:27: Removing file /tmp/filelKic0k
2013-12-04 23:30:27: Successfully removed file: /tmp/filelKic0k
2013-12-04 23:30:27: /bin/su successfully executed

2013-12-04 23:30:27: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:30:27: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch prereq CheckConflictAgainstOH -ph //oragrid/16619898 -oh /oragrid/product/11.2.0
2013-12-04 23:30:27: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch prereq CheckConflictAgainstOH -ph //oragrid/16619898 -oh /oragrid/product/11.2.0 '
2013-12-04 23:30:31: Removing file /tmp/filegbypr6
2013-12-04 23:30:31: Successfully removed file: /tmp/filegbypr6
2013-12-04 23:30:31: /bin/su successfully executed

2013-12-04 23:30:31: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch prereq CheckConflictAgainstOH -ph //oragrid/16619892 -oh /oragrid/product/11.2.0
2013-12-04 23:30:31: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch prereq CheckConflictAgainstOH -ph //oragrid/16619892 -oh /oragrid/product/11.2.0 '
2013-12-04 23:30:35: Removing file /tmp/fileSlfK71
2013-12-04 23:30:35: Successfully removed file: /tmp/fileSlfK71
2013-12-04 23:30:35: /bin/su successfully executed

2013-12-04 23:30:35: Status of component/conflict check  for /oragrid/product/11.2.0 is 1
2013-12-04 23:30:35:  Conflict check passes for oracle home  /oragrid/product/11.2.0
2013-12-04 23:30:35: Conflict check passed  for all oracle homes
2013-12-04 23:30:35: Processing oracle home /oragrid/product/11.2.0
2013-12-04 23:30:35: Opening file /etc/oracle/ocr.loc
2013-12-04 23:30:35: Value (FALSE) is set for key=local_only
2013-12-04 23:30:35: Home type of /oragrid/product/11.2.0 is CRS
2013-12-04 23:30:35: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:30:35: Running as user oracle: /oragrid/product/11.2.0/bin/emctl stop dbconsole
2013-12-04 23:30:35: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/bin/emctl stop dbconsole '
2013-12-04 23:30:36: Removing file /tmp/fileyedrRa
2013-12-04 23:30:36: Successfully removed file: /tmp/fileyedrRa
2013-12-04 23:30:36: /bin/su exited with rc=2

2013-12-04 23:30:36: Unlock crshome...
2013-12-04 23:30:36: crshome passed is
2013-12-04 23:30:36: Exclude file used is /oragrid/product/11.2.0/OPatch/crs/installPatch.excl
2013-12-04 23:30:36: Home location in olr.loc is /oragrid/product/11.2.0
2013-12-04 23:30:36: unlock crs home is /oragrid/product/11.2.0
2013-12-04 23:30:36: Executing /oragrid/product/11.2.0/bin/crsctl stop crs -f
2013-12-04 23:30:36: Executing cmd: /oragrid/product/11.2.0/bin/crsctl stop crs -f
2013-12-04 23:32:32: Command output:
>  CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
>  CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
>  CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
>  CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'
>  CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
>  CRS-2673: Attempting to stop 'ora.rac.db' on 'rac1'
>  CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
>  CRS-2677: Stop of 'ora.rac.db' on 'rac1' succeeded
>  CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
>  CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
>  CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
>  CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
>  CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
>  CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
>  CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
>  CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
>  CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
>  CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
>  CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
>  CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
>  CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded
>  CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
>  CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
>  CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
>  CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
>  CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
>  CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
>  CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
>  CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
>  CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
>  CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
>  CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
>  CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
>  CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
>  CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
>  CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
>  CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
>  CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
>  CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
>  CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
>  CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
>  CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
>  CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
>  CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
>  CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
>  CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
>  CRS-4133: Oracle High Availability Services has been stopped.
>End Command output
2013-12-04 23:32:32: /oragrid/product/11.2.0/bin/crsctl stop crs -f
2013-12-04 23:32:32: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:32:34: Command output:
>  CRS-4639: Could not contact Oracle High Availability Services
>  CRS-4000: Command Check failed, or completed with errors.
>End Command output
2013-12-04 23:32:34: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check has
2013-12-04 23:32:35: Command output:
>  CRS-4639: Could not contact Oracle High Availability Services
>End Command output
2013-12-04 23:32:35: The CRS home passed in is /oragrid/product/11.2.0
2013-12-04 23:34:35: Waiting for complete CRS stack to stop
2013-12-04 23:34:35: Invoking removeproc to clean oracle client procs
2013-12-04 23:34:35: Executing cmd: /sbin/fuser -k /oragrid/product/11.2.0/bin/crsctl.bin
2013-12-04 23:34:35: fuser command output for /oragrid/product/11.2.0/bin/crsctl.bin is
2013-12-04 23:34:35: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:34:35: Opening file /etc/oracle/ocr.loc
2013-12-04 23:34:35: Value (FALSE) is set for key=local_only
2013-12-04 23:34:35: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch prereq CheckApplicable -ph //oragrid/16619898 -oh /oragrid/product/11.2.0
2013-12-04 23:34:35: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch prereq CheckApplicable -ph //oragrid/16619898 -oh /oragrid/product/11.2.0 '
2013-12-04 23:34:39: Removing file /tmp/filenBeWwV
2013-12-04 23:34:39: Successfully removed file: /tmp/filenBeWwV
2013-12-04 23:34:39: /bin/su successfully executed

2013-12-04 23:34:39: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch prereq CheckApplicable -ph //oragrid/16619892 -oh /oragrid/product/11.2.0
2013-12-04 23:34:39: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch prereq CheckApplicable -ph //oragrid/16619892 -oh /oragrid/product/11.2.0 '
2013-12-04 23:34:44: Removing file /tmp/file5N7kpS
2013-12-04 23:34:44: Successfully removed file: /tmp/file5N7kpS
2013-12-04 23:34:44: /bin/su successfully executed

2013-12-04 23:34:44: Status of Applicable  check  for /oragrid/product/11.2.0 is 1
2013-12-04 23:34:44: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:34:44: Executing command /oragrid/product/11.2.0/OPatch/opatch napply //oragrid/16619898 -local -silent -ocmrf /oragrid/product/11.2.0/ocm.rsp -oh /oragrid/product/11.2.0 -invPtrLoc /oragrid/product/11.2.0/oraInst.loc as oracle
2013-12-04 23:34:44: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch napply //oragrid/16619898 -local -silent -ocmrf /oragrid/product/11.2.0/ocm.rsp -oh /oragrid/product/11.2.0 -invPtrLoc /oragrid/product/11.2.0/oraInst.loc
2013-12-04 23:34:44: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch napply //oragrid/16619898 -local -silent -ocmrf /oragrid/product/11.2.0/ocm.rsp -oh /oragrid/product/11.2.0 -invPtrLoc /oragrid/product/11.2.0/oraInst.loc '
2013-12-04 23:41:02: Removing file /tmp/filesPciS4
2013-12-04 23:41:02: Successfully removed file: /tmp/filesPciS4
2013-12-04 23:41:02: /bin/su successfully executed

2013-12-04 23:41:02: status of apply patch is 0
2013-12-04 23:41:02: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.5
 Copyright (c) 2013, Oracle Corporation.  All rights reserved.

 Oracle Home       : /oragrid/product/11.2.0
 Central Inventory : /oragrid/oraInventory
    from           : /oragrid/product/11.2.0/oraInst.loc
 OPatch version    : 11.2.0.3.5
 OUI version       : 11.2.0.3.0
 Log file location : /oragrid/product/11.2.0/cfgtoollogs/opatch/opatch2013-12-04_23-34-44PM_1.log

 Verifying environment and performing prerequisite checks...
 OPatch continues with these patches:   16619898

 Do you want to proceed? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 All checks passed.

 Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
 (Oracle Home = '/oragrid/product/11.2.0')

 Is the local system ready for patching? [y|n]
 Y (auto-answered by -silent)
 User Responded with: Y
 Backing up files...
 Applying interim patch '16619898' to OH '/oragrid/product/11.2.0'

 Patching component oracle.crs, 11.2.0.3.0...

 Patching component oracle.usm, 11.2.0.3.0...

 Verifying the update...
 Patch 16619898 successfully applied.
 Log file location: /oragrid/product/11.2.0/cfgtoollogs/opatch/opatch2013-12-04_23-34-44PM_1.log

 OPatch succeeded.

2013-12-04 23:41:02: patch //oragrid/16619898  apply successful for home  /oragrid/product/11.2.0
2013-12-04 23:41:02: Executing command /oragrid/product/11.2.0/OPatch/opatch napply //oragrid/16619892 -local -silent -ocmrf /oragrid/product/11.2.0/ocm.rsp -oh /oragrid/product/11.2.0 -invPtrLoc /oragrid/product/11.2.0/oraInst.loc as oracle
2013-12-04 23:41:02: Running as user oracle: /oragrid/product/11.2.0/OPatch/opatch napply //oragrid/16619892 -local -silent -ocmrf /oragrid/product/11.2.0/ocm.rsp -oh /oragrid/product/11.2.0 -invPtrLoc /oragrid/product/11.2.0/oraInst.loc
2013-12-04 23:41:02: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/OPatch/opatch napply //oragrid/16619892 -local -silent -ocmrf /oragrid/product/11.2.0/ocm.rsp -oh /oragrid/product/11.2.0 -invPtrLoc /oragrid/product/11.2.0/oraInst.loc '
2013-12-04 23:41:13: Removing file /tmp/filek536M4
2013-12-04 23:41:13: Successfully removed file: /tmp/filek536M4
2013-12-04 23:41:13: /bin/su exited with rc=73

2013-12-04 23:41:13: status of apply patch is 18688
2013-12-04 23:41:13: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.5
 Copyright (c) 2013, Oracle Corporation.  All rights reserved.

 Oracle Home       : /oragrid/product/11.2.0
 Central Inventory : /oragrid/oraInventory
    from           : /oragrid/product/11.2.0/oraInst.loc
 OPatch version    : 11.2.0.3.5
 OUI version       : 11.2.0.3.0
 Log file location : /oragrid/product/11.2.0/cfgtoollogs/opatch/opatch2013-12-04_23-41-03PM_1.log

 Verifying environment and performing prerequisite checks...
 Prerequisite check "CheckActiveFilesAndExecutables" failed.
 The details are:

 Following executables are active :
 /oragrid/product/11.2.0/lib/libclntsh.so.11.1
 UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
 Log file location: /oragrid/product/11.2.0/cfgtoollogs/opatch/opatch2013-12-04_23-41-03PM_1.log

 OPatch failed with error code 73

2013-12-04 23:41:13: patch //oragrid/16619892  apply failed  for home  /oragrid/product/11.2.0
2013-12-04 23:41:13: Performing Post patch actions
2013-12-04 23:41:13: norestart flag is set to
2013-12-04 23:41:13: Opening file /etc/oracle/ocr.loc
2013-12-04 23:41:13: Value (FALSE) is set for key=local_only
2013-12-04 23:41:13: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:41:13: Oracle user for /oragrid/product/11.2.0 is oracle
2013-12-04 23:41:13: Running as user oracle: /oragrid/product/11.2.0/bin/emctl start dbconsole
2013-12-04 23:41:13: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/bin/emctl start dbconsole '
2013-12-04 23:41:14: Removing file /tmp/filedZAZaE
2013-12-04 23:41:14: Successfully removed file: /tmp/filedZAZaE
2013-12-04 23:41:14: /bin/su exited with rc=2

2013-12-04 23:41:14: Performing Post patch actions for Grid Home /oragrid/product/11.2.0
2013-12-04 23:41:14: Executing cmd: /oragrid/product/11.2.0/rdbms/install/rootadd_rdbms.sh
2013-12-04 23:41:14: setrdbmsfileperms succeeded
2013-12-04 23:41:14: Patching Oracle Clusterware
2013-12-04 23:41:14: norestart flag is set to 0
2013-12-04 23:41:15: Executing cmd: /bin/rpm -q sles-release
2013-12-04 23:41:15: Command output:
>  package sles-release is not installed
>End Command output
2013-12-04 23:41:15: init file = /oragrid/product/11.2.0/crs/init/init.ohasd
2013-12-04 23:41:15: Copying file /oragrid/product/11.2.0/crs/init/init.ohasd to /etc/init.d directory
2013-12-04 23:41:15: Setting init.ohasd permission in /etc/init.d directory
2013-12-04 23:41:15: init file = /oragrid/product/11.2.0/crs/init/ohasd
2013-12-04 23:41:15: Copying file /oragrid/product/11.2.0/crs/init/ohasd to /etc/init.d directory
2013-12-04 23:41:15: Setting ohasd permission in /etc/init.d directory
2013-12-04 23:41:15: Executing cmd: /bin/rpm -q sles-release
2013-12-04 23:41:15: Command output:
>  package sles-release is not installed
>End Command output
2013-12-04 23:41:15: Removing "/etc/rc.d/rc3.d/S96ohasd"
2013-12-04 23:41:15: Removing file /etc/rc.d/rc3.d/S96ohasd
2013-12-04 23:41:15: Successfully removed file: /etc/rc.d/rc3.d/S96ohasd
2013-12-04 23:41:15: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
2013-12-04 23:41:15: Removing "/etc/rc.d/rc5.d/S96ohasd"
2013-12-04 23:41:15: Removing file /etc/rc.d/rc5.d/S96ohasd
2013-12-04 23:41:15: Successfully removed file: /etc/rc.d/rc5.d/S96ohasd
2013-12-04 23:41:15: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
2013-12-04 23:41:15: Removing "/etc/rc.d/rc0.d/K15ohasd"
2013-12-04 23:41:15: Removing file /etc/rc.d/rc0.d/K15ohasd
2013-12-04 23:41:15: Successfully removed file: /etc/rc.d/rc0.d/K15ohasd
2013-12-04 23:41:15: Creating a link "/etc/rc.d/rc0.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-12-04 23:41:15: Removing "/etc/rc.d/rc1.d/K15ohasd"
2013-12-04 23:41:15: Removing file /etc/rc.d/rc1.d/K15ohasd
2013-12-04 23:41:15: Successfully removed file: /etc/rc.d/rc1.d/K15ohasd
2013-12-04 23:41:15: Creating a link "/etc/rc.d/rc1.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-12-04 23:41:15: Removing "/etc/rc.d/rc2.d/K15ohasd"
2013-12-04 23:41:15: Removing file /etc/rc.d/rc2.d/K15ohasd
2013-12-04 23:41:15: Successfully removed file: /etc/rc.d/rc2.d/K15ohasd
2013-12-04 23:41:15: Creating a link "/etc/rc.d/rc2.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-12-04 23:41:15: Removing "/etc/rc.d/rc3.d/K15ohasd"
2013-12-04 23:41:15: Removing file /etc/rc.d/rc3.d/K15ohasd
2013-12-04 23:41:15: Successfully removed file: /etc/rc.d/rc3.d/K15ohasd
2013-12-04 23:41:15: Creating a link "/etc/rc.d/rc3.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-12-04 23:41:15: Removing "/etc/rc.d/rc4.d/K15ohasd"
2013-12-04 23:41:15: Removing file /etc/rc.d/rc4.d/K15ohasd
2013-12-04 23:41:15: Successfully removed file: /etc/rc.d/rc4.d/K15ohasd
2013-12-04 23:41:15: Creating a link "/etc/rc.d/rc4.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-12-04 23:41:15: Removing "/etc/rc.d/rc6.d/K15ohasd"
2013-12-04 23:41:15: Removing file /etc/rc.d/rc6.d/K15ohasd
2013-12-04 23:41:15: Successfully removed file: /etc/rc.d/rc6.d/K15ohasd
2013-12-04 23:41:15: Creating a link "/etc/rc.d/rc6.d/K15ohasd" pointing to /etc/init.d/ohasd
2013-12-04 23:41:15: The file ohasd has been successfully linked to the RC directories
2013-12-04 23:41:15: Executing cmd: /oragrid/product/11.2.0/bin/acfsdriverstate supported
2013-12-04 23:41:16: Command output:
>  ACFS-9200: Supported
>End Command output
2013-12-04 23:41:16: acfs is supported
2013-12-04 23:41:16: Executing '/oragrid/product/11.2.0/bin/acfsroot install'
2013-12-04 23:41:16: Executing cmd: /oragrid/product/11.2.0/bin/acfsroot install
2013-12-04 23:41:27: Command output:
>  ACFS-9300: ADVM/ACFS distribution files found.
>  ACFS-9312: Existing ADVM/ACFS installation detected.
>  ACFS-9314: Removing previous ADVM/ACFS installation.
>  ACFS-9315: Previous ADVM/ACFS components successfully removed.
>  ACFS-9307: Installing requested ADVM/ACFS software.
>  ACFS-9308: Loading installed ADVM/ACFS drivers.
>  ACFS-9321: Creating udev for ADVM/ACFS.
>  ACFS-9323: Creating module dependencies - this may take some time.
>  ACFS-9154: Loading 'oracleoks.ko' driver.
>  ACFS-9154: Loading 'oracleadvm.ko' driver.
>  ACFS-9154: Loading 'oracleacfs.ko' driver.
>  ACFS-9327: Verifying ADVM/ACFS devices.
>  ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
>  ACFS-9156: Detecting control device '/dev/ofsctl'.
>  ACFS-9309: ADVM/ACFS installation correctness verified.
>End Command output
2013-12-04 23:41:27: /oragrid/product/11.2.0/bin/acfsroot install ... success
2013-12-04 23:41:27: USM driver install status is 1
2013-12-04 23:41:27: ACFS drivers installation completed
2013-12-04 23:41:27: Starting Oracle Clusterware
2013-12-04 23:41:27: Executing /oragrid/product/11.2.0/bin/crsctl start crs
2013-12-04 23:41:27: Executing cmd: /oragrid/product/11.2.0/bin/crsctl start crs
2013-12-04 23:41:41: Command output:
>  CRS-4123: Oracle High Availability Services has been started.
>End Command output
2013-12-04 23:41:41: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check crs
2013-12-04 23:41:42: Command output:
>  CRS-4638: Oracle High Availability Services is online
>  CRS-4535: Cannot communicate with Cluster Ready Services
>  CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
>  CRS-4534: Cannot communicate with Event Manager
>End Command output
2013-12-04 23:41:42: Running /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:41:42: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:41:42: Checking the status of cluster
2013-12-04 23:41:47: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:41:47: Checking the status of cluster
2013-12-04 23:41:52: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:41:52: Checking the status of cluster
2013-12-04 23:41:57: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:41:57: Checking the status of cluster
2013-12-04 23:42:02: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:02: Checking the status of cluster
2013-12-04 23:42:07: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:07: Checking the status of cluster
2013-12-04 23:42:12: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:12: Checking the status of cluster
2013-12-04 23:42:17: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:17: Checking the status of cluster
2013-12-04 23:42:22: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:22: Checking the status of cluster
2013-12-04 23:42:27: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:27: Checking the status of cluster
2013-12-04 23:42:32: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:32: Checking the status of cluster
2013-12-04 23:42:37: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:38: Checking the status of cluster
2013-12-04 23:42:43: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:43: Checking the status of cluster
2013-12-04 23:42:48: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:48: Checking the status of cluster
2013-12-04 23:42:53: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:53: Checking the status of cluster
2013-12-04 23:42:58: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:42:58: Checking the status of cluster
2013-12-04 23:43:03: Executing cmd: /oragrid/product/11.2.0/bin/crsctl check cluster -n rac1
2013-12-04 23:43:03: Oracle CRS stack installed and running
2013-12-04 23:43:03: Running as user oracle: /oragrid/product/11.2.0/bin/crsctl status resource ora.oc4j
2013-12-04 23:43:03: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/bin/crsctl status resource ora.oc4j '
2013-12-04 23:43:03: Removing file /tmp/fileSPHMTZ
2013-12-04 23:43:03: Successfully removed file: /tmp/fileSPHMTZ
2013-12-04 23:43:03: /bin/su successfully executed

2013-12-04 23:43:03: Running as user oracle: /oragrid/product/11.2.0/bin/srvctl stop oc4j
2013-12-04 23:43:03: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/bin/srvctl stop oc4j '
2013-12-04 23:43:21: Removing file /tmp/fileRPEcmm
2013-12-04 23:43:21: Successfully removed file: /tmp/fileRPEcmm
2013-12-04 23:43:21: /bin/su successfully executed

2013-12-04 23:43:21: Running as user oracle: /oragrid/product/11.2.0/bin/crsctl get jazn /oragrid/product/11.2.0/oc4j/j2ee/home/OC4J_DBWLM_config/system-jazn-data.xml -f
2013-12-04 23:43:21: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/bin/crsctl get jazn /oragrid/product/11.2.0/oc4j/j2ee/home/OC4J_DBWLM_config/system-jazn-data.xml -f '
2013-12-04 23:43:21: Removing file /tmp/filev0qBjC
2013-12-04 23:43:21: Successfully removed file: /tmp/filev0qBjC
2013-12-04 23:43:21: /bin/su successfully executed

2013-12-04 23:43:21: Running as user oracle: /oragrid/product/11.2.0/bin/qosctl -autogenerate
2013-12-04 23:43:21: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/bin/qosctl -autogenerate '
2013-12-04 23:43:24: Removing file /tmp/fileDijqgT
2013-12-04 23:43:24: Successfully removed file: /tmp/fileDijqgT
2013-12-04 23:43:24: /bin/su successfully executed

2013-12-04 23:43:24: Running as user oracle: /oragrid/product/11.2.0/bin/crsctl set jazn /oragrid/product/11.2.0/oc4j/j2ee/home/OC4J_DBWLM_config/system-jazn-data.xml -f
2013-12-04 23:43:24: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/bin/crsctl set jazn /oragrid/product/11.2.0/oc4j/j2ee/home/OC4J_DBWLM_config/system-jazn-data.xml -f '
2013-12-04 23:43:24: Removing file /tmp/fileqkXrxk
2013-12-04 23:43:24: Successfully removed file: /tmp/fileqkXrxk
2013-12-04 23:43:24: /bin/su successfully executed

2013-12-04 23:43:24: Running as user oracle: /oragrid/product/11.2.0/bin/srvctl start oc4j
2013-12-04 23:43:24: s_run_as_user2: Running /bin/su oracle -c ' /oragrid/product/11.2.0/bin/srvctl start oc4j '
2013-12-04 23:43:56: Removing file /tmp/filehsRb5L
2013-12-04 23:43:56: Successfully removed file: /tmp/filehsRb5L
2013-12-04 23:43:56: /bin/su successfully executed

2013-12-04 23:43:56: QoS users were patched correctly.