Exadata Cell Upgrade 과정

Exadata Cell Upgrade 하기 

11.2.3.3.0.131014.1 -> 11.2.3.3.1.140708

Cell Upgrade 는 DB node 에서 진행한다.

[EXA1]root@exa1:/root# dcli -g cell_group -l root /opt/oracle.cellos/ipconf -verify

gtceladm01: Verifying of Exadata configuration file /opt/oracle.cellos/cell.conf

gtceladm01: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checks

gtceladm02: Verifying of Exadata configuration file /opt/oracle.cellos/cell.conf

gtceladm02: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checks

gtceladm03: Verifying of Exadata configuration file /opt/oracle.cellos/cell.conf

gtceladm03: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checks

압축해제

[EXA1]root@exa1:/root# unzip p19166601_112331_Linux-x86-64.zip

Archive:  p19166601_112331_Linux-x86-64.zip

   creating: patch_11.2.3.3.1.140708/

  inflating: patch_11.2.3.3.1.140708/sundcs_36p_repository_2.1.3_4.pkg  

  inflating: patch_11.2.3.3.1.140708/upgradeIBSwitch.sh  

   creating: patch_11.2.3.3.1.140708/linux.db.rpms/

  inflating: patch_11.2.3.3.1.140708/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm  

   creating: patch_11.2.3.3.1.140708/plugins/

  inflating: patch_11.2.3.3.1.140708/plugins/010-check_17854520.sh  

  inflating: patch_11.2.3.3.1.140708/plugins/000-check_dummy_bash  

  inflating: patch_11.2.3.3.1.140708/plugins/000-check_dummy_perl  

  inflating: patch_11.2.3.3.1.140708/11.2.3.3.1.140708.iso  

....................

[EXA1]root@exa1:/root# crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'exa1'

CRS-2673: Attempting to stop 'ora.crsd' on 'exa1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'exa1'

CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exa1'

CRS-2673: Attempting to stop 'ora.exa.db' on 'exa1'

CRS-2673: Attempting to stop 'dbfs_mount' on 'exa1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'exa1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'exa1'

CRS-2677: Stop of 'dbfs_mount' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.fsdb.db' on 'exa1'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on 'exa1'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.exa1.vip' on 'exa1'

CRS-2677: Stop of 'ora.registry.acfs' on 'exa1' succeeded

CRS-2677: Stop of 'ora.fsdb.db' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.DBFS.dg' on 'exa1'

CRS-2677: Stop of 'ora.exa.db' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg' on 'exa1'

CRS-2673: Attempting to stop 'ora.RECO.dg' on 'exa1'

CRS-2677: Stop of 'ora.exa1.vip' on 'exa1' succeeded

CRS-2672: Attempting to start 'ora.exa1.vip' on 'exa2'

CRS-2677: Stop of 'ora.RECO.dg' on 'exa1' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'exa1' succeeded

CRS-2677: Stop of 'ora.scan1.vip' on 'exa1' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on 'exa2'

CRS-2676: Start of 'ora.exa1.vip' on 'exa2' succeeded

CRS-2676: Start of 'ora.scan1.vip' on 'exa2' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'exa2'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'exa2' succeeded

CRS-2677: Stop of 'ora.DBFS.dg' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on 'exa1'

CRS-2677: Stop of 'ora.asm' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.ons' on 'exa1'

CRS-2677: Stop of 'ora.ons' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on 'exa1'

CRS-2677: Stop of 'ora.net1.network' on 'exa1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'exa1' has completed

CRS-2677: Stop of 'ora.crsd' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.mdnsd' on 'exa1'

CRS-2673: Attempting to stop 'ora.ctssd' on 'exa1'

CRS-2673: Attempting to stop 'ora.evmd' on 'exa1'

CRS-2673: Attempting to stop 'ora.asm' on 'exa1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'exa1'

CRS-2677: Stop of 'ora.ctssd' on 'exa1' succeeded

CRS-2677: Stop of 'ora.evmd' on 'exa1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'exa1' succeeded

CRS-2677: Stop of 'ora.asm' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'exa1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'exa1'

CRS-2677: Stop of 'ora.drivers.acfs' on 'exa1' succeeded

CRS-2677: Stop of 'ora.cssd' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.diskmon' on 'exa1'

CRS-2673: Attempting to stop 'ora.crf' on 'exa1'

CRS-2677: Stop of 'ora.crf' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'exa1'

CRS-2677: Stop of 'ora.diskmon' on 'exa1' succeeded

CRS-2677: Stop of 'ora.gipcd' on 'exa1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'exa1'

CRS-2677: Stop of 'ora.gpnpd' on 'exa1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'exa1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[EXA1]root@exa1:/root# dcli -g /root/cell_group -l root 'service celld stop'

gtceladm01: 

gtceladm01: Stopping the RS, CELLSRV, and MS services...

gtceladm01: The SHUTDOWN of services was successful.

gtceladm02: 

gtceladm02: Stopping the RS, CELLSRV, and MS services...

gtceladm02: The SHUTDOWN of services was successful.

gtceladm03: 

gtceladm03: Stopping the RS, CELLSRV, and MS services...

gtceladm03: The SHUTDOWN of services was successful.

[EXA1]root@exa1:/root# dcli -g /root/cell_group -l root 'service celld status'

gtceladm01: rsStatus:            stopped

gtceladm01: msStatus:            unknown

gtceladm01: cellsrvStatus:       unknown

gtceladm02: rsStatus:            stopped

gtceladm02: msStatus:            unknown

gtceladm02: cellsrvStatus:       unknown

gtceladm03: rsStatus:            stopped

gtceladm03: msStatus:            unknown

gtceladm03: cellsrvStatus:       unknown

[EXA1]root@exa1:/root# 

[EXA1]root@exa1:/root#  dcli -g /root/cell_group -l root 'sync;sync;sync'

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708#  ./patchmgr -cells /root/cell_group -reset_force

2015-01-20 13:36:41 +0900 :DONE: reset_force

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708#  ./patchmgr -cells /root/cell_group -cleanup

2015-01-20 13:37:54 +0900        :Working: DO: Cleanup ...

2015-01-20 13:37:55 +0900        :SUCCESS: DONE: Cleanup

-ignore_alerts 옵션 없이

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708#  ./patchmgr -cells /root/cell_group -patch_check_prereq

2015-01-20 13:39:03 +0900        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...

FAILED for following cells

gtceladm01:  gtceladm01 172.16.40.101 2015-01-20 13:39:38 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.

gtceladm02:  gtceladm02 172.16.40.102 2015-01-20 13:20:53 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.

gtceladm03:  gtceladm03 172.16.40.103 2015-01-20 13:21:40 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.

2015-01-20 13:39:36 +0900        :FAILED: Details in files <cell_name>.log /root/patch_11.2.3.3.1.140708/patchmgr.stdout, /root/patch_11.2.3.3.1.140708/patchmgr.stderr

2015-01-20 13:39:36 +0900        :FAILED: DONE: Initialize files, check space and state of cell services.

[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.

-ignore_alerts 옵션 주고

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708#  ./patchmgr -cells /root/cell_group -patch_check_prereq -ignore_alerts

2015-01-20 13:46:10 +0900        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...

2015-01-20 13:46:27 +0900        :SUCCESS: DONE: Initialize files, check space and state of cell services.

2015-01-20 13:46:27 +0900        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...

2015-01-20 13:46:41 +0900 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.

                                          

2015-01-20 13:46:42 +0900        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.

2015-01-20 13:46:42 +0900        :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...

2015-01-20 13:47:28 +0900        :SUCCESS: DONE: Check prerequisites on all cells.

2015-01-20 13:47:28 +0900        :Working: DO: Execute plugin check for Patch Check Prereq ...

2015-01-20 13:47:28 +0900 :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.1. Details in logfile /root/patch_11.2.3.3.1.140708/patchmgr.stdout.

2015-01-20 13:47:28 +0900 :INFO: This plugin checks dbhomes across all nodes with oracle-user ssh equivalence, but only for those known to the local system. dbhomes that exist only on remote nodes must be checked manually.

2015-01-20 13:47:28 +0900 :SUCCESS: No exposure to bug 17854520 with non-rolling patching

2015-01-20 13:47:28 +0900        :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708#  ./patchmgr -cells /root/cell_group -patch

NOTE Cells will reboot during the patch or rollback process.

NOTE For non-rolling patch or rollback, ensure all ASM instances using

NOTE the cells are shut down for the duration of the patch or rollback.

NOTE For rolling patch or rollback, ensure all ASM instances using

NOTE the cells are up for the duration of the patch or rollback.

WARNING Do not start more than one instance of patchmgr.

WARNING Do not interrupt the patchmgr session.

WARNING Do not alter state of ASM instances during patch or rollback.

WARNING Do not resize the screen. It may disturb the screen layout.

WARNING Do not reboot cells or alter cell services during patch or rollback.

WARNING Do not open log files in editor in write mode or try to alter them.

NOTE All time estimates are approximate. Timestamps on the left are real.

NOTE You may interrupt this patchmgr run in next 60 seconds with control-c.

2015-01-20 13:51:22 +0900        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...

FAILED for following cells

gtceladm01:  gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.

gtceladm02:  gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.

gtceladm03:  gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.

2015-01-20 13:51:30 +0900        :FAILED: Details in files <cell_name>.log /root/patch_11.2.3.3.1.140708/patchmgr.stdout, /root/patch_11.2.3.3.1.140708/patchmgr.stderr

2015-01-20 13:51:30 +0900        :FAILED: DONE: Initialize files, check space and state of cell services.

[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# 

================PatchMgr run started 2015-01-20 13:50:15 +0900 ===========

With arguments: -cells /root/cell_group -patch

2015-01-20 13:51:22 +0900        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...

2015-01-20 13:51:29 +0900 ++++++++++++++++++ Logs so far begin ++++++++++

gtceladm01: 

gtceladm01: 2015-01-20 13:51:26 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.

gtceladm01: 2015-01-20 13:51:26 +0900 [INFO]: dostep called: prechk:no:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce

gtceladm01: _EXIT_ERROR_Cell gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.

gtceladm02: 

gtceladm02: 2015-01-20 13:32:41 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.

gtceladm02: 2015-01-20 13:32:41 +0900 [INFO]: dostep called: prechk:no:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce

gtceladm02: _EXIT_ERROR_Cell gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.

gtceladm03: 

gtceladm03: 2015-01-20 13:33:28 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.

gtceladm03: 2015-01-20 13:33:28 +0900 [INFO]: dostep called: prechk:no:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce

gtceladm03: _EXIT_ERROR_Cell gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.

gtceladm01: 

gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101

gtceladm02: 

gtceladm02: _EXIT_PASS_Cell gtceladm02 172.16.40.102

gtceladm03: 

gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103

gtceladm01: 

gtceladm01: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb

gtceladm01: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek

gtceladm01: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek

gtceladm01: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb

gtceladm01: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb

gtceladm01: [INFO] Required free space on /boot is 46Mb

gtceladm01: [ERROR] Detected open hardware alert(s) on the Exadata cell

gtceladm01: [ERROR] 3_1  2014-03-08T04:00:01+09:00       critical        "A power supply component is suspected of causing a fault.  Component Name : /SYS/PS0  Fault class    : fault.chassis.env.power.loss  Fault message  : http://www.sun.com/msg/SPX86-8000-55"

gtceladm01: [ERROR] Check for open hardware alerts on Exadata cell by running "cellcli -e list alerthistory where endTime=null and alertShortName=Hardware and alertType=stateful and severity=critical"

gtceladm01: [ERROR] Specific alerts can be dropped using "cellcli -e drop alerthistory X_Y" where X_Y is the alert ID

gtceladm01: Cell gtceladm01 172.16.40.101

gtceladm01: _EXIT_ERROR_Cell gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.

gtceladm02: 

gtceladm02: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb

gtceladm02: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek

gtceladm02: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek

gtceladm02: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb

gtceladm02: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb

gtceladm02: [INFO] Required free space on /boot is 46Mb

gtceladm02: [ERROR] Detected open hardware alert(s) on the Exadata cell

gtceladm02: [ERROR] 3_1  2014-03-10T10:12:43+09:00       critical        "A power supply component is suspected of causing a fault.  Component Name : /SYS/PS0  Fault class    : fault.chassis.env.power.loss  Fault message  : http://www.sun.com/msg/SPX86-8000-55"

gtceladm02: [ERROR] Check for open hardware alerts on Exadata cell by running "cellcli -e list alerthistory where endTime=null and alertShortName=Hardware and alertType=stateful and severity=critical"

gtceladm02: [ERROR] Specific alerts can be dropped using "cellcli -e drop alerthistory X_Y" where X_Y is the alert ID

gtceladm02: Cell gtceladm02 172.16.40.102

gtceladm02: _EXIT_ERROR_Cell gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.

gtceladm03: 

gtceladm03: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb

gtceladm03: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek

gtceladm03: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek

gtceladm03: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb

gtceladm03: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb

gtceladm03: [INFO] Required free space on /boot is 46Mb

gtceladm03: [ERROR] Detected open hardware alert(s) on the Exadata cell

gtceladm03: [ERROR] 3_1  2014-03-10T11:50:46+09:00       critical        "A power supply component is suspected of causing a fault.  Component Name : /SYS/PS0  Fault class    : fault.chassis.env.power.loss  Fault message  : http://www.sun.com/msg/SPX86-8000-55"

gtceladm03: [ERROR] Check for open hardware alerts on Exadata cell by running "cellcli -e list alerthistory where endTime=null and alertShortName=Hardware and alertType=stateful and severity=critical"

gtceladm03: [ERROR] Specific alerts can be dropped using "cellcli -e drop alerthistory X_Y" where X_Y is the alert ID

gtceladm03: Cell gtceladm03 172.16.40.103

gtceladm03: _EXIT_ERROR_Cell gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.

2015-01-20 13:51:30 +0900 ++++++++++++++++++ Logs so far end ++++++++++

FAILED for following cells

gtceladm01:  gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.

gtceladm02:  gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.

gtceladm03:  gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.

2015-01-20 13:51:30 +0900        :FAILED: Details in files <cell_name>.log /root/patch_11.2.3.3.1.140708/patchmgr.stdout, /root/patch_11.2.3.3.1.140708/patchmgr.stderr

2015-01-20 13:51:30 +0900        :FAILED: DONE: Initialize files, check space and state of cell services.

[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.

================PatchMgr run ended 2015-01-20 13:51:30 +0900 ===========

gtceladm01:  gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.

gtceladm02:  gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.

gtceladm03:  gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.

2015-01-20 13:51:30 +0900        :FAILED: Details in files <cell_name>.log /root/patch_11.2.3.3.1.140708/patchmgr.stdout, /root/patch_11.2.3.3.1.140708/patchmgr.stderr

2015-01-20 13:51:30 +0900        :FAILED: DONE: Initialize files, check space and state of cell services.

[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# 

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# 

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# 

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -patch -ignore_alerts

NOTE Cells will reboot during the patch or rollback process.

NOTE For non-rolling patch or rollback, ensure all ASM instances using

NOTE the cells are shut down for the duration of the patch or rollback.

NOTE For rolling patch or rollback, ensure all ASM instances using

NOTE the cells are up for the duration of the patch or rollback.

WARNING Do not start more than one instance of patchmgr.

WARNING Do not interrupt the patchmgr session.

WARNING Do not alter state of ASM instances during patch or rollback.

WARNING Do not resize the screen. It may disturb the screen layout.

WARNING Do not reboot cells or alter cell services during patch or rollback.

WARNING Do not open log files in editor in write mode or try to alter them.

NOTE All time estimates are approximate. Timestamps on the left are real.

NOTE You may interrupt this patchmgr run in next 60 seconds with control-c.

2015-01-20 13:55:14 +0900        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...

2015-01-20 13:55:31 +0900        :SUCCESS: DONE: Initialize files, check space and state of cell services.

2015-01-20 13:55:31 +0900        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...

2015-01-20 13:55:44 +0900 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.

                                          

2015-01-20 13:55:45 +0900        :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.

2015-01-20 13:55:45 +0900        :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...

2015-01-20 13:56:31 +0900        :SUCCESS: DONE: Check prerequisites on all cells.

2015-01-20 13:56:31 +0900        :Working: DO: Copy the patch to all cells. Up to 3 minutes ...

2015-01-20 13:57:13 +0900        :SUCCESS: DONE: Copy the patch to all cells.

2015-01-20 13:57:15 +0900        :Working: DO: Execute plugin check for Patch Check Prereq ...

2015-01-20 13:57:15 +0900 :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.1. Details in logfile /root/patch_11.2.3.3.1.140708/patchmgr.stdout.

2015-01-20 13:57:15 +0900 :INFO: This plugin checks dbhomes across all nodes with oracle-user ssh equivalence, but only for those known to the local system. dbhomes that exist only on remote nodes must be checked manually.

2015-01-20 13:57:15 +0900 :SUCCESS: No exposure to bug 17854520 with non-rolling patching

2015-01-20 13:57:16 +0900        :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.

2015-01-20 13:57:16 +0900 1 of 5 :Working: DO: Initiate patch on cells. Cells will remain up. Up to 5 minutes ...

2015-01-20 13:57:26 +0900 1 of 5 :SUCCESS: DONE: Initiate patch on cells.

2015-01-20 13:57:26 +0900 2 of 5 :Working: DO: Waiting to finish pre-reboot patch actions. Cells will remain up. Up to 45 minutes ...

2015-01-20 13:58:26 +0900 Wait for patch pre-reboot procedures

||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ 

.....................................

\\\ Minutes left 083||||| Minutes left 083||||| Minutes left 083///// Minutes left 083----- Minutes left 083\\\\\ Minutes left 083||||| Minutes left 083||||| Minutes left 083///// Minutes left 083----- Minutes left 083\\\\\ Minutes left 083||||| Minutes left 083||||| Minutes left 083///// Minutes left 083----- Minutes left 083\\\\\ Minutes left 083||||| Minutes left 083                                          

2015-01-20 15:01:23 +0900 4 of 5 :SUCCESS: DONE: Wait for cells to reboot and come online.

2015-01-20 15:01:23 +0900 5 of 5 :Working: DO: Check the state of patch on cells. Up to 5 minutes ...

2015-01-20 15:02:04 +0900 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.

2015-01-20 15:02:04 +0900        :Working: DO: Execute plugin check for Post Patch ...

2015-01-20 15:02:05 +0900        :SUCCESS: DONE: Execute plugin check for Post Patch.

================PatchMgr run started 2015-01-20 13:54:08 +0900 ===========

With arguments: -cells /root/cell_group -patch -ignore_alerts

2015-01-20 13:55:14 +0900        :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...

2015-01-20 13:55:31 +0900 ++++++++++++++++++ Logs so far begin ++++++++++

gtceladm01: 

gtceladm01: 2015-01-20 13:55:19 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.

gtceladm01: 2015-01-20 13:55:19 +0900 [INFO]: dostep called: prechk:yes:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce

gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101 2015-01-20 13:55:33 +0900:

gtceladm02: 

gtceladm02: 2015-01-20 13:36:34 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.

gtceladm02: 2015-01-20 13:36:34 +0900 [INFO]: dostep called: prechk:yes:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce

gtceladm02: _EXIT_PASS_Cell gtceladm02 172.16.40.102 2015-01-20 13:36:48 +0900:

gtceladm03: 

gtceladm03: 2015-01-20 13:37:21 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.

gtceladm03: 2015-01-20 13:37:21 +0900 [INFO]: dostep called: prechk:yes:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce

gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 13:37:35 +0900:

gtceladm01: 

gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101

gtceladm02: 

gtceladm02: _EXIT_PASS_Cell gtceladm02 172.16.40.102

gtceladm03: 

gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103

gtceladm01: 

gtceladm01: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb

gtceladm01: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek

gtceladm01: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek

gtceladm01: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb

gtceladm01: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb

gtceladm01: [INFO] Required free space on /boot is 46Mb

gtceladm01: [INFO][check_allowed_version] Target: 11.2.3.3.1.140708 Current: 11.2.3.3.0.131014.1 Patch or rollback: 1486 Rolling: non_rolling

gtceladm01: Cell gtceladm01 172.16.40.101

gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101 2015-01-20 13:55:33 +0900:

gtceladm02: 

gtceladm02: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb

gtceladm02: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek

gtceladm02: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek

gtceladm02: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb

gtceladm02: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb

gtceladm02: [INFO] Required free space on /boot is 46Mb

gtceladm02: [INFO][check_allowed_version] Target: 11.2.3.3.1.140708 Current: 11.2.3.3.0.131014.1 Patch or rollback: 1486 Rolling: non_rolling

gtceladm02: Cell gtceladm02 172.16.40.102

gtceladm02: _EXIT_PASS_Cell gtceladm02 172.16.40.102 2015-01-20 13:36:48 +0900:

gtceladm03: 

gtceladm03: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb

gtceladm03: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek

gtceladm03: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek

gtceladm03: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb

gtceladm03: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb

gtceladm03: [INFO] Required free space on /boot is 46Mb

gtceladm03: [INFO][check_allowed_version] Target: 11.2.3.3.1.140708 Current: 11.2.3.3.0.131014.1 Patch or rollback: 1486 Rolling: non_rolling

gtceladm03: Cell gtceladm03 172.16.40.103

gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 13:37:35 +0900:

2015-01-20 13:55:31 +0900 ++++++++++++++++++ Logs so far end ++++++++++

2015-01-20 13:55:31 +0900        :SUCCESS: DONE: Initialize files, check space and state of cell services.

2015-01-20 13:55:31 +0900        :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...

2015-01-20 13:55:45 +0900 ++++++++++++++++++ Logs so far begin ++++++++++

gtceladm01: 

gtceladm01: 2015-01-20 13:55:37 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.

gtceladm01: 2015-01-20 13:55:37 +0900 [INFO]: dostep called: 4p patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default

gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101 2015-01-20 13:55:37 +0900:

gtceladm01: 

gtceladm01: 2015-01-20 13:55:48 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.

gtceladm01: 2015-01-20 13:55:48 +0900 [INFO]: dostep called: 4p_md11_fix_pw patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default

gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101 2015-01-20 13:55:48 +0900:

...............................................

gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI disk timeout: 360 for /sys/block/sdj

gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI block device: /sys/block/sdk

gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI timeout is 60s for /sys/block/sdk. Change it to 300s

gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI disk timeout: 360 for /sys/block/sdk

gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI block device: /sys/block/sdl

gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI timeout is 60s for /sys/block/sdl. Change it to 300s

gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI disk timeout: 360 for /sys/block/sdl

gtceladm03: 2015-01-20 13:39:25 +0900 [INFO] Execute: nohup ./install.sh -patch -execute -standby -no-auto-rollback -first-boot-usb -use-upgrade-usb  -ignore-celld-services -force -force -force

gtceladm03: 2015-01-20 13:39:30 +0900 [INFO] pid file: /root/_patch_hctap_/_p_/install.pid : 15227 0 install.sh

gtceladm03: 2015-01-20 13:39:30 +0900 [INFO] pid 15227 is running as "0 S     0 15227     1  2  80   0 -  3437 pipe_w ?          0:00 /bin/bash ./install.sh -patch -execute -standby -no-auto-rollback -first-boot-usb -use-upgrade-usb -ignore-celld-services -force -force -force"

gtceladm03: Cell gtceladm03 172.16.40.103

gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 13:39:30 +0900:

2015-01-20 13:57:26 +0900 ++++++++++++++++++ Logs so far end ++++++++++

2015-01-20 13:57:26 +0900 1 of 5 :SUCCESS: DONE: Initiate patch on cells.

2015-01-20 13:57:26 +0900 2 of 5 :Working: DO: Waiting to finish pre-reboot patch actions. Cells will remain up. Up to 45 minutes ...

.......................................

gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 14:44:08 +0900: Found: 11.2.3.3.1.140708 state success Expected: 11.2.3.3.1.140708 state success.

2015-01-20 15:02:04 +0900 ++++++++++++++++++ Logs so far end ++++++++++

2015-01-20 15:02:04 +0900 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.

2015-01-20 15:02:04 +0900        :Working: DO: Execute plugin check for Post Patch ...

2015-01-20 15:02:04 +0900 ++++++++++++++++++ Logs so far begin ++++++++++

2015-01-20 15:02:05 +0900 ++++++++++++++++++ Logs so far end ++++++++++

2015-01-20 15:02:05 +0900        :SUCCESS: DONE: Execute plugin check for Post Patch.

================PatchMgr run ended 2015-01-20 15:02:05 +0900 ===========

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# dcli -g /root/cell_group -l root imageinfo

gtceladm01: 

gtceladm01: Kernel version: 2.6.39-400.128.17.el5uek #1 SMP Tue May 27 13:20:24 PDT 2014 x86_64

gtceladm01: Cell version: OSS_11.2.3.3.1_LINUX.X64_140708

gtceladm01: Cell rpm version: cell-11.2.3.3.1_LINUX.X64_140708-1

gtceladm01: 

gtceladm01: Active image version: 11.2.3.3.1.140708

gtceladm01: Active image activated: 2015-01-20 15:00:37 +0900

gtceladm01: Active image status: success

gtceladm01: Active system partition on device: /dev/md6

gtceladm01: Active software partition on device: /dev/md8

gtceladm01: 

gtceladm01: In partition rollback: Impossible

gtceladm01: 

gtceladm01: Cell boot usb partition: /dev/sdm1

gtceladm01: Cell boot usb version: 11.2.3.3.1.140708

gtceladm01: 

gtceladm01: Inactive image version: 11.2.3.3.0.131014.1

gtceladm01: Inactive image activated: 2015-01-06 17:03:56 +0900

gtceladm01: Inactive image status: success

gtceladm01: Inactive system partition on device: /dev/md5

gtceladm01: Inactive software partition on device: /dev/md7

gtceladm01: 

gtceladm01: Boot area has rollback archive for the version: 11.2.3.3.0.131014.1

gtceladm01: Rollback to the inactive partitions: Possible

gtceladm02: 

gtceladm02: Kernel version: 2.6.39-400.128.17.el5uek #1 SMP Tue May 27 13:20:24 PDT 2014 x86_64

gtceladm02: Cell version: OSS_11.2.3.3.1_LINUX.X64_140708

gtceladm02: Cell rpm version: cell-11.2.3.3.1_LINUX.X64_140708-1

gtceladm02: 

gtceladm02: Active image version: 11.2.3.3.1.140708

gtceladm02: Active image activated: 2015-01-20 14:42:04 +0900

gtceladm02: Active image status: success

gtceladm02: Active system partition on device: /dev/md6

gtceladm02: Active software partition on device: /dev/md8

gtceladm02: 

gtceladm02: In partition rollback: Impossible

gtceladm02: 

gtceladm02: Cell boot usb partition: /dev/sdm1

gtceladm02: Cell boot usb version: 11.2.3.3.1.140708

gtceladm02: 

gtceladm02: Inactive image version: 11.2.3.3.0.131014.1

gtceladm02: Inactive image activated: 2015-01-06 17:53:13 +0900

gtceladm02: Inactive image status: success

gtceladm02: Inactive system partition on device: /dev/md5

gtceladm02: Inactive software partition on device: /dev/md7

gtceladm02: 

gtceladm02: Boot area has rollback archive for the version: 11.2.3.3.0.131014.1

gtceladm02: Rollback to the inactive partitions: Possible

gtceladm03: 

gtceladm03: Kernel version: 2.6.39-400.128.17.el5uek #1 SMP Tue May 27 13:20:24 PDT 2014 x86_64

gtceladm03: Cell version: OSS_11.2.3.3.1_LINUX.X64_140708

gtceladm03: Cell rpm version: cell-11.2.3.3.1_LINUX.X64_140708-1

gtceladm03: 

gtceladm03: Active image version: 11.2.3.3.1.140708

gtceladm03: Active image activated: 2015-01-20 14:42:37 +0900

gtceladm03: Active image status: success

gtceladm03: Active system partition on device: /dev/md6

gtceladm03: Active software partition on device: /dev/md8

gtceladm03: 

gtceladm03: In partition rollback: Impossible

gtceladm03: 

gtceladm03: Cell boot usb partition: /dev/sdm1

gtceladm03: Cell boot usb version: 11.2.3.3.1.140708

gtceladm03: 

gtceladm03: Inactive image version: 11.2.3.3.0.131014.1

gtceladm03: Inactive image activated: 2015-01-06 16:57:02 +0900

gtceladm03: Inactive image status: success

gtceladm03: Inactive system partition on device: /dev/md5

gtceladm03: Inactive software partition on device: /dev/md7

gtceladm03: 

gtceladm03: Boot area has rollback archive for the version: 11.2.3.3.0.131014.1

gtceladm03: Rollback to the inactive partitions: Possible

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# dcli -g /root/cell_group -l root imagehistory

gtceladm01: Version                              : 11.2.3.3.0.131014.1

gtceladm01: Image activation date                : 2015-01-06 17:03:56 +0900

gtceladm01: Imaging mode                         : fresh

gtceladm01: Imaging status                       : success

gtceladm01: 

gtceladm01: Version                              : 11.2.3.3.1.140708

gtceladm01: Image activation date                : 2015-01-20 15:00:37 +0900

gtceladm01: Imaging mode                         : out of partition upgrade

gtceladm01: Imaging status                       : success

gtceladm01: 

gtceladm02: Version                              : 11.2.3.3.0.131014.1

gtceladm02: Image activation date                : 2015-01-06 17:53:13 +0900

gtceladm02: Imaging mode                         : fresh

gtceladm02: Imaging status                       : success

gtceladm02: 

gtceladm02: Version                              : 11.2.3.3.1.140708

gtceladm02: Image activation date                : 2015-01-20 14:42:04 +0900

gtceladm02: Imaging mode                         : out of partition upgrade

gtceladm02: Imaging status                       : success

gtceladm02: 

gtceladm03: Version                              : 11.2.3.3.0.131014.1

gtceladm03: Image activation date                : 2015-01-06 16:57:02 +0900

gtceladm03: Imaging mode                         : fresh

gtceladm03: Imaging status                       : success

gtceladm03: 

gtceladm03: Version                              : 11.2.3.3.1.140708

gtceladm03: Image activation date                : 2015-01-20 14:42:37 +0900

gtceladm03: Imaging mode                         : out of partition upgrade

gtceladm03: Imaging status                       : success