Exadata Cell Upgrade 하기
11.2.3.3.0.131014.1 -> 11.2.3.3.1.140708
Cell Upgrade 는 DB node 에서 진행한다.
[EXA1]root@exa1:/root# dcli -g cell_group -l root /opt/oracle.cellos/ipconf -verify
gtceladm01: Verifying of Exadata configuration file /opt/oracle.cellos/cell.conf
gtceladm01: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checks
gtceladm02: Verifying of Exadata configuration file /opt/oracle.cellos/cell.conf
gtceladm02: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checks
gtceladm03: Verifying of Exadata configuration file /opt/oracle.cellos/cell.conf
gtceladm03: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checks
압축해제
[EXA1]root@exa1:/root# unzip p19166601_112331_Linux-x86-64.zip
Archive: p19166601_112331_Linux-x86-64.zip
creating: patch_11.2.3.3.1.140708/
inflating: patch_11.2.3.3.1.140708/sundcs_36p_repository_2.1.3_4.pkg
inflating: patch_11.2.3.3.1.140708/upgradeIBSwitch.sh
creating: patch_11.2.3.3.1.140708/linux.db.rpms/
inflating: patch_11.2.3.3.1.140708/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm
creating: patch_11.2.3.3.1.140708/plugins/
inflating: patch_11.2.3.3.1.140708/plugins/010-check_17854520.sh
inflating: patch_11.2.3.3.1.140708/plugins/000-check_dummy_bash
inflating: patch_11.2.3.3.1.140708/plugins/000-check_dummy_perl
inflating: patch_11.2.3.3.1.140708/11.2.3.3.1.140708.iso
....................
[EXA1]root@exa1:/root# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'exa1'
CRS-2673: Attempting to stop 'ora.crsd' on 'exa1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'exa1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exa1'
CRS-2673: Attempting to stop 'ora.exa.db' on 'exa1'
CRS-2673: Attempting to stop 'dbfs_mount' on 'exa1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'exa1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'exa1'
CRS-2677: Stop of 'dbfs_mount' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.fsdb.db' on 'exa1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'exa1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.exa1.vip' on 'exa1'
CRS-2677: Stop of 'ora.registry.acfs' on 'exa1' succeeded
CRS-2677: Stop of 'ora.fsdb.db' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.DBFS.dg' on 'exa1'
CRS-2677: Stop of 'ora.exa.db' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'exa1'
CRS-2673: Attempting to stop 'ora.RECO.dg' on 'exa1'
CRS-2677: Stop of 'ora.exa1.vip' on 'exa1' succeeded
CRS-2672: Attempting to start 'ora.exa1.vip' on 'exa2'
CRS-2677: Stop of 'ora.RECO.dg' on 'exa1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'exa1' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'exa1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'exa2'
CRS-2676: Start of 'ora.exa1.vip' on 'exa2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'exa2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'exa2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'exa2' succeeded
CRS-2677: Stop of 'ora.DBFS.dg' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'exa1'
CRS-2677: Stop of 'ora.asm' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'exa1'
CRS-2677: Stop of 'ora.ons' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'exa1'
CRS-2677: Stop of 'ora.net1.network' on 'exa1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'exa1' has completed
CRS-2677: Stop of 'ora.crsd' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'exa1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'exa1'
CRS-2673: Attempting to stop 'ora.evmd' on 'exa1'
CRS-2673: Attempting to stop 'ora.asm' on 'exa1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'exa1'
CRS-2677: Stop of 'ora.ctssd' on 'exa1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'exa1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'exa1' succeeded
CRS-2677: Stop of 'ora.asm' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'exa1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'exa1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'exa1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'exa1'
CRS-2673: Attempting to stop 'ora.crf' on 'exa1'
CRS-2677: Stop of 'ora.crf' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'exa1'
CRS-2677: Stop of 'ora.diskmon' on 'exa1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'exa1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'exa1'
CRS-2677: Stop of 'ora.gpnpd' on 'exa1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'exa1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[EXA1]root@exa1:/root# dcli -g /root/cell_group -l root 'service celld stop'
gtceladm01:
gtceladm01: Stopping the RS, CELLSRV, and MS services...
gtceladm01: The SHUTDOWN of services was successful.
gtceladm02:
gtceladm02: Stopping the RS, CELLSRV, and MS services...
gtceladm02: The SHUTDOWN of services was successful.
gtceladm03:
gtceladm03: Stopping the RS, CELLSRV, and MS services...
gtceladm03: The SHUTDOWN of services was successful.
[EXA1]root@exa1:/root# dcli -g /root/cell_group -l root 'service celld status'
gtceladm01: rsStatus: stopped
gtceladm01: msStatus: unknown
gtceladm01: cellsrvStatus: unknown
gtceladm02: rsStatus: stopped
gtceladm02: msStatus: unknown
gtceladm02: cellsrvStatus: unknown
gtceladm03: rsStatus: stopped
gtceladm03: msStatus: unknown
gtceladm03: cellsrvStatus: unknown
[EXA1]root@exa1:/root#
[EXA1]root@exa1:/root# dcli -g /root/cell_group -l root 'sync;sync;sync'
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -reset_force
2015-01-20 13:36:41 +0900 :DONE: reset_force
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -cleanup
2015-01-20 13:37:54 +0900 :Working: DO: Cleanup ...
2015-01-20 13:37:55 +0900 :SUCCESS: DONE: Cleanup
-ignore_alerts 옵션 없이
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -patch_check_prereq
2015-01-20 13:39:03 +0900 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
FAILED for following cells
gtceladm01: gtceladm01 172.16.40.101 2015-01-20 13:39:38 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.
gtceladm02: gtceladm02 172.16.40.102 2015-01-20 13:20:53 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.
gtceladm03: gtceladm03 172.16.40.103 2015-01-20 13:21:40 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.
2015-01-20 13:39:36 +0900 :FAILED: Details in files <cell_name>.log /root/patch_11.2.3.3.1.140708/patchmgr.stdout, /root/patch_11.2.3.3.1.140708/patchmgr.stderr
2015-01-20 13:39:36 +0900 :FAILED: DONE: Initialize files, check space and state of cell services.
[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.
-ignore_alerts 옵션 주고
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -patch_check_prereq -ignore_alerts
2015-01-20 13:46:10 +0900 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2015-01-20 13:46:27 +0900 :SUCCESS: DONE: Initialize files, check space and state of cell services.
2015-01-20 13:46:27 +0900 :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
2015-01-20 13:46:41 +0900 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
2015-01-20 13:46:42 +0900 :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
2015-01-20 13:46:42 +0900 :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
2015-01-20 13:47:28 +0900 :SUCCESS: DONE: Check prerequisites on all cells.
2015-01-20 13:47:28 +0900 :Working: DO: Execute plugin check for Patch Check Prereq ...
2015-01-20 13:47:28 +0900 :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.1. Details in logfile /root/patch_11.2.3.3.1.140708/patchmgr.stdout.
2015-01-20 13:47:28 +0900 :INFO: This plugin checks dbhomes across all nodes with oracle-user ssh equivalence, but only for those known to the local system. dbhomes that exist only on remote nodes must be checked manually.
2015-01-20 13:47:28 +0900 :SUCCESS: No exposure to bug 17854520 with non-rolling patching
2015-01-20 13:47:28 +0900 :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -patch
NOTE Cells will reboot during the patch or rollback process.
NOTE For non-rolling patch or rollback, ensure all ASM instances using
NOTE the cells are shut down for the duration of the patch or rollback.
NOTE For rolling patch or rollback, ensure all ASM instances using
NOTE the cells are up for the duration of the patch or rollback.
WARNING Do not start more than one instance of patchmgr.
WARNING Do not interrupt the patchmgr session.
WARNING Do not alter state of ASM instances during patch or rollback.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot cells or alter cell services during patch or rollback.
WARNING Do not open log files in editor in write mode or try to alter them.
NOTE All time estimates are approximate. Timestamps on the left are real.
NOTE You may interrupt this patchmgr run in next 60 seconds with control-c.
2015-01-20 13:51:22 +0900 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
FAILED for following cells
gtceladm01: gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.
gtceladm02: gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.
gtceladm03: gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.
2015-01-20 13:51:30 +0900 :FAILED: Details in files <cell_name>.log /root/patch_11.2.3.3.1.140708/patchmgr.stdout, /root/patch_11.2.3.3.1.140708/patchmgr.stderr
2015-01-20 13:51:30 +0900 :FAILED: DONE: Initialize files, check space and state of cell services.
[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708#
================PatchMgr run started 2015-01-20 13:50:15 +0900 ===========
With arguments: -cells /root/cell_group -patch
2015-01-20 13:51:22 +0900 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2015-01-20 13:51:29 +0900 ++++++++++++++++++ Logs so far begin ++++++++++
gtceladm01:
gtceladm01: 2015-01-20 13:51:26 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.
gtceladm01: 2015-01-20 13:51:26 +0900 [INFO]: dostep called: prechk:no:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce
gtceladm01: _EXIT_ERROR_Cell gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.
gtceladm02:
gtceladm02: 2015-01-20 13:32:41 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.
gtceladm02: 2015-01-20 13:32:41 +0900 [INFO]: dostep called: prechk:no:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce
gtceladm02: _EXIT_ERROR_Cell gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.
gtceladm03:
gtceladm03: 2015-01-20 13:33:28 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.
gtceladm03: 2015-01-20 13:33:28 +0900 [INFO]: dostep called: prechk:no:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce
gtceladm03: _EXIT_ERROR_Cell gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.
gtceladm01:
gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101
gtceladm02:
gtceladm02: _EXIT_PASS_Cell gtceladm02 172.16.40.102
gtceladm03:
gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103
gtceladm01:
gtceladm01: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb
gtceladm01: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek
gtceladm01: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek
gtceladm01: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb
gtceladm01: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb
gtceladm01: [INFO] Required free space on /boot is 46Mb
gtceladm01: [ERROR] Detected open hardware alert(s) on the Exadata cell
gtceladm01: [ERROR] 3_1 2014-03-08T04:00:01+09:00 critical "A power supply component is suspected of causing a fault. Component Name : /SYS/PS0 Fault class : fault.chassis.env.power.loss Fault message : http://www.sun.com/msg/SPX86-8000-55"
gtceladm01: [ERROR] Check for open hardware alerts on Exadata cell by running "cellcli -e list alerthistory where endTime=null and alertShortName=Hardware and alertType=stateful and severity=critical"
gtceladm01: [ERROR] Specific alerts can be dropped using "cellcli -e drop alerthistory X_Y" where X_Y is the alert ID
gtceladm01: Cell gtceladm01 172.16.40.101
gtceladm01: _EXIT_ERROR_Cell gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.
gtceladm02:
gtceladm02: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb
gtceladm02: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek
gtceladm02: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek
gtceladm02: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb
gtceladm02: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb
gtceladm02: [INFO] Required free space on /boot is 46Mb
gtceladm02: [ERROR] Detected open hardware alert(s) on the Exadata cell
gtceladm02: [ERROR] 3_1 2014-03-10T10:12:43+09:00 critical "A power supply component is suspected of causing a fault. Component Name : /SYS/PS0 Fault class : fault.chassis.env.power.loss Fault message : http://www.sun.com/msg/SPX86-8000-55"
gtceladm02: [ERROR] Check for open hardware alerts on Exadata cell by running "cellcli -e list alerthistory where endTime=null and alertShortName=Hardware and alertType=stateful and severity=critical"
gtceladm02: [ERROR] Specific alerts can be dropped using "cellcli -e drop alerthistory X_Y" where X_Y is the alert ID
gtceladm02: Cell gtceladm02 172.16.40.102
gtceladm02: _EXIT_ERROR_Cell gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.
gtceladm03:
gtceladm03: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb
gtceladm03: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek
gtceladm03: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek
gtceladm03: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb
gtceladm03: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb
gtceladm03: [INFO] Required free space on /boot is 46Mb
gtceladm03: [ERROR] Detected open hardware alert(s) on the Exadata cell
gtceladm03: [ERROR] 3_1 2014-03-10T11:50:46+09:00 critical "A power supply component is suspected of causing a fault. Component Name : /SYS/PS0 Fault class : fault.chassis.env.power.loss Fault message : http://www.sun.com/msg/SPX86-8000-55"
gtceladm03: [ERROR] Check for open hardware alerts on Exadata cell by running "cellcli -e list alerthistory where endTime=null and alertShortName=Hardware and alertType=stateful and severity=critical"
gtceladm03: [ERROR] Specific alerts can be dropped using "cellcli -e drop alerthistory X_Y" where X_Y is the alert ID
gtceladm03: Cell gtceladm03 172.16.40.103
gtceladm03: _EXIT_ERROR_Cell gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.
2015-01-20 13:51:30 +0900 ++++++++++++++++++ Logs so far end ++++++++++
FAILED for following cells
gtceladm01: gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.
gtceladm02: gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.
gtceladm03: gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.
2015-01-20 13:51:30 +0900 :FAILED: Details in files <cell_name>.log /root/patch_11.2.3.3.1.140708/patchmgr.stdout, /root/patch_11.2.3.3.1.140708/patchmgr.stderr
2015-01-20 13:51:30 +0900 :FAILED: DONE: Initialize files, check space and state of cell services.
[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.
================PatchMgr run ended 2015-01-20 13:51:30 +0900 ===========
gtceladm01: gtceladm01 172.16.40.101 2015-01-20 13:51:32 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm01.log for details. Correct this before retrying the patch.
gtceladm02: gtceladm02 172.16.40.102 2015-01-20 13:32:46 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm02.log for details. Correct this before retrying the patch.
gtceladm03: gtceladm03 172.16.40.103 2015-01-20 13:33:33 +0900: Detected open alerts in the alert history for the Exadata cell. Check gtceladm03.log for details. Correct this before retrying the patch.
2015-01-20 13:51:30 +0900 :FAILED: Details in files <cell_name>.log /root/patch_11.2.3.3.1.140708/patchmgr.stdout, /root/patch_11.2.3.3.1.140708/patchmgr.stderr
2015-01-20 13:51:30 +0900 :FAILED: DONE: Initialize files, check space and state of cell services.
[ERROR] Patch prerequisite checks failed. Please run cleanup before retrying.
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708#
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708#
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708#
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -patch -ignore_alerts
NOTE Cells will reboot during the patch or rollback process.
NOTE For non-rolling patch or rollback, ensure all ASM instances using
NOTE the cells are shut down for the duration of the patch or rollback.
NOTE For rolling patch or rollback, ensure all ASM instances using
NOTE the cells are up for the duration of the patch or rollback.
WARNING Do not start more than one instance of patchmgr.
WARNING Do not interrupt the patchmgr session.
WARNING Do not alter state of ASM instances during patch or rollback.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot cells or alter cell services during patch or rollback.
WARNING Do not open log files in editor in write mode or try to alter them.
NOTE All time estimates are approximate. Timestamps on the left are real.
NOTE You may interrupt this patchmgr run in next 60 seconds with control-c.
2015-01-20 13:55:14 +0900 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2015-01-20 13:55:31 +0900 :SUCCESS: DONE: Initialize files, check space and state of cell services.
2015-01-20 13:55:31 +0900 :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
2015-01-20 13:55:44 +0900 Wait correction of degraded md11 due to md partner size mismatch. Up to 30 minutes.
2015-01-20 13:55:45 +0900 :SUCCESS: DONE: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction.
2015-01-20 13:55:45 +0900 :Working: DO: Check prerequisites on all cells. Up to 2 minutes ...
2015-01-20 13:56:31 +0900 :SUCCESS: DONE: Check prerequisites on all cells.
2015-01-20 13:56:31 +0900 :Working: DO: Copy the patch to all cells. Up to 3 minutes ...
2015-01-20 13:57:13 +0900 :SUCCESS: DONE: Copy the patch to all cells.
2015-01-20 13:57:15 +0900 :Working: DO: Execute plugin check for Patch Check Prereq ...
2015-01-20 13:57:15 +0900 :INFO: Patchmgr plugin start: Prereq check for exposure to bug 17854520 v1.1. Details in logfile /root/patch_11.2.3.3.1.140708/patchmgr.stdout.
2015-01-20 13:57:15 +0900 :INFO: This plugin checks dbhomes across all nodes with oracle-user ssh equivalence, but only for those known to the local system. dbhomes that exist only on remote nodes must be checked manually.
2015-01-20 13:57:15 +0900 :SUCCESS: No exposure to bug 17854520 with non-rolling patching
2015-01-20 13:57:16 +0900 :SUCCESS: DONE: Execute plugin check for Patch Check Prereq.
2015-01-20 13:57:16 +0900 1 of 5 :Working: DO: Initiate patch on cells. Cells will remain up. Up to 5 minutes ...
2015-01-20 13:57:26 +0900 1 of 5 :SUCCESS: DONE: Initiate patch on cells.
2015-01-20 13:57:26 +0900 2 of 5 :Working: DO: Waiting to finish pre-reboot patch actions. Cells will remain up. Up to 45 minutes ...
2015-01-20 13:58:26 +0900 Wait for patch pre-reboot procedures
||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\
.....................................
\\\ Minutes left 083||||| Minutes left 083||||| Minutes left 083///// Minutes left 083----- Minutes left 083\\\\\ Minutes left 083||||| Minutes left 083||||| Minutes left 083///// Minutes left 083----- Minutes left 083\\\\\ Minutes left 083||||| Minutes left 083||||| Minutes left 083///// Minutes left 083----- Minutes left 083\\\\\ Minutes left 083||||| Minutes left 083
2015-01-20 15:01:23 +0900 4 of 5 :SUCCESS: DONE: Wait for cells to reboot and come online.
2015-01-20 15:01:23 +0900 5 of 5 :Working: DO: Check the state of patch on cells. Up to 5 minutes ...
2015-01-20 15:02:04 +0900 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.
2015-01-20 15:02:04 +0900 :Working: DO: Execute plugin check for Post Patch ...
2015-01-20 15:02:05 +0900 :SUCCESS: DONE: Execute plugin check for Post Patch.
================PatchMgr run started 2015-01-20 13:54:08 +0900 ===========
With arguments: -cells /root/cell_group -patch -ignore_alerts
2015-01-20 13:55:14 +0900 :Working: DO: Initialize files, check space and state of cell services. Up to 1 minute ...
2015-01-20 13:55:31 +0900 ++++++++++++++++++ Logs so far begin ++++++++++
gtceladm01:
gtceladm01: 2015-01-20 13:55:19 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.
gtceladm01: 2015-01-20 13:55:19 +0900 [INFO]: dostep called: prechk:yes:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce
gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101 2015-01-20 13:55:33 +0900:
gtceladm02:
gtceladm02: 2015-01-20 13:36:34 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.
gtceladm02: 2015-01-20 13:36:34 +0900 [INFO]: dostep called: prechk:yes:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce
gtceladm02: _EXIT_PASS_Cell gtceladm02 172.16.40.102 2015-01-20 13:36:48 +0900:
gtceladm03:
gtceladm03: 2015-01-20 13:37:21 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.
gtceladm03: 2015-01-20 13:37:21 +0900 [INFO]: dostep called: prechk:yes:1486 patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default noforce
gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 13:37:35 +0900:
gtceladm01:
gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101
gtceladm02:
gtceladm02: _EXIT_PASS_Cell gtceladm02 172.16.40.102
gtceladm03:
gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103
gtceladm01:
gtceladm01: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb
gtceladm01: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek
gtceladm01: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek
gtceladm01: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb
gtceladm01: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb
gtceladm01: [INFO] Required free space on /boot is 46Mb
gtceladm01: [INFO][check_allowed_version] Target: 11.2.3.3.1.140708 Current: 11.2.3.3.0.131014.1 Patch or rollback: 1486 Rolling: non_rolling
gtceladm01: Cell gtceladm01 172.16.40.101
gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101 2015-01-20 13:55:33 +0900:
gtceladm02:
gtceladm02: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb
gtceladm02: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek
gtceladm02: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek
gtceladm02: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb
gtceladm02: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb
gtceladm02: [INFO] Required free space on /boot is 46Mb
gtceladm02: [INFO][check_allowed_version] Target: 11.2.3.3.1.140708 Current: 11.2.3.3.0.131014.1 Patch or rollback: 1486 Rolling: non_rolling
gtceladm02: Cell gtceladm02 172.16.40.102
gtceladm02: _EXIT_PASS_Cell gtceladm02 172.16.40.102 2015-01-20 13:36:48 +0900:
gtceladm03:
gtceladm03: [INFO] Free space in /boot (/dev/md4) before clean up is 81Mb
gtceladm03: [INFO] Kernel version on runtime: 2.6.39-400.126.1.el5uek
gtceladm03: [INFO] Kernel version in /opt/oracle.cellos/image.id: 2.6.39-400.126.1.el5uek
gtceladm03: [INFO] Free space in /boot (/dev/md4) after clean up is 81Mb
gtceladm03: [INFO] Size for all files for the kernel 2.6.39-400.126.1.el5uek in /boot is 23Mb
gtceladm03: [INFO] Required free space on /boot is 46Mb
gtceladm03: [INFO][check_allowed_version] Target: 11.2.3.3.1.140708 Current: 11.2.3.3.0.131014.1 Patch or rollback: 1486 Rolling: non_rolling
gtceladm03: Cell gtceladm03 172.16.40.103
gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 13:37:35 +0900:
2015-01-20 13:55:31 +0900 ++++++++++++++++++ Logs so far end ++++++++++
2015-01-20 13:55:31 +0900 :SUCCESS: DONE: Initialize files, check space and state of cell services.
2015-01-20 13:55:31 +0900 :Working: DO: Copy, extract prerequisite check archive to cells. If required start md11 mismatched partner size correction. Up to 40 minutes ...
2015-01-20 13:55:45 +0900 ++++++++++++++++++ Logs so far begin ++++++++++
gtceladm01:
gtceladm01: 2015-01-20 13:55:37 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.
gtceladm01: 2015-01-20 13:55:37 +0900 [INFO]: dostep called: 4p patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default
gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101 2015-01-20 13:55:37 +0900:
gtceladm01:
gtceladm01: 2015-01-20 13:55:48 +0900 [INFO]: patchmgr launch attempt from exa1.gtplus.co.kr_172.16.20.1_root_patch_11.2.3.3.1.140708.
gtceladm01: 2015-01-20 13:55:48 +0900 [INFO]: dostep called: 4p_md11_fix_pw patch_prereq -no-auto-rollback non_rolling 900 3600 900 36000 default
gtceladm01: _EXIT_PASS_Cell gtceladm01 172.16.40.101 2015-01-20 13:55:48 +0900:
...............................................
gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI disk timeout: 360 for /sys/block/sdj
gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI block device: /sys/block/sdk
gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI timeout is 60s for /sys/block/sdk. Change it to 300s
gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI disk timeout: 360 for /sys/block/sdk
gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI block device: /sys/block/sdl
gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI timeout is 60s for /sys/block/sdl. Change it to 300s
gtceladm03: 2015-01-20 13:39:21 +0900 [INFO] dostep_fix_12545272: LSI disk timeout: 360 for /sys/block/sdl
gtceladm03: 2015-01-20 13:39:25 +0900 [INFO] Execute: nohup ./install.sh -patch -execute -standby -no-auto-rollback -first-boot-usb -use-upgrade-usb -ignore-celld-services -force -force -force
gtceladm03: 2015-01-20 13:39:30 +0900 [INFO] pid file: /root/_patch_hctap_/_p_/install.pid : 15227 0 install.sh
gtceladm03: 2015-01-20 13:39:30 +0900 [INFO] pid 15227 is running as "0 S 0 15227 1 2 80 0 - 3437 pipe_w ? 0:00 /bin/bash ./install.sh -patch -execute -standby -no-auto-rollback -first-boot-usb -use-upgrade-usb -ignore-celld-services -force -force -force"
gtceladm03: Cell gtceladm03 172.16.40.103
gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 13:39:30 +0900:
2015-01-20 13:57:26 +0900 ++++++++++++++++++ Logs so far end ++++++++++
2015-01-20 13:57:26 +0900 1 of 5 :SUCCESS: DONE: Initiate patch on cells.
2015-01-20 13:57:26 +0900 2 of 5 :Working: DO: Waiting to finish pre-reboot patch actions. Cells will remain up. Up to 45 minutes ...
.......................................
gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 14:44:08 +0900: Found: 11.2.3.3.1.140708 state success Expected: 11.2.3.3.1.140708 state success.
2015-01-20 15:02:04 +0900 ++++++++++++++++++ Logs so far end ++++++++++
2015-01-20 15:02:04 +0900 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.
2015-01-20 15:02:04 +0900 :Working: DO: Execute plugin check for Post Patch ...
2015-01-20 15:02:04 +0900 ++++++++++++++++++ Logs so far begin ++++++++++
2015-01-20 15:02:05 +0900 ++++++++++++++++++ Logs so far end ++++++++++
2015-01-20 15:02:05 +0900 :SUCCESS: DONE: Execute plugin check for Post Patch.
================PatchMgr run ended 2015-01-20 15:02:05 +0900 ===========
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# dcli -g /root/cell_group -l root imageinfo
gtceladm01:
gtceladm01: Kernel version: 2.6.39-400.128.17.el5uek #1 SMP Tue May 27 13:20:24 PDT 2014 x86_64
gtceladm01: Cell version: OSS_11.2.3.3.1_LINUX.X64_140708
gtceladm01: Cell rpm version: cell-11.2.3.3.1_LINUX.X64_140708-1
gtceladm01:
gtceladm01: Active image version: 11.2.3.3.1.140708
gtceladm01: Active image activated: 2015-01-20 15:00:37 +0900
gtceladm01: Active image status: success
gtceladm01: Active system partition on device: /dev/md6
gtceladm01: Active software partition on device: /dev/md8
gtceladm01:
gtceladm01: In partition rollback: Impossible
gtceladm01:
gtceladm01: Cell boot usb partition: /dev/sdm1
gtceladm01: Cell boot usb version: 11.2.3.3.1.140708
gtceladm01:
gtceladm01: Inactive image version: 11.2.3.3.0.131014.1
gtceladm01: Inactive image activated: 2015-01-06 17:03:56 +0900
gtceladm01: Inactive image status: success
gtceladm01: Inactive system partition on device: /dev/md5
gtceladm01: Inactive software partition on device: /dev/md7
gtceladm01:
gtceladm01: Boot area has rollback archive for the version: 11.2.3.3.0.131014.1
gtceladm01: Rollback to the inactive partitions: Possible
gtceladm02:
gtceladm02: Kernel version: 2.6.39-400.128.17.el5uek #1 SMP Tue May 27 13:20:24 PDT 2014 x86_64
gtceladm02: Cell version: OSS_11.2.3.3.1_LINUX.X64_140708
gtceladm02: Cell rpm version: cell-11.2.3.3.1_LINUX.X64_140708-1
gtceladm02:
gtceladm02: Active image version: 11.2.3.3.1.140708
gtceladm02: Active image activated: 2015-01-20 14:42:04 +0900
gtceladm02: Active image status: success
gtceladm02: Active system partition on device: /dev/md6
gtceladm02: Active software partition on device: /dev/md8
gtceladm02:
gtceladm02: In partition rollback: Impossible
gtceladm02:
gtceladm02: Cell boot usb partition: /dev/sdm1
gtceladm02: Cell boot usb version: 11.2.3.3.1.140708
gtceladm02:
gtceladm02: Inactive image version: 11.2.3.3.0.131014.1
gtceladm02: Inactive image activated: 2015-01-06 17:53:13 +0900
gtceladm02: Inactive image status: success
gtceladm02: Inactive system partition on device: /dev/md5
gtceladm02: Inactive software partition on device: /dev/md7
gtceladm02:
gtceladm02: Boot area has rollback archive for the version: 11.2.3.3.0.131014.1
gtceladm02: Rollback to the inactive partitions: Possible
gtceladm03:
gtceladm03: Kernel version: 2.6.39-400.128.17.el5uek #1 SMP Tue May 27 13:20:24 PDT 2014 x86_64
gtceladm03: Cell version: OSS_11.2.3.3.1_LINUX.X64_140708
gtceladm03: Cell rpm version: cell-11.2.3.3.1_LINUX.X64_140708-1
gtceladm03:
gtceladm03: Active image version: 11.2.3.3.1.140708
gtceladm03: Active image activated: 2015-01-20 14:42:37 +0900
gtceladm03: Active image status: success
gtceladm03: Active system partition on device: /dev/md6
gtceladm03: Active software partition on device: /dev/md8
gtceladm03:
gtceladm03: In partition rollback: Impossible
gtceladm03:
gtceladm03: Cell boot usb partition: /dev/sdm1
gtceladm03: Cell boot usb version: 11.2.3.3.1.140708
gtceladm03:
gtceladm03: Inactive image version: 11.2.3.3.0.131014.1
gtceladm03: Inactive image activated: 2015-01-06 16:57:02 +0900
gtceladm03: Inactive image status: success
gtceladm03: Inactive system partition on device: /dev/md5
gtceladm03: Inactive software partition on device: /dev/md7
gtceladm03:
gtceladm03: Boot area has rollback archive for the version: 11.2.3.3.0.131014.1
gtceladm03: Rollback to the inactive partitions: Possible
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# dcli -g /root/cell_group -l root imagehistory
gtceladm01: Version : 11.2.3.3.0.131014.1
gtceladm01: Image activation date : 2015-01-06 17:03:56 +0900
gtceladm01: Imaging mode : fresh
gtceladm01: Imaging status : success
gtceladm01:
gtceladm01: Version : 11.2.3.3.1.140708
gtceladm01: Image activation date : 2015-01-20 15:00:37 +0900
gtceladm01: Imaging mode : out of partition upgrade
gtceladm01: Imaging status : success
gtceladm01:
gtceladm02: Version : 11.2.3.3.0.131014.1
gtceladm02: Image activation date : 2015-01-06 17:53:13 +0900
gtceladm02: Imaging mode : fresh
gtceladm02: Imaging status : success
gtceladm02:
gtceladm02: Version : 11.2.3.3.1.140708
gtceladm02: Image activation date : 2015-01-20 14:42:04 +0900
gtceladm02: Imaging mode : out of partition upgrade
gtceladm02: Imaging status : success
gtceladm02:
gtceladm03: Version : 11.2.3.3.0.131014.1
gtceladm03: Image activation date : 2015-01-06 16:57:02 +0900
gtceladm03: Imaging mode : fresh
gtceladm03: Imaging status : success
gtceladm03:
gtceladm03: Version : 11.2.3.3.1.140708
gtceladm03: Image activation date : 2015-01-20 14:42:37 +0900
gtceladm03: Imaging mode : out of partition upgrade
gtceladm03: Imaging status : success