pat98's always & forever

oracle 12c 12.1.0.2 버전 설치시에

- grid 설치시 1번노드만 설치가 성공하고 나머지 2,3번은 실패함.

  자료확인해 보면 아래와 같은 버그임.

CLSRSC-507: The root script cannot proceed on this node <node-n> because either the first-node operations have not completed on node <node-1> or there was an error in obtaining the status of the first-node operations. (문서 ID 1919825.1)

* oracle AMDU (ASM Metadata Dump Utility) 가 디스크를 읽지 못해서 나는 버그임.

우선 확인 및 해결절차

1. 1번노드에서 성공적으로 root script가 수행되었는지 확인

<NEW_GI_HOME>/cfgtoollogs/crsconfig/rootcrs_<node>_<timestamp>.log

CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

2. 다른 노드를 살펴보면 root script 가 ocrdump 가 실행되지 못해서 실패했다.

<NEW_GI_HOME>/cfgtoollogs/crsconfig/rootcrs_<node>_<timestamp>.log

2014-09-04 13:45:34: ASM_DISKS=ORCL:OCR01,ORCL:OCR02,ORCL:OCR03
....
2014-09-04 13:46:04: Check the existence of global ckpt 'checkpoints.firstnode'
2014-09-04 13:46:04: setting ORAASM_UPGRADE to 1
2014-09-04 13:46:04: Invoking "/product/app/12.1.0.2/grid/bin/cluutil -exec -keyexists -key checkpoints.firstnode"
2014-09-04 13:46:04: trace file=/product/app/grid/crsdata/sipr0-db04/crsconfig/cluutil8.log
2014-09-04 13:46:04: Running as user grid: /product/app/12.1.0.2/grid/bin/cluutil -exec -keyexists -key checkpoints.firstnode
2014-09-04 13:46:04: s_run_as_user2: Running /bin/su grid -c ' echo CLSRSC_START; /product/app/12.1.0.2/grid/bin/cluutil -exec -keyexists -key checkpoints.firstnode '
2014-09-04 13:46:05: Removing file /tmp/fileRiu5NI
2014-09-04 13:46:05: Successfully removed file: /tmp/fileRiu5NI
2014-09-04 13:46:05: pipe exit code: 256
2014-09-04 13:46:05: /bin/su exited with rc=1

2014-09-04 13:46:05: oracle.ops.mgmt.rawdevice.OCRException: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]

2014-09-04 13:46:05: Cannot get OCR key with CLUUTIL, try using OCRDUMP.
2014-09-04 13:46:05: Check OCR key using ocrdump
2014-09-04 13:46:22: ocrdump output: PROT-302: Failed to initialize ocrdump

2014-09-04 13:46:22: The key pair with keyname: SYSTEM.rootcrs.checkpoints.firstnode does not exist in OCR.
2014-09-04 13:46:22: Checking a remote host sipr0-db03 for reachability...

3. ocrdump fails due to error AMDU-00201 and AMDU-00200 를 확인다.

<ADR_HOME>/crs/<node>/crs/trace/ocrdump_<pid>.trc

2014-09-04 13:46:14.044274 : OCRASM: proprasmo: ASM instance is down. Proceed to open the file in dirty mode.

CLWAL: clsw_Initialize: Error [32] from procr_init_ext
CLWAL: clsw_Initialize: Error [PROCL-32: Oracle High Availability Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]] from procr_init_ext
2014-09-04 13:46:14.050831 : GPNP: clsgpnpkww_initclswcx: [at clsgpnpkww.c:351] Result: (56) CLSGPNP_OCR_INIT. (:GPNP01201:)Failed to init CLSW-OLR context. CLSW Error (3): CLSW-3: Error in the cluster registry (OCR) layer. [32] [PROCL-32: Oracle High Availability Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]]
2014-09-04 13:46:14.093544 : OCRASM: proprasmo: Error [13] in opening the GPNP profile. Try to get offline profile
2014-09-04 13:46:16.210708 : OCRRAW: kgfo_kge2slos error stack at kgfolclcpi1: AMDU-00200: Unable to read [32768] bytes from Disk N0050 at offset [140737488355328]
AMDU-00201: Disk N0050: '/dev/sdg'
AMDU-00200: Unable to read [32768] bytes from Disk N0049 at offset [140737488355328]
AMDU-00201: Disk N0049: '/dev/sdf'
AMDU-00200: Unable to read [32768] bytes from Disk N0048 at offset [140737488355328]
AMDU-00201: Disk N0048: '/dev/sde'
AMDU-00200: Unable to read [32768] bytes from Disk N0035 at offset [140737488355328]
AMDU-00201: Disk N0035: '/dev/sdaw'
AMDU-00200: Unable to read [32768] bytes from Disk N0024 at offset [140737488355328]
AMDU-00201: Disk N0024: '/dev/sdaq'
....

2014-09-04 13:46:16.212934 : OCRASM: proprasmo: Failed to open file in dirty mode

2014-09-04 13:46:16.212964 : OCRASM: proprasmo: dgname is [OCRVOTE] : discoverystring []
2014-09-04 13:46:16.212990 : OCRASM: proprasmo: Error in open/create file in dg [OCRVOTE]
OCRASM: SLOS : SLOS: cat=8, opn=kgfolclcpi1, dep=200, loc=kgfokge

2014-09-04 13:46:16.213075 : OCRASM: ASM Error Stack :

....
2014-09-04 13:46:22.690905 : OCRASM: proprasmo: kgfoCheckMount returned [7]
2014-09-04 13:46:22.690933 : OCRASM: proprasmo: The ASM instance is down
2014-09-04 13:46:22.692150 : OCRRAW: proprioo: Failed to open [+OCRVOTE/sipr0-dbhv1/OCRFILE/registry.255.857389203]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2014-09-04 13:46:22.692204 : OCRRAW: proprioo: No OCR/OLR devices are usable
2014-09-04 13:46:22.692239 : OCRRAW: proprinit: Could not open raw device
2014-09-04 13:46:22.692561 : default: a_init:7!: Backend init unsuccessful : [26]
2014-09-04 13:46:22.692777 : OCRDUMP: Failed to initailized OCR context. Error [PROC-26: Error while accessing the physical storage
] [26].
2014-09-04 13:46:22.692822 : OCRDUMP: Failed to initialize ocrdump stage 2
2014-09-04 13:46:22.692864 : OCRDUMP: Exiting [status=failed]...

4. Case가 2~3가지 정도 되는데 해당 Case 인 경우 patch 18456643 를 받아서 적용해준다.

  patch 는 모든 노드에 다 적용해 주어야 한다.(이게 거지같은게 이미 grid가 설치되어 있어야 패치가 적용되므로 어차피 한번은 설치를
  실패하고 진행해야 함)
   
5.  이미 작업이 실패해서 1번 노드에 grid 가 올라와 있으므로 deconfig 해주고 패치 일괄적용.

root.sh 를 자동으로 모든노드에서 수행하게 했을 경우 2번,3번 노드에 실패한 정보가 있으므로 역시 2번,3번도 deconfig 해줘야 한다.

(설치시 root.sh 를 자동으로 수행하는 옵션이 있는데 절대 하지 말자..노드순서가 꼬임)

2번,3번에서
$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

1번에서
$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode

deconfig 안해주면 cluster patch 정보가 맞지 않아 기동시 다음과 같은 에러 발생

>  CRS-6706: Oracle Clusterware Release patch level ('4137922036') does not match Software patch level ('0'). Oracle Clusterware cannot be started.

(문서 ID 1639285.1)
해당 명령어로 각 노드의 패치를 확인한다.

<GI_HOME>/bin/kfod op=patches
<GI_HOME>/bin/kfod op=patchlvl

export ORACLE_HOME=/oragrid/product/12.1.0.2
opatch apply /oragrid/18456643

opatch rollback -id 18456643

버전 같은데 동일한 에러날 경우 reboot 하면 해결된다.

6. 다시 순서대로 root.sh 를 돌려서 실행한다.

root.sh 이후의 실패해서 GUI 화면이 사라졌으므로 config 작업은

$GRID_HOME/cfgtoollogs/configToolAllCommands 을 수행해서 마무리 한다.