pat98's always & forever

패치작업 19.17.0.0.221018 (GI PSU 34416665)

Database Patch Set Update : 19.17.0.0.221018 (34419443)
OCW Patch Set Update      : 19.17.0.0.221018 (34444834)
ACFS Patch Set Update     : 19.17.0.0.221018 (34428761)
Tomcat Release Update     : 19.0.0.0.0       (34580338)
DBWLM Release Update      : 19.0.0.0.0       (33575402)

Oracle Grid Infrastructure Patch Set Update 19.17.0.0.221018 
-------------------------------------
GI_HOME, ORACLE_HOME 을 개별로 각각 할때

- grid 유저
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34416665/34419443
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34416665/34444834
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34416665/34428761
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34416665/33575402
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34416665/34580338

- oracle 유저
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34416665/34419443
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34416665/34444834

(oracle)
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>

(root 유저)
export GI_HOME=/u01/app/19.3.0.0/grid
$GI_HOME/crs/install/rootcrs.sh -prepatch 

(grid 유저)
export GI_HOME=/u01/app/19.3.0.0/grid
cd /u01/install
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34416665/34444834 -silent
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34416665/34428761 -silent
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34416665/34419443 -silent
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34416665/33575402 -silent
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34416665/34580338 -silent

(oracle 유저)
export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_1
cd /u01/install
./34416665/34444834/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local ./34416665/34444834 -silent
$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local ./34416665/34419443 -silent
./34416665/34444834/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME 

(root 유저)
export GI_HOME=/u01/app/19.3.0.0/grid
$GI_HOME/rdbms/install/rootadd_rdbms.sh
$GI_HOME/crs/install/rootcrs.sh -postpatch 

(oracle)
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name> 

- Loading Modified SQL Files into the Database
sqlplus /nolog
SQL> conect / as sysdba
SQL> startup
SQL> quit
cd $ORACLE_HOME/OPatch
./datapatch -verbose

### [롤백하는 경우] ###########

(oracle)
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>

GI Home
(root로)
$GI_HOME/crs/install/rootcrs.sh -prepatch -rollback

(grid 유저로)
export GI_HOME=/u01/app/19.3.0.0/grid
cd /u01/install
$GI_HOME/OPatch/opatch nrollback -local -id 34444834,34428761,34419443,33575402,34580338 -oh $GI_HOME -silent

(oracle 유저로)
export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_1
cd /u01/install
./34416665/34444834/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME 
$ORACLE_HOME/OPatch/opatch nrollback -local -id 34444834,34419443 -oh /u01/app/oracle/product/19.3.0.0/dbhome_1 -silent
./34416665/34444834/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME

Run post script
(root로)
export GI_HOME=/u01/app/19.3.0.0/grid
$GI_HOME/rdbms/install/rootadd_rdbms.sh
$GI_HOME/crs/install/rootcrs.sh -postpatch -rollback

(oracle)
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>

sqlplus /nolog
SQL> conect / as sysdba
SQL> startup
SQL> quit
cd $ORACLE_HOME/OPatch
./datapatch -verbose

갑자기 잘 되면 OEDA 가 화면이 깨지면서 못쓰게 됐다. 어떤 버전은 잘 되고 어떤 버전은 안되고..

Chrome 등의 캐시 문제인줄 알았으나 삽질 하다가 포트를 변경하니 정상적으로 된다...어휴...식빵#$%@RD^%$%

원래 기본 포트로는 7072번이 사용된다.

D:\onecommand\p30640393_2121100_WINNT\windows-i586>installOedaServer.cmd 
Log of actions is located in file : oeda-2022-08-29-225046.log
 Starting HTTP Server ...
 Server URL: http://localhost:7072/oeda
 Server started successfully ...
 Jetty server is accessible locally. To make the server accessible remotely re-install oedaServer using -g flag

이것저것 삽질하다 안되서 7000번으로 바꿔 봄...

성공!!

D:\onecommand\p30640393_2121100_WINNT\windows-i586>installOedaServer.cmd -p 7000
Log of actions is located in file : oeda-2022-08-29-225046.log
 Starting HTTP Server ...
 Server URL: http://localhost:7000/oeda
 Server started successfully ...
 Jetty server is accessible locally. To make the server accessible remotely re-install oedaServer using -g flag

PDU Default Password Changed After Firmware 1.06 and Above on Sun Rack II platforms and Engineered Systems (Doc ID 1570252.1)

-> 가용성 테스트 때문에 한쪽 PDU 만 껏다 켰더니 PDU 양쪽의 암호가 서로 다른 현상으로 고생했다.

Firmware 업그레이드 한 후에 Factory Reset 이 되게 되는데 암호가 달라져서 로그인이 안되는 문제임.

기존의

PUD Firmware 1.05 및 이하는 admin/admin 

-> 문서에는 이렇게 나왔는데 실제로는 admin/welcome1 으로 접속해야지 정상 로그인이 된다.

업데이트 된 

PUD Firmware 1.06 및 이상은 admin/adm1n (알파벳 i 가 아니라 숫자 1로 바뀜)

별로 중요하지도 않은 건데 왜 자꾸 암호를 변경하는지 이해불가하다..절레절레

패치작업 11.2.0.4.180717 (GI PSU 27967757)

Database Patch Set Update : 11.2.0.4.180717 (27734982)
OCW Patch Set Update      : 11.2.0.4.180717 (27441052)
ACFS PATCH SET UPDATE     : 11.2.0.4.180717 (27959254)


Oracle Grid Infrastructure Patch Set Update 11.2.0.4.180717 (Includes Database PSU 11.2.0.4.180717) 
-------------------------------------
GRID_HOME, ORACLE_HOME 을 개별로 각각 할때

(root 유저)
$GRID_HOME/crs/install/rootcrs.pl -unlock (CRS 가 떠 있으면 script로 자동으로 내려버림)

(grid 유저)

export GRID_HOME=/oragrid/product/11.2.0.4
cd <UNZIPPED_PATCH_LOCATION>

$GRID_HOME/OPatch/opatch napply -oh $GRID_HOME -local ./27967757/27441052
$GRID_HOME/OPatch/opatch napply -oh $GRID_HOME -local ./27967757/27959254
$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local  ./27967757/27734982

(oracle 유저)

export ORACLE_HOME=$ORACLE_BASE/product/11.2.0.4
cd <UNZIPPED_PATCH_LOCATION>

./27967757/23054319/custom/server/23054319/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME (권한체크)
$ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local ./27967757/27441052/custom/server/27441052
$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local ./27967757/27734982
./27967757/23054319/custom/server/23054319/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME (권한체크)


(root 유저)
$GRID_HOME/rdbms/install/rootadd_rdbms.sh
$GRID_HOME/crs/install/rootcrs.pl -patch (script로 자동으로 CRS를 올려 버림)



### [롤백하는 경우] ###########
GI Home
(root로)
$GRID_HOME/crs/install/rootcrs.pl -unlock

(grid 유저로)

$GRID_HOME/OPatch/opatch rollback -local -id 27441052 -oh $GRID_HOME
$GRID_HOME/OPatch/opatch rollback -local -id 27959254 -oh $GRID_HOME
$GRID_HOME/OPatch/opatch rollback -local -id 27734982 -oh $GRID_HOME

(oracle 유저로)

cd <UNZIPPED_PATCH_LOCATION>
./27967757/27441052/custom/server/27441052/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME (권한체크)
$ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local ./27967757/27441052/custom/server/27441052
$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local ./27967757/27734982
./27967757/27441052/custom/server/27441052/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME (권한체크)

Run post script
(root로)
$GRID_HOME/rdbms/install/rootadd_rdbms.sh
$GRID_HOME/crs/install/rootcrs.pl -patch

Exachk 가 자동 Scheduling 되지 않는 Bug가 있음

해당 버전은 : 22.1.1.0.0 임

Bug 34276627 - EXACHK 22.1.1.0.0 SCHEDULED RUN NOT RUNNING

On TFAMain Startup the cron schedule for EXAchk/ORAchk is loaded on
the TFA Master node allowing for EXAchk/ORAchk to manage its own processing across the cluster.
If the master node changes (e.g. Restart AHF on the master) the cron schedule
is not loaded on the new master resulting in future EXAchk runs not executing

Workaround 해결방법

tfactl refreshconfig cron

또는 

AHF 22.2.1 으로 upgrade 할것.

최신 OECA 를 살펴보다가 우연히 발견된 X10M-2 그림인데

DB NODE 모델은 앞면은 아래와 2U 로 바뀌고 Cell Node 쪽은 22 TB 로 업그레이드 될것으로 예상됨.

아님말고..

패치작업 12.2.0.1.220118 (GI PSU 33583921 )

DB RU    : 12.2.0.1.220118 (33587128)
OCW RU   : 12.2.0.1.220118 (33678030)
ACFS RU  : 12.2.0.1.210720 (33116894)
DBWLM    : 12.2.0.1.170913 (26839277)
TOMCAT   : 12.2.0.1.0      (33610989)

Oracle Grid Infrastructure Patch Set Update 12.2.0.1.220118 (Includes Database PSU 12.2.0.1.220118) 
-------------------------------------
GRID_HOME, ORACLE_HOME 을 개별로 각각 할때

- grid 
export ORACLE_HOME=$ORACLE_BASE/product/12.2.0.1/dbhome_1
$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/33583921/33587128
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/33583921/33678030
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/33583921/33116894
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/33583921/26839277
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/33583921/33610989

- oracle 
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/33583921/33587128
% $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/33583921/33678030

- oracle 
$<ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>

(root)
export GRID_HOME=/u01/app/12.2.0.1/grid
$GRID_HOME/crs/install/rootcrs.sh -prepatch 

$GRID_HOME/crs/install/rootcrs.sh -prepatch -nonrolling (non 롤링으로 하고자 할때)

(grid)
export OPATCH_DEBUG=true

export GRID_HOME=/u01/app/12.2.0.1/grid
cd /home/oracle

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local ./33583921/33678030 -silent
$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local ./33583921/33116894 -silent
$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local ./33583921/26839277 -silent
$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local ./33583921/33587128 -silent
$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local ./33583921/33610989 -silent  

(oracle)
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/12.2.0.1/dbhome_1
cd <UNZIPPED_PATCH_LOCATION>

./33583921/33678030/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local ./33583921/33678030 -silent
$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local ./33583921/33587128 -silent
./33583921/33678030/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME 

(root)
export GRID_HOME=/u01/app/12.2.0.1/grid
$GRID_HOME/rdbms/install/rootadd_rdbms.sh
$GRID_HOME/crs/install/rootcrs.sh -postpatch 

$GRID_HOME/crs/install/rootcrs.sh -postpatch -nonrolling (non 롤링으로 하고자 할때)

(oracle)
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>

- Loading Modified SQL Files into the Database

sqlplus /nolog
SQL> conect / as sysdba
SQL> startup
SQL> quit
cd $ORACLE_HOME/OPatch
./datapatch -verbose

* EM 13.5 OMS 및 agent 패치작업

월별로 주기적으로 패치가 Release 되기 때문에 작업전 항상 패치툴과 패치파일 최신 리스트를 확인 해야함.

또한 Database의 opatch 와는 별도로 OMS Patcher/AgentPatcher 가 존재하며 이를 이용해서 각각 패치작업을 진행하여야 함.

13.5의 경우는 해당 문서를 참조 Overview of the Enterprise Manager Proactive Patch Program (Doc ID 822485.1)

- OMSPatcher 업그레이드

p19999993_135000_Generic.zip (OMS Patcher 13.9.5.5화일)
p34003602_135000_Generic.zip (OMS 13.5.0.7 화일) 사이즈는 1.2G

1. p19999993_135000_Generic.zip 을 OMS 홈으로 복사 $OMS_HOME/OMSPatcher
2. 기존꺼 백업
   mv $OMS_HOME/OMSPatcher $OMS_HOME/OMSPatcher_old
   chown oraem:pkggrp p19999993_135000_Generic.zip
3. 압축 해제 p19999993_135000_Generic.zip

4. 버전 확인
$OMS_HOME/OMSPatcher> ./omspatcher version

OMSPatcher Version: 13.9.5.5.0
OPlan Version: 12.2.0.1.16
OsysModel build: <timestamp>

OMSPatcher succeeded.

- OMSPatch 작업
화일 업로드
1. 압축 풀고
p34003602_135000_Generic.zip

Admin 서버 때문에 OMS 내리지 않고 작업함.

export ORACLE_HOME=/engn/oraem/em13c/mw
cd /engn/oraem/em13c/mw/OMSPatcher
./omspatcher apply /engn/oraem/em13c/mw/34003602

weblogic유저/sys유저 암호입력후 
Do you want to proceed? [y|n] Y

패치가 끝나면 자동으로 올림

* 작업은 OMS Patch 가 선행되어 있어야 함.

- AgentPatcher 업그레이드 (13.5.0.7)

p34129921_135000_Generic.zip (패치파일, 42M)
p33355570_135000_Generic.zip (AgentPatcher)

1. p33355570_135000_Generic.zip 을 Agent 홈으로 복사 <Agent_Base_Directory>/agent_13.5.0.0.0

2. 기존꺼 백업
   mv <Agent ORACLE_HOME>/AgentPatcher <Agent ORACLE_HOME>/AgentPatcher_old

3. 압축 해제 unzip p33355570_135000_Generic.zip

4. 버전 확인
cd $ORACLE_HOME/AgentPatcher
$ ./agentpatcher version
AgentPatcher Version: 13.9.5.4.0
OPlan Version: 12.2.0.1.16

- AgentPatch 작업

1. 압축 풀고
unzip p34129921_135000_Generic.zip
cd <PATCH_TOP_DIR>/34129921
emctl stop agent
agentpatcher apply
emctl start agent 

_ksipc_service_mask

. 의미 : MGA 는 12c 부터 도입된 NF. 12c default 는 disable 이나, 18c부터 Default로 Enable 되었음.
        process 별로 사용하였던 pga memory 를 ipc 를 통해 Shared 영역을 일정부분 공유해서 사용 가능
        메모리 공유가 필요할 때 ipc 를 이용하여 namespace 를 attach 하는데,
        이를 latch: MGA shared context root latch/ latch: MGA shared context latch로 관리
        . latch: MGA shared context root latch - mga 를 생성, 삭제, attach, detach 하는데 필요
        . latch: MGA shared context latch - mga 를 관리하는데 (namespace handler 처리, pid 체크 및 shared context operation/ metadata allocation) 필요

. 권고 : 운영 중 / reconfiguration / Restart 이후  이후 CLxx failed to acquire latch 대량 발생으로
        Process clear가 되지 않아, DB Hang 및 Session Failove가 되지 않는 현상 발생으로 Disable (0) 권고 (성능 영향 없음)
. 참조 : Bug 31410321 21.1 RDBMS 21.1 VOS PRODID-5 PORTID-226 ORA-445 (현재 분석 진행 중인 Bug)
       Bug 30293345 - Waits for latch: MGA Shared Context Latch After Migration to 18c (Interim 존재)
       Database Connection is Delayed by latch: MGA shared context latch (Doc ID 2706666.1) - RAC 환경에서 connection delay 발생

. 참고 : 0 설정 후 값이 1024 (1K)로 조회가 되는데 정상적으로 Disable 적용된 것임

Database Patch Set Update : 19.16.0.0.220719 (34133642)
OCW Patch Set Update      : 19.16.0.0.220719 (34160635)
ACFS Patch Set Update     : 19.16.0.0.220719 (34139601)
Tomcat Release Update     : 19.0.0.0.0       (34318175)
DBWLM Release Update      : 19.0.0.0.0       (33575402)

Oracle Grid Infrastructure Patch Set Update 19.16.0.0.220719 
-------------------------------------
GI_HOME, ORACLE_HOME 을 개별로 각각 할때

- grid 유저
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34130714/34133642
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34130714/34160635
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34130714/34139601
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34130714/33575402
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34130714/34318175

- oracle 유저
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34130714/34133642
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir <UNZIPPED_PATCH_LOCATION>/34130714/34160635

(oracle)
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>

(root 유저)
export GI_HOME=/u01/app/19.3.0.0/grid
$GI_HOME/crs/install/rootcrs.sh -prepatch 

(grid 유저)
export GI_HOME=/u01/app/19.3.0.0/grid
cd /u01/install
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34130714/34160635 -silent
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34130714/34139601 -silent
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34130714/34133642 -silent
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34130714/33575402 -silent
$GI_HOME/OPatch/opatch apply -oh $GI_HOME -local ./34130714/34318175 -silent

(oracle 유저)
export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_1
cd /u01/install
./34130714/34160635/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME
$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local ./34130714/34160635 -silent
$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local ./34130714/34133642 -silent
./34130714/34160635/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME 

(root 유저)
export GI_HOME=/u01/app/19.3.0.0/grid
$GI_HOME/rdbms/install/rootadd_rdbms.sh
$GI_HOME/crs/install/rootcrs.sh -postpatch 

(oracle)
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name> 

- Loading Modified SQL Files into the Database
sqlplus /nolog
SQL> conect / as sysdba
SQL> startup
SQL> quit
cd $ORACLE_HOME/OPatch
./datapatch -verbose

### [롤백하는 경우] ###########

(oracle)
$ <ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>

GI Home
(root로)
$GI_HOME/crs/install/rootcrs.sh -prepatch -rollback

(grid 유저로)
export GI_HOME=/u01/app/19.3.0.0/grid
cd /u01/install
$GI_HOME/OPatch/opatch nrollback -local -id 34160635,34139601,34133642,33575402,34318175 -oh $GI_HOME -silent

(oracle 유저로)
export ORACLE_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_1
cd /u01/install
./34130714/34160635/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME 
$ORACLE_HOME/OPatch/opatch nrollback -local -id 34160635,34133642 -oh /u01/app/oracle/product/19.3.0.0/dbhome_1 -silent
./34130714/34160635/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME

Run post script
(root로)
export GI_HOME=/u01/app/19.3.0.0/grid
$GI_HOME/rdbms/install/rootadd_rdbms.sh
$GI_HOME/crs/install/rootcrs.sh -postpatch -rollback

(oracle)
$ <ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>

sqlplus /nolog
SQL> conect / as sysdba
SQL> startup
SQL> quit
cd $ORACLE_HOME/OPatch
./datapatch -verbose