問題描述
填寫問題的基礎信息。
系統名稱 | - |
IP地址 | - |
操作系統 | HP-UNIX |
數據庫 | Oracle 11.2.0.4 兩節點RAC |
癥狀表現
問題的癥狀表現如下
集群的OCR磁盤組掉了一塊盤(/dev/rdisk/disk52):
查詢集群仲裁盤發現只有兩塊(原來是有三塊):
$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------1. ONLINE 08338bbafc7a4fd5bffe759528ad58a0 (/dev/rdisk/disk51) [OCR_VOTE]2. ONLINE 311f8eb857bd4f18bf862055a7f22a0c (/dev/rdisk/disk53) [OCR_VOTE]
Located 2 voting disk(s).
$ ocrcheck
Status of Oracle Cluster Registry is as follows :Version : 3Total space (kbytes) : 262120Used space (kbytes) : 2952Available space (kbytes) : 259168ID : 112220174Device/File Name : +OCR_VOTEDevice/File integrity check succeededDevice/File not configuredDevice/File not configuredDevice/File not configuredDevice/File not configuredCluster registry integrity check succeededLogical corruption check bypassed due to non-privileged user$
查看ASM告警日志發現/dev/rdisk/disk52已經被offline drop了:
處理過程
處理過程推薦按照時間以列表形式,將處理過程時間點,處理內容。
現狀分析:
既然已經被offline_drop了,日常lsdg也不會看到對應磁盤組有offline disk計數信息:
在這樣的情況下,/dev/rhsidk/disk52雖然被踢出ocr磁盤組,但是該磁盤頭信息還有關于ocr_vote磁盤組的元數據信息,如果這個時候再把/dev/rhdisk/disk52添加到OCR_VOTE磁盤組,會報錯如下:
SQL> alter diskgroup OCR_VOTE add disk '/dev/rhsidk/disk52';
alter diskgroup OCR_VOTE add disk '/dev/rhsidk/disk52'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15033: disk '/dev/rhsidk/disk52' belongs to diskgroup "OCR_VOTE"
如果嘗試強制添加該磁盤到OCR_VOTE磁盤組,可能會導致實例終止(來自網絡上的測試):
ocr磁盤組掉盤故障處理 – 提供7*24專業數據庫(Oracle,SQL Server,MySQL,PostgreSQL等)恢復和技術支持@Tel:17813235971 - 惜分飛
解決思路:
1、將/dev/rdisk/hdisk52的磁盤頭信息清除之后再加入到OCR_VOTE磁盤組,保證集群仲裁三副本冗余。
2、存儲上再劃分一小塊大小相同的磁盤,添加進OCR_VOTE磁盤組中,保證集群仲裁三副本冗余。
問題原因
問題原因如下
由于磁盤或者操作系統IO故障導致/dev/rdisk/hdisk52被踢出OCR_VOTE磁盤組,目前集群仲裁盤不能保證三副本冗余。
問題解決
問題解決如下
建議選擇方案2,服務器上盡量不要做清理磁盤頭的操作(不怕萬一,就怕一萬)。等待新的磁盤加入到OCR_VOTE磁盤組后,磁盤組內部會自動進行refresh,新磁盤會被自動同步集群仲裁信息,以下是測試環境中的測試:
測試驗證
先找一個測試環境驗證一下流程
測試環境:Oracle 11.2.0.4 RAC Centos7
準備一個正常的環境:
[grid@centos73 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------1. ONLINE 2654cf6472314f07bfefab8fc83311e4 (/dev/raw/raw1) [OCR]2. ONLINE 9d3d26e93ac14fb2bf7c8b2e2df979bc (/dev/raw/raw2) [OCR]3. ONLINE 0ad3fa5c46874fc4bf5e3e70df47cd3a (/dev/raw/raw3) [OCR]
Located 3 voting disk(s).
[grid@centos73 ~]$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.4.0 Production on Tue Jun 10 16:53:48 2025Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management optionsSQL> set lines 900
col path for a65
select g.name,d.header_status,g.state,d.path from v$asm_disk d,v$asm_diskgroup g where
g.group_number=d.group_number order by g.name;SQL> SQL> 2 NAME HEADER_STATU STATE PATH
------------------------------ ------------ ----------- -----------------------------------------------------------------
DATA MEMBER MOUNTED /dev/raw/raw4
OCR MEMBER MOUNTED /dev/raw/raw2
OCR MEMBER MOUNTED /dev/raw/raw3
OCR MEMBER MOUNTED /dev/raw/raw1
模擬OCR磁盤組掉盤
#正常情況下,磁盤組內磁盤第0個au和第510個au互為鏡像,asm實例主要通過第一個塊來判斷那塊盤屬于哪個磁盤組
[grid@centos73 ~]$ kfed read /dev/raw/raw2 blknum=510 |egrep 'ausize|dsknum|dskname|grpname|fgname|grpnum|dsksize|blksize'
kfdhdb.dsknum: 1 ; 0x024: 0x0001
kfdhdb.dskname: OCR_0001 ; 0x028: length=8
kfdhdb.grpname: OCR ; 0x048: length=3
kfdhdb.fgname: OCR_0001 ; 0x068: length=8
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.dsksize: 2048 ; 0x0c4: 0x00000800
[grid@centos73 ~]$ kfed read /dev/raw/raw2 blknum=0 |egrep 'ausize|dsknum|dskname|grpname|fgname|grpnum|dsksize|blksize'
kfdhdb.dsknum: 1 ; 0x024: 0x0001
kfdhdb.dskname: OCR_0001 ; 0x028: length=8
kfdhdb.grpname: OCR ; 0x048: length=3
kfdhdb.fgname: OCR_0001 ; 0x068: length=8
kfdhdb.blksize: 4096 ; 0x0ba: 0x1000
kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000
kfdhdb.dsksize: 2048 ; 0x0c4: 0x00000800
先將磁盤的前50個au進行備份:
dd if=/dev/raw/raw2 of=/tmp/raw2.dump bs=1048576 count=50
然后破壞一下該磁盤(記得使用conv=notrunc用法):
dd if=/dev/zero of=/dev/raw/raw2 bs=1048576 count=50 conv=notrunc
問題復現:
重啟集群之后,ASM實例報告OCR掉了一塊盤(/dev/raw/raw2):
OTE: Diskgroup used for Voting files is:OCR
Diskgroup with spfile:OCR
Diskgroup used for OCR is:OCR
NOTE: cache registered group DATA number=1 incarn=0x8e3f07c5
NOTE: cache began mount (first) of group DATA number=1 incarn=0x8e3f07c5
NOTE: cache registered group OCR number=2 incarn=0x8e3f07c6
NOTE: cache began mount (first) of group OCR number=2 incarn=0x8e3f07c6
NOTE: Assigning number (1,0) to disk (/dev/raw/raw4)
NOTE: Assigning number (2,0) to disk (/dev/raw/raw1)
NOTE: Assigning number (2,2) to disk (/dev/raw/raw3)
WARNING: GMON has insufficient disks to maintain consensus. Minimum required is 2: updating 2 PST copies from a total of 3.
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 4 for pid 23, osid 9601
NOTE: cache opening disk 0 of grp 1: DATA_0000 path:/dev/raw/raw4
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache mounting (first) external redundancy group 1/0x8E3F07C5 (DATA)
* allocate domain 1, invalid = TRUE
NOTE: attached to recovery domain 1
NOTE: cache recovered group 1 to fcn 0.29494
NOTE: redo buffer size is 256 blocks (1053184 bytes)
NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATA)
Process LGWR (pid 9581) is running at high priority QoS for Exadata I/O
NOTE: LGWR found thread 1 closed at ABA 25.4191
NOTE: LGWR mounted thread 1 for diskgroup 1 (DATA)
NOTE: LGWR opening thread 1 at fcn 0.29494 ABA 26.4192
NOTE: cache mounting group 1/0x8E3F07C5 (DATA) succeeded
NOTE: cache ending mount (success) of group DATA number=1 incarn=0x8e3f07c5
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: GMON heartbeating for grp 2
GMON querying group 2 at 6 for pid 23, osid 9601
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: Assigning number (2,1) to disk ()
GMON querying group 2 at 7 for pid 23, osid 9601
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: cache dismounting (clean) group 2/0x8E3F07C6 (OCR)
NOTE: messaging CKPT to quiesce pins Unix process pid: 9601, image: oracle@centos73 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 2/0x8E3F07C6 (OCR)
NOTE: cache ending mount (fail) of group OCR number=2 incarn=0x8e3f07c6
NOTE: cache deleting context for group OCR 2/0x8e3f07c6
GMON dismounting group 2 at 8 for pid 23, osid 9601
NOTE: Disk OCR_0000 in mode 0x7f marked for de-assignment
NOTE: Disk in mode 0x7f marked for de-assignment
NOTE: Disk OCR_0002 in mode 0x7f marked for de-assignment
ERROR: diskgroup OCR was not mounted
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup DATA was mounted
WARNING: Disk Group OCR containing spfile for this instance is not mounted
WARNING: Disk Group OCR containing configured OCR is not mounted
WARNING: Disk Group OCR containing voting files is not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "1" is missing from group number "2"
這個時候集群是啟動失敗的,卡在了ocr啟動失敗:
[grid@centos73 ~]$ crsctl status res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[grid@centos73 ~]$ crsctl status res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm1 ONLINE INTERMEDIATE centos73 OCR not started
ora.cluster_interconnect.haip1 ONLINE ONLINE centos73
ora.crf1 ONLINE ONLINE centos73
ora.crsd1 ONLINE OFFLINE
ora.cssd1 ONLINE ONLINE centos73
ora.cssdmonitor1 ONLINE ONLINE centos73
ora.ctssd1 ONLINE ONLINE centos73 ACTIVE:0
ora.diskmon1 OFFLINE OFFLINE
ora.evmd1 ONLINE INTERMEDIATE centos73
ora.gipcd1 ONLINE ONLINE centos73
ora.gpnpd1 ONLINE ONLINE centos73
ora.mdnsd1 ONLINE ONLINE centos73
這個時候需要強制mount一下ocr磁盤組:
強制mount之后,集群會以兩個仲裁盤形式運行,同樣的ocr會掉一塊盤:
[grid@centos73 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------1. ONLINE 2654cf6472314f07bfefab8fc83311e4 (/dev/raw/raw1) [OCR]2. ONLINE 0ad3fa5c46874fc4bf5e3e70df47cd3a (/dev/raw/raw3) [OCR]
Located 2 voting disk(s).
[grid@centos73 ~]$ sqlplus / as sysasmSQL*Plus: Release 11.2.0.4.0 Production on Tue Jun 10 17:29:32 2025Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management optionsSQL> set lines 900
col path for a65
select g.name,d.header_status,g.state,d.path from v$asm_disk d,v$asm_diskgroup g where
g.group_number=d.group_number order by g.name;SQL> SQL> 2 NAME HEADER_STATU STATE PATH
------------------------------ ------------ ----------- -----------------------------------------------------------------
DATA MEMBER MOUNTED /dev/raw/raw4
OCR MEMBER MOUNTED /dev/raw/raw3
OCR MEMBER MOUNTED /dev/raw/raw1
asm告警日志也會顯示先前故障磁盤被強制drop了
SUCCESS: diskgroup OCR was mounted
SUCCESS: alter diskgroup ocr mount force
Tue Jun 10 17:26:28 2025
ASM Health Checker found 1 new failures
Tue Jun 10 17:26:29 2025
WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 3 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
ERROR: failed to update diskgroup resource ora.OCR.dg
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
Tue Jun 10 17:26:45 2025
WARNING: PST-initiated drop of 1 disk(s) in group 2(.1618937807))
SQL> alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=2
Tue Jun 10 17:26:48 2025
GMON updating for reconfiguration, group 2 at 23 for pid 26, osid 10010
NOTE: cache closing disk 1 of grp 2: (not open) OCR_0001
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: group 2 PST updated.
Tue Jun 10 17:26:48 2025
NOTE: membership refresh pending for group 2/0x607f07cf (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
GMON querying group 2 at 24 for pid 18, osid 9587
NOTE: cache closing disk 1 of grp 2: (not open) _DROPPED_0001_OCR
SUCCESS: refreshed membership for 2/0x607f07cf (OCR)
NOTE: starting rebalance of group 2/0x607f07cf (OCR) at power 1
SUCCESS: alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */
SUCCESS: PST-initiated drop disk in group 2(1618937807))
Starting background process ARB0
Tue Jun 10 17:26:51 2025
ARB0 started with pid=27, OS id=10023
NOTE: assigning ARB0 to group 2/0x607f07cf (OCR) with 1 parallel I/O
cellip.ora not found.
NOTE: F1X0 copy 2 relocating from 1:2 to 2:2 for diskgroup 2 (OCR)
NOTE: F1X0 copy 3 relocating from 2:2 to 65534:4294967294 for diskgroup 2 (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
Tue Jun 10 17:26:54 2025
NOTE: [crsd.bin@centos73 (TNS V1-V3) 10028] opening OCR file
Starting background process ASMB
Tue Jun 10 17:26:54 2025
ASMB started with pid=29, OS id=10042
NOTE: Successful voting file relocation on diskgroup OCR
Tue Jun 10 17:26:54 2025
NOTE: client +ASM1:+ASM registered, osid 10044, mbr 0x0
Tue Jun 10 17:26:58 2025
NOTE: Rebalance has restored redundancy for any existing control file or redo log in disk group OCR
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 2/0x607f07cf (OCR)
Tue Jun 10 17:27:00 2025
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=2
Tue Jun 10 17:27:03 2025
GMON updating for reconfiguration, group 2 at 25 for pid 27, osid 10142
NOTE: cache closing disk 1 of grp 2: (not open) _DROPPED_0001_OCR
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: group 2 PST updated.
SUCCESS: grp 2 disk _DROPPED_0001_OCR going offline
GMON updating for reconfiguration, group 2 at 26 for pid 27, osid 10142
NOTE: cache closing disk 1 of grp 2: (not open) _DROPPED_0001_OCR
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: group 2 PST updated.
NOTE: membership refresh pending for group 2/0x607f07cf (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
GMON querying group 2 at 27 for pid 18, osid 9587
GMON querying group 2 at 28 for pid 18, osid 9587
NOTE: Disk _DROPPED_0001_OCR in mode 0x0 marked for de-assignment
SUCCESS: refreshed membership for 2/0x607f07cf (OCR)
Tue Jun 10 17:27:09 2025
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Successful voting file relocation on diskgroup OCR
Tue Jun 10 17:28:01 2025
ALTER SYSTEM SET local_listener=' (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.3.113)(PORT=1521))))' SCOPE=MEMORY SID='+ASM1';
Tue Jun 10 17:31:32 2025
這個時候我們再看一下v$asm_disk中的磁盤信息發現/dev/raw/raw2變成了CANDIDATE,這是因為我們先前是把/dev/raw/raw2磁盤頭信息都清除了。
SQL> set lines 900
col path for a65
select g.name,d.header_status,g.state,d.path from v$asm_disk d,v$asm_diskgroup g where
g.group_number(+)=d.group_number order by g.name;SQL> SQL> 2 NAME HEADER_STATU STATE PATH
------------------------------ ------------ ----------- -----------------------------------------------------------------
DATA MEMBER MOUNTED /dev/raw/raw4
OCR MEMBER MOUNTED /dev/raw/raw1
OCR MEMBER MOUNTED /dev/raw/raw3CANDIDATE /dev/raw/raw2
嘗試將/dev/raw/raw2的磁盤頭信息恢復試試,使用下面的方法恢復(一定記得用conv=notrunc):
dd if=/tmp/raw2.dump of=/dev/raw/raw2 bs=1048576 count=50 conv=notrunc
再次查看/dev/raw/raw2對應的header_status變成member了,但是現在它已經不是ocr磁盤組的一部分了
SQL> set lines 900
col path for a65
select g.name,d.header_status,d.mount_status,g.state,d.path from v$asm_disk d,v$asm_diskgroup g where
g.group_number(+)=d.group_number order by g.name;SQL> SQL> 2 NAME HEADER_STATU MOUNT_S STATE PATH
------------------------------ ------------ ------- ----------- -----------------------------------------------------------------
DATA MEMBER CACHED MOUNTED /dev/raw/raw4
OCR MEMBER CACHED MOUNTED /dev/raw/raw1
OCR MEMBER CACHED MOUNTED /dev/raw/raw3MEMBER CLOSED /dev/raw/raw2
嘗試添加/dev/raw/raw2到ocr磁盤組:
SQL> alter diskgroup ocr add disk '/dev/raw/raw2';
alter diskgroup ocr add disk '/dev/raw/raw2'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15033: disk '/dev/raw/raw2' belongs to diskgroup "OCR"
直接報錯,因為該磁盤頭信息顯示該盤屬于ocr磁盤組。
解決方案:
1、將/dev/raw/raw2磁盤頭信息清除再嘗試加入,這個操作要格外小心,不能清除錯盤了。
2、使用add disk force命令強制添加,原理和1差不多,但是網上有強制添加導致實例crash的案例。
3、存儲再重新劃分一塊磁盤添加進去,回收之前的磁盤(比較推薦)。
最終解決效果:
最終選擇add disk force的方式將/dev/raw/raw2加入到ocr磁盤組,生產環境有條件最好還是選擇方案3。
添加完成之后,ocr自動做refresh,將新加入的盤上同步仲裁相關信息:
Wed Jun 11 09:58:50 2025
SQL> alter diskgroup ocr add disk '/dev/raw/raw2' force
NOTE: GroupBlock outside rolling migration privileged region
NOTE: Assigning number (2,1) to disk (/dev/raw/raw2)
NOTE: requesting all-instance membership refresh for group=2
NOTE: initializing header on grp 2 disk OCR_0001
NOTE: requesting all-instance disk validation for group=2
Wed Jun 11 09:58:51 2025
NOTE: skipping rediscovery for group 2/0x8c294763 (OCR) on local instance.
NOTE: requesting all-instance disk validation for group=2
NOTE: skipping rediscovery for group 2/0x8c294763 (OCR) on local instance.
Wed Jun 11 09:58:51 2025
GMON updating for reconfiguration, group 2 at 19 for pid 31, osid 2491
NOTE: group 2 PST updated.
NOTE: initiating PST update: grp = 2
GMON updating group 2 at 20 for pid 31, osid 2491
NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)
NOTE: group OCR: updated PST location: disk 0001 (PST copy 2)
NOTE: PST update grp = 2 completed successfully
NOTE: membership refresh pending for group 2/0x8c294763 (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
NOTE: Failed voting file relocation on diskgroup OCR
GMON querying group 2 at 21 for pid 18, osid 1742
NOTE: cache opening disk 1 of grp 2: OCR_0001 path:/dev/raw/raw2
GMON querying group 2 at 22 for pid 18, osid 1742
SUCCESS: refreshed membership for 2/0x8c294763 (OCR)
NOTE: starting rebalance of group 2/0x8c294763 (OCR) at power 1
SUCCESS: alter diskgroup ocr add disk '/dev/raw/raw2' force
Starting background process ARB0
Wed Jun 11 09:58:54 2025
ARB0 started with pid=28, OS id=4645
NOTE: assigning ARB0 to group 2/0x8c294763 (OCR) with 1 parallel I/O
cellip.ora not found.
NOTE: F1X0 copy 3 relocating from 65534:4294967294 to 1:2 for diskgroup 2 (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup OCR
NOTE: Attempting voting file relocation on diskgroup OCR
Wed Jun 11 09:59:01 2025
NOTE: voting file allocation on grp 2 disk OCR_0001
NOTE: Successful voting file relocation on diskgroup OCR
Wed Jun 11 09:59:06 2025
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=2
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 2/0x8c294763 (OCR)
NOTE: membership refresh pending for group 2/0x8c294763 (OCR)
Wed Jun 11 09:59:12 2025
GMON querying group 2 at 23 for pid 18, osid 1742
Wed Jun 11 09:59:12 2025
SUCCESS: refreshed membership for 2/0x8c294763 (OCR)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 3 voting file(s).
集群仲裁信息恢復三副本: