解決
把服務器名修改為原來的,重啟服務器。
故障
建表空間失敗。
分析
查看告警日志
ORA-1119 signalled during: create tablespace splex datafile ‘+DATA’ size 2000M…
Tue May 20 18:04:28 2025
create tablespace splex datafile ‘+DATA/option/datafile/splex01.dbf’ size 2G
WARNING: ASM communication error: op 11 state 0x50 (3113)
ERROR: slave communication error with ASM
ORA-1119 signalled during: create tablespace splex datafile ‘+DATA/option/datafile/splex01.dbf’ size 2G…
往前查,有其他ORA報錯,最早發生在Mar 19.
Wed Mar 19 11:25:50 2025
NOTE: ASMB terminating
Errors in file /u01/app/oracle/diag/rdbms/option/Option/trace/Option_asmb_3250.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 1135 Serial number: 27
Errors in file /u01/app/oracle/diag/rdbms/option/Option/trace/Option_asmb_3250.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 1135 Serial number: 27
ASMB (ospid: 3250): terminating the instance due to error 15064
Wed Mar 19 11:25:51 2025
System state dump requested by (instance=1, osid=3250 (ASMB)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/option/Option/trace/Option_diag_3224_20250319112551.trc
Dumping diagnostic data in directory=[cdmp_20250319112551], requested by (instance=1, osid=3250 (ASMB)), summary=[abnormal instance termination].
Instance terminated by ASMB, pid = 3250
偵聽看不到asm實例
lsnrctl status
正常情況可以看到asm實例和數據庫實例。但是看不到asm實例。判斷
asm運行故障。
懷疑存儲故障
查看硬盤沒有故障燈。
查MOS錯誤代碼ORA-29701
- Preliminary Checks When “ERROR: unrecoverable error ORA-29701 raised in ASM I/O path” in Oracle Restart Setup (Doc ID 1473701.1)
提到的oracle,grid的權限,group等, 經檢查,都沒有問題。
溝通
跟客戶溝通,確認客戶前段時間修改了服務器名,從option修改為 db-option。
嘗試重啟數據庫和has
關閉數據庫時報錯,直接kill -9殺死數據庫和asm的pmon進程
手動啟動has報錯
[root@db-option init.d]# /u01/app/11.2.0/grid/bin/crsctl start has
CLSU-00100: Operating System function: opendir failed with error data: 2
CLSU-00101: Operating System error message: No such file or directory
CLSU-00103: error location: scrsearch1
CLSU-00104: additional error information: cant open scr home dir scls_scr_getval
CRS-4000: Command Start failed, or completed with errors
chatgpt分析:
ls -ld /etc/oracle/scls_scr 這個目錄有問題
檢查發現目錄名字是原來的主機名, 啟動時讀取這個而目錄失敗。
所以考慮改名會解決。
服務器改名并重啟
#hostnamectl set-hostname option
重啟服務器。
asm和數據庫自動啟動正常。
建表空間正常。
總結
單實例+ASM數據庫,管理和RAC類似,對OS的IP, 主機名的修改不像文件系統那么簡單。
一般來說,在部署前應該確定服務器名,數據庫名,IP等。修改要慎重。如果要修改,參考文檔
- How to change Hostname / IP for a Grid Infrastructure Oracle Restart
Standalone Configuration (SIHA) 11.2 and Later (Doc ID 1552810.1)
1) Configure the CSS & OHAS services as root user as follows:
# <Grid Infrastructure Oracle Home>/crs/install/roothas.pl -deconfig -force
Please note that from 12.1.0.2 onwards need to run roothas.sh instead of roothas.pl
2) Now Change the hostname
3) Reconfigure the cluster
# cd <11.2 Grid infrastructure Oracle Home>
# ./root.sh
For 12.1.0.2 and later releases ,
Please run <Grid infrastructure Oracle Home>/crs/install/roothas.sh
4) Go to the grid home bin directory.
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=eon8x73rq_280&id=1552810.1
2/8Use the srvctl add database command to add the database in an Oracle Restart configuration. If you have
role separation then use the DB home owner for this.
5) Please perform the next steps as oracle or grid OS user (as the Grid Infrastructure OS owner):
5/21/25, 11:00 AM
Document 1552810.1
$> <Grid Infrastructure Oracle Home>/bin/crsctl modify resource "ora.cssd" -
init -attr "AUTO_START=1" -unsupported
NOTE: "-unsupported" is not required for 11.2 version
6) Restart the OHAS stack as grid or oracle OS user (as the Grid Infrastructure OS owner):
$> <Grid Infrastructure Oracle Home>/bin/crsctl stop has
$> <Grid Infrastructure Oracle Home>/bin/crsctl start has
7) Check the CSS & OHAS state as grid or oracle OS user (as the Grid Infrastructure OS owner):
$> <Grid Infrastructure Oracle Home>/bin/crsctl check has
$> <Grid Infrastructure Oracle Home>/bin/crsctl check css
$> <Grid Infrastructure Oracle Home>/bin/ crsctl stat resource
$> <Grid Infrastructure Oracle Home>/bin/crsctl stat res -t
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=eon8x73rq_280&id=1552810.1
3/85/21/25, 11:00 AM
Document 1552810.1
Note: If the CSS & OHAS service did NOT start, then you will need to reboot the Linux/unix box and check
them again.
8) Recreate the default listener (LISTENER) using port 1521 (or using your desired port), thru the NETCA GUI
located on the new Grid Infrastructure Oracle Home (or manually if you do not have graphical access) as grid
or oracle OS user (as the Grid Infrastructure OS owner):
$> srvctl add listener
$> srvctl start listener
9) Please create the init+ASM.ora file on the <Grid Infrastructure Oracle Home>/dbs directory with the next
parameters:
asm_diskgroups= <list of diskgroups>
asm_diskstring= '/dev/oracleasm/disks/*' ?= use the existing asm_diskstring
value. This can be found with "asmcmd dsget" command.
instance_type='asm'
large_pool_size=12M
10) Add the ASM instance as grid or oracle OS user (as the Grid Infrastructure OS owner):
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=eon8x73rq_280&id=1552810.1
4/85/21/25, 11:00 AM
Document 1552810.1
$> <Grid Infrastructure Oracle Home>/bin/srvctl add asm
$> <12.1 Grid Infrastructure Oracle Home>/bin/srvctl add asm
11) Enable ASM instance Auto Start as grid or oracle OS user (as the Grid Infrastructure OS owner) as
follow:
$> <Grid Infrastructure Oracle Home>/bin/crsctl modify resource "ora.asm" -init
-attr "AUTO_START=1" -unsupported
NOTE: "-unsupported" is not required for 11.2 version
12) Make sure the disks are discovered by kfod as grid or oracle OS user (as the Grid Infrastructure OS
owner) as follow:
Example:
$> <Grid Infrastructure Oracle Home>/bin/kfod asm_diskstring='ORCL:*' disks=all
?= if using ASMlib, otherwise use below.
Or
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=eon8x73rq_280&id=1552810.1
5/85/21/25, 11:00 AM
Document 1552810.1
$> <Grid Infrastructure Oracle Home>/bin/kfod asm_diskstring='<full path ASM
disks location>/*' disks=all
13) If so, then startup the ASM instance as grid or oracle OS user (as the Grid Infrastructure OS owner) as
follow:
$> export ORACLE_SID=+ASM
$> <Grid Infrastructure Oracle Home>/bin/sqlplus “/as sysasm”
SQL> startup pfile=init+ASM.ora --#init file from point #6
SQL> show parameter asm
14) Validate that the candidate disks are being discovered:
SQL> select path from v$asm_disk;
15) Create a new ASM instance spfile:
SQL> create spfile from pfile;
16) Add the new ASM spfile and listener to the new ASM instance resource:
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=eon8x73rq_280&id=1552810.1
6/85/21/25, 11:00 AM
Document 1552810.1
$> <Grid Infrastructure Oracle Home>/bin/srvctl modify asm -p <spfile full
path>
$> <Grid Infrastructure Oracle Home>/bin/srvctl modify asm -l LISTENER
17) Validate the OHAS (Oracle Restart) services start as follows:
$> <Grid Infrastructure Oracle Home>/bin/crsctl stop has
$> <Grid Infrastructure Oracle Home>/bin/crsctl start has
$> <Grid Infrastructure Oracle Home>/bin/crsctl stat res
$> <Grid Infrastructure Oracle Home>/bin/crsctl stat res -t