Oracle 19.20未知BUG導致oraagent進程內存泄漏

故障現象

查詢操作系統進程的使用排序，這里看到oraagent的物理內存達到16G，遠遠超過正常環境（正常環境在19.20大概就是100M多一點）

[root@orastd tmp]# ./hmem|more
PID      NAME                     VIRT(kB)   SHARED(kB)      RSS(kB)  PRIVATE(kB)
2102     oraagent.bin             18212328        47160     16215456     16215456

數據庫版本

獲得數據庫的版本為19.20

[grid@orastd trace]$ $ORACLE_HOME/OPatch/opatch lspatches35332537;ACFS RELEASE UPDATE 19.20.0.0.0 (35332537)
35320081;Database Release Update : 19.20.0.0.230718 (35320081)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

分析過程

獲得內存使用詳細信息

獲得內存的詳細信息

[root@orastd tmp]# ./hmem -l|more
PID      NAME                     VIRT(kB)   SHARED(kB)      RSS(kB)  PRIVATE(kB)    RssAnon    RssFile   RssShmem      VmRSS     VmData
2102     oraagent.bin             18212328        47160     16215456     16215456   16168296      47160          0   16215456   17850384

RssAnon標識進程的私有內存，常住在內存中，其實可以理解為DATA部分的內存，這里可以肯定oraagent存在內存泄漏了。

查看oraagent日志

oraagent部分一直不斷的在重復顯示下面內容，這里面有個奇怪的現象就是ora.LISTENER.lsnr，但是我集群中并沒有這個資源

2025-07-24 23:04:11.272 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateEndPoints 2040 Listener ResName:ora.LISTENER.lsnr
2025-07-24 23:04:11.272 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateHostName entry lsnrResName:ora.LISTENER.lsnr
2025-07-24 23:04:11.272 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::getLsnrResNameAddrType &s_lsnrResNametoAddrTypeXLock:0x55555a209718get resname:ora.LISTENER.lsnr addressType
2025-07-24 23:04:11.272 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::getAddressTypeValue 000 entry { actx:(nil) lsnrName:ora.LISTENER.lsnr
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} Agent::getValueFromNetwork 000 { get netParam:ADDRESS_TYPE from vipResName: defaultNetRes: tag:
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} Agent::getValueFromNetwork 999 } netParam:ADDRESS_TYPE vipResName: defaultNetRes: networkResName: netParamValue:IPV4
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::getAddressTypeValue 220 lsnrResNametoAddrType lsnrNetworkMap lsnrName:ora.LISTENER.lsnr networkResName:
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::getLsnrResNameAddrType &s_lsnrResNametoAddrTypeXLock:0x55555a209718lsnrNetworkMap[ora.LISTENER.lsnr]: networkAddressTypeMap[]:IPV4
2025-07-24 23:04:11.277 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateHostName getNodeName 1
2025-07-24 23:04:11.278 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateHostName exit hostIP:192.168.1.223  hostIPV6:
2025-07-24 23:04:11.281 : USRTHRD:3858700032: [     INFO] {0:0:2} checkCrsRet clscrsret: 5
2025-07-24 23:04:11.281 : USRTHRD:3858700032: [     INFO] {0:0:2} CrsCmd::ClscrsCmdData:statGetAttr 040 error getting attribute:ENDPOINTS@SERVERNAME(orastd)
2025-07-24 23:04:11.281 : USRTHRD:3858700032: [     INFO] {0:0:2} CrsCmd::statGetAttr 120 error
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateEndPoints 2311 lsnrResName:ora.LISTENER.lsnr endpAttr:TCP:1521
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::ifFirewallParam entry
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::ifFirewallParam exit
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateEndPoints 2511 type:2 vipVector endpString:(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.223)(PORT=1521))
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::removeDuplicates
2025-07-24 23:04:11.284 : USRTHRD:3858700032: [     INFO] {0:0:2} LsnrAgent::generateEndPoints 1999 exit }

因為是備庫環境，監聽用的1522端口，所以沒有默認的ora.LISTENER.lsnr，難道跟這個有關系？

監控oraagent內存的變換

可以看到每個10分鐘，大概新增加如下的物理內存的使用量。

[root@orastd tmp]# sh ./count_diff_rss.sh  /tmp/ps_2102.log
2025-07-24 14:13:14: 164
2025-07-24 14:23:15: 96
2025-07-24 14:33:15: 272
2025-07-24 14:43:15: 164
2025-07-24 14:53:15: 184
2025-07-24 15:03:15: 72
2025-07-24 15:13:15: 180
2025-07-24 15:23:15: 140
2025-07-24 15:33:15: 192
2025-07-24 15:43:15: 156
2025-07-24 15:53:16: 184
2025-07-24 16:03:16: 196
2025-07-24 16:13:16: 164
2025-07-24 16:23:16: 148
2025-07-24 16:33:16: 228
2025-07-24 16:43:16: 164
2025-07-24 16:53:16: 196
2025-07-24 17:03:16: 92
2025-07-24 17:13:16: 264
2025-07-24 17:23:16: 176
2025-07-24 17:33:16: 180
2025-07-24 17:43:16: 160
2025-07-24 17:53:16: 200
2025-07-24 18:03:16: 164
2025-07-24 18:13:16: 76
2025-07-24 18:23:17: 292
2025-07-24 18:33:17: 48
2025-07-24 18:43:17: 184
2025-07-24 18:53:17: 168
2025-07-24 19:03:17: 128
2025-07-24 19:13:17: 196
2025-07-24 19:23:17: 192
2025-07-24 19:33:17: 144
2025-07-24 19:43:17: 220
2025-07-24 19:53:17: 164
2025-07-24 20:03:18: 200
2025-07-24 20:13:18: 112
2025-07-24 20:23:18: 232
2025-07-24 20:33:18: 192
2025-07-24 20:43:18: 184
2025-07-24 20:53:18: 160
2025-07-24 21:03:18: 192
2025-07-24 21:13:18: 164
2025-07-24 21:23:18: 84

解決方案

從上面的日志中可以懷疑oraagent在不斷判斷對listener的資源時可能異常，導致申請的內存一直不釋放，并且不斷重復上述過程，所以使得內存不斷增加。

新建監聽

新建監聽，并啟動監聽

[grid@orastd ~]$ srvctl add listener
[grid@orastd ~]$ srvctl start listener -l listener

觀察內存使用量

新建監聽后，內存的使用未有明顯變化，所以這里大概就猜測是的原因就是oraagent內部判斷監聽資源時異常，導致內存不釋放

手動釋放oraagent的內存資源

oraagent進程負責grid用戶下面資源的啟停，是一個可以手動kill的進程，所以這里采用手動的方式。

kill oragent進程

[grid@orastd ~]$ ps -ef|grep oraagent
grid      2102     1 18  2024 ?        102-00:56:17 /oracle/app/19.3.0/grid/bin/oraagent.bin
grid     41122 40551  0 23:05 pts/0    00:00:00 tail -1000f ohasd_oraagent_grid.trc
grid     41317 40748  0 23:06 pts/1    00:00:00 grep --color=auto oraagent
[grid@orastd ~]$ kill -9 2102

查看啟動日志信息

ohasd進程負責oraagent進程的啟動，有如下的信息：

2025-07-24 23:06:38.834 : CRSCOMM:3643938560: [     INFO]  IpcL: connection to member 4 has been removed
2025-07-24 23:06:38.834 :CLSFRAME:3643938560: [     INFO]  Removing IPC Member:{Relative|Node:0|Process:4|Type:3}
2025-07-24 23:06:38.834 :CLSFRAME:3643938560: [     INFO]  Disconnected from AGENT process: {Relative|Node:0|Process:4|Type:3}
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} Agfw Proxy Server received process disconnected notification, count=1
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} /oracle/app/19.3.0/grid/bin/oraagent_grid disconnected.
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} Agent /oracle/app/19.3.0/grid/bin/oraagent_grid[2102] stopped!
2025-07-24 23:06:38.835 : CRSCOMM:3637634816: [     INFO] {0:0:23496} IpcL: removeConnection: Member 4 does not exist in pending or formed connections.
2025-07-24 23:06:38.835 : CRSCOMM:3637634816: [    ERROR] [FFAIL]{0:0:23496} IpcL: Failed to disconnect member 4 rc = 4
2025-07-24 23:06:38.835 :   CRSPE:1879045888: [     INFO] {0:0:23497} Disconnected from server:
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} Restarting the agent /oracle/app/19.3.0/grid/bin/oraagent_grid
2025-07-24 23:06:38.835 :    AGFW:3637634816: [     INFO] {0:0:23496} Starting the agent: /oracle/app/19.3.0/grid/bin/oraagent with user id: grid and incarnation:3
2025-07-24 23:06:38.844 :    AGFW:3637634816: [     INFO] {0:0:23496} Starting the HB [Interval =  30000, misscount = 6kill allowed=1] for agent: /oracle/app/19.3.0/grid/bin/oraagent_grid
2025-07-24 23:06:38.930 : CRSCOMM:3643938560: [     INFO]  IpcL: Accepted connection 16497 from user grid member number 7

oraagent進程自己的日志：

2025-07-24 23:06:30.319 :    AGFW:3846092544: [     INFO] {0:0:2} Agent received the message: AGENT_HB[Engine] ID 12293:687689824
Trace file /oracle/app/grid/diag/crs/orastd/crs/trace/ohasd_oraagent_grid.trc
Oracle Database 19c Clusterware Release 19.0.0.0.0 - Production
Version 19.20.0.0.0 Copyright 1996, 2023 Oracle. All rights reserved.CLSB:4160626432: [     INFO] Argument count (argc) for this daemon is 1CLSB:4160626432: [     INFO] Argument 0 is: /oracle/app/19.3.0/grid/bin/oraagent.bin
2025-07-24 23:06:38.927 : default:4160626432: clsu_load_ENV_levels: Failure to retrieve ORA_DAEMON_LOGGING_LEVELS environment variable [-1] [21104].
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdadr  2
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdstat  2
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdnreg  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdynam  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: allcomp  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: default  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: clsdimt  0
2025-07-24 23:06:38.927 :   AGENT:4160626432: [     INFO]  Logging level for Module: CLSCAL  0

crs的alet日志如下：

2024-01-06 20:37:59.966 [OCSSD(2281)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
2025-07-24 23:06:38.917 [ORAAGENT(41326)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 41326

總結

本案例展示了在Oracle 19.20環境下，oraagent進程出現內存泄漏的實際處理過程。通過對系統進程、內存使用、日志信息的細致排查，發現oraagent不斷嘗試處理一個并不存在的監聽資源（ora.LISTENER.lsnr），導致內存持續增長，最終遠超正常值。雖然嘗試新建監聽未能緩解問題，但通過手動重啟oraagent進程，內存占用恢復正常，間接驗證了進程本身存在異常分配和釋放內存的問題。

提示：

即使在較新版本的Oracle 19C中，依然可能遇到未被官方收錄的BUG，遇到異常現象時要善于通過日志和系統工具定位問題根源。
oraagent進程的內存異常增長，往往與資源配置或內部異常循環有關，及時關注日志中的異常提示（如不存在的監聽資源）有助于快速定位問題。
對于oraagent等可安全重啟的進程，適當的重啟操作可以臨時緩解內存泄漏帶來的影響，但建議后續持續關注官方補丁和MOS文檔，及時獲取修復方案。
日常巡檢中，建議定期監控關鍵進程的內存使用情況，發現異常及時處理，避免影響數據庫集群的穩定性。
最重要的就是環境搭建的規范化，這次的內存泄漏，明顯是由于在部署備庫環境時，未完全按照Oracle環境的最佳實踐來部署，最佳實踐是一個永無休止的根據大量真實案例來不斷的更新。
建議將數據庫版本升級到最新的補丁包，不要想到當前沒有出現故障，就不升級。當前沒有出現故障，并不代表著環境中不存在隱患，甚至有可能是已經出現了隱患，但是沒有被發現，就如這個案例一樣，如果不是朋友的細心，估計備庫環境中oraagent泄漏的問題不會提前發現，等生產系統容災切換到備庫后，到時出現內存不足，影響數據庫穩定性和性能，此時估計就是重大故障了。最新的補丁包其實都是在修復BUG，并沒有引入新的功能，修復的BUG越多，也就意味著環境越穩定。