【銀河麒麟高級服務器操作系統】服務器外掛存儲ioerror分析及處理分享

更多銀河麒麟操作系統產品及技術討論，歡迎加入銀河麒麟操作系統官方論壇

forum.kylinos.cn

了解更多銀河麒麟操作系統全新產品，請點擊訪問

麒麟軟件產品專區：product.kylinos.cn

開發者專區：developer.kylinos.cn

文檔中心：document.kylinos.cn

服務器環境以及配置

系統環境	物理機/虛擬機/云/容器	物理機
網絡環境	外網/私有網絡/無網絡	私有網絡
硬件環境	處理器：	S2500
	內存：	512GB
	機器型號	擎天EF860
	整機類型/架構：	arm64
	BIOS版本：	?Great Wall BIOS V3.0
軟件環境	具體操作系統版本	銀河麒麟高級服務器操作系統 Kylin Linux Advanced Server release V10 (Lance)
軟件環境	內核版本	4.19.90-52.30.v2207.ky10.aarch64

現象描述

服務器巡檢告顯示有io error，需要進行分析。

現象分析

查看磁盤存儲情況

根據串口日志，報IO Error錯誤的是dm-4和dm-5磁盤設備，對應的是/dev/mpathxsky02blk01和/dev/mpathxsky02blk02兩塊多路徑盤。

分析串口日志

查看串口日志，系統出現過三次I/O error相關的報錯，第一次導致系統發生了hung task，后面兩次出現IO error報錯后便出現了shutdown相關的日志打印。

分析第一次出現I/O error

日志中多次出現print_req_error: I/O error, dev sdb和print_req_error: I/O error, dev sdc的錯誤。這表明設備sdb和sdc發生了I/O錯誤。另外，日志中還出現了rejecting I/O to offline device的消息（例如：sd 3:0:0:1: rejecting I/O to offline device），這通常意味著設備已經離線，無法再進行I/O操作。這些I/O錯誤可能是由于硬盤故障、連接問題（例如，SATA線纜故障）或控制器問題引起的。

日志中有多個任務被報告為掛起超過1200秒，如xfsaild/dm-4和containerd。這些任務的掛起是由于無法完成的磁盤I/O請求導致的，因為設備已經離線或不可用。出現了任務掛起（通常和無法訪問存儲設備相關），導致最后內核觸發了恐慌（Kernel panic - not syncing: hung_task: blocked tasks）。

sdb和sdc設備應該對應sd 3:0:0:0和sd 3:0:0:1，查看當前收集的sosreport中的lsscsi命令，由于相隔的時間太過久遠，沒有3:0:0:0和3:0:0:1相關的設備，變為了5:0:0:0和5:0:0:1。后續也未再出現過rejecting I/O to offline device，問題應該已經修復。

[30551.689416][ 86] print_req_error: I/O error, dev sdb, sector 278120^M
[30551.690643][ 86] print_req_error: I/O error, dev sdb, sector 526730880^M
[30551.691874][ 86] print_req_error: I/O error, dev sdb, sector 794630784^M
[30551.693092][ 86] print_req_error: I/O error, dev sdb, sector 3239552^M
[30601.823314][ 24] print_req_error: I/O error, dev sdb, sector 267177832^M
[30622.174613][ 23] print_req_error: I/O error, dev sdb, sector 267177832^M
[30627.418715][ 86] sd 3:0:0:1: rejecting I/O to offline device^M
[30627.420021][ 86] print_req_error: I/O error, dev sdc, sector 209772368^M
[30627.421253][ 86] print_req_error: I/O error, dev sdc, sector 209772376^M
[30627.438674][ 86] sd 3:0:0:0: rejecting I/O to offline device^M
[30627.439942][ 86] print_req_error: I/O error, dev sdb, sector 1745360^M
[30627.441180][ 86] print_req_error: I/O error, dev sdb, sector 788207504^M
[30627.442345][ 86] print_req_error: I/O error, dev sdb, sector 526731600^M
[30627.443517][ 86] print_req_error: I/O error, dev sdb, sector 263186528^M
[30627.444683][ 86] print_req_error: I/O error, dev sdb, sector 3239552^M
[31446.409648][ 28] print_req_error: I/O error, dev sdb, sector 267177832^M
[31467.420538][ 28] print_req_error: I/O error, dev sdb, sector 267177832^M
[31476.890512][ 61] print_req_error: I/O error, dev sdc, sector 209772368^M
[31476.891710][ 61] print_req_error: I/O error, dev sdc, sector 209772376^M
[31482.488033][ 84] sd 3:0:0:1: rejecting I/O to offline device^M
[31482.489254][ 84] print_req_error: I/O error, dev sdc, sector 209772392^M
[31482.490507][ 84] print_req_error: I/O error, dev sdc, sector 209772384^M
[31482.491733][ 84] print_req_error: I/O error, dev sdc, sector 209772400^M
[31482.492918][ 84] print_req_error: I/O error, dev sdc, sector 209772408^M
[31482.494051][ 84] sd 3:0:0:0: rejecting I/O to offline device^M
[31482.495054][ 84] print_req_error: I/O error, dev sdb, sector 270107808^M
[31482.496109][ 84] print_req_error: I/O error, dev sdb, sector 270106784^M
[31482.497193][ 84] print_req_error: I/O error, dev sdb, sector 788207504^M
[31871.417820][ ?4] INFO: task xfsaild/dm-4:402347 blocked for more than 1200 seconds.^M
[31871.419496][ ?4] ??????Tainted: G ??????????OE ????4.19.90-52.30.v2207.ky10.aarch64 #1^M
[31871.421055][ ?4] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M
[31871.422705][ ?4] xfsaild/dm-4 ???D ???0 402347 ?????2 0x00000628^M
[31871.423726][ ?4] Call trace:^M
[31871.424436][ ?4] ?__switch_to+0xe8/0x150^M
[31871.425293][ ?4] ?__schedule+0x2b0/0x768^M
[31871.426152][ ?4] ?schedule+0x30/0xf0^M
[31871.426917][ ?4] ?xfs_log_force+0x170/0x358^M
[31871.427809][ ?4] ?xfsaild_push+0x5a8/0x6c0^M
[31871.428766][ ?4] ?xfsaild+0x11c/0x238^M
[31871.429627][ ?4] ?kthread+0x134/0x138^M
[31871.430447][ ?4] ?ret_from_fork+0x10/0x18^M
[31871.431339][ ?4] INFO: task containerd:409512 blocked for more than 1200 seconds.^M
[31871.433064][ ?4] ??????Tainted: G ??????????OE ????4.19.90-52.30.v2207.ky10.aarch64 #1^M
[31871.434889][ ?4] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M
[31871.436710][ ?4] containerd ?????D ???0 409512 ?????1 0x00000608^M
[31871.437906][ ?4] Call trace:^M
[31871.438822][ ?4] ?__switch_to+0xe8/0x150^M
[31871.439785][ ?4] ?__schedule+0x2b0/0x768^M
[31871.440741][ ?4] ?schedule+0x30/0xf0^M
[31871.441658][ ?4] ?io_schedule+0x20/0x90^M
[31871.442570][ ?4] ?wait_on_page_bit+0x134/0x178^M
[31871.443548][ ?4] ?__filemap_fdatawait_range+0xd0/0x120^M
[31871.444618][ ?4] ?file_write_and_wait_range+0xb0/0xd8^M
[31871.445655][ ?4] ?xfs_file_fsync+0x58/0x1d8^M
[31871.446584][ ?4] ?vfs_fsync_range+0x4c/0x90^M
[31871.447743][ ?4] ?do_fsync+0x48/0x78^M
[31871.448623][ ?4] ?sys_fdatasync+0x24/0x38^M
[31871.449503][ ?4] ?__sys_trace_return+0x0/0x4^M
[31871.450841][ ?4] Kernel panic - not syncing: hung_task: blocked tasks^M
[31871.451999][ ?4] CPU: 4 PID: 748 Comm: khungtaskd Tainted: G ??????????OE ????4.19.90-52.30.v2207.ky10.aarch64 #1^M
[31871.453937][ ?4] Source Version: ccdbfc2c55f0fb8dde14fae29155446fc8a7e941^M
[31871.455059][ ?4] Hardware name: GreatWall ?\x93\x8e?¤?EF860/GW-748F2A-FTG, BIOS Great Wall BIOS V3.0 2022-11-17^M
[31871.457098][ ?4] Call trace:^M
[31871.457908][ ?4] ?dump_backtrace+0x0/0x1b0^M
[31871.458911][ ?4] ?show_stack+0x24/0x30^M
[31871.459845][ ?4] ?dump_stack+0xb4/0xf0^M
[31871.460748][ ?4] ?panic+0x130/0x310^M
[31871.461654][ ?4] ?watchdog+0x2b8/0x468^M
[31871.462536][ ?4] ?kthread+0x134/0x138^M
[31871.463337][ ?4] ?ret_from_fork+0x10/0x18^M
[31871.464354][ ?4] SMP: stopping secondary CPUs^M
[31871.584509][ ?4] Kernel Offset: 0x37db061e0000 from 0xffff000048000000^M
[31871.585577][ ?4] CPU features: 0x10,a0000008^M
[31871.586428][ ?4] Memory Limit: none^M
[31871.587456][ ?4] Rebooting in 10 seconds..^M

分析第二次出現I/O error

日志中先出現了多個print_req_error: I/O error, dev dm-4/5/6, sector xxxx，說明lvm邏輯卷輸入輸出出現錯誤。而后出現了XFS (dm-4): writeback error on sector xxxx，XFS 文件系統報告了writeback error和metadata I/O error，特別是在xlog_iodone操作中，錯誤發生在不同的扇區，涉及數據和元數據的讀寫操作。錯誤代碼為error 5，對應于輸入/輸出錯誤（EIO），這表示底層存儲設備無法完成讀寫請求。

而后馬上出現了shutdown及reboot: Restarting system相關日志，這應是之前執行了關機操作。雖然是先出現了IO error的日志，而后出現的shutdown日志，但是shutdown操作并不會立即寫到日志，推測是關機操作使得多路徑存儲設備下線，文件系統無法寫入，而后出現的IO error。

[4931293.526231][ 71] print_req_error: I/O error, dev dm-4, sector 5428128^M
[4931293.526436][ 70] print_req_error: I/O error, dev dm-5, sector 15354992^M
[4931293.527010][ 71] XFS (dm-4): writeback error on sector 5428240^M
[4931293.527704][ 70] XFS (dm-5): writeback error on sector 15355120^M
[4931293.528811][ 70] print_req_error: I/O error, dev dm-5, sector 16527680^M
[4931293.529516][ 70] XFS (dm-5): writeback error on sector 16527808^M
[4931293.530124][ 70] print_req_error: I/O error, dev dm-5, sector 24740872^M
[4931293.530762][ 70] print_req_error: I/O error, dev dm-5, sector 27572552^M
[4931293.531550][ 70] XFS (dm-5): writeback error on sector 272373048^M
[4931293.532123][ 70] XFS (dm-5): writeback error on sector 276753512^M
[4931293.532702][ 70] XFS (dm-5): writeback error on sector 531646336^M
[4931293.533384][ 70] XFS (dm-5): writeback error on sector 24740992^M
[4931293.533961][ 70] XFS (dm-5): writeback error on sector 27572656^M
[4931293.534656][ 70] XFS (dm-5): writeback error on sector 288670704^M
[4931293.535240][ 70] XFS (dm-5): writeback error on sector 288670720^M
[4931293.538236][ 66] XFS (dm-4): metadata I/O error in "xlog_iodone" at daddr 0xc8086b0 len 64 error 5^M
[4931293.539266][ 66] XFS (dm-4): Log I/O Error Detected. ?Shutting down filesystem^M
[4931293.539586][ 50] XFS (dm-5): metadata I/O error in "xlog_iodone" at daddr 0x1f402070 len 64 error 5^M
[4931293.540003][ 50] XFS (dm-4): Please umount the filesystem and rectify the problem(s)^M
[4931293.541683][ 50] XFS (dm-5): Log I/O Error Detected. ?Shutting down filesystem^M
[4931293.542370][ 50] XFS (dm-5): Please umount the filesystem and rectify the problem(s)^M
[4931294.409340][ 99] XFS (dm-6): metadata I/O error in "xlog_iodone" at daddr 0x1f45ad90 len 64 error 5^M
[4931314.007532][ 42] shutdown: 9 output lines suppressed due to ratelimiting^M
[4931314.118056][ 47] dracut Warning: Killing all remaining processes^M
[4931314.508042][ 47] dracut Warning: Unmounted /oldroot.^M
[4931315.028993][ 75] kauditd_printk_skb: 8 callbacks suppressed^M
[4931328.960882][ ?0] reboot: Restarting system^M

分析第三次出現I/O error

和第二次出現IO error相同，出現IO error后，也立馬出現了shutdown相關的日志。

[2001517.911050][102] print_req_error: I/O error, dev dm-6, sector 62984704^M
[2001517.911712][100] print_req_error: I/O error, dev dm-5, sector 218187936^M
[2001517.912383][102] XFS (dm-6): writeback error on sector 62984832^M
[2001517.913028][100] print_req_error: I/O error, dev dm-5, sector 216240008^M
[2001517.913138][100] print_req_error: I/O error, dev dm-5, sector 6241168^M
[2001517.913144][100] print_req_error: I/O error, dev dm-5, sector 7116376^M
[2001517.913166][100] XFS (dm-5): writeback error on sector 6241240^M
[2001517.913170][100] XFS (dm-5): writeback error on sector 7116472^M
[2001517.913174][100] XFS (dm-5): writeback error on sector 7116568^M
[2001517.913229][100] print_req_error: I/O error, dev dm-5, sector 6772344^M
[2001517.913254][100] XFS (dm-5): writeback error on sector 6772472^M
[2001517.913258][100] XFS (dm-5): writeback error on sector 6772520^M
[2001517.913263][100] XFS (dm-5): writeback error on sector 338549088^M
[2001517.913392][100] XFS (dm-5): writeback error on sector 110130576^M
[2001517.913398][100] XFS (dm-5): writeback error on sector 110130584^M
[2001517.923828][101] XFS (dm-5): metadata I/O error in "xlog_iodone" at daddr 0xc82f840 len 64 error 5^M
[2001517.925072][101] XFS (dm-5): Log I/O Error Detected. ?Shutting down filesystem^M
[2001517.925237][103] XFS (dm-6): metadata I/O error in "xlog_iodone" at daddr 0x1f44c278 len 64 error 5^M
[2001517.925257][101] XFS (dm-6): metadata I/O error in "xlog_iodone" at daddr 0x1f44c2b8 len 64 error 5^M
[2001517.925419][103] XFS (dm-6): Log I/O Error Detected. ?Shutting down filesystem^M
[2001517.925421][103] XFS (dm-6): Please umount the filesystem and rectify the problem(s)^M
[2001517.925840][103] XFS (dm-5): Please umount the filesystem and rectify the problem(s)^M
[2001518.084461][ 59] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M
[2001518.134555][ 29] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M
[2001518.134679][ ?8] overlayfs: failed to get metacopy (-5)^M
[2001518.135513][ 29] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M
[2001518.137716][ 17] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M
[2001518.138605][ 17] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M
[2001518.141203][ 71] overlayfs: failed to get metacopy (-5)^M
[2001518.143645][ 54] overlayfs: failed to get metacopy (-5)^M
[2001518.144968][ 95] overlayfs: failed to get metacopy (-5)^M
[2001518.145177][ 46] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M
[2001518.146489][ 46] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M
[2001518.148168][ 72] overlayfs: failed to get metacopy (-5)^M
[2001518.148170][ 34] overlayfs: failed to get metacopy (-5)^M
[2001518.148259][ 34] overlayfs: failed to get metacopy (-5)^M
[2001518.150749][ 96] overlayfs: failed to get metacopy (-5)^M
[2001518.152389][116] overlayfs: failed to get metacopy (-5)^M
[2001528.063127][ 97] systemd-shutdown[1]: Waiting for process: local-path-prov^M
[2001600.084921][123] kauditd_printk_skb: 23 callbacks suppressed^M
[2001613.949707][ 99] shutdown: 7 output lines suppressed due to ratelimiting^M
[2001614.020310][100] dracut Warning: Killing all remaining processes^M
[2001614.364201][100] dracut Warning: Unmounted /oldroot.^M
[2001628.676636][ ?0] reboot: Restarting system^M

查看tuned.log，出現了兩次關機操作，應該是對應這兩次IO error。

分析結果

第一次出現I/O錯誤時，日志中顯示了 rejecting I/O to offline device的消息。這通常意味著設備已經離線，無法繼續進行I/O操作，從而導致了I/O錯誤。由于這個錯誤的日志已經存在一段時間，問題應該已經得到解決。

第二次和第三次出現I/O錯誤時，伴隨著關機相關的日志信息。具體來說，關機日志（shutdown 和 reboot）表明系統在I/O錯誤發生前，應該有執行關機操作。系統關機通常會觸發文件系統的卸載和數據的回寫，但由于多路徑存儲設備已經下線，XFS 無法完成日志寫回和其他數據的持久化操作。日志中的XFS writeback error和metadata I/O error表明XFS文件系統在嘗試進行數據寫回時失敗。尤其是在元數據操作過程中，XFS 在執行xlog_iodone操作時未能完成I/O請求，導致元數據無法成功寫回。這些錯誤的原因是，XFS嘗試將數據寫入物理存儲設備，但由于設備處于離線狀態或無法訪問，文件系統無法完成這些操作。