參考：https://access.redhat.com/solutions/1257953

問題

nccl-test容器docker.io/library/nccl-tests:24.12中跑mpirun，buff設置為NCCL_BUFFSIZE=503316480
提示out of memory：

pod-1:78:91 [0] include/alloc.h:114 NCCL WARN Cuda failure 'out of memory'pod-1:78:91 [0] include/alloc.h:119 NCCL WARN Failed to CUDA host alloc -268435456 bytes
pod-1:78:91 [0] NCCL INFO transport/net.cc:517 -> 1
pod-1:78:91 [0] NCCL INFO transport/net.cc:719 -> 1
pod-1:78:93 [0] NCCL INFO transport.cc:193 -> 1
pod-1:78:93 [0] NCCL INFO group.cc:133 -> 1
pod-1:78:93 [0] NCCL INFO group.cc:75 -> 1 [Async thread]pod-1:78:91 [0] proxy.cc:1620 NCCL WARN [Service thread] Error encountered progressing operation=Connect, res=3, closing connection
pod-1:78:78 [0] NCCL INFO group.cc:426 -> 1
pod-1:78:78 [0] NCCL INFO group.cc:566 -> 1
pod-1:78:78 [0] NCCL INFO group.cc:106 -> 1
pod-1: Test NCCL failure sendrecv.cu:57 'unhandled cuda error (run with NCCL_DEBUG=INFO for details) / '.. pod-1 pid 78: Test failure common.cu:383.. pod-1 pid 78: Test failure common.cu:592.. pod-1 pid 78: Test failure sendrecv.cu:103.. pod-1 pid 78: Test failure common.cu:625.. pod-1 pid 78: Test failure common.cu:1123.. pod-1 pid 78: Test failure common.cu:893

問題確認

容器內執行ulimit -a顯示max locked memory只有64k

放開容器max locked memory限制

在 /etc/systemd/system/docker.service中增加LimitMEMLOCK=infinity

然后重啟docker：

systemctl daemon-reloadsystemctl restart docker