背景

裝有nvlink的h20機器上配置好驅動和cuda之后使用pytorch報錯
A800機器同樣

(pytorch2.4) root@xx-dev-H20:~# python
Python 3.12.0 | packaged by Anaconda, Inc. | (main, Oct 2 2023, 17:29:18) [GCC 11.2.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import torch
torch.>>> torch.cuda.is_available()
/root/anaconda3/envs/pytorch2.4/lib/python3.12/site-packages/torch/cuda/init.py:128: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at /opt/conda/conda-bld/pytorch_1724789220573/work/c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0
False

解決

在nvidia fabricmanager官網找到和H20機器上的驅動版本相對應的fabricmanager版本安裝，啟動即可

(pytorch2.4) root@xx-dev-H20:/opt/fabricmanager-linux-x86_64-550.163.01-archive# python
Python 3.12.0 | packaged by Anaconda, Inc. | (main, Oct  2 2023, 17:29:18) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> 
>>> 
>>> import torch
>>> torch.
KeyboardInterrupt
>>> torch.cuda.is_available()
True

查看nvlink吞吐量
nvidia-smi nvlink --getthroughput d
watch -n 1 ‘nvidia-smi nvlink -gt d’

reference

fabricmanager下載地址
https://developer.download.nvidia.cn/compute/nvidia-driver/redist/fabricmanager/linux-x86_64/
nccl使用nvlink通信
https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html