前言

本文主要借鑒：VLLM部署deepseek，結合自身進行整理

下載模型

from modelscope import snapshot_download
model_dir = snapshot_download('OpenGVLab/InternVL3-8B', local_dir="xxx/OpenGVLab/InternVL2_5-1B")

環境配置

auto-dl上選擇單卡910B即可，Pytorch框架只有一個CANN版本選擇，選這個即可。

安裝vllm和vllm ascend

git clone --depth 1 --branch v0.7.3 https://github.com/vllm-project/vllm
cd vllm
VLLM_TARGET_DEVICE=empty pip install . --extra-index https://download.pytorch.org/whl/cpu/git clone  --depth 1 --branch v0.7.3rc1 https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
pip install -e . --extra-index https://download.pytorch.org/whl/cpu/

手動安裝torch-npu

mkdir pta
cd pta
wget https://pytorch-package.obs.cn-north-4.myhuaweicloud.com/pta/Daily/v2.5.1/20250320.3/pytorch_v2.5.1_py310.tar.gz
tar -xvf pytorch_v2.5.1_py310.tar.gz
pip install ./torch_npu-2.5.1.dev20250320-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

pip install torchvision==0.20.1

啟動指令

vllm serve /root/xxx/xxx/InternVL3-8B --max-model-len 4096 --port 8000 --tensor-parallel-size 1 --trust-remote-code --served-model-name InternVL3-8B

啟動成功如下圖所示：

通過curl去進行驗證

curl http://localhost:8000/v1/chat/completions \-H "Content-Type: application/json" \-d '{"model": "InternVL3-8B","messages": [{"role": "user", "content": "你是誰?"}]}'