使用vllm部署neo4j的text2cypher-gemma-2-9b-it-finetuned-2024v1模型

系統環境準備

由于使用的基于 nvcr.io/nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04 的 workbench，需要進行以下準備（其他系統環境可忽略）

ldconfig -p | grep libcudnn 找到 libcudnn 的so庫，然后建立軟鏈接：

ln -s /lib/x86_64-linux-gnu/libcudnn.so.8 /usr/local/cuda/lib64/libcudnn.so

將以下添加到 ~/.bashrc

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/cuda/lib64/:$LD_LIBRARY_PATH

安裝可能用到的CUDA相關包：

apt-get install -y --no-install-recommends \cuda-cudart-dev-12-1=12.1.105-1 \cuda-command-line-tools-12-1=12.1.1-1 \cuda-minimal-build-12-1=12.1.1-1 \cuda-libraries-dev-12-1=12.1.1-1 \cuda-nvml-dev-12-1=12.1.105-1 \cuda-nvprof-12-1=12.1.105-1 \libnpp-dev-12-1=12.1.0.40-1 \libcusparse-dev-12-1=12.1.0.106-1 \libcublas-dev-12-1=12.1.3.1-1 \libnccl-dev=2.17.1-1+cuda12.1 \cuda-nsight-compute-12-1=12.1.1-1

避免驅動更新：

apt-mark hold cuda-cudart-dev-12-1 cuda-command-line-tools-12-1 cuda-minimal-build-12-1 cuda-libraries-dev-12-1 cuda-nvml-dev-12-1 cuda-nvprof-12-1 libnpp-dev-12-1 libcusparse-dev-12-1 libcublas-dev-12-1 libnccl-dev cuda-nsight-compute-12-1

安裝系統依賴：

apt-get update
apt-get install wget unzip
apt-get install ffmpeg libsm6 libxext6

設置 git 代理

git config --system url."https://githubfast.com/".insteadOf https://github.com/

創建conda環境

pytorch

conda create -n py310torch python=3.10
conda activate py310torch
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121

bitsandbytes

參考官方文檔 https://huggingface.co/docs/bitsandbytes/v0.45.0/en/installation#cuda-compile 安裝 bitsandbytes

# bitsandbytes require gcc>=12 for cuda>=12.0
apt update
apt install gcc-12 g++-12
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12 --slave /usr/bin/g++ g++ /usr/bin/g++-12
gcc --versiongit clone --depth 1 --branch 0.45.0 https://github.com/bitsandbytes-foundation/bitsandbytes
cd bitsandbytes
pip install -r requirements-dev.txt -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
cmake -DCOMPUTE_BACKEND=cuda -S .
make -j 16
pip install .   # `-e` for "editable" install, when developing BNB (otherwise leave that out)

添加 export BNB_CUDA_VERSION=121 到 ~/.bashrc 后，打開新的shell，執行