DeepSeek模型量化

技術背景

大語言模型（Large Language Model，LLM），可以通過量化（Quantization）操作來節約內存/顯存的使用，并且降低了通訊開銷，進而達到加速模型推理的效果。常見的就是把Float16的浮點數，轉換成低精度的整數，例如Int4整數。最極限的情況下，可以把參數轉化成二值Bool變量，也就是只有0和1，但是這種大幅度的量化有可能導致模型的推理效果不佳。常用的是，在70B以下的模型用Q8，70B以上可以用Q4。具體的原理，包括對稱量化和非對稱量化等，這里就不作介紹了，主要看看工程上怎么實現，主要使用了llama.cpp來完成量化。

安裝llama.cpp

這里我們在Ubuntu上使用本地編譯構建的方法進行安裝，首先從github上面clone下來：

$ git clone https://github.com/ggerganov/llama.cpp.git
正克隆到 'llama.cpp'...
remote: Enumerating objects: 43657, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 43657 (delta 3), reused 5 (delta 1), pack-reused 43642 (from 3)
接收對象中: 100% (43657/43657), 88.26 MiB | 8.30 MiB/s, 完成.
處理 delta 中: 100% (31409/31409), 完成.

最好創建一個虛擬環境，以避免各種軟件依賴的問題，推薦Python3.10：

# 創建虛擬環境
$ conda create -n llama python=3.10
# 激活虛擬環境
$ conda activate llama

進入下載好的llama.cpp路徑，安裝所有的依賴項：

$ cd llama.cpp/
$ python3 -m pip install -e .

創建一個編譯目錄，執行編譯指令：

$ mkdir build
$ cd build/
$ cmake ..
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.25.1") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE  
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with GGML_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- Including CPU backend
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native 
-- Configuring done
-- Generating done
-- Build files have been written to: /datb/DeepSeek/llama/llama.cpp/build
$ cmake --build . --config Release
Scanning dependencies of target ggml-base
[  0%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml.c.o
[  1%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-alloc.c.o
[100%] Linking CXX executable ../../bin/llama-vdot
[100%] Built target llama-vdot

到這里，就成功構建了cpu版本的llama.cpp，可以直接使用了。如果需要安裝gpu加速的版本，可以參考下面這一小節，如果嫌麻煩建議直接跳過。

llama.cpp之CUDA加速

安裝GPU版本llama.cpp需要先安裝一些依賴：

$ sudo apt install curl libcurl4-openssl-dev

跟cpu版本不同的地方，主要在于cmake的編譯指令（如果已經編譯了cpu的版本，最好先清空build路徑下的文件）：

$ cmake .. -DCMAKE_CUDA_COMPILER=/usr/bin/nvcc -DGGML_CUDA=ON -DBUILD_SHARED_LIBS=OFF -DLLAMA_CURL=ON -DCMAKE_CUDA_STANDARD=17

這里加的一個FLAG：-DCMAKE_CUDA_STANDARD=17可以解決Llama.cpp倉庫里面的Issue，如果不加這個Flag，有可能出現下面這種報錯：

Make Error in ggml/src/ggml-cuda/CMakeLists.txt:Target "ggml-cuda" requires the language dialect "CUDA17" (with compilerextensions), but CMake does not know the compile flags to use to enable it.

如果順利的話，執行下面這個指令，成功編譯通過的話就是成功了：

$ cmake --build . --config Release

但是如果像我這樣有報錯信息，那就得單獨處理以下。

/datb/DeepSeek/llama/llama.cpp/ggml/src/ggml-cuda/vendors/cuda.h:6:10: fatal error: cuda_bf16.h: 沒有那個文件或目錄#include <cuda_bf16.h>^~~~~~~~~~~~~
compilation terminated.

這個報錯是說找不到頭文件，于是在環境里面find / -name cuda_bf16.h了一下，發現其實是有這個頭文件的：

/home/dechin/anaconda3/envs/llama/lib/python3.10/site-packages/nvidia/cuda_runtime/include/cuda_bf16.h
/home/dechin/anaconda3/envs/llama/lib/python3.10/site-packages/triton/backends/nvidia/include/cuda_bf16.h

處理方式是把這個路徑加到CPATH里面：

$ export CPATH=$CPATH:/home/dechin/anaconda3/envs/llama/lib/python3.10/site-packages/nvidia/cuda_runtime/include/

如果是出現這個報錯：

/home/dechin/anaconda3/envs/llama/lib/python3.10/site-packages/nvidia/cuda_runtime/include/cuda_fp16.h:4100:10: fatal error: nv/target: 沒有那個文件或目錄#include <nv/target>^~~~~~~~~~~
compilation terminated.

那就是找不到target目錄的路徑，如果本地有target路徑的話，也可以直接配置到CPATH里面：

$ export CPATH=/home/dechin/anaconda3/pkgs/cupy-core-13.3.0-py310h5da974a_2/lib/python3.10/site-packages/cupy/_core/include/cupy/_cccl/libcudacxx/:$CPATH

如果是下面這些報錯：

/datb/DeepSeek/llama/llama.cpp/ggml/src/ggml-cuda/common.cuh(138): error: identifier "cublasGetStatusString" is undefined/datb/DeepSeek/llama/llama.cpp/ggml/src/ggml-cuda/common.cuh(417): error: A __device__ variable cannot be marked constexpr/datb/DeepSeek/llama/llama.cpp/ggml/src/ggml-cuda/common.cuh(745): error: identifier "CUBLAS_TF32_TENSOR_OP_MATH" is undefined3 errors detected in the compilation of "/tmp/tmpxft_000a126f_00000000-9_acc.compute_75.cpp1.ii".
make[2]: *** [ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/build.make:82：ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/acc.cu.o] 錯誤 1
make[1]: *** [CMakeFiles/Makefile2:1964：ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/all] 錯誤 2
make: *** [Makefile:160：all] 錯誤 2

那么很有可能是cuda-toolkit的版本問題，嘗試安裝cuda-12：

$ conda install nvidia::cuda-toolkit

如果使用conda安裝過程有這種問題：

Collecting package metadata (current_repodata.json): failed# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<Traceback (most recent call last):File "/home/dechin/anaconda3/lib/python3.8/site-packages/conda/gateways/repodata/__init__.py", line 132, in conda_http_errorsyieldFile "/home/dechin/anaconda3/lib/python3.8/site-packages/conda/gateways/repodata/__init__.py", line 101, in repodataresponse.raise_for_status()File "/home/dechin/anaconda3/lib/python3.8/site-packages/requests/models.py", line 1024, in raise_for_statusraise HTTPError(http_error_msg, response=self)requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://conda.anaconda.org/defaults/linux-64/current_repodata.json

那應該是conda源的問題，可以刪掉舊的channels，使用默認channels或者找一個國內可以用的鏡像源進行配置：

$ conda config --remove-key channels
$ conda config --remove-key default_channels
$ conda config --append channels conda-forge

重新安裝以后，nvcc的路徑發生了變化，要注意修改下編譯時的DCMAKE_CUDA_COMPILER參數配置：

$ cmake .. -DCMAKE_CUDA_COMPILER=/home/dechin/anaconda3/envs/llama/bin/nvcc -DGGML_CUDA=ON -DBUILD_SHARED_LIBS=OFF -DLLAMA_CURL=ON -DCMAKE_CUDA_STANDARD=17

如果出現如下報錯：

-- Unable to find cuda_runtime.h in "/home/dechin/anaconda3/envs/llama/include" for CUDAToolkit_INCLUDE_DIR.
-- Could NOT find CUDAToolkit (missing: CUDAToolkit_INCLUDE_DIR) 
CMake Error at ggml/src/ggml-cuda/CMakeLists.txt:151 (message):CUDA Toolkit not found-- Configuring incomplete, errors occurred!
See also "/datb/DeepSeek/llama/llama.cpp/build/CMakeFiles/CMakeOutput.log".
See also "/datb/DeepSeek/llama/llama.cpp/build/CMakeFiles/CMakeError.log".

這是找不到CUDAToolkit_INCLUDE_DIR的路徑配置，只要在cmake的指令里面加上一個include路徑即可：

$ cmake .. -DCMAKE_CUDA_COMPILER=/home/dechin/anaconda3/envs/llama/bin/nvcc -DGGML_CUDA=ON -DBUILD_SHARED_LIBS=OFF -DLLAMA_CURL=ON -DCMAKE_CUDA_STANDARD=17 -DCUDAToolkit_INCLUDE_DIR=/home/dechin/anaconda3/envs/llama/targets/x86_64-linux/include/ -DCURL_LIBRARY=/usr/lib/x86_64-linux-gnu/

如果經過以上的一串處理，依然有報錯信息，那我建議還是用個Docker吧，或者直接用CPU版本執行quantize，模型調用使用Ollama，這樣方便一些。

下載Hugging Face模型

由于很多已經完成量化的GGUF模型文件，無法被二次量化，所以建議直接從Hugging Face下載safetensors模型文件。然后用llama.cpp里面的一個Python腳本將hf模型轉為gguf模型，然后再使用llama.cpp進行模型quantize。

關于模型下載這部分，因為Hugging Face的訪問有時候也會受限，所以這里首推的還是國內的ModelScope平臺。從ModelScope平臺下載模型，可以裝一個這種Python形式的modelscope：

$ python3 -m pip install modelscope
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: modelscope in /home/dechin/anaconda3/lib/python3.8/site-packages (1.22.3)
Requirement already satisfied: requests>=2.25 in /home/dechin/.local/lib/python3.8/site-packages (from modelscope) (2.25.1)
Requirement already satisfied: urllib3>=1.26 in /home/dechin/.local/lib/python3.8/site-packages (from modelscope) (1.26.5)
Requirement already satisfied: tqdm>=4.64.0 in /home/dechin/anaconda3/lib/python3.8/site-packages (from modelscope) (4.67.1)
Requirement already satisfied: certifi>=2017.4.17 in /home/dechin/.local/lib/python3.8/site-packages (from requests>=2.25->modelscope) (2021.5.30)
Requirement already satisfied: chardet<5,>=3.0.2 in /home/dechin/.local/lib/python3.8/site-packages (from requests>=2.25->modelscope) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in /home/dechin/.local/lib/python3.8/site-packages (from requests>=2.25->modelscope) (2.10)

然后使用modelcope下載模型：

$ modelscope download --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B

如果出現報錯（如果沒有報錯就不用理會，等待模型下載完成即可）：

safetensors integrity check failed, expected sha256 signature is xxx

可以嘗試另一種安裝方式：

$ sudo apt install git-lfs

下載模型：

$ git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B.git
正克隆到 'DeepSeek-R1-Distill-Qwen-32B'...
remote: Enumerating objects: 52, done.
remote: Counting objects: 100% (52/52), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 52 (delta 17), reused 42 (delta 13), pack-reused 0
展開對象中: 100% (52/52), 2.27 MiB | 2.62 MiB/s, 完成.
過濾內容: 100% (8/8), 5.02 GiB | 912.00 KiB/s, 完成.
Encountered 8 file(s) that may not have been copied correctly on Windows:model-00005-of-000008.safetensorsmodel-00004-of-000008.safetensorsmodel-00008-of-000008.safetensorsmodel-00002-of-000008.safetensorsmodel-00007-of-000008.safetensorsmodel-00003-of-000008.safetensorsmodel-00006-of-000008.safetensorsmodel-00001-of-000008.safetensorsSee: `git lfs help smudge` for more details.

這個過程會消耗很多時間，請耐心等待模型下載完成為止。下載完成后查看路徑：

$ cd DeepSeek-R1-Distill-Qwen-32B/
$ ll
總用量 63999072
drwxrwxr-x 4 dechin dechin       4096 2月  12 19:22 ./
drwxrwxr-x 3 dechin dechin       4096 2月  12 17:46 ../
-rw-rw-r-- 1 dechin dechin        664 2月  12 17:46 config.json
-rw-rw-r-- 1 dechin dechin         73 2月  12 17:46 configuration.json
drwxrwxr-x 2 dechin dechin       4096 2月  12 17:46 figures/
-rw-rw-r-- 1 dechin dechin        181 2月  12 17:46 generation_config.json
drwxrwxr-x 9 dechin dechin       4096 2月  12 19:22 .git/
-rw-rw-r-- 1 dechin dechin       1519 2月  12 17:46 .gitattributes
-rw-rw-r-- 1 dechin dechin       1064 2月  12 17:46 LICENSE
-rw-rw-r-- 1 dechin dechin 8792578462 2月  12 19:22 model-00001-of-000008.safetensors
-rw-rw-r-- 1 dechin dechin 8776906899 2月  12 19:03 model-00002-of-000008.safetensors
-rw-rw-r-- 1 dechin dechin 8776906927 2月  12 19:18 model-00003-of-000008.safetensors
-rw-rw-r-- 1 dechin dechin 8776906927 2月  12 18:56 model-00004-of-000008.safetensors
-rw-rw-r-- 1 dechin dechin 8776906927 2月  12 18:38 model-00005-of-000008.safetensors
-rw-rw-r-- 1 dechin dechin 8776906927 2月  12 19:19 model-00006-of-000008.safetensors
-rw-rw-r-- 1 dechin dechin 8776906927 2月  12 19:15 model-00007-of-000008.safetensors
-rw-rw-r-- 1 dechin dechin 4073821536 2月  12 19:02 model-00008-of-000008.safetensors
-rw-rw-r-- 1 dechin dechin      64018 2月  12 17:46 model.safetensors.index.json
-rw-rw-r-- 1 dechin dechin      18985 2月  12 17:46 README.md
-rw-rw-r-- 1 dechin dechin       3071 2月  12 17:46 tokenizer_config.json
-rw-rw-r-- 1 dechin dechin    7031660 2月  12 17:46 tokenizer.json

這就是下載成功了。

HF模型轉GGUF模型

找到編譯好的llama/llama.cpp/下的python腳本文件，可以先看下其用法：

$ python3 convert_hf_to_gguf.py --help
usage: convert_hf_to_gguf.py [-h] [--vocab-only] [--outfile OUTFILE] [--outtype {f32,f16,bf16,q8_0,tq1_0,tq2_0,auto}] [--bigendian] [--use-temp-file] [--no-lazy][--model-name MODEL_NAME] [--verbose] [--split-max-tensors SPLIT_MAX_TENSORS] [--split-max-size SPLIT_MAX_SIZE] [--dry-run][--no-tensor-first-split] [--metadata METADATA] [--print-supported-models][model]Convert a huggingface model to a GGML compatible filepositional arguments:model                 directory containing model fileoptions:-h, --help            show this help message and exit--vocab-only          extract only the vocab--outfile OUTFILE     path to write to; default: based on input. {ftype} will be replaced by the outtype.--outtype {f32,f16,bf16,q8_0,tq1_0,tq2_0,auto}output format - use f32 for float32, f16 for float16, bf16 for bfloat16, q8_0 for Q8_0, tq1_0 or tq2_0 for ternary, and auto for the highest-fidelity 16-bit float type depending on the first loaded tensor type--bigendian           model is executed on big endian machine--use-temp-file       use the tempfile library while processing (helpful when running out of memory, process killed)--no-lazy             use more RAM by computing all outputs before writing (use in case lazy evaluation is broken)--model-name MODEL_NAMEname of the model--verbose             increase output verbosity--split-max-tensors SPLIT_MAX_TENSORSmax tensors in each split--split-max-size SPLIT_MAX_SIZEmax size per split N(M|G)--dry-run             only print out a split plan and exit, without writing any new files--no-tensor-first-splitdo not add tensors to the first split (disabled by default)--metadata METADATA   Specify the path for an authorship metadata override file--print-supported-modelsPrint the supported models

然后執行構建GGUF：

$ python3 convert_hf_to_gguf.py /datb/DeepSeek/models/DeepSeek-R1-Distill-Qwen-32B --outfile /datb/DeepSeek/models/DeepSeek-R1-Distill-Qwen-32B.gguf
INFO:hf-to-gguf:Set model quantization version
INFO:gguf.gguf_writer:Writing the following files:
INFO:gguf.gguf_writer:/datb/DeepSeek/models/DeepSeek-R1-Distill-Qwen-32B.gguf: n_tensors = 771, total_size = 65.5G
Writing: 100%|██████████████████████████████████████████████████████████████| 65.5G/65.5G [19:42<00:00, 55.4Mbyte/s]
INFO:hf-to-gguf:Model successfully exported to /datb/DeepSeek/models/DeepSeek-R1-Distill-Qwen-32B.gguf

完成轉化后，會在指定的路徑下生成一個gguf文件，也就是all-in-one的模型文件。默認是fp32的精度，可以用于執行下一步的量化操作。

GGUF模型量化

在編譯好的llama.cpp的build/bin/路徑下，可以找到量化的可執行文件：

$ ./llama-quantize --help
usage: ./llama-quantize [--help] [--allow-requantize] [--leave-output-tensor] [--pure] [--imatrix] [--include-weights] [--exclude-weights] [--output-tensor-type] [--token-embedding-type] [--override-kv] model-f32.gguf [model-quant.gguf] type [nthreads]--allow-requantize: Allows requantizing tensors that have already been quantized. Warning: This can severely reduce quality compared to quantizing from 16bit or 32bit--leave-output-tensor: Will leave output.weight un(re)quantized. Increases model size but may also increase quality, especially when requantizing--pure: Disable k-quant mixtures and quantize all tensors to the same type--imatrix file_name: use data in file_name as importance matrix for quant optimizations--include-weights tensor_name: use importance matrix for this/these tensor(s)--exclude-weights tensor_name: use importance matrix for this/these tensor(s)--output-tensor-type ggml_type: use this ggml_type for the output.weight tensor--token-embedding-type ggml_type: use this ggml_type for the token embeddings tensor--keep-split: will generate quantized model in the same shards as input--override-kv KEY=TYPE:VALUEAdvanced option to override model metadata by key in the quantized model. May be specified multiple times.
Note: --include-weights and --exclude-weights cannot be used togetherAllowed quantization types:2  or  Q4_0    :  4.34G, +0.4685 ppl @ Llama-3-8B3  or  Q4_1    :  4.78G, +0.4511 ppl @ Llama-3-8B8  or  Q5_0    :  5.21G, +0.1316 ppl @ Llama-3-8B9  or  Q5_1    :  5.65G, +0.1062 ppl @ Llama-3-8B19  or  IQ2_XXS :  2.06 bpw quantization20  or  IQ2_XS  :  2.31 bpw quantization28  or  IQ2_S   :  2.5  bpw quantization29  or  IQ2_M   :  2.7  bpw quantization24  or  IQ1_S   :  1.56 bpw quantization31  or  IQ1_M   :  1.75 bpw quantization36  or  TQ1_0   :  1.69 bpw ternarization37  or  TQ2_0   :  2.06 bpw ternarization10  or  Q2_K    :  2.96G, +3.5199 ppl @ Llama-3-8B21  or  Q2_K_S  :  2.96G, +3.1836 ppl @ Llama-3-8B23  or  IQ3_XXS :  3.06 bpw quantization26  or  IQ3_S   :  3.44 bpw quantization27  or  IQ3_M   :  3.66 bpw quantization mix12  or  Q3_K    : alias for Q3_K_M22  or  IQ3_XS  :  3.3 bpw quantization11  or  Q3_K_S  :  3.41G, +1.6321 ppl @ Llama-3-8B12  or  Q3_K_M  :  3.74G, +0.6569 ppl @ Llama-3-8B13  or  Q3_K_L  :  4.03G, +0.5562 ppl @ Llama-3-8B25  or  IQ4_NL  :  4.50 bpw non-linear quantization30  or  IQ4_XS  :  4.25 bpw non-linear quantization15  or  Q4_K    : alias for Q4_K_M14  or  Q4_K_S  :  4.37G, +0.2689 ppl @ Llama-3-8B15  or  Q4_K_M  :  4.58G, +0.1754 ppl @ Llama-3-8B17  or  Q5_K    : alias for Q5_K_M16  or  Q5_K_S  :  5.21G, +0.1049 ppl @ Llama-3-8B17  or  Q5_K_M  :  5.33G, +0.0569 ppl @ Llama-3-8B18  or  Q6_K    :  6.14G, +0.0217 ppl @ Llama-3-8B7  or  Q8_0    :  7.96G, +0.0026 ppl @ Llama-3-8B1  or  F16     : 14.00G, +0.0020 ppl @ Mistral-7B32  or  BF16    : 14.00G, -0.0050 ppl @ Mistral-7B0  or  F32     : 26.00G              @ 7BCOPY    : only copy tensors, no quantizing

這里可以看到完整的可以執行量化操作的精度。例如我們可以量化一個q4_0精度的32B模型：

$ ./llama-quantize /datb/DeepSeek/models/DeepSeek-R1-Distill-Qwen-32B.gguf /datb/DeepSeek/models/DeepSeek-R1-Distill-Qwen-32B-Q4_0.gguf q4_0

輸出結果對比（這里的Q8_0是直接從模型倉庫里面下載的別人量化出來的Q8_0模型）：

-rw-rw-r-- 1 dechin dechin 65535969184 2月  13 09:33 DeepSeek-R1-Distill-Qwen-32B.gguf
-rw-rw-r-- 1 dechin dechin 18640230304 2月  13 09:51 DeepSeek-R1-Distill-Qwen-32B-Q4_0.gguf
-rw-rw-r-- 1 dechin dechin 34820884384 2月   9 01:44 DeepSeek-R1-Distill-Qwen-32B-Q8_0.gguf

從F32到Q8再到Q4，可以看到有一個很明顯的內存占用的下降。我們可以根據自己本地的計算機資源來決定要做多少精度的量化操作。

量化完成后，導入模型成功以后，可以用ollama list查看到所有的本地模型：

$ ollama list
NAME                            ID              SIZE      MODIFIED      
deepseek-r1:32b-q2k             8d2a0c19f6e0    12 GB     5 seconds ago    
deepseek-r1:32b-q40             13c7c287f615    18 GB     3 minutes ago    
deepseek-r1:32b                 91f2de3dd7fd    34 GB     42 hours ago     
nomic-embed-text-v1.5:latest    5b3683392ccb    274 MB    43 hours ago     
deepseek-r1:14b                 ea35dfe18182    9.0 GB    7 days ago

這里q2k也是本地量化的Q2_K的模型。只是從Q4_0到Q2_k已經沒有太大的參數內存縮減了，所以很多人量化一般就到Q4_0這個級別，可以兼具性能與精確性。

其他報錯處理

如果運行llama-quantize這個可執行文件出現這種報錯：

./xxx/llama-quantize: error while loading shared libraries: libllama.so: cannot open shared object file: No such file or directory

動態鏈接庫路徑LD_LIBRARY_PATH沒有設置，也可以選擇直接進入到bin/路徑下運行該可執行文件。

總結概要

這篇文章主要介紹了llama.cpp這一大模型工具的使用。因為已經使用Ollama來run大模型，因此僅介紹了llama.cpp在HF模型轉GGUF模型中的應用，及其在大模型量化中的使用。大模型的參數量化技術，使得我們可以在本地有限預算的硬件條件下，也能夠運行DeepSeek的蒸餾模型。

文章轉載自：Dechin的博客

原文鏈接：DeepSeek模型量化 - DECHIN - 博客園

體驗地址：引邁 - JNPF快速開發平臺_低代碼開發平臺_零代碼開發平臺_流程設計器_表單引擎_工作流引擎_軟件架構