一、安裝xinference

pip install xinference

二、啟動xinference

./xinference-local  --host=0.0.0.0  --port=5544

三、注冊本地模型

1、注冊embedding模型
curl -X POST "http://localhost:5544/v1/models" \
-H "Content-Type: application/json" \
-d '{"model_type": "embedding","model_name": "bce-embedding-base_v1",  "model_uid": "bce-embedding-base_v1",   "model_path": "/root/embed_rerank/bce-embedding-base_v1/" 
}'驗證：
curl -X POST "http://localhost:5544/v1/embeddings" \
-H "Content-Type: application/json" \
-d '{"model": "bce-embedding-base_v1","input": ["需要嵌入的文本1", "這是第二個句子"]
}'2、注冊rerank模型curl -X POST "http://localhost:5544/v1/models" \
-H "Content-Type: application/json" \
-d '{"model_type": "rerank",              "model_name": "bce-reranker-base_v1",     "model_uid": "bce-reranker-base_v1",   "model_path": "/root/embed_rerank/bce-reranker-base_v1" 
}'驗證
curl -X POST "http://localhost:5544/v1/rerank" \
-H "Content-Type: application/json" \
-d '{"model": "bge-reranker-v2-m3","query": "What is Python?","documents": ["Python is a programming language.","Java is another language.","Python is used for web development."]
}'3、執行./xinference list 查看運行模型

四、刪除模型

curl -X DELETE "http://localhost:5544/v1/models/bge-reranker-v2-m3"

五、備注

1、在cpu運行

• 服務器有顯卡但是選擇用cpu加載

? ? ? ? ? ? ?啟動xinference之前設置

? ? ? ? ? ? ? export CUDA_VISIBLE_DEVICES=""

• 服務器無顯卡會自動在cpu加載模型

2、在gpu運行

啟動服務器前設置環境變量

export CUDA_VISIBLE_DEVICES=""

curl -X POST "http://localhost:5544/v1/models" \
-H "Content-Type: application/json" \
-d '{"model_type": "embedding","model_name": "bce-embedding-base_v1",  "model_uid": "bce-embedding-base_v1",   "model_path": "/root/zml/embed_rerank/bce-embedding-base_v1/" "gpu_idx": 1"n_gpu" : 1
}'備注：
gpu_idx :選用的顯卡index
n_gpu:選定的顯卡總張數