1、創建虛擬環境

conda create -n myenv python=3.12 -y

2、激活虛擬環境

conda activate myenv

3、安裝相關庫

pip install vllm fastapi uvicorn

4、編寫腳本（test.py）

from fastapi import FastAPI, Request
from vllm import LLM, SamplingParams
import uvicorn# Initialize FastAPI
app = FastAPI()# Load the model once at startup with adjusted parameters
model_path = "/home/zhengyihan/.cache/modelscope/hub/LLM-Research/Llama-3___2-3B-Instruct"
llm = LLM(model=model_path,max_model_len=8192,  # Reduced from defaultgpu_memory_utilization=0.95  # Increase memory allocation
)@app.post("/generate")
async def generate(request: Request):# Parse the request bodybody = await request.json()# Extract parameters from the requestprompt = body.get("prompt", "")temperature = body.get("temperature", 0.7)top_p = body.get("top_p", 0.95)max_tokens = body.get("max_tokens", 512)  # Reduced default# Set up sampling parameterssampling_params = SamplingParams(temperature=temperature,top_p=top_p,max_tokens=max_tokens)# Generate the responseoutputs = llm.generate(prompt, sampling_params)# Extract the generated textresults = []for output in outputs:results.append({"generated_text": output.outputs[0].text,"prompt": output.prompt})return {"results": results}if __name__ == "__main__":uvicorn.run(app, host="0.0.0.0", port=8000)

調用腳本

python test.py

5、bash中測試通信

curl -X POST http://localhost:8000/generate -H "Content-Type: application/json" -d '{"prompt": "Once upon a time"}'

完美結果