序

本文主要研究一下如何本地運行local-ai并通過langchain4j集成調用。

步驟

curl安裝

curl https://localai.io/install.sh | sh% Total    % Received % Xferd  Average Speed   Time    Time     Time  CurrentDload  Upload   Total   Spent    Left  Speed
100 21509    0 21509    0     0  45959      0 --:--:-- --:--:-- --:--:-- 45861Downloading local-ai...
######################################################################## 100.0%Installing local-ai to /usr/local/bin...Creating local-ai user...Adding current user to local-ai group...Creating local-ai systemd service...The LocalAI API is now available at 127.0.0.1:8080.Install complete. Run "local-ai" from the command line.
[WARN]  No NVIDIA/AMD GPU detected. LocalAI will run in CPU-only mode.

運行

local-ai
9:49AM INF env file found, loading environment variables from file envFile=/etc/localai.env
9:49AM INF Setting logging to info
9:49AM INF Starting LocalAI using 16 threads, with models path: /usr/share/local-ai/models
9:49AM INF LocalAI version: v2.26.0 (09941c0bfb9119bb01a04b2a0a16897ecf2cd087)
9:49AM INF Preloading models from /usr/share/local-ai/models
9:49AM INF core/startup process completed!
9:49AM INF LocalAI API is listening! Please connect to the endpoint for API documentation. endpoint=http://0.0.0.0:8080

訪問http://localhost:8080

安裝model

訪問http://localhost:8080/browse 瀏覽模型并選擇安裝
或者使用命令行，這里選擇一個小模型

local-ai models install llama-3.2-1b-instruct:q4_k_m

https://hf-mirror.com/hugging-quants/Llama-3.2-1B-Instruct-Q4_K_M-GGUF

pom.xml

        <dependency><groupId>dev.langchain4j</groupId><artifactId>langchain4j-local-ai</artifactId><version>1.0.0-beta1</version></dependency>

controller

@RequestMapping("/local-ai")
@RestController
@Slf4j
public class LocalAiController {ChatLanguageModel model = LocalAiChatModel.builder().baseUrl("http://localhost:8080/v1").modelName("llama-3.2-1b-instruct:q4_k_m")
//            .maxTokens(3).logRequests(true).logResponses(true).build();/*** curl -i http://localhost:8080/local-ai/chat?prompt=who are you* @param prompt* @return*/@GetMapping("/chat")public String chat(@RequestParam("prompt") String prompt) {return model.chat(prompt);}}

請求

http://localhost:8080/local-ai/chat?prompt=who%20are%20you
返回

I'm an AI assistant, which means I'm a computer program designed to understand and respond to human language. I'm here to help with a wide range of topics, answer questions, provide information, and even chat with you in a friendly and helpful way. I'm a large language model, which means I've been trained on a massive database of text from the internet, books, and other sources, which allows me to generate responses that are often surprisingly accurate and insightful. Some common ways I can assist you include: * Answering questions on a particular topic * Providing definitions and explanations * Offering suggestions and ideas * Summarizing long pieces of text * Translating languages (I can translate text from one language to another) * Offering writing and proofreading services * Generating text on a given topic or topic I'm always learning and improving, so please bear with me if I make any mistakes or don't quite understand the question. I'm here to help, so feel free to ask me anything that's on your mind!

小結

LocalAI是OpenAI 的免費開源替代產品，提供了與 OpenAI 兼容的直接替代 REST API，可以在本地直接運行大語言模型LLM、生成圖像、音頻等。它提供了docker安裝及腳本安裝。langchain4j提供了langchain4j-local-ai用于集成local-ai.

doc

• langchain4j local-ai
• localai.io