開源項目介紹：Native-LLM-for-Android

項目地址：Native-LLM-for-Android
創作活動時間：2025年

支持在 Android 設備上運行大型語言模型（LLM），具體支持的模型包括：
DeepSeek-R1-Distill-Qwen: 1.5B
Qwen2.5-Instruct: 0.5B, 1.5B
Qwen2/2.5VL: 2B, 3B
MiniCPM-DPO/SFT: 1B, 2.7B
Gemma2-it: 2B
Phi3.5-mini-instruct: 3.8B
Llama-3.2-Instruct: 1B

Native-LLM-for-Android項目主要提供2個參考點，1、將LLM模型導出為onnx模型，2、在安卓端實現LLL模型的運行，本博文主要關注將llm導出為onnx推理（對現有的llm模型進行局部修改并導出），并以miniCPM模型為例進行測試。同時，Native-LLM-for-Android項目還有一些列模型量化代碼可以學習。

1、模型運行性能

運行最快的模型是Llama3.2-1B-Instruct q8f32，達到25 token每秒，相應的硬件與os為 Nubia Z50（Android 13、8_Gen2-CPU）；其次是Distill-Qwen-1.5B q8f32，達到22 token每秒，相應的硬件與os為Xiaomi-14T-Pro （HyperOS 2、MediaTek_9300±CPU）；

在這里插入圖片描述

2、核心功能

這里主要關注將llm導出為onnx脫離torch環境運行，因此對Android運行代碼不予理會

2.1、分詞器

分詞器也就是Tokenizer ，一共兩個功能：
1、將輸入模型的文本，分為多個短詞，并轉換為token，
2、將模型輸出的token轉換為文本
需要注意的是，不同的llm模型分詞規則是不一樣的，同時對于的token編碼規則也不同

一般運行onnx模型，都是基于Transformer庫中的Tokenizer，這無法脫離torch環境。應該自行實現。

Native-LLM-for-Android項目中Tokenizer 依賴的是mnn-llm倉庫中的實現.
具體代碼鏈接為：
https://github.com/wangzhaode/mnn-llm/blob/master/src/tokenizer.cpp，是純c++代碼，與torch環境毫無關聯

同時，在每一種模型的Android-onnx代碼路徑下，都有對于的Tokenizer的c++實現代碼
在這里插入圖片描述

2.2、導出onnx模型

在Native-LLM-for-Android項目下Export_ONNX目錄中，每一個模型都有單獨的導出代碼
在這里插入圖片描述
如Gemma模型的導出，分別執行A-B-C步驟，導出3個模型，在最后的導出代碼中含onnx推理代碼

其中關于QwenVL模型的導出較為復雜，需要對transformers庫中modeling_qwen2_vl.py文件進行改寫覆蓋，將單個模型拆分為5個模型運行。其中A模型是VIT的主體部分，E模型是llm的主體部分，BCD模型是一些切片索引操作，被單獨導出為模型。關于E模型導出有報錯，可以參考https://github.com/DakeQQ/Native-LLM-for-Android/issues/10
在這里插入圖片描述

如果導出模型報錯

RuntimeError: The serialized model is larger than the 2GiB limit imposed by the protobuf library. Therefore the output file must be a file path, so that the ONNX external data can be written to the same directory. Please specify the output file name.

則嘗試將torch版本降低到2.4.1
pip install torch2.4.1 torchvision0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121

2.3、onnx模型量化

關于onnx模型量化，可以參考：https://blog.csdn.net/m0_63642362/article/details/124741589，根據介紹，onnx量化可以分為動態量化與靜態量化，動態量化在推理時根據輸入數據動態計算縮放因子與零點；靜態量化，使用校準數據集離線計算縮放因子（Scale）和零點（Zero Point）。通常，建議對 RNN 和基于 Transformer 的模型使用動態量化，對 CNN 模型使用靜態量化

在Native-LLM-for-Android-main\Do_Quantize\Dynamic_Quant 目錄下有多個模型量化代碼，具體如下圖所示
在這里插入圖片描述
q8_f16的量化代碼如下所示，可以看到對于大尺寸的模型的量化有一個關鍵參數項 is_large_model

import os
import gc
import glob
import sys
import onnx
import torch
import subprocess
import onnx.version_converter
from onnxsim import simplify
from onnxslim import slim
from onnxruntime.quantization import QuantType, quantize_dynamic
from onnxruntime.transformers.optimizer import optimize_model
from transformers import AutoModelForCausalLM# Path Setting
original_folder_path = r"C:\Users\Downloads\Model_ONNX"                          # The original folder.
quanted_folder_path = r"C:\Users\Downloads\Model_ONNX_Optimized"                 # The optimized folder.
model_path = os.path.join(original_folder_path, "Model.onnx")                    # The original fp32 model path.
quanted_model_path = os.path.join(quanted_folder_path, "Model_Optimized.onnx")   # The optimized model stored path.
download_path = r'C:\Users\Downloads\Qwen2-1.5B-Instruct'                        # Set the folder path where the LLM whole project downloaded, otherwise set "NONE".
use_gpu = True                                                                   # If true, the transformers.optimizer will remain the FP16 processes.
provider = 'CPUExecutionProvider'                                                # ['CPUExecutionProvider', 'CUDAExecutionProvider']
use_low_memory_mode_in_Android = False                                           # If you need to use low memory mode on Android, please set it to True.# Preprocess, it also cost alot of memory during preprocess, you can close this command and keep quanting. Call subprocess may get permission failed on Windows system.
# (optional process)
# subprocess.run([f'python -m onnxruntime.quantization.preprocess --auto_merge --all_tensors_to_one_file --input {model_path} --output {quanted_folder_path}'], shell=True)# Start Quantize
quantize_dynamic(model_input=model_path,model_output=quanted_model_path,per_channel=True,                                        # True for model accuracy but cost a lot of time during quanting process.reduce_range=False,                                      # True for some x86_64 platform.weight_type=QuantType.QInt8,                            # It is recommended using uint8 + Symmetric Falseextra_options={'ActivationSymmetric': False,             # True for inference speed. False may keep more accuracy.'WeightSymmetric': False,                 # True for inference speed. False may keep more accuracy.'EnableSubgraph': True,                   # True for more quant.'ForceQuantizeNoInputCheck': False,       # True for more quant.'MatMulConstBOnly': True                  # False for more quant. Sometime, the inference speed may get worse.},nodes_to_exclude=None,                                   # Specify the node names to exclude quant process. Example: nodes_to_exclude={'/Gather'}use_external_data_format=True                            # Save the model into two parts.
)model_size_bytes = sys.getsizeof(onnx.load(model_path).SerializeToString())
model_size_gb = model_size_bytes * 9.31322575e-10            # 1 / (1024 * 1024 * 1024)
if model_size_gb > 2.0:is_large_model = True
else:is_large_model = True if use_low_memory_mode_in_Android else False# ONNX Model Optimizer
slim(model=quanted_model_path,output_model=quanted_model_path,no_shape_infer=False,   # True for more optimize but may get errors.skip_fusion_patterns=False,no_constant_folding=False,save_as_external_data=is_large_model,verbose=False
)if download_path == "NONE":num_heads = 0    # defaulthidden_size = 0  # default
else:if ('vl' in download_path.lower()) & ('qwen' in download_path.lower()):if "2.5" in download_path or "3b" in download_path.lower():from transformers import Qwen2_5_VLForConditionalGenerationmodel = Qwen2_5_VLForConditionalGeneration.from_pretrained(download_path, torch_dtype=torch.float16, device_map='cpu', trust_remote_code=True, low_cpu_mem_usage=True).eval()else:from transformers import Qwen2VLForConditionalGenerationmodel = Qwen2VLForConditionalGeneration.from_pretrained(download_path, torch_dtype=torch.float16, device_map='cpu', trust_remote_code=True, low_cpu_mem_usage=True).eval()else:model = AutoModelForCausalLM.from_pretrained(download_path, torch_dtype=torch.float16, device_map='cpu', trust_remote_code=True, low_cpu_mem_usage=True).eval()num_heads = model.config.num_attention_headshidden_size = model.config.hidden_sizedel modelgc.collect()# transformers.optimizer
model = optimize_model(quanted_model_path,use_gpu=use_gpu,opt_level=2,num_heads=num_heads,hidden_size=hidden_size,provider=provider,verbose=False,model_type='bert')
model.convert_float_to_float16(keep_io_types=True,force_fp16_initializers=True,use_symbolic_shape_infer=True,  # True for more optimize but may get errors.op_block_list=['DynamicQuantizeLinear', 'DequantizeLinear', 'DynamicQuantizeMatMul', 'Range', 'MatMulIntegerToFloat']
)
model.save_model_to_file(quanted_model_path, use_external_data_format=is_large_model)
del model
gc.collect()# onnxslim 2nd
slim(model=quanted_model_path,output_model=quanted_model_path,no_shape_infer=False,   # True for more optimize but may get errors.skip_fusion_patterns=False,no_constant_folding=False,save_as_external_data=is_large_model,verbose=False
)# Upgrade the Opset version. (optional process)
model = onnx.load(quanted_model_path)
model = onnx.version_converter.convert_version(model, 21)
onnx.save(model, quanted_model_path, save_as_external_data=is_large_model)if is_large_model:pattern = os.path.join(quanted_folder_path, '*.data')files_to_delete = glob.glob(pattern)for file_path in files_to_delete:try:os.remove(file_path)except Exception as e:print(f"Error deleting {file_path}: {e}")# It is not recommended to convert an FP16 ONNX model to the ORT format because this process adds a Cast operation to convert the FP16 process back to FP32.

3、導出minicpm模型onnx推理

3.1 下載模型

pip install modelscope

基于modelscope 庫可以下載MiniCPM-2B-dpo-fp16模型

from modelscope import snapshot_download
model_dir = snapshot_download('OpenBMB/MiniCPM-2B-dpo-fp16',cache_dir=".cache_dir")

3.2 導出onnx模型

這里以MiniCPM-2B-split導出方式為例

先在命令行進入 F:\Native-LLM-for-Android-main\Export_ONNX\MiniCPM\MiniCPM-2B-split 目錄

然后創建，model_a，model_b兩個目錄，用于存儲2個onnx模型，并將代碼修改為以下形式
在這里插入圖片描述

最后在命令行中執行 python .\MiniCPM_Export.py 即可實現模型導出為onnx，并進行推理測試
在這里插入圖片描述
這里可以發現代碼的推理速度居然為0.375token/s，簡直巨慢。

按照單個模型導出，并進行推理測試，發現效果如下所示，可以發現性能有6倍的提升，這表明數據通信也占據了大量的耗時。
在這里插入圖片描述

3.3 單獨運行onnx

基于以下代碼可以運行onnx模型，但無法脫離transformers庫，除非手寫tokenizer實現分詞，并實現token與文本的對應關系。


import numpy as np
import onnxruntime
from transformers import AutoModelForCausalLM, AutoTokenizer
import timepath = 'F:\DMT\.cache_dir\OpenBMB\MiniCPM-2B-dpo-fp16'  # Set the folder path where the MiniCPM whole project downloaded.
onnx_model_A = r'F:\Native-LLM-for-Android-main\Export_ONNX\MiniCPM\MiniCPM-2B-single\model_q8_f16\MiniCPM_part_A_Optimized.onnx'  # Assign a path where the exported MiniCPM_part_A stored.# Run the exported model by ONNX Runtime
query = "山東省最高的山是哪座山, 它比黃山高還是矮？差距多少？"
max_seq_len = 1024  # Please modify the same variable, which declared in the modified modeling_minicpm.py on line 1008, at the same time.
num_heads = 36
head_dim = 64
num_key_value_heads = 36
num_layers = 40
hidden_size = 2304max_single_chat_length = 341  # It a adjustable value, but must less than max_seq_len.
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)# ONNX Runtime settings
session_opts = onnxruntime.SessionOptions()
session_opts.log_severity_level = 3  # error level, it a adjustable value.
session_opts.inter_op_num_threads = 0  # Run different nodes with num_threads. Set 0 for auto.
session_opts.intra_op_num_threads = 0  # Under the node, execute the operators with num_threads. Set 0 for auto.
session_opts.enable_cpu_mem_arena = True  # True for execute speed; False for less memory usage.
session_opts.execution_mode = onnxruntime.ExecutionMode.ORT_SEQUENTIAL
session_opts.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
session_opts.add_session_config_entry("session.intra_op.allow_spinning", "1")
session_opts.add_session_config_entry("session.inter_op.allow_spinning", "1")ort_session_A = onnxruntime.InferenceSession(onnx_model_A, sess_options=session_opts, providers=['CPUExecutionProvider'])
in_name_A = ort_session_A.get_inputs()
out_name_A = ort_session_A.get_outputs()
in_name_A0 = in_name_A[0].name
in_name_A1 = in_name_A[1].name
in_name_A2 = in_name_A[2].name
in_name_A3 = in_name_A[3].name
in_name_A4 = in_name_A[4].name
in_name_A5 = in_name_A[5].name
out_name_A0 = out_name_A[0].name
out_name_A1 = out_name_A[1].name
out_name_A2 = out_name_A[2].name# Pre-process inputs
prompt = tokenizer.apply_chat_template([{"role": 'user', "content": query}], tokenize=False, add_generation_prompt=False)
token = tokenizer(prompt, return_tensors='pt')['input_ids']
ids_len = token.shape[1] + np.zeros(1, dtype=np.int64)
input_ids = np.zeros(max_seq_len, dtype=np.int32)
input_ids[:ids_len[0]] = token[0, :]
attention_mask = np.array([-65504.0], dtype=np.float32)
history_len = np.zeros(1, dtype=np.int64)
past_key_states_A = np.zeros((num_layers, num_key_value_heads, max_seq_len, head_dim), dtype=np.float16)
past_values_states_A = past_key_states_A
num_decode = 0
print('\nTest Question: ' + query + "\n\nMiniCPM Answering:\n")# Start to run LLM
start_time = time.time()
while history_len < max_single_chat_length:token_id, past_key_states_A, past_values_states_A = ort_session_A.run([out_name_A0, out_name_A1, out_name_A2],{in_name_A0: input_ids,in_name_A1: attention_mask,in_name_A2: past_key_states_A,in_name_A3: past_values_states_A,in_name_A4: history_len,in_name_A5: ids_len})if token_id == 2:  # the stop_id in MiniCPM is "2"breakelse:history_len[0] += ids_len[0]ids_len[0] = 1num_decode += 1attention_mask[0] = 0.0input_ids[0] = token_idprint(tokenizer.decode(token_id), end="", flush=True)
end_time = time.time()
print(f"\n\nDecode: {(num_decode / (end_time - start_time)):.3f} token/s")