1.背景

上一篇文章寫到，【個人開發】macbook m1 Lora微調qwen大模型

該微調方式，同樣適用于GPU，只不過在train.py腳本中，針對device，調整為cuda即可。

如果數據量過大的話，單卡微調會存在瓶頸，因此考慮多GPU進行微調。

網上搜羅了一圈，多卡微調的常用方案：deepspeed+Llama-factory。

本文主要記錄該方式的微調情況，僅為個人學習記錄

2.微調方式

2.1 關鍵環境版本信息

模塊	版本
python	3.10
CUDA	12.6
torch	2.5.1
peft	0.12.0
transformers	4.46.2
accelerate	1.1.1
trl	0.9.6
deepspeed	0.15.4

2.2 步驟

2.2.1 下載llama-factory

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]"

2.2.2 準備數據集

數據集采用網上流傳的《甄嬛傳》。

數據源地址：huanhuan.json

數據集結構如下。

// 文件命名：huanhuan.json
[{"instruction": "小姐，別的秀女都在求中選，唯有咱們小姐想被撂牌子，菩薩一定記得真真兒的——","input": "","output": "噓——都說許愿說破是不靈的。"},...
]

其次，還得準備數據集信息【dataset_info.json】,因為是本地微調，所以微調時現訪問dataset_info，再指定到具體的數據集中。

{"identity": {"file_name": "test_data.json"}
}

注意文本的數據集的格式必須為，json，不然會報錯。

2.2.3 微調模式

2.2.3.1 zero-1微調

配置參考zero-3的配置，修改了一下zero_optimization.stage的參數。

// 文件命名：ds_config_zero1.json
{"fp16": {"enabled": "auto","loss_scale": 0,"loss_scale_window": 1000,"initial_scale_power": 16,"hysteresis": 2,"min_loss_scale": 1},"bf16": {"enabled": "auto"},"optimizer": {"type": "AdamW","params": {"lr": "auto","betas": "auto","eps": "auto","weight_decay": "auto"}},"scheduler": {"type": "WarmupLR","params": {"warmup_min_lr": "auto","warmup_max_lr": "auto","warmup_num_steps": "auto"}},"zero_optimization": {"stage": 1,"offload_optimizer": {"device": "none","pin_memory": true},"offload_param": {"device": "none","pin_memory": true},"overlap_comm": true,"contiguous_gradients": true,"sub_group_size": 1e9,"reduce_bucket_size": "auto","stage3_prefetch_bucket_size": "auto","stage3_param_persistence_threshold": "auto","stage3_max_live_parameters": 1e9,"stage3_max_reuse_distance": 1e9,"stage3_gather_16bit_weights_on_model_save": true},"gradient_accumulation_steps": 4,"gradient_clipping": "auto","steps_per_print": 100,"train_batch_size": "auto","train_micro_batch_size_per_gpu": "auto","wall_clock_breakdown": false
}

微調腳本

# run_train_bash_zero_1.sh
#!/bin/bash
# 記錄開始時間
START=$(date +%s.%N)CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch  src/train.py \--deepspeed ds_config_zero1.json \--stage sft \--do_train True \--model_name_or_path /root/ai_project/fine-tuning-by-lora/models/model/qwen/Qwen2___5-7B-Instruct \--finetuning_type lora \--template qwen \--dataset_dir /root/ai_project/fine-tuning-by-lora/dataset/ \--dataset identity \--cutoff_len 1024 \--num_train_epochs 30 \--max_samples 100000 \--learning_rate 5e-05 \--lr_scheduler_type cosine \--warmup_steps 10 \--per_device_train_batch_size 4 \--gradient_accumulation_steps 4 \--max_grad_norm 1.0 \--logging_steps 10 \--save_steps 100 \--neftune_noise_alpha 0 \--lora_rank 8 \--lora_dropout 0.1 \--lora_alpha 32 \--lora_target q_proj,v_proj,k_proj,gate_proj,up_proj,o_proj,down_proj \--output_dir ./output/qwen_7b_ft/zero1/ \--bf16 True \--plot_loss True# 記錄結束時間
END=$(date +%s.%N)# 計算運行時間
DUR=$(echo "$END - $START" | bc)# 輸出運行時間
printf "Execution time: %.6f seconds\n" $DUR

2.2.3.2 zero-2微調

zero-2下述的配置中，調度器使用了AdamW，學習率在訓練時候可以逐步下降。

// 文件命名：ds_config_zero2.json
{"fp16": {"enabled": "auto","loss_scale": 0,"loss_scale_window": 1000,"initial_scale_power": 16,"hysteresis": 2,"min_loss_scale": 1},"bf16": {"enabled": "auto"},"optimizer": {"type": "AdamW","params": {"lr": "auto","betas": "auto","eps": "auto","weight_decay": "auto"}},"zero_optimization": {"stage": 2,"offload_optimizer": {"device": "cpu","pin_memory": true}},"gradient_accumulation_steps": 4,"gradient_clipping": "auto","steps_per_print": 100,"train_batch_size": "auto","train_micro_batch_size_per_gpu": "auto","wall_clock_breakdown": false
}

2.2.3.3 zero-3微調

本次微調采用zero-3的方式，因此在LLaMa-Factory目錄下，新增配置文件。
相關配置可參考Llama-Factory提供的文件樣例[./LLaMA-Factory/examples/deepspeed/]

// 文件命名：ds_config_zero3.json
{"fp16": {"enabled": "auto","loss_scale": 0,"loss_scale_window": 1000,"initial_scale_power": 16,"hysteresis": 2,"min_loss_scale": 1},"bf16": {"enabled": "auto"},"optimizer": {"type": "AdamW","params": {"lr": "auto","betas": "auto","eps": "auto","weight_decay": "auto"}},"scheduler": {"type": "WarmupLR","params": {"warmup_min_lr": "auto","warmup_max_lr": "auto","warmup_num_steps": "auto"}},"zero_optimization": {"stage": 3,"offload_optimizer": {"device": "none","pin_memory": true},"offload_param": {"device": "none","pin_memory": true},"overlap_comm": true,"contiguous_gradients": true,"sub_group_size": 1e9,"reduce_bucket_size": "auto","stage3_prefetch_bucket_size": "auto","stage3_param_persistence_threshold": "auto","stage3_max_live_parameters": 1e9,"stage3_max_reuse_distance": 1e9,"stage3_gather_16bit_weights_on_model_save": true},"gradient_accumulation_steps": "auto","gradient_clipping": "auto","steps_per_print": 100,"train_batch_size": "auto","train_micro_batch_size_per_gpu": "auto","wall_clock_breakdown": false
}

微調腳本

# run_train_bash.sh 
#!/bin/bash
# 記錄開始時間
START=$(date +%s.%N)
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch  src/train.py \--deepspeed ds_config_zero3.json \--stage sft \--do_train True \--model_name_or_path /root/ai_project/fine-tuning-by-lora/models/model/qwen/Qwen2___5-7B-Instruct \--finetuning_type lora \--template qwen \--dataset_dir /root/ai_project/fine-tuning-by-lora/dataset/ \--dataset identity \--cutoff_len 1024 \--num_train_epochs 5 \--max_samples 100000 \--per_device_train_batch_size 4 \--gradient_accumulation_steps 4 \--lr_scheduler_type cosine \--learning_rate 5e-04 \--lr_scheduler_type cosine \--max_grad_norm 1.0 \--logging_steps 5 \--save_steps 100 \--neftune_noise_alpha 0 \--lora_rank 8 \--lora_dropout 0.1 \--lora_alpha 32 \--lora_target q_proj,v_proj,k_proj,gate_proj,up_proj,o_proj,down_proj \--output_dir ./output/qwen_7b_ds/train_2025_02_13 \--bf16 True \--plot_loss True# 記錄結束時間
END=$(date +%s.%N)
# 計算運行時間
DUR=$(echo "$END - $START" | bc)
# 輸出運行時間
printf "Execution time: %.6f seconds\n" $DUR

說明一下上述一些關鍵參數：

參數	版本
–deepspeed	指定deepspeed加速微調方式
–model_name_or_path	微調模型路徑
–finetuning_type	微調方式，這里用lora微調
–template	訓練和推理時構造 prompt 的模板，不同大語言模型的模板不一樣，這里用的是qwen
–dataset_dir	本地的數據集路徑
–dataset	指定dataset_info.json中哪個數據集
–lora_target	應用 LoRA 方法的模塊名稱。
–output_dir	模型輸出路徑。

模型微調參數可以參考：Llama-Factory參數介紹

其他參數，其實就是常規使用peft進行lora微調的常見參數，以及常見的微調參數，可以對照如下。

lora_config = LoraConfig(task_type=TaskType.CAUSAL_LM,target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],inference_mode=False,r=8,lora_alpha=32,lora_dropout=0.1
)

2.2.3.4 單卡Lora微調

具體使用可以參考上一篇文章：【個人開發】macbook m1 Lora微調qwen大模型
也可以參考github項目：fine-tuning-by-Lora

微調代碼如下。

torch_dtype = torch.halflora_config = LoraConfig(task_type=TaskType.CAUSAL_LM,target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],inference_mode=False,r=8,lora_alpha=32,lora_dropout=0.1
)def train():# 加載模型model_dir = snapshot_download(model_id=model_id, cache_dir=f"{models_dir}/model", revision='master')if model_path != model_dir:raise Exception(f"model_path:{model_path} != model_dir:{model_dir}")model = AutoModelForCausalLM.from_pretrained(model_path,device_map=device, torch_dtype=torch_dtype)model.enable_input_require_grads()  # 開啟梯度檢查點時，要執行該方法# 加載數據df = pd.read_json(dataset_file)ds = Dataset.from_pandas(df)print(ds[:3])# 處理數據tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=False, trust_remote_code=True)tokenizer.pad_token = tokenizer.eos_tokendef process_func(item):MAX_LENGTH = 384  # Llama分詞器會將一個中文字切分為多個token，因此需要放開一些最大長度，保證數據的完整性input_ids, attention_mask, labels = [], [], []instruction = tokenizer(f"<|start_header_id|>user<|end_header_id|>\n\n{item['instruction'] + item['input']}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",add_special_tokens=False)  # add_special_tokens 不在開頭加 special_tokensresponse = tokenizer(f"{item['output']}<|eot_id|>", add_special_tokens=False)input_ids = instruction["input_ids"] + response["input_ids"] + [tokenizer.pad_token_id]attention_mask = instruction["attention_mask"] + response["attention_mask"] + [1]  # 因為eos token咱們也是要關注的所以 補充為1labels = [-100] * len(instruction["input_ids"]) + response["input_ids"] + [tokenizer.pad_token_id]if len(input_ids) > MAX_LENGTH:  # 做一個截斷input_ids = input_ids[:MAX_LENGTH]attention_mask = attention_mask[:MAX_LENGTH]labels = labels[:MAX_LENGTH]return {"input_ids": input_ids,"attention_mask": attention_mask,"labels": labels}tokenized_id = ds.map(process_func, remove_columns=ds.column_names)tokenizer.decode(list(filter(lambda x: x != -100, tokenized_id[1]["labels"])))# 加載lora權重model = get_peft_model(model, lora_config)# 訓練模型training_args = TrainingArguments(output_dir=checkpoint_dir,per_device_train_batch_size=4,gradient_accumulation_steps=4,logging_steps=5,num_train_epochs=30,save_steps=100,learning_rate=5e-04,save_on_each_node=True,gradient_checkpointing=True,)trainer = Trainer(model=model,args=training_args,train_dataset=tokenized_id,data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True),)trainer.train()# 保存模型trainer.model.save_pretrained(lora_dir)tokenizer.save_pretrained(lora_dir)

2.2.4 實驗

本次測試使用多GPU微調，測試多GPU微調跟單GPU微調的性能對比。

使用2,030條數據，epoch = 30 ，batch size = 4，Gradient Accumulation steps = 4

實驗組	實驗類別	步數	耗時	最終loss
實驗1	zero1微調	480	09:00	0.0101
實驗2	zero2微調	480	09:59	0.4757
實驗3	zero3微調	480	1:49:11	0.0746
實驗4	單卡lora微調	3810	1:07:57	0.0009

初步結論：
1.基于實驗1，實驗3的對照，使用zero3微調，耗時明顯提升的原因還是資源使用不合理【沒充分使用GPU】。
2.基于實驗1，實驗3跟實驗2的對照，實驗2的損失下降比較慢的一個原因是因為使用的學習率調度器的問題。

2.2.4.1 實驗1：多GPU微調-zero1

日志如下

[INFO|trainer.py:2369] 2025-02-18 09:44:50,875 >> ***** Running training *****
[INFO|trainer.py:2370] 2025-02-18 09:44:50,875 >>   Num examples = 2,030
[INFO|trainer.py:2371] 2025-02-18 09:44:50,875 >>   Num Epochs = 30
[INFO|trainer.py:2372] 2025-02-18 09:44:50,875 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2375] 2025-02-18 09:44:50,875 >>   Total train batch size (w. parallel, distributed & accumulation) = 128
[INFO|trainer.py:2376] 2025-02-18 09:44:50,875 >>   Gradient Accumulation steps = 4
[INFO|trainer.py:2377] 2025-02-18 09:44:50,875 >>   Total optimization steps = 480
[INFO|trainer.py:2378] 2025-02-18 09:44:50,878 >>   Number of trainable parameters = 20,185,088
.....
***** train metrics *****epoch                    =        30.0total_flos               = 234733999GFtrain_loss               =      1.0322train_runtime            =  0:09:00.75train_samples_per_second =     112.619train_steps_per_second   =       0.888
Figure saved at: ./output/qwen_7b_ft/zero1/training_loss.png

GPU使用情況。

loss下降情況如下：

2.2.4.2 實驗2：多GPU微調-zero2

使用2,030條數據，8卡微調，微調參數如下,總共480步，耗時09:59。

[INFO|trainer.py:2369] 2025-02-17 12:53:54,461 >> ***** Running training *****
[INFO|trainer.py:2370] 2025-02-17 12:53:54,461 >>   Num examples = 2,030
[INFO|trainer.py:2371] 2025-02-17 12:53:54,461 >>   Num Epochs = 30
[INFO|trainer.py:2372] 2025-02-17 12:53:54,461 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2375] 2025-02-17 12:53:54,461 >>   Total train batch size (w. parallel, distributed & accumulation) = 128
[INFO|trainer.py:2376] 2025-02-17 12:53:54,461 >>   Gradient Accumulation steps = 4
[INFO|trainer.py:2377] 2025-02-17 12:53:54,461 >>   Total optimization steps = 480
[INFO|trainer.py:2378] 2025-02-17 12:53:54,465 >>   Number of trainable parameters = 20,185,088***** train metrics *****epoch                    =        30.0total_flos               = 234733999GFtrain_loss               =      1.6736train_runtime            =  0:09:59.38train_samples_per_second =     101.605train_steps_per_second   =       0.801
Figure saved at: ./output/qwen_7b_ft/zero2/training_loss.png

GPU使用情況如下：

損失下降情況：

2.2.4.3 實驗3：多GPU微調-zero3

使用2,030條數據，8卡微調，微調參數如下,總共480步，耗時1:49:11。

[INFO|trainer.py:2369] 2025-02-17 13:07:48,438 >> ***** Running training *****
[INFO|trainer.py:2370] 2025-02-17 13:07:48,438 >>   Num examples = 2,030
[INFO|trainer.py:2371] 2025-02-17 13:07:48,438 >>   Num Epochs = 30
[INFO|trainer.py:2372] 2025-02-17 13:07:48,438 >>   Instantaneous batch size per device = 4
[INFO|trainer.py:2375] 2025-02-17 13:07:48,438 >>   Total train batch size (w. parallel, distributed & accumulation) = 128
[INFO|trainer.py:2376] 2025-02-17 13:07:48,438 >>   Gradient Accumulation steps = 4
[INFO|trainer.py:2377] 2025-02-17 13:07:48,438 >>   Total optimization steps = 480
[INFO|trainer.py:2378] 2025-02-17 13:07:48,442 >>   Number of trainable parameters = 20,185,088...***** train metrics *****epoch                    =       30.0total_flos               =   257671GFtrain_loss               =     0.3719train_runtime            = 1:49:11.88train_samples_per_second =      9.295train_steps_per_second   =      0.073
Figure saved at: ./output/qwen_7b_ft/zero3/training_loss.png
[WARNING|2025-02-17 14:57:11] llamafactory.extras.ploting:162 >> No metric eval_loss to plot.
[WARNING|2025-02-17 14:57:11] llamafactory.extras.ploting:162 >> No metric eval_accuracy to plot.
[INFO|modelcard.py:449] 2025-02-17 14:57:11,629 >> Dropping the following result as it does not have all the necessary fields:

GPU使用情況如下：

損失下降情況：

2.2.4.4 實驗4：Lora單卡微調

單卡微調，總共需要3810步。

2.2.5 合并大模型并啟動

2.2.5.1 方法一：Llama-factory合并，并使用ollama調用大模型

模型合并

利用Llama-factory的框架，配置llama3_lora_sft_qwen.yaml 文件，進行模型合并。

# llama3_lora_sft_qwen.yaml
### model
model_name_or_path: /root/ai_project/fine-tuning-by-lora/models/model/qwen/Qwen2___5-7B-Instruct
adapter_name_or_path: /root/ai_project/LLaMA-Factory/output/qwen_7b_ds/zero2/
template: qwen
trust_remote_code: true### export
export_dir: output/llama3_lora_sft_qwen
export_size: 5
export_device: gpu
export_legacy_format: false

llamafactory-cli export llama3_lora_sft_qwen.yaml

模型打包

合并完成后，會有直接生成Modelfile文件，可以直接打包到ollama中。

# ollama modelfile auto-generated by llamafactory
FROM .TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ range .Messages }}{{ if eq .Role "user" }}<|im_start|>user
{{ .Content }}<|im_end|>
<|im_start|>assistant
{{ else if eq .Role "assistant" }}{{ .Content }}<|im_end|>
{{ end }}{{ end }}"""SYSTEM """You are a helpful assistant."""PARAMETER stop "<|im_end|>"
PARAMETER num_ctx 4096

模型啟動
ollama啟動

ollama create llama3_lora_sft_qwen -f Modelfile

參考文章：大模型開發和微調工具Llama-Factory–＞LoRA合并

2.2.5.2 方法二：Llama-factory合并，并使用vllm啟動模型服務

模型的合并同方法一，之后使用vllm命令啟動。

vllm命令啟動模型服務

# 內置了vllm的qwen的template。
CUDA_VISIBLE_DEVICES=1,2,3,4 python3 -m vllm.entrypoints.openai.api_server \--model "/root/ai_project/LLaMA-Factory/output/merge/" \--port 6006 \--tensor-parallel-size 4 \--served-model-name Qwen2.5-7B-sft \--max-model-len 8192 \--dtype half \--host 0.0.0.0

模型服務接口調用

import requestsdef chat_with_vllm(prompt, port=6006):url = f"http://localhost:{port}/v1/chat/completions"headers = {"Content-Type": "application/json"}data = {"model": "Qwen2.5-7B-sft",  # 模型名稱或路徑"messages": [{"role": "user", "content": prompt}],"max_tokens": 512,"temperature": 0.7}response = requests.post(url, headers=headers, json=data)if response.status_code == 200:result = response.json()generated_text = result["choices"][0]["message"]["content"]print(generated_text.strip())else:print("Error:", response.status_code, response.text)# 示例調用
chat_with_vllm("你是誰？", port=6006)

服務日志：

說明：日志中可以看到template。

調用結果：

3 踩坑經驗

3.1 微調踩坑

3.1.1 問題一：ValueError: Undefined dataset xxxx in dataset_info.json.

如果你腳本的啟動參數，–dataset identity。而dataset_info.json中的數據信息，沒有“identity”這個key，則會出現這個報錯，只要確保你dataset_info.json中存在該key即可。

3.1.2 問題二： ValueError: Target modules {‘c_attn’} not found in the base model. Please check the target modules and try again.

如果你腳本的啟動參數，–lora_target參數設為常見的c_attn參數，則會報此錯。處理方式還是調整參數，使用Lora微調時的常見參數，q_proj,v_proj,k_proj,gate_proj,up_proj,o_proj,down_proj。注意格式，如果格式不對，還是會報錯。

3.1.3 問題三： RuntimeError: The size of tensor a (1060864) must match the size of tensor b (315392) at non-singleton dimension 0。

這種tensor的問題，很可能是模型沖突的問題，比如調到一半，然后重新提調，指到相同的路徑。重新指定output路徑即可。

3.1.4 問題四： 訓練效率問題

在GPU充分的情況下，使用zero_2的訓練效率，很明顯比zero_3的訓練效率更快！

【后續，持續更新。。。】