開源模型應用落地-DeepSeek-R1-Distill-Qwen-7B-LoRA微調-LLaMA-Factory-單機單卡-V100（一）

一、前言

??? 如今，大語言模型領域熱鬧非凡，各種模型不斷涌現。DeepSeek-R1-Distill-Qwen-7B 模型憑借其出色的效果和性能，吸引了眾多開發者的目光。而 LLaMa-Factory 作為強大的微調工具，能讓模型更好地滿足個性化需求。

??? 在本篇中，將深入探討如何運用 LLaMa-Factory 對 DeepSeek-R1-Distill-Qwen-7B 模型進行微調，探索如何通過微調，讓模型更好地為我們所用。

二、術語介紹

2.1. LoRA微調

? ? LoRA (Low-Rank Adaptation) 用于微調大型語言模型 (LLM)。 ?是一種有效的自適應策略，它不會引入額外的推理延遲，并在保持模型質量的同時顯著減少下游任務的可訓練參數數量。

2.2. 參數高效微調(PEFT)?

? ? 僅微調少量 (額外) 模型參數，同時凍結預訓練 LLM 的大部分參數，從而大大降低了計算和存儲成本。

2.3. LLaMA-Factory

? ? 是一個與 LLaMA（Large Language Model Meta AI）相關的項目，旨在為用戶提供一種簡化和優化的方式來訓練、微調和部署大型語言模型。該工具通常包括一系列功能，如數據處理、模型配置、訓練監控等，以幫助研究人員和開發者更高效地使用 LLaMA 模型。

? ? LLaMA-Factory支持的模型列表：

2.4. DeepSeek-R1-Distill-Qwen-7B

? ? 是一個由DeepSeek開發的模型，它是通過蒸餾技術將Qwen-7B大型模型的一部分知識精華提取出來，以適應更小型的模型需求。

三、前置條件

?3.1. 基礎環境及前置條件

?? ? 1. 操作系統：centos7

? ? ?2. NVIDIA Tesla V100 32GB ? CUDA Version: 12.2?

? ? ?3. 提前下載好DeepSeek-R1-Distill-Qwen-7B模型? ? ? ? ?

?通過以下兩個地址進行下載，優先推薦魔搭? ? ? ??

huggingface：

https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/tree/main

ModelScope：

魔搭社區

?按需選擇SDK或者Git方式下載

? 使用git-lfs方式下載示例：

3.2.?Anaconda安裝

1、Update Systemsudo yum update -ysudo yum upgrade -y2、Download Anacondawget https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh3、Verify Data Integritysha256sum Anaconda3-2022.10-Linux-x86_64.sh4、Run Anaconda Installation Scriptbash Anaconda3-2022.10-Linux-x86_64.sh安裝目錄：/opt/anaconda3注：安裝位置可以在執行安裝腳本的時候直接指定，可以這樣修改執行內容bash Anaconda3-2022.10-Linux-x86_64.sh -p /opt/anaconda3Do you wish the installer to initialize Anaconda3 by running conda init?yes如果沒有執行初始化，可以執行：/opt/anaconda3/bin/conda init注：初始化時，anaconda將配置寫入了~/.bashrc 文件，直接執行source ~/.bashrc5、Verify Installationconda --version6、配置鏡像源conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/conda config --set show_channel_urls yes

3.3.下載LLaMA-Factory

方式一：直接下載

地址：GitHub - hiyouga/LLaMA-Factory: Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

方式二：使用git克隆項目

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git

下載好的項目放置在/data/service目錄下

3.4. 安裝依賴

conda create --name llama_factory  python=3.10
conda activate llama_factory
cd /data/service/LLaMA-Factory
pip install -e ".[torch,metrics]" -i https://pypi.tuna.tsinghua.edu.cn/simple

? PS：軟硬件要求

四、技術實現

4.1.數據準備

有兩種格式選擇，包括alpaca和sharegpt

alpaca示例格式：

[{"instruction": "人類指令（必填）","input": "人類輸入（選填）","output": "模型回答（必填）","system": "系統提示詞（選填）","history": [["第一輪指令（選填）", "第一輪回答（選填）"],["第二輪指令（選填）", "第二輪回答（選填）"]]}
]

對于上述格式的數據，dataset_info.json?中的數據集描述應為：

"數據集名稱": {"file_name": "data.json","columns": {"prompt": "instruction","query": "input","response": "output","system": "system","history": "history"}
}

sharegpt示例格式：

相比 alpaca 格式的數據集，sharegpt 格式支持更多的角色種類，例如 human、gpt、observation、function 等等。它們構成一個對象列表呈現在?conversations?列中。
注意其中 human 和 observation 必須出現在奇數位置，gpt 和 function 必須出現在偶數位置。

[{"conversations": [{"from": "human","value": "人類指令"},{"from": "function_call","value": "工具參數"},{"from": "observation","value": "工具結果"},{"from": "gpt","value": "模型回答"}],"system": "系統提示詞（選填）","tools": "工具描述（選填）"}
]

對于上述格式的數據，dataset_info.json?中的數據集描述應為：

"數據集名稱": {"file_name": "data.json","formatting": "sharegpt","columns": {"messages": "conversations","system": "system","tools": "tools"}
}

以下采用sharegpt格式，在/data/service/LLaMA-Factory/data/目錄上傳測試數據，文件名為ds_zh_demo.json，測試數據示例如下：

[{"conversations": [{"from": "user","value": "什么是深度學習，它與傳統機器學習有什么區別？"},{"from": "assistant","value": "深度學習是機器學習的一個子領域，使用人工神經網絡，尤其是深度神經網絡，來進行數據建模。與傳統機器學習方法相比，深度學習能夠自動提取特征，無需手動特征工程，適合處理大規模數據集和復雜模式。"}]},{"conversations": [{"from": "user","value": "如何評估一個機器學習模型的性能？"},{"from": "assistant","value": "評估機器學習模型的性能通常使用多種指標，包括準確率、精確率、召回率、F1-score、ROC曲線和AUC值。選擇合適的指標取決于具體任務的性質和目標。"}]}
]

修改數據集描述文件dataset_info.json

vi /data/service/LLaMA-Factory/data/dataset_info.json

增加以下內容：

"ds_zh_demo": {"file_name": "ds_zh_demo.json","formatting": "sharegpt","columns": {"messages": "conversations"},"tags": {"role_tag": "from","content_tag": "value","user_tag": "user","assistant_tag": "assistant"}
}

4.2.配置文件準備

1) 備份原有的配置文件

cp /data/service/LLaMA-Factory/examples/train_lora/llama3_lora_sft.yaml /data/service/LLaMA-Factory/examples/train_lora/llama3_lora_sft.yaml.bak

2) 創建新的配置文件

mv /data/service/LLaMA-Factory/examples/train_lora/llama3_lora_sft.yaml /data/service/LLaMA-Factory/examples/train_lora/ds_qwen7b_lora_sft.yaml

3) 修改配置文件內容

vi /data/service/LLaMA-Factory/examples/train_lora/ds_qwen7b_lora_sft.yaml

? 內容如下：

### model
model_name_or_path: /data/model/DeepSeek-R1-Distill-Qwen-7B
trust_remote_code: true### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all### dataset
dataset: ds_zh_demo
template: deepseek3
cutoff_len: 4096
max_samples: 4019
overwrite_cache: true
preprocessing_num_workers: 16### output
output_dir: /data/model/sft/DeepSeek-R1-Distill-Qwen-7B
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 500

? 需要關注以下參數

model_name_or_path：模型路徑
dataset：數據集名稱，對應上面聲明的qwen_zh_demo
template：模版
cutoff_len：控制輸入序列的最大長度
output_dir：微調后權重保存路徑
gradient_accumulation_steps：梯度累積的步數，GPU資源不足時需要減少該值
num_train_epochs：訓練的輪數

4.3.啟動微調

conda activate llama_factory
cd /data/service/LLaMA-Factory
llamafactory-cli train /data/service/LLaMA-Factory/examples/train_lora/ds_qwen7b_lora_sft.yaml# 后臺運行
nohup llamafactory-cli train /data/service/LLaMA-Factory/examples/train_lora/ds_qwen7b_lora_sft.yaml > output.log 2>&1 &

4.4.微調結果

[INFO|configuration_utils.py:1052] 2025-02-18 16:39:55,400 >> loading configuration file /data/model/DeepSeek-R1-Distill-Qwen-7B/generation_config.json
[INFO|configuration_utils.py:1099] 2025-02-18 16:39:55,400 >> Generate config GenerationConfig {"bos_token_id": 151646,"do_sample": true,"eos_token_id": 151643,"temperature": 0.6,"top_p": 0.95
}[INFO|2025-02-18 16:39:55] llamafactory.model.model_utils.checkpointing:157 >> Gradient checkpointing enabled.
[INFO|2025-02-18 16:39:55] llamafactory.model.model_utils.attention:157 >> Using torch SDPA for faster training and inference.
[INFO|2025-02-18 16:39:55] llamafactory.model.adapter:157 >> Upcasting trainable params to float32.
[INFO|2025-02-18 16:39:55] llamafactory.model.adapter:157 >> Fine-tuning method: LoRA
[INFO|2025-02-18 16:39:55] llamafactory.model.model_utils.misc:157 >> Found linear modules: down_proj,o_proj,up_proj,k_proj,v_proj,q_proj,gate_proj
[INFO|2025-02-18 16:39:55] llamafactory.model.loader:157 >> trainable params: 20,185,088 || all params: 7,635,801,600 || trainable%: 0.2643
Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
[INFO|trainer.py:667] 2025-02-18 16:39:55,807 >> Using auto half precision backend
[INFO|trainer.py:2243] 2025-02-18 16:39:56,634 >> ***** Running training *****
[INFO|trainer.py:2244] 2025-02-18 16:39:56,634 >>   Num examples = 3,617
[INFO|trainer.py:2245] 2025-02-18 16:39:56,634 >>   Num Epochs = 1
[INFO|trainer.py:2246] 2025-02-18 16:39:56,634 >>   Instantaneous batch size per device = 1
[INFO|trainer.py:2249] 2025-02-18 16:39:56,634 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:2250] 2025-02-18 16:39:56,634 >>   Gradient Accumulation steps = 8
[INFO|trainer.py:2251] 2025-02-18 16:39:56,634 >>   Total optimization steps = 452
[INFO|trainer.py:2252] 2025-02-18 16:39:56,638 >>   Number of trainable parameters = 20,185,0880%|          | 0/452 [00:00<?, ?it/s]/usr/local/miniconda3/envs/llama_factory/lib/python3.10/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):  # type: ignore[attr-defined]
100%|██████████| 452/452 [4:06:28<00:00, 31.87s/it][INFO|trainer.py:3705] 2025-02-18 20:46:24,795 >> Saving model checkpoint to /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/checkpoint-452
[INFO|configuration_utils.py:670] 2025-02-18 20:46:24,819 >> loading configuration file /data/model/DeepSeek-R1-Distill-Qwen-7B/config.json
[INFO|configuration_utils.py:739] 2025-02-18 20:46:24,820 >> Model config Qwen2Config {"architectures": ["Qwen2ForCausalLM"],"attention_dropout": 0.0,"bos_token_id": 151643,"eos_token_id": 151643,"hidden_act": "silu","hidden_size": 3584,"initializer_range": 0.02,"intermediate_size": 18944,"max_position_embeddings": 131072,"max_window_layers": 28,"model_type": "qwen2","num_attention_heads": 28,"num_hidden_layers": 28,"num_key_value_heads": 4,"rms_norm_eps": 1e-06,"rope_scaling": null,"rope_theta": 10000,"sliding_window": null,"tie_word_embeddings": false,"torch_dtype": "bfloat16","transformers_version": "4.45.0","use_cache": true,"use_mrope": false,"use_sliding_window": false,"vocab_size": 152064
}[INFO|tokenization_utils_base.py:2649] 2025-02-18 20:46:25,042 >> tokenizer config file saved in /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/checkpoint-452/tokenizer_config.json
[INFO|tokenization_utils_base.py:2658] 2025-02-18 20:46:25,043 >> Special tokens file saved in /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/checkpoint-452/special_tokens_map.json
[INFO|trainer.py:2505] 2025-02-18 20:46:25,377 >> Training completed. Do not forget to share your model on huggingface.co/models =)100%|██████████| 452/452 [4:06:28<00:00, 32.72s/it]
[INFO|trainer.py:3705] 2025-02-18 20:46:25,379 >> Saving model checkpoint to /data/model/sft/DeepSeek-R1-Distill-Qwen-7B
[INFO|configuration_utils.py:670] 2025-02-18 20:46:25,401 >> loading configuration file /data/model/DeepSeek-R1-Distill-Qwen-7B/config.json
[INFO|configuration_utils.py:739] 2025-02-18 20:46:25,401 >> Model config Qwen2Config {"architectures": ["Qwen2ForCausalLM"],"attention_dropout": 0.0,"bos_token_id": 151643,"eos_token_id": 151643,"hidden_act": "silu","hidden_size": 3584,"initializer_range": 0.02,"intermediate_size": 18944,"max_position_embeddings": 131072,"max_window_layers": 28,"model_type": "qwen2","num_attention_heads": 28,"num_hidden_layers": 28,"num_key_value_heads": 4,"rms_norm_eps": 1e-06,"rope_scaling": null,"rope_theta": 10000,"sliding_window": null,"tie_word_embeddings": false,"torch_dtype": "bfloat16","transformers_version": "4.45.0","use_cache": true,"use_mrope": false,"use_sliding_window": false,"vocab_size": 152064
}[INFO|tokenization_utils_base.py:2649] 2025-02-18 20:46:25,556 >> tokenizer config file saved in /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/tokenizer_config.json
[INFO|tokenization_utils_base.py:2658] 2025-02-18 20:46:25,556 >> Special tokens file saved in /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/special_tokens_map.json
{'loss': 3.6592, 'grad_norm': 0.38773563504219055, 'learning_rate': 2.173913043478261e-05, 'epoch': 0.02}
{'loss': 3.667, 'grad_norm': 0.698821485042572, 'learning_rate': 4.347826086956522e-05, 'epoch': 0.04}
{'loss': 3.4784, 'grad_norm': 0.41371676325798035, 'learning_rate': 6.521739130434783e-05, 'epoch': 0.07}
{'loss': 3.2962, 'grad_norm': 0.4966348111629486, 'learning_rate': 8.695652173913044e-05, 'epoch': 0.09}
{'loss': 3.0158, 'grad_norm': 0.333425909280777, 'learning_rate': 9.997605179330019e-05, 'epoch': 0.11}
{'loss': 3.2221, 'grad_norm': 0.3786776065826416, 'learning_rate': 9.970689785771798e-05, 'epoch': 0.13}
{'loss': 2.8439, 'grad_norm': 0.3683229386806488, 'learning_rate': 9.914027086842322e-05, 'epoch': 0.15}
{'loss': 3.0528, 'grad_norm': 0.42745739221572876, 'learning_rate': 9.82795618288397e-05, 'epoch': 0.18}
{'loss': 2.9092, 'grad_norm': 0.45462721586227417, 'learning_rate': 9.712992168898436e-05, 'epoch': 0.2}
{'loss': 3.1055, 'grad_norm': 0.5547119379043579, 'learning_rate': 9.56982305193869e-05, 'epoch': 0.22}
{'loss': 2.9412, 'grad_norm': 0.5830215811729431, 'learning_rate': 9.399305633701373e-05, 'epoch': 0.24}
{'loss': 2.7873, 'grad_norm': 0.5862609148025513, 'learning_rate': 9.202460382960448e-05, 'epoch': 0.27}
{'loss': 2.8255, 'grad_norm': 0.5828853845596313, 'learning_rate': 8.980465328528219e-05, 'epoch': 0.29}
{'loss': 2.6266, 'grad_norm': 0.6733331084251404, 'learning_rate': 8.734649009291585e-05, 'epoch': 0.31}
{'loss': 2.8745, 'grad_norm': 0.6904928684234619, 'learning_rate': 8.46648252351431e-05, 'epoch': 0.33}
{'loss': 2.8139, 'grad_norm': 0.7874809503555298, 'learning_rate': 8.177570724986628e-05, 'epoch': 0.35}
{'loss': 2.7818, 'grad_norm': 0.8345168232917786, 'learning_rate': 7.86964261870916e-05, 'epoch': 0.38}
{'loss': 2.7198, 'grad_norm': 0.8806198239326477, 'learning_rate': 7.544541013588645e-05, 'epoch': 0.4}
{'loss': 2.7231, 'grad_norm': 0.9481658935546875, 'learning_rate': 7.204211494069292e-05, 'epoch': 0.42}
{'loss': 2.7371, 'grad_norm': 0.9718573093414307, 'learning_rate': 6.850690776699573e-05, 'epoch': 0.44}
{'loss': 2.6862, 'grad_norm': 1.2056019306182861, 'learning_rate': 6.486094521315022e-05, 'epoch': 0.46}
{'loss': 2.4661, 'grad_norm': 1.200085163116455, 'learning_rate': 6.112604669781572e-05, 'epoch': 0.49}
{'loss': 2.4841, 'grad_norm': 1.1310691833496094, 'learning_rate': 5.732456388071247e-05, 'epoch': 0.51}
{'loss': 2.3755, 'grad_norm': 1.1279083490371704, 'learning_rate': 5.3479246898159063e-05, 'epoch': 0.53}
{'loss': 2.5552, 'grad_norm': 1.2654848098754883, 'learning_rate': 4.96131082139099e-05, 'epoch': 0.55}
{'loss': 2.6197, 'grad_norm': 1.3887016773223877, 'learning_rate': 4.574928490008264e-05, 'epoch': 0.58}
{'loss': 2.3773, 'grad_norm': 1.3009178638458252, 'learning_rate': 4.1910900172361764e-05, 'epoch': 0.6}
{'loss': 2.3881, 'grad_norm': 1.346793532371521, 'learning_rate': 3.812092500812646e-05, 'epoch': 0.62}
{'loss': 2.4821, 'grad_norm': 1.7273674011230469, 'learning_rate': 3.440204067565511e-05, 'epoch': 0.64}
{'loss': 2.3563, 'grad_norm': 1.529177188873291, 'learning_rate': 3.077650299710653e-05, 'epoch': 0.66}
{'loss': 2.1308, 'grad_norm': 1.5957469940185547, 'learning_rate': 2.7266009157601224e-05, 'epoch': 0.69}
{'loss': 2.1709, 'grad_norm': 1.4444897174835205, 'learning_rate': 2.3891567857490372e-05, 'epoch': 0.71}
{'loss': 2.275, 'grad_norm': 1.5686719417572021, 'learning_rate': 2.067337358489085e-05, 'epoch': 0.73}
{'loss': 2.2075, 'grad_norm': 1.5931408405303955, 'learning_rate': 1.7630685760908622e-05, 'epoch': 0.75}
{'loss': 2.1727, 'grad_norm': 1.7681787014007568, 'learning_rate': 1.4781713480810184e-05, 'epoch': 0.77}
{'loss': 2.3562, 'grad_norm': 1.742925763130188, 'learning_rate': 1.2143506540914128e-05, 'epoch': 0.8}
{'loss': 2.1187, 'grad_norm': 1.6716198921203613, 'learning_rate': 9.731853403356705e-06, 'epoch': 0.82}
{'loss': 2.2564, 'grad_norm': 1.915489912033081, 'learning_rate': 7.561186709365653e-06, 'epoch': 0.84}
{'loss': 2.261, 'grad_norm': 2.132519245147705, 'learning_rate': 5.644496906502233e-06, 'epoch': 0.86}
{'loss': 2.1632, 'grad_norm': 1.591231107711792, 'learning_rate': 3.9932545067728366e-06, 'epoch': 0.88}
{'loss': 2.1266, 'grad_norm': 1.584917664527893, 'learning_rate': 2.6173414408598827e-06, 'epoch': 0.91}
{'loss': 2.2944, 'grad_norm': 1.5982666015625, 'learning_rate': 1.524991919285429e-06, 'epoch': 0.93}
{'loss': 2.3799, 'grad_norm': 2.1475727558135986, 'learning_rate': 7.227431544266194e-07, 'epoch': 0.95}
{'loss': 2.1196, 'grad_norm': 1.6714484691619873, 'learning_rate': 2.153962382888841e-07, 'epoch': 0.97}
{'loss': 2.1427, 'grad_norm': 1.7334465980529785, 'learning_rate': 5.987410165758656e-09, 'epoch': 1.0}
{'train_runtime': 14788.7396, 'train_samples_per_second': 0.245, 'train_steps_per_second': 0.031, 'train_loss': 2.6206856934370193, 'epoch': 1.0}
***** train metrics *****epoch                    =      0.9997total_flos               = 100517734GFtrain_loss               =      2.6207train_runtime            =  4:06:28.73train_samples_per_second =       0.245train_steps_per_second   =       0.031
Figure saved at: /data/model/sft/DeepSeek-R1-Distill-Qwen-7B/training_loss.png
[WARNING|2025-02-18 20:46:25] llamafactory.extras.ploting:162 >> No metric eval_loss to plot.
[WARNING|2025-02-18 20:46:25] llamafactory.extras.ploting:162 >> No metric eval_accuracy to plot.
[INFO|trainer.py:4021] 2025-02-18 20:46:25,781 >> 
***** Running Evaluation *****
[INFO|trainer.py:4023] 2025-02-18 20:46:25,781 >>   Num examples = 402
[INFO|trainer.py:4026] 2025-02-18 20:46:25,781 >>   Batch size = 1
100%|██████████| 402/402 [09:03<00:00,  1.35s/it]t]
[INFO|modelcard.py:449] 2025-02-18 20:55:30,409 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}
***** eval metrics *****epoch                   =     0.9997eval_loss               =     2.2648eval_runtime            = 0:09:04.62eval_samples_per_second =      0.738eval_steps_per_second   =      0.738

生成的權重文件：

五、附帶說明

5.1. dataset_info.json

包含了所有可用的數據集。如果您希望使用自定義數據集，請務必在?dataset_info.json?文件中添加數據集描述，并通過修改?dataset: 數據集名稱?配置來使用數據集。

"數據集名稱": {"hf_hub_url": "Hugging Face 的數據集倉庫地址（若指定，則忽略 script_url 和 file_name）","ms_hub_url": "ModelScope 的數據集倉庫地址（若指定，則忽略 script_url 和 file_name）","script_url": "包含數據加載腳本的本地文件夾名稱（若指定，則忽略 file_name）","file_name": "該目錄下數據集文件夾或文件的名稱（若上述參數未指定，則此項必需）","formatting": "數據集格式（可選，默認：alpaca，可以為 alpaca 或 sharegpt）","ranking": "是否為偏好數據集（可選，默認：False）","subset": "數據集子集的名稱（可選，默認：None）","split": "所使用的數據集切分（可選，默認：train）","folder": "Hugging Face 倉庫的文件夾名稱（可選，默認：None）","num_samples": "該數據集所使用的樣本數量。（可選，默認：None）","columns（可選）": {"prompt": "數據集代表提示詞的表頭名稱（默認：instruction）","query": "數據集代表請求的表頭名稱（默認：input）","response": "數據集代表回答的表頭名稱（默認：output）","history": "數據集代表歷史對話的表頭名稱（默認：None）","messages": "數據集代表消息列表的表頭名稱（默認：conversations）","system": "數據集代表系統提示的表頭名稱（默認：None）","tools": "數據集代表工具描述的表頭名稱（默認：None）","images": "數據集代表圖像輸入的表頭名稱（默認：None）","videos": "數據集代表視頻輸入的表頭名稱（默認：None）","audios": "數據集代表音頻輸入的表頭名稱（默認：None）","chosen": "數據集代表更優回答的表頭名稱（默認：None）","rejected": "數據集代表更差回答的表頭名稱（默認：None）","kto_tag": "數據集代表 KTO 標簽的表頭名稱（默認：None）"},"tags（可選，用于 sharegpt 格式）": {"role_tag": "消息中代表發送者身份的鍵名（默認：from）","content_tag": "消息中代表文本內容的鍵名（默認：value）","user_tag": "消息中代表用戶的 role_tag（默認：human）","assistant_tag": "消息中代表助手的 role_tag（默認：gpt）","observation_tag": "消息中代表工具返回結果的 role_tag（默認：observation）","function_tag": "消息中代表工具調用的 role_tag（默認：function_call）","system_tag": "消息中代表系統提示的 role_tag（默認：system，會覆蓋 system column）"}
}

5.2. 自定義對話模版

在 template.py 中添加自己的對話模板。

https://github.com/hiyouga/LLaMA-Factory/blob/main/src/llamafactory/data/template.py

# Copyright 2025 the LlamaFactory team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.from dataclasses import dataclass
from typing import TYPE_CHECKING, Dict, List, Optional, Sequence, Tuple, Type, Unionfrom typing_extensions import overridefrom ..extras import logging
from ..extras.misc import check_version
from .data_utils import Role
from .formatter import EmptyFormatter, FunctionFormatter, StringFormatter, ToolFormatter
from .mm_plugin import get_mm_pluginif TYPE_CHECKING:from transformers import PreTrainedTokenizerfrom ..hparams import DataArgumentsfrom .formatter import SLOTS, Formatterfrom .mm_plugin import BasePluginfrom .tool_utils import FunctionCalllogger = logging.get_logger(__name__)@dataclass
class Template:format_user: "Formatter"format_assistant: "Formatter"format_system: "Formatter"format_function: "Formatter"format_observation: "Formatter"format_tools: "Formatter"format_prefix: "Formatter"default_system: strstop_words: List[str]thought_words: Tuple[str, str]efficient_eos: boolreplace_eos: boolreplace_jinja_template: boolmm_plugin: "BasePlugin"def encode_oneturn(self,tokenizer: "PreTrainedTokenizer",messages: Sequence[Dict[str, str]],system: Optional[str] = None,tools: Optional[str] = None,) -> Tuple[List[int], List[int]]:r"""Returns a single pair of token ids representing prompt and response respectively."""encoded_messages = self._encode(tokenizer, messages, system, tools)prompt_ids = []for encoded_ids in encoded_messages[:-1]:prompt_ids += encoded_idsresponse_ids = encoded_messages[-1]return prompt_ids, response_idsdef encode_multiturn(self,tokenizer: "PreTrainedTokenizer",messages: Sequence[Dict[str, str]],system: Optional[str] = None,tools: Optional[str] = None,) -> List[Tuple[List[int], List[int]]]:r"""Returns multiple pairs of token ids representing prompts and responses respectively."""encoded_messages = self._encode(tokenizer, messages, system, tools)return [(encoded_messages[i], encoded_messages[i + 1]) for i in range(0, len(encoded_messages), 2)]def extract_tool(self, content: str) -> Union[str, List["FunctionCall"]]:r"""Extracts tool message."""return self.format_tools.extract(content)def get_stop_token_ids(self, tokenizer: "PreTrainedTokenizer") -> List[int]:r"""Returns stop token ids."""stop_token_ids = {tokenizer.eos_token_id}for token in self.stop_words:stop_token_ids.add(tokenizer.convert_tokens_to_ids(token))return list(stop_token_ids)def _convert_elements_to_ids(self, tokenizer: "PreTrainedTokenizer", elements: "SLOTS") -> List[int]:r"""Converts elements to token ids."""token_ids = []for elem in elements:if isinstance(elem, str):if len(elem) != 0:token_ids += tokenizer.encode(elem, add_special_tokens=False)elif isinstance(elem, dict):token_ids += [tokenizer.convert_tokens_to_ids(elem.get("token"))]elif isinstance(elem, set):if "bos_token" in elem and tokenizer.bos_token_id is not None:token_ids += [tokenizer.bos_token_id]elif "eos_token" in elem and tokenizer.eos_token_id is not None:token_ids += [tokenizer.eos_token_id]else:raise ValueError(f"Input must be string, set[str] or dict[str, str], got {type(elem)}")return token_idsdef _encode(self,tokenizer: "PreTrainedTokenizer",messages: Sequence[Dict[str, str]],system: Optional[str],tools: Optional[str],) -> List[List[int]]:r"""Encodes formatted inputs to pairs of token ids.Turn 0: prefix + system + query        respTurn t: query                          resp"""system = system or self.default_systemencoded_messages = []for i, message in enumerate(messages):elements = []if i == 0:elements += self.format_prefix.apply()if system or tools:tool_text = self.format_tools.apply(content=tools)[0] if tools else ""elements += self.format_system.apply(content=(system + tool_text))if message["role"] == Role.USER.value:elements += self.format_user.apply(content=message["content"], idx=str(i // 2))elif message["role"] == Role.ASSISTANT.value:elements += self.format_assistant.apply(content=message["content"])elif message["role"] == Role.OBSERVATION.value:elements += self.format_observation.apply(content=message["content"])elif message["role"] == Role.FUNCTION.value:elements += self.format_function.apply(content=message["content"])else:raise NotImplementedError("Unexpected role: {}".format(message["role"]))encoded_messages.append(self._convert_elements_to_ids(tokenizer, elements))return encoded_messages@staticmethoddef _add_or_replace_eos_token(tokenizer: "PreTrainedTokenizer", eos_token: str) -> None:r"""Adds or replaces eos token to the tokenizer."""is_added = tokenizer.eos_token_id is Nonenum_added_tokens = tokenizer.add_special_tokens({"eos_token": eos_token})if is_added:logger.info_rank0(f"Add eos token: {tokenizer.eos_token}.")else:logger.info_rank0(f"Replace eos token: {tokenizer.eos_token}.")if num_added_tokens > 0:logger.warning_rank0("New tokens have been added, make sure `resize_vocab` is True.")def fix_special_tokens(self, tokenizer: "PreTrainedTokenizer") -> None:r"""Adds eos token and pad token to the tokenizer."""stop_words = self.stop_wordsif self.replace_eos:if not stop_words:raise ValueError("Stop words are required to replace the EOS token.")self._add_or_replace_eos_token(tokenizer, eos_token=stop_words[0])stop_words = stop_words[1:]if tokenizer.eos_token_id is None:self._add_or_replace_eos_token(tokenizer, eos_token="<|endoftext|>")if tokenizer.pad_token_id is None:tokenizer.pad_token = tokenizer.eos_tokenlogger.info_rank0(f"Add pad token: {tokenizer.pad_token}")if stop_words:num_added_tokens = tokenizer.add_special_tokens(dict(additional_special_tokens=stop_words), replace_additional_special_tokens=False)logger.info_rank0("Add {} to stop words.".format(",".join(stop_words)))if num_added_tokens > 0:logger.warning_rank0("New tokens have been added, make sure `resize_vocab` is True.")@staticmethoddef _jinja_escape(content: str) -> str:r"""Escape single quotes in content."""return content.replace("'", r"\'")@staticmethoddef _convert_slots_to_jinja(slots: "SLOTS", tokenizer: "PreTrainedTokenizer", placeholder: str = "content") -> str:r"""Converts slots to jinja template."""slot_items = []for slot in slots:if isinstance(slot, str):slot_pieces = slot.split("{{content}}")if slot_pieces[0]:slot_items.append("'" + Template._jinja_escape(slot_pieces[0]) + "'")if len(slot_pieces) > 1:slot_items.append(placeholder)if slot_pieces[1]:slot_items.append("'" + Template._jinja_escape(slot_pieces[1]) + "'")elif isinstance(slot, set):  # do not use {{ eos_token }} since it may be replacedif "bos_token" in slot and tokenizer.bos_token_id is not None:slot_items.append("'" + tokenizer.bos_token + "'")elif "eos_token" in slot and tokenizer.eos_token_id is not None:slot_items.append("'" + tokenizer.eos_token + "'")elif isinstance(slot, dict):raise ValueError("Dict is not supported.")return " + ".join(slot_items)def _get_jinja_template(self, tokenizer: "PreTrainedTokenizer") -> str:r"""Returns the jinja template."""prefix = self._convert_slots_to_jinja(self.format_prefix.apply(), tokenizer)system = self._convert_slots_to_jinja(self.format_system.apply(), tokenizer, placeholder="system_message")user = self._convert_slots_to_jinja(self.format_user.apply(), tokenizer)assistant = self._convert_slots_to_jinja(self.format_assistant.apply(), tokenizer)jinja_template = ""if prefix:jinja_template += "{{ " + prefix + " }}"if self.default_system:jinja_template += "{% set system_message = '" + self._jinja_escape(self.default_system) + "' %}"jinja_template += ("{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}""{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% endif %}""{% if system_message is defined %}{{ " + system + " }}{% endif %}""{% for message in loop_messages %}""{% set content = message['content'] %}""{% if message['role'] == 'user' %}""{{ " + user + " }}""{% elif message['role'] == 'assistant' %}""{{ " + assistant + " }}""{% endif %}""{% endfor %}")return jinja_templatedef fix_jinja_template(self, tokenizer: "PreTrainedTokenizer") -> None:r"""Replaces the jinja template in the tokenizer."""if tokenizer.chat_template is None or self.replace_jinja_template:try:tokenizer.chat_template = self._get_jinja_template(tokenizer)except ValueError as e:logger.info_rank0(f"Cannot add this chat template to tokenizer: {e}.")@staticmethoddef _convert_slots_to_ollama(slots: "SLOTS", tokenizer: "PreTrainedTokenizer", placeholder: str = "content") -> str:r"""Converts slots to ollama template."""slot_items = []for slot in slots:if isinstance(slot, str):slot_pieces = slot.split("{{content}}")if slot_pieces[0]:slot_items.append(slot_pieces[0])if len(slot_pieces) > 1:slot_items.append("{{ " + placeholder + " }}")if slot_pieces[1]:slot_items.append(slot_pieces[1])elif isinstance(slot, set):  # do not use {{ eos_token }} since it may be replacedif "bos_token" in slot and tokenizer.bos_token_id is not None:slot_items.append(tokenizer.bos_token)elif "eos_token" in slot and tokenizer.eos_token_id is not None:slot_items.append(tokenizer.eos_token)elif isinstance(slot, dict):raise ValueError("Dict is not supported.")return "".join(slot_items)def _get_ollama_template(self, tokenizer: "PreTrainedTokenizer") -> str:r"""Returns the ollama template."""prefix = self._convert_slots_to_ollama(self.format_prefix.apply(), tokenizer)system = self._convert_slots_to_ollama(self.format_system.apply(), tokenizer, placeholder=".System")user = self._convert_slots_to_ollama(self.format_user.apply(), tokenizer, placeholder=".Content")assistant = self._convert_slots_to_ollama(self.format_assistant.apply(), tokenizer, placeholder=".Content")return (f"{prefix}{{{{ if .System }}}}{system}{{{{ end }}}}"f"""{{{{ range .Messages }}}}{{{{ if eq .Role "user" }}}}{user}"""f"""{{{{ else if eq .Role "assistant" }}}}{assistant}{{{{ end }}}}{{{{ end }}}}""")def get_ollama_modelfile(self, tokenizer: "PreTrainedTokenizer") -> str:r"""Returns the ollama modelfile.TODO: support function calling."""modelfile = "# ollama modelfile auto-generated by llamafactory\n\n"modelfile += f'FROM .\n\nTEMPLATE """{self._get_ollama_template(tokenizer)}"""\n\n'if self.default_system:modelfile += f'SYSTEM """{self.default_system}"""\n\n'for stop_token_id in self.get_stop_token_ids(tokenizer):modelfile += f'PARAMETER stop "{tokenizer.convert_ids_to_tokens(stop_token_id)}"\n'modelfile += "PARAMETER num_ctx 4096\n"return modelfile@dataclass
class Llama2Template(Template):@overridedef _encode(self,tokenizer: "PreTrainedTokenizer",messages: Sequence[Dict[str, str]],system: str,tools: str,) -> List[List[int]]:system = system or self.default_systemencoded_messages = []for i, message in enumerate(messages):elements = []system_text = ""if i == 0:elements += self.format_prefix.apply()if system or tools:tool_text = self.format_tools.apply(content=tools)[0] if tools else ""system_text = self.format_system.apply(content=(system + tool_text))[0]if message["role"] == Role.USER.value:elements += self.format_user.apply(content=system_text + message["content"])elif message["role"] == Role.ASSISTANT.value:elements += self.format_assistant.apply(content=message["content"])elif message["role"] == Role.OBSERVATION.value:elements += self.format_observation.apply(content=message["content"])elif message["role"] == Role.FUNCTION.value:elements += self.format_function.apply(content=message["content"])else:raise NotImplementedError("Unexpected role: {}".format(message["role"]))encoded_messages.append(self._convert_elements_to_ids(tokenizer, elements))return encoded_messagesdef _get_jinja_template(self, tokenizer: "PreTrainedTokenizer") -> str:prefix = self._convert_slots_to_jinja(self.format_prefix.apply(), tokenizer)system_message = self._convert_slots_to_jinja(self.format_system.apply(), tokenizer, placeholder="system_message")user_message = self._convert_slots_to_jinja(self.format_user.apply(), tokenizer)assistant_message = self._convert_slots_to_jinja(self.format_assistant.apply(), tokenizer)jinja_template = ""if prefix:jinja_template += "{{ " + prefix + " }}"if self.default_system:jinja_template += "{% set system_message = '" + self._jinja_escape(self.default_system) + "' %}"jinja_template += ("{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}""{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% endif %}""{% for message in loop_messages %}""{% if loop.index0 == 0 and system_message is defined %}""{% set content = " + system_message + " + message['content'] %}""{% else %}{% set content = message['content'] %}{% endif %}""{% if message['role'] == 'user' %}""{{ " + user_message + " }}""{% elif message['role'] == 'assistant' %}""{{ " + assistant_message + " }}""{% endif %}""{% endfor %}")return jinja_templateTEMPLATES: Dict[str, "Template"] = {}def register_template(name: str,format_user: Optional["Formatter"] = None,format_assistant: Optional["Formatter"] = None,format_system: Optional["Formatter"] = None,format_function: Optional["Formatter"] = None,format_observation: Optional["Formatter"] = None,format_tools: Optional["Formatter"] = None,format_prefix: Optional["Formatter"] = None,default_system: str = "",stop_words: Optional[Sequence[str]] = None,thought_words: Optional[Tuple[str, str]] = None,efficient_eos: bool = False,replace_eos: bool = False,replace_jinja_template: bool = False,mm_plugin: "BasePlugin" = get_mm_plugin(name="base"),template_class: Type["Template"] = Template,
) -> None:r"""Registers a chat template.To add the following chat template:```<s><user>user prompt here<model>model response here</s><user>user prompt here<model>model response here</s>```The corresponding code should be:```register_template(name="custom",format_user=StringFormatter(slots=["<user>{{content}}\n<model>"]),format_assistant=StringFormatter(slots=["{{content}}</s>\n"]),format_prefix=EmptyFormatter("<s>"),)```"""if name in TEMPLATES:raise ValueError(f"Template {name} already exists.")default_slots = ["{{content}}"] if efficient_eos else ["{{content}}", {"eos_token"}]default_user_formatter = StringFormatter(slots=["{{content}}"])default_assistant_formatter = StringFormatter(slots=default_slots)default_function_formatter = FunctionFormatter(slots=default_slots, tool_format="default")default_tool_formatter = ToolFormatter(tool_format="default")default_prefix_formatter = EmptyFormatter()TEMPLATES[name] = template_class(format_user=format_user or default_user_formatter,format_assistant=format_assistant or default_assistant_formatter,format_system=format_system or default_user_formatter,format_function=format_function or default_function_formatter,format_observation=format_observation or format_user or default_user_formatter,format_tools=format_tools or default_tool_formatter,format_prefix=format_prefix or default_prefix_formatter,default_system=default_system,stop_words=stop_words or [],thought_words=thought_words or ("<think>", "</think>"),efficient_eos=efficient_eos,replace_eos=replace_eos,replace_jinja_template=replace_jinja_template,mm_plugin=mm_plugin,)def parse_template(tokenizer: "PreTrainedTokenizer") -> "Template":r"""Extracts a chat template from the tokenizer."""def find_diff(short_str: str, long_str: str) -> str:i, j = 0, 0diff = ""while i < len(short_str) and j < len(long_str):if short_str[i] == long_str[j]:i += 1j += 1else:diff += long_str[j]j += 1return diffprefix = tokenizer.decode(tokenizer.encode(""))messages = [{"role": "system", "content": "{{content}}"}]system_slot = tokenizer.apply_chat_template(messages, add_generation_prompt=False, tokenize=False)[len(prefix) :]messages = [{"role": "system", "content": ""}, {"role": "user", "content": "{{content}}"}]user_slot_empty_system = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)user_slot_empty_system = user_slot_empty_system[len(prefix) :]messages = [{"role": "user", "content": "{{content}}"}]user_slot = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)user_slot = user_slot[len(prefix) :]messages = [{"role": "user", "content": "{{content}}"}, {"role": "assistant", "content": "{{content}}"}]assistant_slot = tokenizer.apply_chat_template(messages, add_generation_prompt=False, tokenize=False)assistant_slot = assistant_slot[len(prefix) + len(user_slot) :]if len(user_slot) > len(user_slot_empty_system):default_system = find_diff(user_slot_empty_system, user_slot)sole_system = system_slot.replace("{{content}}", default_system, 1)user_slot = user_slot[len(sole_system) :]else:  # if defaut_system is empty, user_slot_empty_system will be longer than user_slotdefault_system = ""return Template(format_user=StringFormatter(slots=[user_slot]),format_assistant=StringFormatter(slots=[assistant_slot]),format_system=StringFormatter(slots=[system_slot]),format_function=FunctionFormatter(slots=[assistant_slot], tool_format="default"),format_observation=StringFormatter(slots=[user_slot]),format_tools=ToolFormatter(tool_format="default"),format_prefix=EmptyFormatter(slots=[prefix]) if prefix else EmptyFormatter(),default_system=default_system,stop_words=[],thought_words=("<think>", "</think>"),efficient_eos=False,replace_eos=False,replace_jinja_template=False,mm_plugin=get_mm_plugin(name="base"),)def get_template_and_fix_tokenizer(tokenizer: "PreTrainedTokenizer", data_args: "DataArguments") -> "Template":r"""Gets chat template and fixes the tokenizer."""if data_args.template is None:if isinstance(tokenizer.chat_template, str):logger.warning_rank0("`template` was not specified, try parsing the chat template from the tokenizer.")template = parse_template(tokenizer)else:logger.warning_rank0("`template` was not specified, use `empty` template.")template = TEMPLATES["empty"]  # placeholderelse:if data_args.template not in TEMPLATES:raise ValueError(f"Template {data_args.template} does not exist.")template = TEMPLATES[data_args.template]if template.mm_plugin.__class__.__name__ != "BasePlugin":check_version("transformers>=4.45.0")if data_args.train_on_prompt and template.efficient_eos:raise ValueError("Current template does not support `train_on_prompt`.")if data_args.tool_format is not None:logger.info_rank0(f"Using tool format: {data_args.tool_format}.")default_slots = ["{{content}}"] if template.efficient_eos else ["{{content}}", {"eos_token"}]template.format_function = FunctionFormatter(slots=default_slots, tool_format=data_args.tool_format)template.format_tools = ToolFormatter(tool_format=data_args.tool_format)template.fix_special_tokens(tokenizer)template.fix_jinja_template(tokenizer)return templateregister_template(name="alpaca",format_user=StringFormatter(slots=["### Instruction:\n{{content}}\n\n### Response:\n"]),format_assistant=StringFormatter(slots=["{{content}}", {"eos_token"}, "\n\n"]),default_system=("Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n"),replace_jinja_template=True,
)register_template(name="aquila",format_user=StringFormatter(slots=["Human: {{content}}###Assistant:"]),format_assistant=StringFormatter(slots=["{{content}}###"]),format_system=StringFormatter(slots=["System: {{content}}###"]),default_system=("A chat between a curious human and an artificial intelligence assistant. ""The assistant gives helpful, detailed, and polite answers to the human's questions."),stop_words=["</s>"],
)register_template(name="atom",format_user=StringFormatter(slots=[{"bos_token"}, "Human: {{content}}\n", {"eos_token"}, {"bos_token"}, "Assistant:"]),format_assistant=StringFormatter(slots=["{{content}}\n", {"eos_token"}]),
)register_template(name="baichuan",format_user=StringFormatter(slots=[{"token": "<reserved_102>"}, "{{content}}", {"token": "<reserved_103>"}]),efficient_eos=True,
)register_template(name="baichuan2",format_user=StringFormatter(slots=["<reserved_106>{{content}}<reserved_107>"]),efficient_eos=True,
)register_template(name="belle",format_user=StringFormatter(slots=["Human: {{content}}\n\nBelle: "]),format_assistant=StringFormatter(slots=["{{content}}", {"eos_token"}, "\n\n"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
)register_template(name="bluelm",format_user=StringFormatter(slots=[{"token": "[|Human|]:"}, "{{content}}", {"token": "[|AI|]:"}]),
)register_template(name="breeze",format_user=StringFormatter(slots=["[INST] {{content}} [/INST] "]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),efficient_eos=True,
)register_template(name="chatglm2",format_user=StringFormatter(slots=["[Round {{idx}}]\n\n問：{{content}}\n\n答："]),format_prefix=EmptyFormatter(slots=[{"token": "[gMASK]"}, {"token": "sop"}]),efficient_eos=True,
)register_template(name="chatglm3",format_user=StringFormatter(slots=[{"token": "<|user|>"}, "\n", "{{content}}", {"token": "<|assistant|>"}]),format_assistant=StringFormatter(slots=["\n", "{{content}}"]),format_system=StringFormatter(slots=[{"token": "<|system|>"}, "\n", "{{content}}"]),format_function=FunctionFormatter(slots=["{{content}}"], tool_format="glm4"),format_observation=StringFormatter(slots=[{"token": "<|observation|>"}, "\n", "{{content}}", {"token": "<|assistant|>"}]),format_tools=ToolFormatter(tool_format="glm4"),format_prefix=EmptyFormatter(slots=[{"token": "[gMASK]"}, {"token": "sop"}]),stop_words=["<|user|>", "<|observation|>"],efficient_eos=True,
)register_template(name="chatml",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),stop_words=["<|im_end|>", "<|im_start|>"],replace_eos=True,replace_jinja_template=True,
)# copied from chatml template
register_template(name="chatml_de",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),default_system="Du bist ein freundlicher und hilfsbereiter KI-Assistent.",stop_words=["<|im_end|>", "<|im_start|>"],replace_eos=True,replace_jinja_template=True,
)register_template(name="codegeex2",format_prefix=EmptyFormatter(slots=[{"token": "[gMASK]"}, {"token": "sop"}]),
)register_template(name="codegeex4",format_user=StringFormatter(slots=["<|user|>\n{{content}}<|assistant|>\n"]),format_system=StringFormatter(slots=["<|system|>\n{{content}}"]),format_function=FunctionFormatter(slots=["{{content}}"], tool_format="glm4"),format_observation=StringFormatter(slots=["<|observation|>\n{{content}}<|assistant|>\n"]),format_tools=ToolFormatter(tool_format="glm4"),format_prefix=EmptyFormatter(slots=["[gMASK]<sop>"]),default_system=("你是一位智能編程助手，你叫CodeGeeX。你會為用戶回答關于編程、代碼、計算機方面的任何問題，""并提供格式規范、可以執行、準確安全的代碼，并在必要時提供詳細的解釋。"),stop_words=["<|user|>", "<|observation|>"],efficient_eos=True,
)register_template(name="cohere",format_user=StringFormatter(slots=[("<|START_OF_TURN_TOKEN|><|USER_TOKEN|>{{content}}<|END_OF_TURN_TOKEN|>""<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>")]),format_system=StringFormatter(slots=["<|START_OF_TURN_TOKEN|><|SYSTEM_TOKEN|>{{content}}<|END_OF_TURN_TOKEN|>"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
)register_template(name="cpm",format_user=StringFormatter(slots=["<用戶>{{content}}<AI>"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
)# copied from chatml template
register_template(name="cpm3",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),stop_words=["<|im_end|>"],
)# copied from chatml template
register_template(name="dbrx",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),default_system=("You are DBRX, created by Databricks. You were last updated in December 2023. ""You answer questions based on information available up to that point.\n""YOU PROVIDE SHORT RESPONSES TO SHORT QUESTIONS OR STATEMENTS, but provide thorough ""responses to more complex and open-ended questions.\nYou assist with various tasks, ""from writing to coding (using markdown for code blocks — remember to use ``` with ""code, JSON, and tables).\n(You do not have real-time data access or code execution ""capabilities. You avoid stereotyping and provide balanced perspectives on ""controversial topics. You do not provide song lyrics, poems, or news articles and ""do not divulge details of your training data.)\nThis is your system prompt, ""guiding your responses. Do not reference it, just respond to the user. If you find ""yourself talking about this message, stop. You should be responding appropriately ""and usually that means not mentioning this.\nYOU DO NOT MENTION ANY OF THIS INFORMATION ""ABOUT YOURSELF UNLESS THE INFORMATION IS DIRECTLY PERTINENT TO THE USER'S QUERY."),stop_words=["<|im_end|>"],
)register_template(name="deepseek",format_user=StringFormatter(slots=["User: {{content}}\n\nAssistant:"]),format_system=StringFormatter(slots=["{{content}}\n\n"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
)register_template(name="deepseek3",format_user=StringFormatter(slots=["<｜User｜>{{content}}<｜Assistant｜>"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
)register_template(name="deepseekcoder",format_user=StringFormatter(slots=["### Instruction:\n{{content}}\n### Response:"]),format_assistant=StringFormatter(slots=["\n{{content}}\n<|EOT|>\n"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),default_system=("You are an AI programming assistant, utilizing the DeepSeek Coder model, ""developed by DeepSeek Company, and you only answer questions related to computer science. ""For politically sensitive questions, security and privacy issues, ""and other non-computer science questions, you will refuse to answer.\n"),
)register_template(name="default",format_user=StringFormatter(slots=["Human: {{content}}\nAssistant:"]),format_assistant=StringFormatter(slots=["{{content}}", {"eos_token"}, "\n"]),format_system=StringFormatter(slots=["System: {{content}}\n"]),
)register_template(name="empty",format_assistant=StringFormatter(slots=["{{content}}"]),
)register_template(name="exaone",format_user=StringFormatter(slots=["[|user|]{{content}}\n[|assistant|]"]),format_assistant=StringFormatter(slots=["{{content}}", {"eos_token"}, "\n"]),format_system=StringFormatter(slots=["[|system|]{{content}}[|endofturn|]\n"]),
)register_template(name="falcon",format_user=StringFormatter(slots=["User: {{content}}\nFalcon:"]),format_assistant=StringFormatter(slots=["{{content}}\n"]),efficient_eos=True,
)register_template(name="fewshot",format_assistant=StringFormatter(slots=["{{content}}\n\n"]),efficient_eos=True,
)register_template(name="gemma",format_user=StringFormatter(slots=["<start_of_turn>user\n{{content}}<end_of_turn>\n<start_of_turn>model\n"]),format_assistant=StringFormatter(slots=["{{content}}<end_of_turn>\n"]),format_observation=StringFormatter(slots=["<start_of_turn>tool\n{{content}}<end_of_turn>\n<start_of_turn>model\n"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
)register_template(name="glm4",format_user=StringFormatter(slots=["<|user|>\n{{content}}<|assistant|>"]),format_assistant=StringFormatter(slots=["\n{{content}}"]),format_system=StringFormatter(slots=["<|system|>\n{{content}}"]),format_function=FunctionFormatter(slots=["{{content}}"], tool_format="glm4"),format_observation=StringFormatter(slots=["<|observation|>\n{{content}}<|assistant|>"]),format_tools=ToolFormatter(tool_format="glm4"),format_prefix=EmptyFormatter(slots=["[gMASK]<sop>"]),stop_words=["<|user|>", "<|observation|>"],efficient_eos=True,
)register_template(name="granite3",format_user=StringFormatter(slots=["<|start_of_role|>user<|end_of_role|>{{content}}<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>"]),format_assistant=StringFormatter(slots=["{{content}}<|end_of_text|>\n"]),format_system=StringFormatter(slots=["<|start_of_role|>system<|end_of_role|>{{content}}<|end_of_text|>\n"]),
)register_template(name="index",format_user=StringFormatter(slots=["reserved_0{{content}}reserved_1"]),format_system=StringFormatter(slots=["<unk>{{content}}"]),efficient_eos=True,
)register_template(name="intern",format_user=StringFormatter(slots=["<|User|>:{{content}}\n<|Bot|>:"]),format_assistant=StringFormatter(slots=["{{content}}<eoa>\n"]),format_system=StringFormatter(slots=["<|System|>:{{content}}\n"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),default_system=("You are an AI assistant whose name is InternLM (書生·浦語).\n""- InternLM (書生·浦語) is a conversational language model that is developed by Shanghai AI Laboratory ""(上海人工智能實驗室). It is designed to be helpful, honest, and harmless.\n""- InternLM (書生·浦語) can understand and communicate fluently in the language ""chosen by the user such as English and 中文."),stop_words=["<eoa>"],
)register_template(name="intern2",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),default_system=("You are an AI assistant whose name is InternLM (書生·浦語).\n""- InternLM (書生·浦語) is a conversational language model that is developed by Shanghai AI Laboratory ""(上海人工智能實驗室). It is designed to be helpful, honest, and harmless.\n""- InternLM (書生·浦語) can understand and communicate fluently in the language ""chosen by the user such as English and 中文."),stop_words=["<|im_end|>"],
)register_template(name="llama2",format_user=StringFormatter(slots=[{"bos_token"}, "[INST] {{content}} [/INST]"]),format_system=StringFormatter(slots=["<<SYS>>\n{{content}}\n<</SYS>>\n\n"]),template_class=Llama2Template,
)# copied from llama2 template
register_template(name="llama2_zh",format_user=StringFormatter(slots=[{"bos_token"}, "[INST] {{content}} [/INST]"]),format_system=StringFormatter(slots=["<<SYS>>\n{{content}}\n<</SYS>>\n\n"]),default_system="You are a helpful assistant. 你是一個樂于助人的助手。",template_class=Llama2Template,
)register_template(name="llama3",format_user=StringFormatter(slots=[("<|start_header_id|>user<|end_header_id|>\n\n{{content}}<|eot_id|>""<|start_header_id|>assistant<|end_header_id|>\n\n")]),format_assistant=StringFormatter(slots=["{{content}}<|eot_id|>"]),format_system=StringFormatter(slots=["<|start_header_id|>system<|end_header_id|>\n\n{{content}}<|eot_id|>"]),format_function=FunctionFormatter(slots=["{{content}}<|eot_id|>"], tool_format="llama3"),format_observation=StringFormatter(slots=[("<|start_header_id|>ipython<|end_header_id|>\n\n{{content}}<|eot_id|>""<|start_header_id|>assistant<|end_header_id|>\n\n")]),format_tools=ToolFormatter(tool_format="llama3"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),stop_words=["<|eot_id|>", "<|eom_id|>"],
)# copied from llama3 template
register_template(name="mllama",format_user=StringFormatter(slots=[("<|start_header_id|>user<|end_header_id|>\n\n{{content}}<|eot_id|>""<|start_header_id|>assistant<|end_header_id|>\n\n")]),format_assistant=StringFormatter(slots=["{{content}}<|eot_id|>"]),format_system=StringFormatter(slots=["<|start_header_id|>system<|end_header_id|>\n\n{{content}}<|eot_id|>"]),format_function=FunctionFormatter(slots=["{{content}}<|eot_id|>"], tool_format="llama3"),format_observation=StringFormatter(slots=[("<|start_header_id|>ipython<|end_header_id|>\n\n{{content}}<|eot_id|>""<|start_header_id|>assistant<|end_header_id|>\n\n")]),format_tools=ToolFormatter(tool_format="llama3"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),stop_words=["<|eot_id|>", "<|eom_id|>"],mm_plugin=get_mm_plugin(name="mllama", image_token="<|image|>"),
)# copied from vicuna template
register_template(name="llava",format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]),default_system=("A chat between a curious user and an artificial intelligence assistant. ""The assistant gives helpful, detailed, and polite answers to the user's questions."),mm_plugin=get_mm_plugin(name="llava", image_token="<image>"),
)# copied from vicuna template
register_template(name="llava_next",format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]),default_system=("A chat between a curious user and an artificial intelligence assistant. ""The assistant gives helpful, detailed, and polite answers to the user's questions."),mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"),
)# copied from llama3 template
register_template(name="llava_next_llama3",format_user=StringFormatter(slots=[("<|start_header_id|>user<|end_header_id|>\n\n{{content}}<|eot_id|>""<|start_header_id|>assistant<|end_header_id|>\n\n")]),format_assistant=StringFormatter(slots=["{{content}}<|eot_id|>"]),format_system=StringFormatter(slots=["<|start_header_id|>system<|end_header_id|>\n\n{{content}}<|eot_id|>"]),format_function=FunctionFormatter(slots=["{{content}}<|eot_id|>"], tool_format="llama3"),format_observation=StringFormatter(slots=[("<|start_header_id|>ipython<|end_header_id|>\n\n{{content}}<|eot_id|>""<|start_header_id|>assistant<|end_header_id|>\n\n")]),format_tools=ToolFormatter(tool_format="llama3"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),stop_words=["<|eot_id|>", "<|eom_id|>"],mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"),
)# copied from mistral template
register_template(name="llava_next_mistral",format_user=StringFormatter(slots=["[INST] {{content}}[/INST]"]),format_assistant=StringFormatter(slots=[" {{content}}", {"eos_token"}]),format_system=StringFormatter(slots=["{{content}}\n\n"]),format_function=FunctionFormatter(slots=["[TOOL_CALLS] {{content}}", {"eos_token"}], tool_format="mistral"),format_observation=StringFormatter(slots=["""[TOOL_RESULTS] {"content": {{content}}}[/TOOL_RESULTS]"""]),format_tools=ToolFormatter(tool_format="mistral"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"),template_class=Llama2Template,
)# copied from qwen template
register_template(name="llava_next_qwen",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_function=FunctionFormatter(slots=["{{content}}<|im_end|>\n"], tool_format="qwen"),format_observation=StringFormatter(slots=["<|im_start|>user\n<tool_response>\n{{content}}\n</tool_response><|im_end|>\n<|im_start|>assistant\n"]),format_tools=ToolFormatter(tool_format="qwen"),default_system="You are a helpful assistant.",stop_words=["<|im_end|>"],mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"),
)# copied from chatml template
register_template(name="llava_next_yi",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),stop_words=["<|im_end|>"],mm_plugin=get_mm_plugin(name="llava_next", image_token="<image>"),
)# copied from vicuna template
register_template(name="llava_next_video",format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]),default_system=("A chat between a curious user and an artificial intelligence assistant. ""The assistant gives helpful, detailed, and polite answers to the user's questions."),mm_plugin=get_mm_plugin(name="llava_next_video", image_token="<image>", video_token="<video>"),
)# copied from mistral template
register_template(name="llava_next_video_mistral",format_user=StringFormatter(slots=["[INST] {{content}}[/INST]"]),format_assistant=StringFormatter(slots=[" {{content}}", {"eos_token"}]),format_system=StringFormatter(slots=["{{content}}\n\n"]),format_function=FunctionFormatter(slots=["[TOOL_CALLS] {{content}}", {"eos_token"}], tool_format="mistral"),format_observation=StringFormatter(slots=["""[TOOL_RESULTS] {"content": {{content}}}[/TOOL_RESULTS]"""]),format_tools=ToolFormatter(tool_format="mistral"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),mm_plugin=get_mm_plugin(name="llava_next_video", image_token="<image>", video_token="<video>"),template_class=Llama2Template,
)# copied from chatml template
register_template(name="llava_next_video_yi",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),stop_words=["<|im_end|>"],mm_plugin=get_mm_plugin(name="llava_next_video", image_token="<image>", video_token="<video>"),
)# copied from chatml template
register_template(name="marco",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),default_system=("你是一個經過良好訓練的AI助手，你的名字是Marco-o1.由阿里國際數字商業集團的AI Business創造.\n## 重要！！！！！\n""當你回答問題時，你的思考應該在<Thought>內完成，<Output>內輸出你的結果。\n""<Thought>應該盡可能是英文，但是有2個特例，一個是對原文中的引用，另一個是是數學應該使用markdown格式，<Output>內的輸出需要遵循用戶輸入的語言。\n"),stop_words=["<|im_end|>"],
)# copied from chatml template
register_template(name="minicpm_v",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),stop_words=["<|im_end|>"],default_system="You are a helpful assistant.",mm_plugin=get_mm_plugin(name="minicpm_v", image_token="<image>", video_token="<video>"),
)# copied from minicpm_v template
register_template(name="minicpm_o",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),stop_words=["<|im_end|>"],default_system="You are Qwen, created by Alibaba Cloud. You are a helpful assistant.",mm_plugin=get_mm_plugin(name="minicpm_v", image_token="<image>", video_token="<video>", audio_token="<audio>"),
)# mistral tokenizer v3 tekken
register_template(name="ministral",format_user=StringFormatter(slots=["[INST]{{content}}[/INST]"]),format_system=StringFormatter(slots=["{{content}}\n\n"]),format_function=FunctionFormatter(slots=["[TOOL_CALLS]{{content}}", {"eos_token"}], tool_format="mistral"),format_observation=StringFormatter(slots=["""[TOOL_RESULTS]{"content": {{content}}}[/TOOL_RESULTS]"""]),format_tools=ToolFormatter(tool_format="mistral"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),template_class=Llama2Template,
)# mistral tokenizer v3
register_template(name="mistral",format_user=StringFormatter(slots=["[INST] {{content}}[/INST]"]),format_assistant=StringFormatter(slots=[" {{content}}", {"eos_token"}]),format_system=StringFormatter(slots=["{{content}}\n\n"]),format_function=FunctionFormatter(slots=["[TOOL_CALLS] {{content}}", {"eos_token"}], tool_format="mistral"),format_observation=StringFormatter(slots=["""[TOOL_RESULTS] {"content": {{content}}}[/TOOL_RESULTS]"""]),format_tools=ToolFormatter(tool_format="mistral"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),template_class=Llama2Template,
)# mistral tokenizer v7 tekken (copied from ministral)
register_template(name="mistral_small",format_user=StringFormatter(slots=["[INST]{{content}}[/INST]"]),format_system=StringFormatter(slots=["[SYSTEM_PROMPT]{{content}}[/SYSTEM_PROMPT]"]),format_function=FunctionFormatter(slots=["[TOOL_CALLS]{{content}}", {"eos_token"}], tool_format="mistral"),format_observation=StringFormatter(slots=["""[TOOL_RESULTS]{"content": {{content}}}[/TOOL_RESULTS]"""]),format_tools=ToolFormatter(tool_format="mistral"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
)register_template(name="olmo",format_user=StringFormatter(slots=["<|user|>\n{{content}}<|assistant|>\n"]),format_prefix=EmptyFormatter(slots=[{"eos_token"}]),
)register_template(name="openchat",format_user=StringFormatter(slots=["GPT4 Correct User: {{content}}", {"eos_token"}, "GPT4 Correct Assistant:"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
)register_template(name="openchat-3.6",format_user=StringFormatter(slots=[("<|start_header_id|>GPT4 Correct User<|end_header_id|>\n\n{{content}}<|eot_id|>""<|start_header_id|>GPT4 Correct Assistant<|end_header_id|>\n\n")]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),stop_words=["<|eot_id|>"],
)# copied from chatml template
register_template(name="opencoder",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_observation=StringFormatter(slots=["<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),default_system="You are OpenCoder, created by OpenCoder Team.",stop_words=["<|im_end|>"],
)register_template(name="orion",format_user=StringFormatter(slots=["Human: {{content}}\n\nAssistant: ", {"eos_token"}]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
)# copied from gemma template
register_template(name="paligemma",format_user=StringFormatter(slots=["<start_of_turn>user\n{{content}}<end_of_turn>\n<start_of_turn>model\n"]),format_assistant=StringFormatter(slots=["{{content}}<end_of_turn>\n"]),format_observation=StringFormatter(slots=["<start_of_turn>tool\n{{content}}<end_of_turn>\n<start_of_turn>model\n"]),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),mm_plugin=get_mm_plugin(name="paligemma", image_token="<image>"),
)register_template(name="phi",format_user=StringFormatter(slots=["<|user|>\n{{content}}<|end|>\n<|assistant|>\n"]),format_assistant=StringFormatter(slots=["{{content}}<|end|>\n"]),format_system=StringFormatter(slots=["<|system|>\n{{content}}<|end|>\n"]),stop_words=["<|end|>"],
)register_template(name="phi_small",format_user=StringFormatter(slots=["<|user|>\n{{content}}<|end|>\n<|assistant|>\n"]),format_assistant=StringFormatter(slots=["{{content}}<|end|>\n"]),format_system=StringFormatter(slots=["<|system|>\n{{content}}<|end|>\n"]),format_prefix=EmptyFormatter(slots=[{"<|endoftext|>"}]),stop_words=["<|end|>"],
)register_template(name="phi4",format_user=StringFormatter(slots=["<|im_start|>user<|im_sep|>{{content}}<|im_end|><|im_start|>assistant<|im_sep|>"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>"]),format_system=StringFormatter(slots=["<|im_start|>system<|im_sep|>{{content}}<|im_end|>"]),stop_words=["<|im_end|>"],
)# copied from ministral template
register_template(name="pixtral",format_user=StringFormatter(slots=["[INST]{{content}}[/INST]"]),format_system=StringFormatter(slots=["{{content}}\n\n"]),format_function=FunctionFormatter(slots=["[TOOL_CALLS]{{content}}", {"eos_token"}], tool_format="mistral"),format_observation=StringFormatter(slots=["""[TOOL_RESULTS]{"content": {{content}}}[/TOOL_RESULTS]"""]),format_tools=ToolFormatter(tool_format="mistral"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),mm_plugin=get_mm_plugin(name="pixtral", image_token="[IMG]"),template_class=Llama2Template,
)# copied from chatml template
register_template(name="qwen",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_function=FunctionFormatter(slots=["{{content}}<|im_end|>\n"], tool_format="qwen"),format_observation=StringFormatter(slots=["<|im_start|>user\n<tool_response>\n{{content}}\n</tool_response><|im_end|>\n<|im_start|>assistant\n"]),format_tools=ToolFormatter(tool_format="qwen"),default_system="You are a helpful assistant.",stop_words=["<|im_end|>"],
)# copied from chatml template
register_template(name="qwen2_audio",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),default_system="You are a helpful assistant.",stop_words=["<|im_end|>"],mm_plugin=get_mm_plugin(name="qwen2_audio", audio_token="<|AUDIO|>"),
)# copied from qwen template
register_template(name="qwen2_vl",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),format_function=FunctionFormatter(slots=["{{content}}<|im_end|>\n"], tool_format="qwen"),format_observation=StringFormatter(slots=["<|im_start|>user\n<tool_response>\n{{content}}\n</tool_response><|im_end|>\n<|im_start|>assistant\n"]),format_tools=ToolFormatter(tool_format="qwen"),default_system="You are a helpful assistant.",stop_words=["<|im_end|>"],mm_plugin=get_mm_plugin(name="qwen2_vl", image_token="<|image_pad|>", video_token="<|video_pad|>"),
)register_template(name="sailor",format_user=StringFormatter(slots=["<|im_start|>question\n{{content}}<|im_end|>\n<|im_start|>answer\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),default_system=("You are an AI assistant named Sailor created by Sea AI Lab. ""Your answer should be friendly, unbiased, faithful, informative and detailed."),stop_words=["<|im_end|>"],
)# copied from llama3 template
register_template(name="skywork_o1",format_user=StringFormatter(slots=[("<|start_header_id|>user<|end_header_id|>\n\n{{content}}<|eot_id|>""<|start_header_id|>assistant<|end_header_id|>\n\n")]),format_assistant=StringFormatter(slots=["{{content}}<|eot_id|>"]),format_system=StringFormatter(slots=["<|start_header_id|>system<|end_header_id|>\n\n{{content}}<|eot_id|>"]),format_function=FunctionFormatter(slots=["{{content}}<|eot_id|>"], tool_format="llama3"),format_observation=StringFormatter(slots=[("<|start_header_id|>ipython<|end_header_id|>\n\n{{content}}<|eot_id|>""<|start_header_id|>assistant<|end_header_id|>\n\n")]),format_tools=ToolFormatter(tool_format="llama3"),format_prefix=EmptyFormatter(slots=[{"bos_token"}]),default_system=("You are Skywork-o1, a thinking model developed by Skywork AI, specializing in solving complex problems ""involving mathematics, coding, and logical reasoning through deep thought. When faced with a user's request, ""you first engage in a lengthy and in-depth thinking process to explore possible solutions to the problem. ""After completing your thoughts, you then provide a detailed explanation of the solution process ""in your response."),stop_words=["<|eot_id|>", "<|eom_id|>"],
)register_template(name="solar",format_user=StringFormatter(slots=["### User:\n{{content}}\n\n### Assistant:\n"]),format_system=StringFormatter(slots=["### System:\n{{content}}\n\n"]),efficient_eos=True,
)register_template(name="starchat",format_user=StringFormatter(slots=["<|user|>\n{{content}}<|end|>\n<|assistant|>"]),format_assistant=StringFormatter(slots=["{{content}}<|end|>\n"]),format_system=StringFormatter(slots=["<|system|>\n{{content}}<|end|>\n"]),stop_words=["<|end|>"],
)register_template(name="telechat",format_user=StringFormatter(slots=["<_user>{{content}}<_bot>"]),format_system=StringFormatter(slots=["<_system>{{content}}<_end>"]),
)register_template(name="telechat2",format_user=StringFormatter(slots=["<_user>{{content}}<_bot>"]),format_system=StringFormatter(slots=["<_system>{{content}}"]),default_system=("你是中國電信星辰語義大模型，英文名是TeleChat，你是由中電信人工智能科技有限公司和中國電信人工智能研究院（TeleAI）研發的人工智能助手。"),
)register_template(name="vicuna",format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]),default_system=("A chat between a curious user and an artificial intelligence assistant. ""The assistant gives helpful, detailed, and polite answers to the user's questions."),replace_jinja_template=True,
)register_template(name="video_llava",format_user=StringFormatter(slots=["USER: {{content}} ASSISTANT:"]),default_system=("A chat between a curious user and an artificial intelligence assistant. ""The assistant gives helpful, detailed, and polite answers to the user's questions."),mm_plugin=get_mm_plugin(name="video_llava", image_token="<image>", video_token="<video>"),
)register_template(name="xuanyuan",format_user=StringFormatter(slots=["Human: {{content}} Assistant:"]),default_system=("以下是用戶和人工智能助手之間的對話。用戶以Human開頭，人工智能助手以Assistant開頭，""會對人類提出的問題給出有幫助、高質量、詳細和禮貌的回答，并且總是拒絕參與與不道德、""不安全、有爭議、政治敏感等相關的話題、問題和指示。\n"),
)register_template(name="xverse",format_user=StringFormatter(slots=["Human: {{content}}\n\nAssistant: "]),
)register_template(name="yayi",format_user=StringFormatter(slots=[{"token": "<|Human|>"}, ":\n{{content}}\n\n", {"token": "<|YaYi|>"}, ":"]),format_assistant=StringFormatter(slots=["{{content}}\n\n"]),format_system=StringFormatter(slots=[{"token": "<|System|>"}, ":\n{{content}}\n\n"]),default_system=("You are a helpful, respectful and honest assistant named YaYi ""developed by Beijing Wenge Technology Co.,Ltd. ""Always answer as helpfully as possible, while being safe.  ""Your answers should not include any harmful, unethical, ""racist, sexist, toxic, dangerous, or illegal content. ""Please ensure that your responses are socially unbiased and positive in nature.\n\n""If a question does not make any sense, or is not factually coherent, ""explain why instead of answering something not correct. ""If you don't know the answer to a question, please don't share false information."),stop_words=["<|End|>"],
)# copied from chatml template
register_template(name="yi",format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n"]),format_assistant=StringFormatter(slots=["{{content}}<|im_end|>\n"]),format_system=StringFormatter(slots=["<|im_start|>system\n{{content}}<|im_end|>\n"]),stop_words=["<|im_end|>"],
)register_template(name="yi_vl",format_user=StringFormatter(slots=["### Human: {{content}}\n### Assistant:"]),format_assistant=StringFormatter(slots=["{{content}}\n"]),default_system=("This is a chat between an inquisitive human and an AI assistant. ""Assume the role of the AI assistant. Read all the images carefully, ""and respond to the human's questions with informative, helpful, detailed and polite answers. ""這是一個好奇的人類和一個人工智能助手之間的對話。假設你扮演這個AI助手的角色。""仔細閱讀所有的圖像，并對人類的問題做出信息豐富、有幫助、詳細的和禮貌的回答。\n\n"),stop_words=["###"],efficient_eos=True,mm_plugin=get_mm_plugin(name="llava", image_token="<image>"),
)register_template(name="yuan",format_user=StringFormatter(slots=["{{content}}", {"token": "<sep>"}]),format_assistant=StringFormatter(slots=["{{content}}<eod>\n"]),stop_words=["<eod>"],
)register_template(name="zephyr",format_user=StringFormatter(slots=["<|user|>\n{{content}}", {"eos_token"}, "<|assistant|>\n"]),format_system=StringFormatter(slots=["<|system|>\n{{content}}", {"eos_token"}]),default_system="You are Zephyr, a helpful assistant.",
)register_template(name="ziya",format_user=StringFormatter(slots=["<human>:{{content}}\n<bot>:"]),format_assistant=StringFormatter(slots=["{{content}}\n"]),
)