OpenCompass面向大模型的開源方和使用者, 提供開源、高效、全面的大模型評測開放平臺。
一、OpenCompass文檔
1.基礎安裝
使用Conda準備 OpenCompass 運行環境:
conda create --name opencompass python=3.10 -y
conda activate opencompass
2. 安裝 OpenCompass
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .
root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace# conda create --name opencompass python=3.10 -y
root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace# conda activate opencompass
(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace# cd opencompass
(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace/opencompass# pip install -e .
(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace/opencompass# pip install -e .
### 報錯如下
Looking in indexes: https://mirrors.cloud.aliyuncs.com/pypi/simple
Obtaining file:///mnt/workspace/opencompassPreparing metadata (setup.py) ... errorerror: subprocess-exited-with-error× python setup.py egg_info did not run successfully.│ exit code: 1╰─> [56 lines of output]/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/_distutils_hack/__init__.py:53: UserWarning: Reliance on distutils from stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.ymlwarnings.warn(/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/__init__.py:94: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.!!********************************************************************************Requirements should be satisfied by a PEP 517 installer.If you are using pip, you can try `pip install --use-pep517`.********************************************************************************!!dist.fetch_build_eggs(dist.setup_requires)WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'mirrors.cloud.aliyuncs.com'. (_ssl.c:1007)"))': /pypi/simple/nltk/ERROR: Could not find a version that satisfies the requirement nltk==3.8 (from versions: none)ERROR: No matching distribution found for nltk==3.8Traceback (most recent call last):File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/installer.py", line 107, in _fetch_build_egg_no_warnsubprocess.check_call(cmd)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/subprocess.py", line 369, in check_callraise CalledProcessError(retcode, cmd)subprocess.CalledProcessError: Command '['/mnt/workspace/Anaconda3/envs/opencompass/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpvvttj4xe', '--quiet', 'nltk==3.8']' returned non-zero exit status 1.The above exception was the direct cause of the following exception:Traceback (most recent call last):File "<string>", line 2, in <module>File "<pip-setuptools-caller>", line 34, in <module>File "/mnt/workspace/opencompass/setup.py", line 164, in <module>do_setup()File "/mnt/workspace/opencompass/setup.py", line 108, in do_setupsetup(File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/__init__.py", line 116, in setup_install_setup_requires(attrs)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/__init__.py", line 89, in _install_setup_requires_fetch_build_eggs(dist)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/__init__.py", line 94, in _fetch_build_eggsdist.fetch_build_eggs(dist.setup_requires)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/dist.py", line 768, in fetch_build_eggsreturn _fetch_build_eggs(self, requires)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/installer.py", line 44, in _fetch_build_eggsresolved_dists = pkg_resources.working_set.resolve(File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/pkg_resources/__init__.py", line 893, in resolvedist = self._resolve_dist(File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/pkg_resources/__init__.py", line 929, in _resolve_distdist = best[req.key] = env.best_match(File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1267, in best_matchreturn self.obtain(req, installer)File "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1303, in obtainreturn installer(requirement) if installer else NoneFile "/mnt/workspace/Anaconda3/envs/opencompass/lib/python3.10/site-packages/setuptools/installer.py", line 109, in _fetch_build_egg_no_warnraise DistutilsError(str(e)) from edistutils.errors.DistutilsError: Command '['/mnt/workspace/Anaconda3/envs/opencompass/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpvvttj4xe', '--quiet', 'nltk==3.8']' returned non-zero exit status 1.[end of output]note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed× Encountered error while generating package metadata.
╰─> See above for output.note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
改用如下安裝成功
(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace/opencompass# pip install -r /mnt/workspace/opencompass/requirements/runtime.txt
3.數據集準備
在 OpenCompass 項目根目錄下運行下面命令,將數據集準備至 ${OpenCompass}/data 目錄下:
wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip
unzip OpenCompassData-core-20240207.zip
4.配置評估任務
命令行(自定義HF模型)
對于 HuggingFace 模型,用戶可以通過命令行直接設置模型參數,無需額外的配置文件。例如,對于
internlm/internlm2-chat-1_8b 模型,可以使用以下命令進行評估:
命令行(自定義hf類型)
windows環境好像不太行
python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen --hf-type chat --hf-path "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\model\Qwen\Qwen1___5-0___5B-Chat" --debug
python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen --hf-path "D:\Program Files\python\PycharmProjects\AiStudyProject\demo07\models\Qwen\Qwen2___5-1___5B-Instruct" --debug
報錯如下:
signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
04/11 15:38:55 - OpenCompass - INFO - Loading demo_gsm8k_chat_gen: D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\configs\./datasets\demo\demo_gsm8k_chat_gen.py
04/11 15:38:55 - OpenCompass - INFO - Loading demo_math_chat_gen: D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\configs\./datasets\demo\demo_math_chat_gen.py
04/11 15:38:55 - OpenCompass - INFO - Loading example: D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\configs\./summarizers\example.py
04/11 15:38:55 - OpenCompass - INFO - Current exp folder: outputs\default\20250411_153855
04/11 15:38:56 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
demo_gsm8k train 7473
demo_gsm8k test 1319
demo_math test 5000
demo_math train 5000
04/11 15:38:59 - OpenCompass - INFO - Partitioned into 1 tasks.
Traceback (most recent call last):File "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\run.py", line 4, in <module>main()File "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\cli\main.py", line 339, in mainrunner(tasks)File "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\runners\base.py", line 38, in __call__status = self.launch(tasks)File "D:\Program Files\python\PycharmProjects\AiStudyProject\demo14\opencompass\opencompass\runners\local.py", line 102, in launchassert len(all_gpu_ids) >= num_gpus
AssertionError
AssertionError: assert len(all_gpu_ids) >= num_gpus。這表明系統檢測到的可用GPU數量不足以滿足任務所需的GPU數量。
安裝cuda
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen --hf-path "D:\Program Files\python\PycharmProjects\AiStudyProject\demo07\models\Qwen\Qwen2___5-1___5B-Instruct" --debug --hf-num-gpus 1 #指定使用gpu的數量
或者直接使用cpu
python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen --hf-path "D:\Program Files\python\PycharmProjects\AiStudyProject\demo07\models\Qwen\Qwen2___5-1___5B-Instruct" --debug --hf-num-gpus 0
運行成功以后有outputs文件
沒有結果,不知道什么原因
日志如下:
D:\envs\opencompass\python.exe: can't open file 'D:\\Program': [Errno 2] No such file or directory
D:\envs\opencompass\python.exe: can't open file 'D:\\Program': [Errno 2] No such file or directory
D:\envs\opencompass\python.exe: can't open file 'D:\\Program': [Errno 2] No such file or directory
但是不管怎么改都是一樣的問題,所以改用linux環境
linux 環境沒問題
(opencompass) root@dsw-990246-5b546cb984-pcfzq:/mnt/workspace/opencompass# python run.py --datasets demo_gsm8k_chat_gen demo_math_chat_gen --hf-type chat --hf-path "/mnt/workspace/llm/Qwen/Qwen1.5-0.5B-Chat" --debug
正常啟動
有正常的outputs文件輸出
有評測結果
請注意,通過這種方式,OpenCompass 一次只評估一個模型,而其他方式可以一次評估多個模型。
命令行
用戶可以使用 --models 和 --datasets 結合想測試的模型和數據集。
models 名為以下兩個文件名,不然會報錯
python run.py --models hf_qwen2_5_0_5b_instruct hf_qwen1_5_0_5b_chat --datasets demo_gsm8k_chat_gen demo_math_chat_gen --debug
在文件opencompass/opencompass/configs/models/qwen2_5或者qwen
最后看到如下得分:opencompass/outputs/default/20250411_174951/summary/summary_20250411_174951.md
模型和數據集的配置文件預存于 configs/models 和 configs/datasets 中。用戶可以使用
tools/list_configs.py 查看或過濾當前可用的模型和數據集配置。
# 列出所有配置
python tools/list_configs.py
# 列出與llama和mmlu相關的所有配置
python tools/list_configs.py llama mmlu
運行 python tools/list_configs.py llama mmlu 將產生如下輸出:
+-----------------+-----------------------------------+
| Model | Config Path |
|-----------------+-----------------------------------|
| hf_llama2_13b | configs/models/hf_llama2_13b.py |
| hf_llama2_70b | configs/models/hf_llama2_70b.py |
| ... | ... |
+-----------------+-----------------------------------+
+-------------------+---------------------------------------------------+
| Dataset | Config Path |
|-------------------+---------------------------------------------------|
| cmmlu_gen | configs/datasets/cmmlu/cmmlu_gen.py |
| cmmlu_gen_ffe7c0 | configs/datasets/cmmlu/cmmlu_gen_ffe7c0.py |
| ... | ... |
+-------------------+---------------------------------------------------+
用戶可以使用第一列中的名稱作為 python run.py 中 --models 和 --datasets 的輸入參數。對于數據集,同一名稱的不同后綴通常表示其提示或評估方法不同。
二、配置文件
除了通過命令行配置實驗外,OpenCompass 還允許用戶在配置文件中編寫實驗的完整配置,并通過run.py 直接運行它。配置文件是以 Python 格式組織的,并且必須包括 datasets 和 models 字段。
from mmengine.config import read_basewith read_base():from .datasets.demo.demo_gsm8k_chat_gen import gsm8k_datasetsfrom .datasets.demo.demo_math_chat_gen import math_datasetsfrom .models.qwen.hf_qwen1_5_0_5b_chat import models as hf_qwen1_5_0_5b_chatfrom .models.qwen2_5.hf_qwen2_5_1_5b_instruct import models as hf_qwen2_5_1_5b_instructdatasets = gsm8k_datasets + math_datasets
models = hf_qwen1_5_0_5b_chat + hf_qwen2_5_1_5b_instruct
運行任務時,我們只需將配置文件的路徑傳遞給 run.py :
python run.py configs/eval_chat_demo.py --debug
三、自定義數據集
數據集格式
選擇題 ( mcq )
對于選擇 ( mcq ) 類型的數據,默認的字段如下:
- question : 表示選擇題的題干
- A , B , C , …: 使用單個大寫字母表示選項,個數不限定。默認只會從 A 開始,解析連續的字母作
為選項。 - answer : 表示選擇題的正確答案,其值必須是上述所選用的選項之一,如 A , B , C 等。
對于非默認字段,我們都會進行讀入,但默認不會使用。如需使用,則需要在 .meta.json 文件中進行
指定。
.jsonl 格式樣例如下:
{"question": "165+833+650+615=", "A": "2258", "B": "2263", "C": "2281", "answer":
"B"}
{"question": "368+959+918+653+978=", "A": "3876", "B": "3878", "C": "3880",
"answer": "A"}
{"question": "776+208+589+882+571+996+515+726=", "A": "5213", "B": "5263", "C":
"5383", "answer": "B"}
{"question": "803+862+815+100+409+758+262+169=", "A": "4098", "B": "4128", "C":
"4178", "answer": "C"}
- .csv 格式樣例如下:
question,A,B,C,answer
127+545+588+620+556+199=,2632,2635,2645,B
735+603+102+335+605=,2376,2380,2410,B
506+346+920+451+910+142+659+850=,4766,4774,4784,C
504+811+870+445=,2615,2630,2750,B
問答題 ( qa )
對于問答 ( qa ) 類型的數據,默認的字段如下:
- question : 表示問答題的題干
- answer : 表示問答題的正確答案。可缺失,表示該數據集無正確答案。
對于非默認字段,我們都會進行讀入,但默認不會使用。如需使用,則需要在 .meta.json 文件中進行
指定。
.json格式樣例如下:
{"question": "752+361+181+933+235+986=", "answer": "3448"}
{"question": "712+165+223+711=", "answer": "1811"}
{"question": "921+975+888+539=", "answer": "3323"}
{"question": "752+321+388+643+568+982+468+397=", "answer": "4519"}
.csv 格式樣例如下:
question,answer
123+147+874+850+915+163+291+604=,3967
149+646+241+898+822+386=,3142
332+424+582+962+735+798+653+214=,4700
649+215+412+495+220+738+989+452=,4170
6.命令行列表
自定義數據集可直接通過命令行來調用開始評測。
python run.py --models hf_llama2_7b --custom-dataset-path xxx/test_mcq.csv --custom-dataset-data-type mcq --custom-dataset-infer-method ppl
python run.py \
--models hf_llama2_7b \
--custom-dataset-path xxx/test_qa.jsonl \
--custom-dataset-data-type qa \
--custom-dataset-infer-method gen
在絕大多數情況下, --custom-dataset-data-type 和 --custom-dataset-infer-method 可以省略,
- OpenCompass 會根據以下邏輯進行設置:
如果從數據集文件中可以解析出選項,如 A , B , C 等,則認定該數據集為 mcq ,否則認定為
qa 。 - 默認 infer_method 為 gen 。