昇思25天學習打卡營第12天 | LLM原理和實踐：MindNLP ChatGLM-6B StreamChat

1. MindNLP ChatGLM-6B StreamChat

本案例基于MindNLP和ChatGLM-6B實現一個聊天應用。

ChatGLM-6B應該是國內第一個發布的可以在消費級顯卡上進行推理部署的國產開源大模型，2023年3月就發布了。我在23年6月份的時候就在自己的筆記本電腦上部署測試過，當時的1代6B模型已經能解雞兔同籠的數學問題，感覺上是真正“理解”了人類語言的語義。我認為和chatgpt相比，也并沒有非常明顯的差距。當然存在的問題也有不少。

1.1 環境配置

安裝mindnlp

!pip install mindnlp

安裝過程:

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting mindnlpDownloading https://pypi.tuna.tsinghua.edu.cn/packages/72/37/ef313c23fd587c3d1f46b0741c98235aecdfd93b4d6d446376f3db6a552c/mindnlp-0.3.1-py3-none-any.whl (5.7 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 16.7 MB/s eta 0:00:00a 0:00:01
Requirement already satisfied: mindspore in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindnlp) (2.2.14)
Requirement already satisfied: tqdm in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindnlp) (4.66.4)
Requirement already satisfied: requests in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindnlp) (2.32.3)
Collecting datasets (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/60/2d/963b266bb8f88492d5ab4232d74292af8beb5b6fdae97902df9e284d4c32/datasets-2.20.0-py3-none-any.whl (547 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 547.8/547.8 kB 16.4 MB/s eta 0:00:00
Collecting evaluate (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c2/d6/ff9baefc8fc679dcd9eb21b29da3ef10c81aa36be630a7ae78e4611588e1/evaluate-0.4.2-py3-none-any.whl (84 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.1/84.1 kB 25.4 MB/s eta 0:00:00
Collecting tokenizers (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ba/26/139bd2371228a0e203da7b3e3eddcb02f45b2b7edd91df00e342e4b55e13/tokenizers-0.19.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.6 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 19.6 MB/s eta 0:00:0000:0100:01
Collecting safetensors (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/02/28e6280ed0f1bde89eed644b80f2ece4e5ae212dc9ee70d7f56fadc93602/safetensors-0.4.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 20.2 MB/s eta 0:00:00a 0:00:01
Collecting sentencepiece (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a3/69/e96ef68261fa5b82379fdedb325ceaf1d353c6e839ec346d8244e0da5f2f/sentencepiece-0.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 15.5 MB/s eta 0:00:00a 0:00:01
Collecting regex (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/70/70/fea4865c89a841432497d1abbfd53878513b55c6543245fabe31cf8df0b8/regex-2024.5.15-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (774 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 774.7/774.7 kB 17.0 MB/s eta 0:00:00a 0:00:01
Collecting addict (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6a/00/b08f23b7d7e1e14ce01419a467b583edbb93c6cdb8654e54a9cc579cd61f/addict-2.4.0-py3-none-any.whl (3.8 kB)
Collecting ml-dtypes (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/50/96/13d7c3cc82d5ef597279216cf56ff461f8b57e7096a3ef10246a83ca80c0/ml_dtypes-0.4.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.2 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 12.6 MB/s eta 0:00:00a 0:00:01
Collecting pyctcdecode (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a5/8a/93e2118411ae5e861d4f4ce65578c62e85d0f1d9cb389bd63bd57130604e/pyctcdecode-0.5.0-py2.py3-none-any.whl (39 kB)
Collecting jieba (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/cb/18eeb235f833b726522d7ebed54f2278ce28ba9438e3135ab0278d9792a2/jieba-0.42.1.tar.gz (19.2 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.2/19.2 MB 19.3 MB/s eta 0:00:0000:0100:01Preparing metadata (setup.py) ... done
Collecting pytest==7.2.0 (from mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/67/68/a5eb36c3a8540594b6035e6cdae40c1ef1b6a2bfacbecc3d1a544583c078/pytest-7.2.0-py3-none-any.whl (316 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.8/316.8 kB 17.8 MB/s eta 0:00:00
Requirement already satisfied: attrs>=19.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (23.2.0)
Requirement already satisfied: iniconfig in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (2.0.0)
Requirement already satisfied: packaging in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (23.2)
Requirement already satisfied: pluggy<2.0,>=0.12 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (1.5.0)
Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (1.2.0)
Requirement already satisfied: tomli>=1.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pytest==7.2.0->mindnlp) (2.0.1)
Requirement already satisfied: filelock in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (3.15.3)
Requirement already satisfied: numpy>=1.17 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (1.26.4)
Collecting pyarrow>=15.0.0 (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/87/60/cc0645eb4ef73f88847e40a7f9d238bae6b7409d6c1f6a5d200d8ade1f09/pyarrow-16.1.0-cp39-cp39-manylinux_2_28_aarch64.whl (38.1 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.1/38.1 MB 18.3 MB/s eta 0:00:0000:0100:01
Collecting pyarrow-hotfix (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e4/f4/9ec2222f5f5f8ea04f66f184caafd991a39c8782e31f5b0266f101cb68ca/pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (0.3.8)
Requirement already satisfied: pandas in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (2.2.2)
Collecting xxhash (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7c/b9/93f860969093d5d1c4fa60c75ca351b212560de68f33dc0da04c89b7dc1b/xxhash-3.4.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (220 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 220.6/220.6 kB 17.8 MB/s eta 0:00:00
Collecting multiprocess (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/da/d9/f7f9379981e39b8c2511c9e0326d212accacb82f12fbfdc1aa2ce2a7b2b6/multiprocess-0.70.16-py39-none-any.whl (133 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.4/133.4 kB 17.5 MB/s eta 0:00:00
Collecting fsspec<=2024.5.0,>=2023.1.0 (from fsspec[http]<=2024.5.0,>=2023.1.0->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ba/a3/16e9fe32187e9c8bc7f9b7bcd9728529faa725231a0c96f2f98714ff2fc5/fsspec-2024.5.0-py3-none-any.whl (316 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.1/316.1 kB 19.4 MB/s eta 0:00:00
Collecting aiohttp (from datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/eb/45/eebe8d2215328434f33ccb44a05d2741ff7ed4b96b56ca507e2ecf598b73/aiohttp-3.9.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 18.2 MB/s eta 0:00:00a 0:00:01
Requirement already satisfied: huggingface-hub>=0.21.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (0.23.4)
Requirement already satisfied: pyyaml>=5.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from datasets->mindnlp) (6.0.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (2.2.2)
Requirement already satisfied: certifi>=2017.4.17 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->mindnlp) (2024.6.2)
Requirement already satisfied: protobuf>=3.13.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (5.27.1)
Requirement already satisfied: asttokens>=2.0.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (2.0.5)
Requirement already satisfied: pillow>=6.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (10.3.0)
Requirement already satisfied: scipy>=1.5.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (1.13.1)
Requirement already satisfied: psutil>=5.6.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (5.9.0)
Requirement already satisfied: astunparse>=1.6.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from mindspore->mindnlp) (1.6.3)
Collecting pygtrie<3.0,>=2.1 (from pyctcdecode->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ec/cd/bd196b2cf014afb1009de8b0f05ecd54011d881944e62763f3c1b1e8ef37/pygtrie-2.5.0-py3-none-any.whl (25 kB)
Collecting hypothesis<7,>=6.14 (from pyctcdecode->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/58/14/a4c621cb713f6053f37afa78ab3809f9d879182422071ca9d4af61c6d1d9/hypothesis-6.105.0-py3-none-any.whl (462 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 462.2/462.2 kB 21.2 MB/s eta 0:00:00
Requirement already satisfied: six in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from asttokens>=2.0.4->mindspore->mindnlp) (1.16.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from astunparse>=1.6.3->mindspore->mindnlp) (0.43.0)
Collecting aiosignal>=1.1.2 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/76/ac/a7305707cb852b7e16ff80eaf5692309bde30e2b1100a1fcacdc8f731d97/aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting frozenlist>=1.1.1 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/57/15/172af60c7e150a1d88ecc832f2590721166ae41eab582172fe1e9844eab4/frozenlist-1.4.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (239 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 kB 19.6 MB/s eta 0:00:00
Collecting multidict<7.0,>=4.5 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d0/10/2ff646c471e84af25fe8111985ffb8ec85a3f6e1ade8643bfcfcc0f4d2b1/multidict-6.0.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (125 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.9/125.9 kB 16.9 MB/s eta 0:00:00
Collecting yarl<2.0,>=1.0 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c6/d6/5b30ae1d8a13104ee2ceb649f28f2db5ad42afbd5697fd0fc61528bb112c/yarl-1.9.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (300 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 300.9/300.9 kB 14.5 MB/s eta 0:00:00
Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/fa/e01228c2938de91d47b307831c62ab9e4001e747789d0b05baf779a6488c/async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.21.2->datasets->mindnlp) (4.11.0)
Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis<7,>=6.14->pyctcdecode->mindnlp)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas->datasets->mindnlp) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas->datasets->mindnlp) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas->datasets->mindnlp) (2024.1)
Building wheels for collected packages: jiebaBuilding wheel for jieba (setup.py) ... doneCreated wheel for jieba: filename=jieba-0.42.1-py3-none-any.whl size=19314459 sha256=b34d50dd74d300723e8f42cfea23a680ef40525b2f94488be07da842e838a41cStored in directory: /home/nginx/.cache/pip/wheels/1a/76/68/b6d79c4db704bb18d54f6a73ab551185f4711f9730c0c15d97
Successfully built jieba
Installing collected packages: sortedcontainers, sentencepiece, pygtrie, jieba, addict, xxhash, safetensors, regex, pytest, pyarrow-hotfix, pyarrow, multiprocess, multidict, ml-dtypes, hypothesis, fsspec, frozenlist, async-timeout, yarl, pyctcdecode, aiosignal, tokenizers, aiohttp, datasets, evaluate, mindnlpAttempting uninstall: pytestFound existing installation: pytest 8.0.0Uninstalling pytest-8.0.0:Successfully uninstalled pytest-8.0.0Attempting uninstall: fsspecFound existing installation: fsspec 2024.6.0Uninstalling fsspec-2024.6.0:Successfully uninstalled fsspec-2024.6.0
Successfully installed addict-2.4.0 aiohttp-3.9.5 aiosignal-1.3.1 async-timeout-4.0.3 datasets-2.20.0 evaluate-0.4.2 frozenlist-1.4.1 fsspec-2024.5.0 hypothesis-6.105.0 jieba-0.42.1 mindnlp-0.3.1 ml-dtypes-0.4.0 multidict-6.0.5 multiprocess-0.70.16 pyarrow-16.1.0 pyarrow-hotfix-0.6 pyctcdecode-0.5.0 pygtrie-2.5.0 pytest-7.2.0 regex-2024.5.15 safetensors-0.4.3 sentencepiece-0.2.0 sortedcontainers-2.4.0 tokenizers-0.19.1 xxhash-3.4.1 yarl-1.9.4[notice] A new release of pip is available: 24.1 -> 24.1.1
[notice] To update, run: python -m pip install --upgrade pip

安裝mdtex2html

!pip install mdtex2html

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting mdtex2htmlDownloading https://pypi.tuna.tsinghua.edu.cn/packages/ff/e8/c5fab9aa5d9254ad7c7e37d33a3c32fd49d82b4c6b54da337bbca378eb5c/mdtex2html-1.3.0-py3-none-any.whl (13 kB)
Requirement already satisfied: gradio in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (4.26.0)
Collecting markdown (from mdtex2html)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fc/b3/0c0c994fe49cd661084f8d5dc06562af53818cc0abefaca35bdc894577c3/Markdown-3.6-py3-none-any.whl (105 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.4/105.4 kB 11.6 MB/s eta 0:00:00
Collecting latex2mathml (from mdtex2html)Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f2/0a/181ed55562ce90179aedf33b09fcd79db31c868a5d480f3cb71a31d19692/latex2mathml-3.77.0-py3-none-any.whl (73 kB)━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.7/73.7 kB 22.7 MB/s eta 0:00:00
Requirement already satisfied: aiofiles<24.0,>=22.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (22.1.0)
Requirement already satisfied: altair<6.0,>=4.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (5.3.0)
Requirement already satisfied: fastapi in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.111.0)
Requirement already satisfied: ffmpy in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.3.2)
Requirement already satisfied: gradio-client==0.15.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.15.1)
Requirement already satisfied: httpx>=0.24.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.27.0)
Requirement already satisfied: huggingface-hub>=0.19.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.23.4)
Requirement already satisfied: importlib-resources<7.0,>=1.3 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (6.4.0)
Requirement already satisfied: jinja2<4.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (3.1.4)
Requirement already satisfied: markupsafe~=2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.1.5)
Requirement already satisfied: matplotlib~=3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (3.9.0)
Requirement already satisfied: numpy~=1.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (1.26.4)
Requirement already satisfied: orjson~=3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (3.10.5)
Requirement already satisfied: packaging in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (23.2)
Requirement already satisfied: pandas<3.0,>=1.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.2.2)
Requirement already satisfied: pillow<11.0,>=8.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (10.3.0)
Requirement already satisfied: pydantic>=2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.7.4)
Requirement already satisfied: pydub in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.25.1)
Requirement already satisfied: python-multipart>=0.0.9 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.0.9)
Requirement already satisfied: pyyaml<7.0,>=5.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (6.0.1)
Requirement already satisfied: ruff>=0.2.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.4.10)
Requirement already satisfied: semantic-version~=2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (2.10.0)
Requirement already satisfied: tomlkit==0.12.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.12.0)
Requirement already satisfied: typer<1.0,>=0.9 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (0.12.3)
Requirement already satisfied: typing-extensions~=4.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (4.11.0)
Requirement already satisfied: uvicorn>=0.14.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio) (0.30.1)
Requirement already satisfied: fsspec in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio-client==0.15.1->gradio) (2024.5.0)
Requirement already satisfied: websockets<12.0,>=10.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from gradio-client==0.15.1->gradio) (11.0.3)
Requirement already satisfied: jsonschema>=3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from altair<6.0,>=4.2.0->gradio) (4.22.0)
Requirement already satisfied: toolz in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from altair<6.0,>=4.2.0->gradio) (0.12.1)
Requirement already satisfied: anyio in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (4.4.0)
Requirement already satisfied: certifi in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (2024.6.2)
Requirement already satisfied: httpcore==1.* in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (1.0.5)
Requirement already satisfied: idna in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (3.7)
Requirement already satisfied: sniffio in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpx>=0.24.1->gradio) (1.3.1)
Requirement already satisfied: h11<0.15,>=0.13 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from httpcore==1.*->httpx>=0.24.1->gradio) (0.14.0)
Requirement already satisfied: filelock in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.19.3->gradio) (3.15.3)
Requirement already satisfied: requests in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.19.3->gradio) (2.32.3)
Requirement already satisfied: tqdm>=4.42.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from huggingface-hub>=0.19.3->gradio) (4.66.4)
Requirement already satisfied: zipp>=3.1.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from importlib-resources<7.0,>=1.3->gradio) (3.17.0)
Requirement already satisfied: contourpy>=1.0.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (1.2.1)
Requirement already satisfied: cycler>=0.10 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (4.53.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (1.4.5)
Requirement already satisfied: pyparsing>=2.3.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (3.1.2)
Requirement already satisfied: python-dateutil>=2.7 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from matplotlib~=3.0->gradio) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas<3.0,>=1.0->gradio) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pandas<3.0,>=1.0->gradio) (2024.1)
Requirement already satisfied: annotated-types>=0.4.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pydantic>=2.0->gradio) (0.7.0)
Requirement already satisfied: pydantic-core==2.18.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from pydantic>=2.0->gradio) (2.18.4)
Requirement already satisfied: click>=8.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (8.1.7)
Requirement already satisfied: shellingham>=1.3.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (1.5.4)
Requirement already satisfied: rich>=10.11.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (13.7.1)
WARNING: typer 0.12.3 does not provide the extra 'all'
Requirement already satisfied: starlette<0.38.0,>=0.37.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (0.37.2)
Requirement already satisfied: fastapi-cli>=0.0.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (0.0.4)
Requirement already satisfied: ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (5.10.0)
Requirement already satisfied: email_validator>=2.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from fastapi->gradio) (2.2.0)
Requirement already satisfied: importlib-metadata>=4.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from markdown->mdtex2html) (7.0.1)
Requirement already satisfied: dnspython>=2.0.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from email_validator>=2.0.0->fastapi->gradio) (2.6.1)
Requirement already satisfied: attrs>=22.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (23.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.35.1)
Requirement already satisfied: rpds-py>=0.7.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.18.1)
Requirement already satisfied: six>=1.5 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio) (1.16.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (2.15.1)
Requirement already satisfied: exceptiongroup>=1.0.2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from anyio->httpx>=0.24.1->gradio) (1.2.0)
Requirement already satisfied: httptools>=0.5.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (0.6.1)
Requirement already satisfied: python-dotenv>=0.13 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (1.0.1)
Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (0.19.0)
Requirement already satisfied: watchfiles>=0.13 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from uvicorn[standard]>=0.12.0->fastapi->gradio) (0.22.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->huggingface-hub>=0.19.3->gradio) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from requests->huggingface-hub>=0.19.3->gradio) (2.2.2)
Requirement already satisfied: mdurl~=0.1 in /home/nginx/miniconda/envs/jupyter/lib/python3.9/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0,>=0.9->typer[all]<1.0,>=0.9; sys_platform != "emscripten"->gradio) (0.1.2)
Installing collected packages: latex2mathml, markdown, mdtex2html
Successfully installed latex2mathml-3.77.0 markdown-3.6 mdtex2html-1.3.0[notice] A new release of pip is available: 24.1 -> 24.1.1
[notice] To update, run: python -m pip install --upgrade pip

安裝gradio

!pip install gradio

配置環境變量

# 設置環境變量 HF_ENDPOINT，其值為 https://hf-mirror.com
# 這個環境變量通常用于指定 Hugging Face Transformers 模型位于國內的鏡像站點
# 以便從該鏡像站點下載模型和相關資源，提高下載速度和穩定性,不需要代理
export HF_ENDPOINT=https://hf-mirror.com

1.2 代碼開發

模型參數量為6B, 磁盤空間大小占用約12G, 下載權重產加載大約需要20分鐘

# 導入 MindNLP 庫中的 AutoModelForSeq2SeqLM 類和 AutoTokenizer 類
# MindNLP 是一個基于 PyTorch 的自然語言處理庫，提供了許多預訓練模型和工具
from mindnlp.transformers import AutoModelForSeq2SeqLM, AutoTokenizer# 導入 Gradio 庫，用于創建一個交互式的 Web 界面
import gradio as gr# 導入 mdtex2html 庫，用于將 Markdown 格式的文本轉換為 HTML 格式
import mdtex2html# 使用 AutoModelForSeq2SeqLM 類從預訓練模型 'ZhipuAI/ChatGLM-6B' 創建一個模型實例
# 'ZhipuAI/ChatGLM-6B' 是一個序列到序列的語言模型，用于文本生成任務
# mirror="modelscope" 參數指定了模型的鏡像站點為 "modelscope"，以提高下載速度和穩定性.half()是將float32轉為float16.
model = AutoModelForSeq2SeqLM.from_pretrained('ZhipuAI/ChatGLM-6B', mirror="modelscope").half()# 將模型設置為評估模式，即不進行訓練
model.set_train(False)# 使用 AutoTokenizer 類從預訓練模型 'ZhipuAI/ChatGLM-6B' 創建一個分詞器實例
# 分詞器用于將輸入文本轉換為模型可以理解的 tokens
tokenizer = AutoTokenizer.from_pretrained('ZhipuAI/ChatGLM-6B', mirror="modelscope")

輸出:

100%773/773?[00:00<00:00,?50.6kB/s]
100%32.6k/32.6k?[00:00<00:00,?2.78MB/s]
Downloading?shards:?100%8/8?[16:43<00:00,?107.69s/it]
100%1.62G/1.62G?[02:14<00:00,?14.2MB/s]
100%1.75G/1.75G?[02:19<00:00,?16.8MB/s]
100%1.84G/1.84G?[02:27<00:00,?17.9MB/s]
100%1.78G/1.78G?[02:21<00:00,?17.5MB/s]
100%1.75G/1.75G?[02:20<00:00,?14.6MB/s]
100%1.75G/1.75G?[02:20<00:00,?18.7MB/s]
100%1.00G/1.00G?[01:18<00:00,?16.7MB/s]
100%1.00G/1.00G?[01:19<00:00,?8.88MB/s]
Loading?checkpoint?shards:?100%8/8?[00:51<00:00,??5.84s/it]
100%441/441?[00:00<00:00,?38.4kB/s]
100%2.58M/2.58M?[00:00<00:00,?6.99MB/s]

1.3 進行推理

# 定義一個字符串變量 prompt，內容為 "你好"，表示用戶的提問或對話輸入
prompt = '你好'# 定義一個列表變量 history，用于存儲對話歷史
# 在這個例子中，對話歷史為空，因為這是新的對話
history = []# 調用 model 的 chat 方法，傳入 tokenizer、prompt、history 和 max_length 參數
# chat 方法是模型的一個函數，用于生成對話響應
# tokenizer 是之前加載的分詞器，用于處理輸入文本
# prompt 是用戶的輸入文本
# history 是對話歷史，用于提供上下文信息
# max_length 是生成響應的最大長度
response, _ = model.chat(tokenizer, prompt, history=history, max_length=20)# 打印出模型的對話響應
print(response)

輸出:

\
The dtype of attention mask (Float32) is not bool
|
'你好👋！我是人工智能助手 ChatGLM-6B'

響應非常地慢, 不知是什么原因.
根據npu監控信息來看, 一開始完全沒有利用到ai core, 而是一直在跑內存. 直到開始調用ai core之后,很快就給出了響應.

prompt = '房間里有7只雞和兔子,共20只腳.問有幾只雞,幾只兔子?'
history = []
response, _ = model.chat(tokenizer, prompt, history=history, max_length=512)
response

輸出:
雞兔同籠

可以看到,他有正確的解題思路. 雖然數學稀爛,二元一次方程組都能解錯.
耗時約200s, 輸出250多個字符.如果算token的話,可能就是每秒1個token, 這個性能是真的拉垮…
當然,這是免費的虛擬資源，并且很大概率是共享的，可能并不能和實體的昇騰芯片相提并論。

2. 小結

本文主要介紹了使用mindnlp下載chatglm-6B預訓練模型，并基于此模型，通過輸入提示詞完成回答文本生成的模型推理任務，從而實現了一個簡單的聊天應用。

昇思25天學習打卡營第12天 | LLM原理和實踐：MindNLP ChatGLM-6B StreamChat

1. MindNLP ChatGLM-6B StreamChat

1.1 環境配置

1.2 代碼開發

1.3 進行推理

2. 小結

相關文章

UI自動化測試框架：PO 模式+數據驅動（超詳細）

Python學習中進行條件判斷（if, else, elif）

一篇讀懂128陷阱

NCCL 中的一些輔助debug 知識點

算法題型歸類整理及同類題型解法思路總結（持續更新）

12種增強Python代碼的函數式編程技術

算法·二分

Docker-11☆ Docker Compose部署RuoYi-Cloud

深入理解計算機系統 CSAPP 家庭作業8.22

新加坡工作和生活指北：工作篇

預訓練對齊：數學理論到工程實踐的橋梁

Java-關鍵字（static，final）

Keepalived+HAProxy 集群及虛IP切換實踐

srs直播內網拉流帶寬飆升問題記錄

數學建模論文寫作文檔word

Milvus lite start 及存儲策略

adb參數詳解

最小二乘支持向量機（Least Squares Support Vector Machine，LSSVM）及其Python和MATLAB實現

redis集群部署（通過redis工具快速部署，手動部署）

mysql異常數據損壞處理，報錯：Operating system error number 2 in a file operation