Local Deployment of Qwen 1.5 with PyTorch: A Step-by-Step Guide

This article presents a practical approach to implementing a wrapper for the official Qwen 1.5 examples (also available on the HuggingFace documentation page). While not highly complex, this documentation aims to provide a valuable resource for technical professionals seeking to deploy large language models locally from scratch without relying on third-party tools.

Although tools like Ollama simplify local model deployement with a single command, integrating these models into existing systems can be challenging. Therefore, this guide focuses on a pure code implementation approach that offers greater flexibility for system integration.

1. Model Download

As previously discussed, models from HuggingFace can be downloaded through the mirror site https://hf-mirror.com/ for users in China. After configuring the necessary environment variables, you can download the Qwen model using the following command:

cd /path/to/your/model/directory
huggingface-cli download --resume-download Qwen/Qwen1.5-7B-Chat --local-dir .

The output will indicate the download progress:

Fetching 14 files:   7%|█▊                       | 1/14 [00:00<00:06,  1.87it/s]
downloading https://hf-mirror.com/Qwen/Qwen1.5-7B-Chat/resolve/294483ad23713036574b30587b186713373f4271/README.md to /Users/username/.cache/huggingface/hub/models--Qwen--Qwen1.5-7B-Chat/blobs/0963c198257a0607c4d2def66a84aec172240afd.incomplete
README.md: 4.26kB [00:00, 6.40MB/s]
Fetching 14 files: 100%|████████████████████████| 14/14 [00:01<00:00, 12.81it/s]

If this is your first time downloading the model, please note that the process may take considerable time due to the model's size.

For macOS users, the model will typically be stored in the following directory structure:

pwd
# Output: /Users/username/.cache/huggingface/hub

ls
# Output: 
# models--BAAI--bge-large-zh-v1.5
# models--Qwen--Qwen1.5-7B-Chat
# version.txt

The downloaded model is stored with a directory name following the pattern "models--OriginalOrganization--OriginalModelName" due to the repository path "Qwen/Qwen1.5-7B-Chat".

Some users may question the purpose of the "--local-dir ." parameter when the model apppears to be stored elsewhere. This parameter actually creates symbolic links and configuration files in the specified local directory. Here's what the directory structure looks like:

tree -l
.
├── LICENSE
├── README.md
├── config.json
├── generation_config.json
├── merges.txt
├── model-00001-of-00004.safetensors -> ../../../../../../../../../.cache/huggingface/hub/models--Qwen--Qwen1.5-7B-Chat/blobs/9e8f7873d7c4c74b8883db207a08bf8a783ec8c26da6b3d660a0929048ce6422
├── model-00002-of-00004.safetensors -> ../../../../../../../../../.cache/huggingface/hub/models--Qwen--Qwen1.5-7B-Chat/blobs/e573fdaf3eba785c4b31b8858288f762f3541f09d75b53dfb1ae4d8ee5011d65
├── model-00003-of-00004.safetensors -> ../../../../../../
└── model-00004-of-00004.safetensors -> ../../../../../../

As shown, the actual model files are stored in the cache directory, while the local directory contains symbolic links pointing to these files, along with necessary configuration files.