LMDeploy is an integrated toolkit for compressing, deploying, and serving large language models (LLMs), offering solutions for efficient inference. This guide demonstrates LMDeploy's deployment process, quantization techniques, and API serving capabilities.

Environment Setup

Conda Environment Creation

studio-conda -t lmdeploy -o pytorch-2.1.2

LMDeploy Installation

conda activate lmdeploy
pip install lmdeploy[all]==0.3.0

Model Deployment with LMDeploy

Model Download

cd ~
ln -s /root/share/new_models/Shanghai_AI_Laboratory/internlm2-chat-1_8b /root/

Interactive Chat

lmdeploy chat /root/internlm2-chat-1_8b

Model Quantization Techniques

KV Cache Configurasion

KV Cache reduces memory bandwidth pressure by caching intermediate results. Adjust cache usage:

lmdeploy chat /root/internlm2-chat-1_8b --cache-max-entry-count 0.4

W4A16 Quantization

lmdeploy lite auto_awq \
   /root/internlm2-chat-1_8b \
  --calib-dataset 'ptb' \
  --calib-samples 128 \
  --calib-seqlen 1024 \
  --w-bits 4 \
  --w-group-size 128 \
  --work-dir /root/internlm2-chat-1_8b-4bit

API Server Deployment

Starting API Server

lmdeploy serve api_server \
    /root/internlm2-chat-1_8b \
    --model-format hf \
    --quant-policy 0 \
    --server-name 0.0.0.0 \
    --server-port 23333 \
    --tp 1

Client Connections

# Command-line client
lmdeploy serve api_client http://localhost:23333

# Web client
lmdeploy serve gradio http://localhost:23333 \
    --server-name 0.0.0.0 \
    --server-port 6006

Python Integration

Basic Pipeline

from lmdeploy import pipeline

pipe = pipeline('/root/internlm2-chat-1_8b')
response = pipe(['Hi, please introduce yourself', 'What is Shanghai?'])
print(response)

Advanced Configuration

from lmdeploy import pipeline, TurbomindEngineConfig

backend_config = TurbomindEngineConfig(cache_max_entry_count=0.2)
pipe = pipeline('/root/internlm2-chat-1_8b',
                backend_config=backend_config)
response = pipe(['Hi, please introduce yourself', 'What is Shanghai?'])
print(response)

Multimodal Model Support

LLaVA Integration

from lmdeploy.vl import load_image
from lmdeploy import pipeline, TurbomindEngineConfig

backend_config = TurbomindEngineConfig(session_len=8192)
pipe = pipeline('/share/new_models/liuhaotian/llava-v1.6-vicuna-7b', 
                backend_config=backend_config)

image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
response = pipe(('describe this image', image))
print(response)

Gradio Interface

import gradio as gr
from lmdeploy import pipeline, TurbomindEngineConfig

backend_config = TurbomindEngineConfig(session_len=8192)
pipe = pipeline('/share/new_models/liuhaotian/llava-v1.6-vicuna-7b', 
                backend_config=backend_config)

def model(image, text):
    if image is None:
        return [(text, "Please upload an image.")]
    else:
        response = pipe((text, image)).text
        return [(text, response)]

demo = gr.Interface(fn=model, inputs=[gr.Image(type="pil"), gr.Textbox()], outputs=gr.Chatbot())
demo.launch()

Performance Benchmarking

Transformer Baseline

import torch
import datetime
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("/root/internlm2-chat-1_8b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("/root/internlm2-chat-1_8b", 
                                              torch_dtype=torch.float16, 
                                              trust_remote_code=True).cuda()
model = model.eval()

# Test implementation
inp = "Please introduce yourself."
times = 10
total_words = 0
start_time = datetime.datetime.now()

for i in range(times):
    response, history = model.chat(tokenizer, inp, history=history)
    total_words += len(response)

end_time = datetime.datetime.now()
delta_time = end_time - start_time
delta_time = delta_time.seconds + delta_time.microseconds / 1000000.0
speed = total_words / delta_time
print(f"Speed: {speed:.3f} words/s")

LMDeploy Performance

import datetime
from lmdeploy import pipeline

pipe = pipeline('/root/internlm2-chat-1_8b')

inp = "Please introduce yourself."
times = 10
total_words = 0
start_time = datetime.datetime.now()

for i in range(times):
    response = pipe([inp])
    total_words += len(response[0].text)

end_time = datetime.datetime.now()
delta_time = end_time - start_time
delta_time = delta_time.seconds + delta_time.microseconds / 1000000.0
speed = total_words / delta_time
print(f"Speed: {speed:.3f} words/s")