LMDeploy Deployment and Quantization Guide for LLMs
LMDeploy is an integrated toolkit for compressing, deploying, and serving large language models (LLMs), offering solutions for efficient inference. This guide demonstrates LMDeploy's deployment process, quantization techniques, and API serving capabilities.
Environment Setup
Conda Environment Creation
studio-conda -t lmdeploy -o pytorch-2.1.2
...
Posted on Fri, 28 Aug 2026 16:55:53 +0000 by Dysan
Deploying Langchain-Chatchat 0.3.0 with Xinference: Setup Walkthrough and Troubleshooting Tips
2024-7-15 Update
The Langchain-Chatchat codebase has advanced to version 0.3.1, which revises CLI execution. The original Step 4 instructions are no longer compatible; follow the project’s official README instead.
The 0.3.0 release of Langchain-Chatchat introduced architectural adjustments, requiring integration with third-party model inferen ...
Posted on Thu, 04 Jun 2026 17:57:22 +0000 by kpetsche20
Configuring a Local LLM Backend for LangChain Applications
Prerequisites and Environment Setup
Establishing a self-hosted foundation for large language models is essential for leveraging LangChain without reliance on external paid APIs. This procedure outlines the initialization process using the Baichuan2-13B-Chat model optimized for 4-bit quantization.
Ensure the server environment supports specific ...
Posted on Thu, 07 May 2026 00:50:33 +0000 by eppievojt