单端口部署多模型最简单解决方案（vllm sglang 等均适用）

TLDR

将不同模型部署在不同端口，使用litellm的proxy功能统一管理转发

步骤1：本地部署好模型在若干端口上

vllm serve Qwen/Qwen3.5-2B --port 8031 vllm serve Qwen/Qwen3.5-4B --port 8032 vllm serve Qwen/Qwen3.5-9B --port 8033

步骤2：安装配置litellm

安装

# 选项1：使用uv安装 uv tool install 'litellm[proxy]' # 选项2：脚本直接安装 curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh

配置litellm

当前目录创建litellm_config.yaml，写入以下内容

model_list: - model_name: Qwen/Qwen3.5-4B litellm_params: model: openai/Qwen/Qwen3.5-4B # add openai/ prefix to route as OpenAI provider api_base: http://localhost:8031/v1 api_key: none - model_name: Qwen/Qwen3.5-2B litellm_params: model: openai/Qwen/Qwen3.5-2B # add openai/ prefix to route as OpenAI provider api_base: http://localhost:8032/v1 api_key: none - model_name: Qwen/Qwen3.5-9B litellm_params: model: openai/Qwen/Qwen3.5-9B # add openai/ prefix to route as OpenAI provider api_base: http://localhost:8033/v1 api_key: none

步骤3：用你需要的端口启动litellm

litellm --config ./litellm_config.yaml --port 8088

然后如下，之后直接使用8088端口就可以访问部署的三个模型了

INFO: Started server process [214508] INFO: Waiting for application startup. ██╗ ██╗████████╗███████╗██╗ ██╗ ███╗ ███╗ ██║ ██║╚══██╔══╝██╔════╝██║ ██║ ████╗ ████║ ██║ ██║ ██║ █████╗ ██║ ██║ ██╔████╔██║ ██║ ██║ ██║ ██╔══╝ ██║ ██║ ██║╚██╔╝██║ ███████╗██║ ██║ ███████╗███████╗███████╗██║ ╚═╝ ██║ ╚══════╝╚═╝ ╚═╝ ╚══════╝╚══════╝╚══════╝╚═╝ ╚═╝ Thank you for using LiteLLM! - Krrish & Ishaan LiteLLM: Proxy initialized with Config, Set models: Qwen/Qwen3.5-4B Qwen/Qwen3.5-2B Qwen/Qwen3.5-9B INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8088 (Press CTRL+C to quit)