qwen3.5 vllm本地部署

VLLM_USE_MODELSCOPE=true vllm serve Qwen/Qwen3.5-9B --port 8000 --tensor-parallel-size 1 --max-model-len 262144 --reasoning-parser qwen3

客户端

from openai import OpenAI import os import time start = time.time() # Configured by environment variables client = OpenAI( api_key="EMPTY", base_url="http://ip:8000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/RealWorld/RealWorld-04.png" # "url": image_url } }, { "type": "text", "text": "描述一下" } ] } ] #lovedheart/Qwen3.5-4B-FP8 #tclf90/Qwen3.5-4B-AWQ chat_response = client.chat.completions.create( model="tclf90/Qwen3.5-4B-AWQ", messages=messages, max_tokens=8192, temperature=0.7, top_p=0.8, presence_penalty=1.5, extra_body={ "top_k": 20, "chat_template_kwargs": {"enable_thinking": False}, }, ) print("Chat response:", chat_response) print("total-time:", time.time() -start)

开机自启动服务

[Unit] Description=vLLM Service After=network.target [Service] Type=simple User=ippbx Environment="VLLM_USE_MODELSCOPE=true" Environment="CUDA_VISIBLE_DEVICES=0" Environment="HF_HUB_OFFLINE=1" Environment="TRANSFORMERS_OFFLINE=1" ExecStart=/bin/bash -c 'source /home/ippbx/anaconda3/bin/activate vllm && VLLM_USE_MODELSCOPE=true vllm serve tclf90/Qwen3.5-4B-AWQ --port 8000 --tensor-parallel-size 1 --max-model-len 262144 --reasoning-parser qwen3' Restart=always RestartSec=10 StandardOutput=journal StandardError=journal [Install] WantedBy=multi-user.target

启动及日志查看

sudo systemctl daemon-reload sudo systemctl start vllm.service sudo journalctl -u vllm.service -f