xinference初探

注册模型:

model.json

  "version": 2,"context_length": 2048,"model_name": "qwen2.5-1.5b-instruct","model_lang": ["en", "zh"],"model_ability": ["chat"],"model_family": "qwen2.5","chat_template": "{% for message in messages %}{{ message['role'] }}: {{ message['content'] }}\n{% endfor %}assistant:","model_specs": [{"model_format": "pytorch","model_file_name_template": "model.safetensors","model_size_in_billions": 2,"quantization": "none","model_hub": "huggingface","model_uri": "file:///data/models/qwen2.5-1.5B/qwen2.5-1.5B-instruct"}],"is_builtin": false
}

注册模型
xinference launch --model-name qwen-2.5-1.5 --model-engine vllm

运行模型
xinference launch --model-name qwen-2.5-1.5b-instruct --model-engine transformers

验证:
curl -X 'POST' 'http://localhost:9997/v1/chat/completions' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{
"model": "qwen2.5-1.5b-instruct",
"messages": [
{
"role": "user",
"content": "hello"
}
]
}'