- 短文本测试
- 中等长度测试
- 长文本测试
短文本测试
vllm bench serve
--model /data/models/Qwen1.5-14B-Chat-AWQ
--backend vllm
--base-url http://localhost:8000
--endpoint /v1/completions
--dataset-name random
--num-prompts 50
--random-input-len 32
--random-output-len 64
中等长度测试
vllm bench serve
--model /data/models/Qwen1.5-14B-Chat-AWQ
--backend vllm
--base-url http://localhost:8000
--endpoint /v1/completions
--dataset-name random
--num-prompts 100
--random-input-len 128
--random-output-len 256
长文本测试
vllm bench serve
--model /data/models/Qwen1.5-14B-Chat-AWQ
--backend vllm
--base-url http://localhost:8000
--endpoint /v1/completions
--dataset-name random
--num-prompts 50
--random-input-len 512
--random-output-len 512
============ Serving Benchmark Result ============
Successful requests: 100
Failed requests: 0
Benchmark duration (s): 83.02
Total input tokens: 12800
Total generated tokens: 25600
Request throughput (req/s): 1.20
Output token throughput (tok/s): 308.35
Peak output token throughput (tok/s): 768.00
Peak concurrent requests: 100.00
Total token throughput (tok/s): 462.53
---------------Time to First Token----------------
Mean TTFT (ms): 2960.13
Median TTFT (ms): 1815.87
P99 TTFT (ms): 50967.20
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 185.77
Median TPOT (ms): 184.23
P99 TPOT (ms): 292.57
---------------Inter-token Latency----------------
Mean ITL (ms): 185.77
Median ITL (ms): 124.29
P99 ITL (ms): 428.14
