[Migration] Adapt to vLLM-V1 engine migration interface by KuilongCui · Pull Request #290 · llumnix-project/llumnix-ray

KuilongCui · 2025-07-31T08:51:02Z

No description provided.

github-actions · 2025-08-07T03:11:54Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.4498	1.4498	1.4498	1.4498	1.4498
across_llumlet_latency	1.0663	1.0663	1.0663	1.0663	1.0663
across_engine_latency	0.2923	0.2923	0.2923	0.2923	0.2923
process_model_outputs_latency	0.0801	0.0739	0.1121	0.0711	0.1137
engine_step_latency	34.2644	34.1207	35.0900	33.9397	35.1382
step_postprocess_latency	0.0239	0.0113	0.1264	0.0109	0.1377
across_async_put_queue_thread_latency	0.0108	0.0102	0.0141	0.0098	0.0144
across_async_put_queue_actor_latency	0.1896	0.1884	0.2043	0.1810	0.2050
across_queue_client_latency	0.0341	0.0333	0.0413	0.0298	0.0419
queue_rpc_latency	0.2550	0.2530	0.2904	0.2329	0.2914
api_server_get_queue_latency	0.0927	0.0928	0.1040	0.0844	0.1048
across_request_streams_latency	0.0694	0.0599	0.1363	0.0532	0.1416

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.4079	1.4079	1.4079	1.4079	1.4079
across_llumlet_latency	1.0242	1.0242	1.0242	1.0242	1.0242
across_engine_latency	0.2988	0.2988	0.2988	0.2988	0.2988
process_model_outputs_latency	0.0913	0.0857	0.1302	0.0815	0.1331
engine_step_latency	34.3617	34.2969	34.8214	33.9995	34.8254
step_postprocess_latency	0.0278	0.0125	0.1527	0.0118	0.1665
across_async_put_queue_thread_latency	0.0118	0.0114	0.0156	0.0106	0.0159
across_async_put_queue_actor_latency	0.2029	0.2059	0.2735	0.0569	0.2752
across_queue_client_latency	0.0390	0.0342	0.0735	0.0312	0.0763
queue_rpc_latency	0.3843	0.2950	0.6495	0.2646	0.6525
api_server_get_queue_latency	0.1698	0.1124	0.4156	0.1036	0.4304
across_request_streams_latency	0.0823	0.0650	0.1952	0.0576	0.2059

github-actions · 2025-08-07T03:17:36Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.0526	1.0526	1.0526	1.0526	1.0526
across_llumlet_latency	0.8333	0.8333	0.8333	0.8333	0.8333
across_engine_latency	0.1005	0.1005	0.1005	0.1005	0.1005
process_model_outputs_latency	0.4203	0.4037	0.5480	0.3953	0.5609
engine_step_latency	33.9681	33.8896	34.9481	33.6944	35.0443
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0140	0.0139	0.0157	0.0130	0.0157
across_async_put_queue_actor_latency	0.0365	0.0360	0.0400	0.0332	0.0401
across_queue_client_latency	0.0307	0.0301	0.0363	0.0269	0.0366
queue_rpc_latency	0.2188	0.2183	0.2528	0.1889	0.2551
api_server_get_queue_latency	0.1088	0.1067	0.1277	0.0988	0.1290
across_request_streams_latency	0.0410	0.0305	0.1178	0.0287	0.1255

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1116	1.1116	1.1116	1.1116	1.1116
across_llumlet_latency	0.8608	0.8608	0.8608	0.8608	0.8608
across_engine_latency	0.0902	0.0902	0.0902	0.0902	0.0902
process_model_outputs_latency	0.4249	0.4077	0.5709	0.3930	0.5852
engine_step_latency	34.0132	33.8372	35.2026	33.7160	35.3091
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0137	0.0133	0.0155	0.0129	0.0156
across_async_put_queue_actor_latency	0.0370	0.0363	0.0423	0.0337	0.0423
across_queue_client_latency	0.0295	0.0279	0.0357	0.0266	0.0358
queue_rpc_latency	0.1942	0.1923	0.2297	0.1714	0.2323
api_server_get_queue_latency	0.1016	0.1004	0.1201	0.0874	0.1214
across_request_streams_latency	0.0382	0.0285	0.1123	0.0252	0.1198

github-actions · 2025-08-07T03:18:24Z

test_simple_benchmark[engine_vLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	60.69	67.19	72.47	89.20	136.84	67.29
prefill	161.89	3488.23	19615.74	38634.64	41669.02	10688.36

test_simple_benchmark[engine_vLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	54.92	68.07	74.50	99.08	153.79	66.57
prefill	367.45	6138.48	18959.60	41179.04	53871.15	12289.37

github-actions · 2025-08-07T03:47:33Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.3583	1.3583	1.3583	1.3583	1.3583
across_llumlet_latency	1.1829	1.1829	1.1829	1.1829	1.1829
across_engine_latency	0.3066	0.3066	0.3066	0.3066	0.3066
process_model_outputs_latency	0.0883	0.0831	0.1128	0.0780	0.1139
engine_step_latency	34.1335	34.1142	34.3133	33.8605	34.3142
step_postprocess_latency	0.0229	0.0127	0.1061	0.0122	0.1153
across_async_put_queue_thread_latency	0.0112	0.0111	0.0125	0.0106	0.0125
across_async_put_queue_actor_latency	0.2058	0.2055	0.2163	0.1932	0.2166
across_queue_client_latency	0.0286	0.0286	0.0304	0.0271	0.0305
queue_rpc_latency	0.2828	0.2783	0.3196	0.2602	0.3208
api_server_get_queue_latency	0.1019	0.0994	0.1285	0.0939	0.1302
across_request_streams_latency	0.0744	0.0611	0.1602	0.0558	0.1671

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.3271	1.3271	1.3271	1.3271	1.3271
across_llumlet_latency	1.0582	1.0582	1.0582	1.0582	1.0582
across_engine_latency	0.3024	0.3024	0.3024	0.3024	0.3024
process_model_outputs_latency	0.0823	0.0771	0.1278	0.0674	0.1311
engine_step_latency	34.2301	34.2223	34.9649	33.7830	35.0269
step_postprocess_latency	0.0248	0.0107	0.1340	0.0098	0.1455
across_async_put_queue_thread_latency	0.0109	0.0106	0.0137	0.0096	0.0139
across_async_put_queue_actor_latency	0.1774	0.1850	0.2331	0.0463	0.2360
across_queue_client_latency	0.0348	0.0336	0.0461	0.0311	0.0470
queue_rpc_latency	0.2832	0.2778	0.3431	0.2543	0.3471
api_server_get_queue_latency	0.1056	0.1039	0.1139	0.0983	0.1139
across_request_streams_latency	0.0784	0.0688	0.1656	0.0586	0.1735

github-actions · 2025-08-07T04:00:55Z

test_simple_benchmark[engine_vLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	57.72	65.67	71.87	94.24	145.08	67.25
prefill	153.03	2978.26	24276.04	32635.05	47564.50	10638.91

test_simple_benchmark[engine_vLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	57.75	68.19	75.25	96.07	147.47	68.21
prefill	404.67	4850.72	18407.30	36087.41	49653.91	10554.67

github-actions · 2025-08-07T05:53:06Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1589	1.1589	1.1589	1.1589	1.1589
across_llumlet_latency	0.9830	0.9830	0.9830	0.9830	0.9830
across_engine_latency	0.1112	0.1112	0.1112	0.1112	0.1112
process_model_outputs_latency	0.4425	0.4263	0.5905	0.3938	0.6042
engine_step_latency	33.9870	33.7879	35.2090	33.7011	35.3184
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0149	0.0147	0.0165	0.0132	0.0165
across_async_put_queue_actor_latency	0.0383	0.0383	0.0446	0.0336	0.0451
across_queue_client_latency	0.0314	0.0322	0.0352	0.0255	0.0353
queue_rpc_latency	0.1977	0.1972	0.2268	0.1815	0.2290
api_server_get_queue_latency	0.1081	0.1058	0.1272	0.0965	0.1282
across_request_streams_latency	0.0402	0.0302	0.1181	0.0276	0.1260

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.0950	1.0950	1.0950	1.0950	1.0950
across_llumlet_latency	0.9526	0.9526	0.9526	0.9526	0.9526
across_engine_latency	0.1014	0.1014	0.1014	0.1014	0.1014
process_model_outputs_latency	0.4068	0.3946	0.5287	0.3809	0.5412
engine_step_latency	34.0609	33.8776	35.2855	33.7795	35.3902
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0132	0.0131	0.0151	0.0122	0.0152
across_async_put_queue_actor_latency	0.0370	0.0358	0.0420	0.0335	0.0421
across_queue_client_latency	0.0316	0.0306	0.0363	0.0277	0.0364
queue_rpc_latency	0.1978	0.1981	0.2248	0.1704	0.2250
api_server_get_queue_latency	0.1052	0.1031	0.1233	0.0907	0.1241
across_request_streams_latency	0.0394	0.0296	0.1117	0.0262	0.1189

github-actions · 2025-08-08T08:19:25Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1744	1.1744	1.1744	1.1744	1.1744
across_llumlet_latency	0.8712	0.8712	0.8712	0.8712	0.8712
across_engine_latency	0.1121	0.1121	0.1121	0.1121	0.1121
process_model_outputs_latency	0.4094	0.3908	0.5424	0.3749	0.5554
engine_step_latency	33.9201	33.7408	35.1045	33.6656	35.2110
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0135	0.0131	0.0158	0.0125	0.0159
across_async_put_queue_actor_latency	0.0378	0.0380	0.0421	0.0346	0.0423
across_queue_client_latency	0.0333	0.0332	0.0386	0.0279	0.0388
queue_rpc_latency	0.2036	0.2080	0.2356	0.1785	0.2376
api_server_get_queue_latency	0.1079	0.1077	0.1233	0.0972	0.1240
across_request_streams_latency	0.0412	0.0322	0.1189	0.0266	0.1270

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.0995	1.0995	1.0995	1.0995	1.0995
across_llumlet_latency	1.0467	1.0467	1.0467	1.0467	1.0467
across_engine_latency	0.0987	0.0987	0.0987	0.0987	0.0987
process_model_outputs_latency	0.4446	0.4128	0.7261	0.3895	0.7557
engine_step_latency	33.9877	33.8068	35.2399	33.7158	35.3476
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0139	0.0132	0.0176	0.0129	0.0178
across_async_put_queue_actor_latency	0.0364	0.0348	0.0430	0.0330	0.0433
across_queue_client_latency	0.0300	0.0291	0.0370	0.0263	0.0372
queue_rpc_latency	0.2040	0.1992	0.2539	0.1837	0.2575
api_server_get_queue_latency	0.1079	0.1054	0.1344	0.0962	0.1358
across_request_streams_latency	0.0428	0.0311	0.1308	0.0276	0.1395

github-actions · 2025-08-08T08:29:05Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	2.0624	2.0624	2.0624	2.0624	2.0624
across_llumlet_latency	1.0538	1.0538	1.0538	1.0538	1.0538
across_engine_latency	0.3075	0.3075	0.3075	0.3075	0.3075
process_model_outputs_latency	0.0888	0.0833	0.1100	0.0766	0.1108
engine_step_latency	34.1025	34.1169	34.3009	33.8237	34.3066
step_postprocess_latency	0.0203	0.0107	0.0979	0.0097	0.1064
across_async_put_queue_thread_latency	0.0106	0.0105	0.0120	0.0097	0.0121
across_async_put_queue_actor_latency	0.1974	0.1959	0.2191	0.1776	0.2202
across_queue_client_latency	0.0345	0.0333	0.0403	0.0318	0.0405
queue_rpc_latency	0.2685	0.2675	0.2931	0.2551	0.2941
api_server_get_queue_latency	0.1007	0.0970	0.1253	0.0925	0.1269
across_request_streams_latency	0.0734	0.0622	0.1534	0.0565	0.1602

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.3536	1.3536	1.3536	1.3536	1.3536
across_llumlet_latency	1.1462	1.1462	1.1462	1.1462	1.1462
across_engine_latency	0.3139	0.3139	0.3139	0.3139	0.3139
process_model_outputs_latency	0.0902	0.0864	0.1253	0.0785	0.1281
engine_step_latency	34.2679	34.2540	34.5051	33.9756	34.5104
step_postprocess_latency	0.0294	0.0132	0.1471	0.0122	0.1590
across_async_put_queue_thread_latency	0.0120	0.0110	0.0176	0.0104	0.0179
across_async_put_queue_actor_latency	0.2055	0.2030	0.2277	0.1926	0.2287
across_queue_client_latency	0.0353	0.0352	0.0423	0.0316	0.0428
queue_rpc_latency	0.2778	0.2770	0.3008	0.2601	0.3016
api_server_get_queue_latency	0.1041	0.1056	0.1161	0.0946	0.1166
across_request_streams_latency	0.0786	0.0653	0.1619	0.0573	0.1680

github-actions · 2025-08-08T08:42:37Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.3783	1.3783	1.3783	1.3783	1.3783
across_llumlet_latency	0.9714	0.9714	0.9714	0.9714	0.9714
across_engine_latency	0.1158	0.1158	0.1158	0.1158	0.1158
process_model_outputs_latency	0.4169	0.4010	0.5569	0.3801	0.5703
engine_step_latency	34.1056	33.9343	35.3035	33.8019	35.4083
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0132	0.0129	0.0151	0.0124	0.0153
across_async_put_queue_actor_latency	0.0370	0.0371	0.0408	0.0336	0.0409
across_queue_client_latency	0.0327	0.0331	0.0357	0.0273	0.0357
queue_rpc_latency	0.2410	0.2361	0.3038	0.2064	0.3087
api_server_get_queue_latency	0.1285	0.1268	0.1678	0.1030	0.1708
across_request_streams_latency	0.0490	0.0346	0.1511	0.0294	0.1606

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.2387	1.2387	1.2387	1.2387	1.2387
across_llumlet_latency	0.8936	0.8936	0.8936	0.8936	0.8936
across_engine_latency	0.1002	0.1002	0.1002	0.1002	0.1002
process_model_outputs_latency	0.4366	0.4251	0.5679	0.4018	0.5789
engine_step_latency	34.0858	33.9246	35.3110	33.8387	35.4262
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0144	0.0140	0.0166	0.0133	0.0167
across_async_put_queue_actor_latency	0.0376	0.0372	0.0428	0.0334	0.0430
across_queue_client_latency	0.0316	0.0316	0.0372	0.0267	0.0373
queue_rpc_latency	0.2353	0.2249	0.2888	0.1947	0.2907
api_server_get_queue_latency	0.1166	0.1119	0.1598	0.0976	0.1614
across_request_streams_latency	0.0453	0.0316	0.1416	0.0263	0.1504

github-actions · 2025-08-08T08:45:27Z

test_simple_benchmark[engine_vLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	58.69	67.04	74.42	96.57	126.97	68.14
prefill	206.69	1176.44	18704.76	43240.25	53391.62	10811.32

test_simple_benchmark[engine_vLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	54.90	67.68	74.82	94.87	150.31	65.71
prefill	363.48	5970.31	19192.73	41885.58	48722.97	11831.41

github-actions · 2025-08-08T08:51:18Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.5415	1.5415	1.5415	1.5415	1.5415
across_llumlet_latency	1.0574	1.0574	1.0574	1.0574	1.0574
across_engine_latency	0.3119	0.3119	0.3119	0.3119	0.3119
process_model_outputs_latency	0.0829	0.0790	0.1012	0.0743	0.1014
engine_step_latency	34.2047	34.1754	34.5670	33.8821	34.5752
step_postprocess_latency	0.0206	0.0113	0.0956	0.0103	0.1038
across_async_put_queue_thread_latency	0.0105	0.0104	0.0115	0.0098	0.0115
across_async_put_queue_actor_latency	0.1886	0.1902	0.1989	0.1723	0.1990
across_queue_client_latency	0.0298	0.0301	0.0344	0.0254	0.0346
queue_rpc_latency	0.2717	0.2672	0.3109	0.2515	0.3134
api_server_get_queue_latency	0.1042	0.1025	0.1263	0.0965	0.1282
across_request_streams_latency	0.0781	0.0632	0.1718	0.0598	0.1801

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.5319	1.5319	1.5319	1.5319	1.5319
across_llumlet_latency	1.5017	1.5017	1.5017	1.5017	1.5017
across_engine_latency	0.3161	0.3161	0.3161	0.3161	0.3161
process_model_outputs_latency	0.0824	0.0788	0.0971	0.0743	0.0971
engine_step_latency	34.2304	34.1654	34.7607	34.0411	34.7991
step_postprocess_latency	0.0229	0.0136	0.1015	0.0117	0.1102
across_async_put_queue_thread_latency	0.0107	0.0106	0.0117	0.0101	0.0118
across_async_put_queue_actor_latency	0.1990	0.1990	0.2146	0.1842	0.2148
across_queue_client_latency	0.0341	0.0340	0.0367	0.0303	0.0367
queue_rpc_latency	0.2750	0.2749	0.3151	0.2422	0.3177
api_server_get_queue_latency	0.1047	0.1066	0.1139	0.0925	0.1142
across_request_streams_latency	0.0777	0.0637	0.1754	0.0566	0.1834

github-actions · 2025-08-08T10:11:39Z

test_simple_benchmark[engine_vLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	58.19	65.73	72.38	92.53	149.48	66.70
prefill	178.80	6146.71	17813.56	42140.76	44102.80	11007.31

test_simple_benchmark[engine_vLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	59.59	68.22	74.13	97.95	138.31	67.68
prefill	365.59	6223.13	20844.73	36004.67	37644.93	11839.80

github-actions · 2025-08-08T10:16:46Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.2688	1.2688	1.2688	1.2688	1.2688
across_llumlet_latency	1.0145	1.0145	1.0145	1.0145	1.0145
across_engine_latency	0.3173	0.3173	0.3173	0.3173	0.3173
process_model_outputs_latency	0.0828	0.0797	0.1006	0.0753	0.1008
engine_step_latency	34.0656	34.0728	34.2733	33.8910	34.2876
step_postprocess_latency	0.0205	0.0110	0.0976	0.0106	0.1061
across_async_put_queue_thread_latency	0.0109	0.0107	0.0118	0.0104	0.0118
across_async_put_queue_actor_latency	0.1935	0.1939	0.2089	0.1731	0.2091
across_queue_client_latency	0.0304	0.0307	0.0342	0.0274	0.0343
queue_rpc_latency	0.2527	0.2512	0.2741	0.2378	0.2749
api_server_get_queue_latency	0.1049	0.1031	0.1187	0.0918	0.1191
across_request_streams_latency	0.0767	0.0641	0.1536	0.0610	0.1594

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.4412	1.4412	1.4412	1.4412	1.4412
across_llumlet_latency	1.6316	1.6316	1.6316	1.6316	1.6316
across_engine_latency	0.3402	0.3402	0.3402	0.3402	0.3402
process_model_outputs_latency	0.1177	0.0997	0.2167	0.0775	0.2225
engine_step_latency	34.1794	34.1914	34.3283	34.0198	34.3284
step_postprocess_latency	0.0278	0.0147	0.1288	0.0119	0.1392
across_async_put_queue_thread_latency	0.0117	0.0115	0.0133	0.0104	0.0134
across_async_put_queue_actor_latency	0.1939	0.2049	0.2344	0.0516	0.2348
across_queue_client_latency	0.0326	0.0321	0.0391	0.0285	0.0393
queue_rpc_latency	0.3409	0.2933	0.5530	0.2573	0.5598
api_server_get_queue_latency	0.1585	0.1059	0.3980	0.1002	0.4043
across_request_streams_latency	0.0915	0.0655	0.2134	0.0602	0.2178

github-actions · 2025-08-08T11:39:38Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1710	1.1710	1.1710	1.1710	1.1710
across_llumlet_latency	0.9421	0.9421	0.9421	0.9421	0.9421
across_engine_latency	0.1155	0.1155	0.1155	0.1155	0.1155
process_model_outputs_latency	0.4343	0.4210	0.5763	0.3811	0.5900
engine_step_latency	34.0865	33.8848	35.3658	33.7908	35.4775
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0141	0.0138	0.0159	0.0126	0.0160
across_async_put_queue_actor_latency	0.0380	0.0387	0.0435	0.0335	0.0436
across_queue_client_latency	0.0340	0.0356	0.0389	0.0284	0.0390
queue_rpc_latency	0.2080	0.2101	0.2501	0.1847	0.2525
api_server_get_queue_latency	0.1088	0.1055	0.1286	0.0969	0.1293
across_request_streams_latency	0.0410	0.0322	0.1119	0.0284	0.1192

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.2819	1.2819	1.2819	1.2819	1.2819
across_llumlet_latency	0.9305	0.9305	0.9305	0.9305	0.9305
across_engine_latency	0.0994	0.0994	0.0994	0.0994	0.0994
process_model_outputs_latency	0.4088	0.3938	0.5131	0.3854	0.5222
engine_step_latency	34.0324	33.8666	35.1702	33.7945	35.2796
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0135	0.0134	0.0148	0.0127	0.0149
across_async_put_queue_actor_latency	0.0354	0.0351	0.0410	0.0312	0.0412
across_queue_client_latency	0.0297	0.0282	0.0376	0.0255	0.0378
queue_rpc_latency	0.1831	0.1759	0.2296	0.1663	0.2319
api_server_get_queue_latency	0.1004	0.0956	0.1239	0.0890	0.1250
across_request_streams_latency	0.0370	0.0284	0.1025	0.0252	0.1089

github-actions · 2025-08-11T06:51:54Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1913	1.1913	1.1913	1.1913	1.1913
across_llumlet_latency	0.8936	0.8936	0.8936	0.8936	0.8936
across_engine_latency	0.1061	0.1061	0.1061	0.1061	0.1061
process_model_outputs_latency	0.4076	0.3923	0.5294	0.3829	0.5412
engine_step_latency	34.0480	33.9000	35.1165	33.8353	35.2210
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0129	0.0127	0.0146	0.0123	0.0147
across_async_put_queue_actor_latency	0.0361	0.0354	0.0418	0.0330	0.0421
across_queue_client_latency	0.0292	0.0283	0.0348	0.0273	0.0352
queue_rpc_latency	0.2751	0.2727	0.2920	0.2656	0.2929
api_server_get_queue_latency	0.1915	0.1916	0.1970	0.1869	0.1971
across_request_streams_latency	0.0642	0.0516	0.1665	0.0501	0.1776

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.0673	1.0673	1.0673	1.0673	1.0673
across_llumlet_latency	0.8348	0.8348	0.8348	0.8348	0.8348
across_engine_latency	0.0978	0.0978	0.0978	0.0978	0.0978
process_model_outputs_latency	0.4177	0.4035	0.5379	0.3917	0.5502
engine_step_latency	34.0001	33.8200	35.2042	33.7228	35.3139
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0143	0.0142	0.0157	0.0134	0.0158
across_async_put_queue_actor_latency	0.0394	0.0389	0.0499	0.0358	0.0508
across_queue_client_latency	0.0333	0.0334	0.0369	0.0298	0.0370
queue_rpc_latency	0.2021	0.2008	0.2327	0.1813	0.2343
api_server_get_queue_latency	0.1078	0.1063	0.1237	0.0972	0.1244
across_request_streams_latency	0.0385	0.0286	0.1128	0.0260	0.1205

github-actions · 2025-08-11T13:01:54Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.4605	1.4605	1.4605	1.4605	1.4605
across_llumlet_latency	1.0811	1.0811	1.0811	1.0811	1.0811
across_engine_latency	0.3092	0.3092	0.3092	0.3092	0.3092
process_model_outputs_latency	0.0842	0.0800	0.1113	0.0740	0.1128
engine_step_latency	34.2243	34.0660	34.7560	33.8653	34.7637
step_postprocess_latency	0.0220	0.0119	0.1034	0.0114	0.1124
across_async_put_queue_thread_latency	0.0099	0.0098	0.0108	0.0092	0.0108
across_async_put_queue_actor_latency	0.1893	0.1875	0.1994	0.1839	0.1998
across_queue_client_latency	0.0322	0.0313	0.0352	0.0290	0.0353
queue_rpc_latency	0.2694	0.2652	0.2988	0.2543	0.2998
api_server_get_queue_latency	0.1069	0.1013	0.1269	0.0946	0.1270
across_request_streams_latency	0.0765	0.0657	0.1538	0.0575	0.1600

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	2.3839	2.3839	2.3839	2.3839	2.3839
across_llumlet_latency	1.0364	1.0364	1.0364	1.0364	1.0364
across_engine_latency	0.3090	0.3090	0.3090	0.3090	0.3090
process_model_outputs_latency	0.0849	0.0757	0.1372	0.0717	0.1403
engine_step_latency	34.2464	34.1975	34.6281	33.9902	34.6413
step_postprocess_latency	0.0229	0.0113	0.1156	0.0106	0.1257
across_async_put_queue_thread_latency	0.0111	0.0106	0.0155	0.0099	0.0159
across_async_put_queue_actor_latency	0.1936	0.1884	0.2204	0.1804	0.2216
across_queue_client_latency	0.0348	0.0326	0.0432	0.0304	0.0433
queue_rpc_latency	0.2840	0.2815	0.3184	0.2630	0.3191
api_server_get_queue_latency	0.1049	0.1037	0.1210	0.0965	0.1218
across_request_streams_latency	0.0781	0.0643	0.1535	0.0587	0.1587

github-actions · 2025-08-11T13:07:38Z

test_simple_benchmark[engine_vLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	63.00	68.62	73.74	103.33	189.12	73.78
prefill	170.90	5453.17	19902.99	39493.63	54592.82	10976.36

test_simple_benchmark[engine_vLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	61.30	69.52	73.43	103.33	182.61	71.27
prefill	256.55	3217.54	21907.38	50873.11	59332.71	12618.43

github-actions · 2025-08-11T13:20:35Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.3188	1.3188	1.3188	1.3188	1.3188
across_llumlet_latency	1.0445	1.0445	1.0445	1.0445	1.0445
across_engine_latency	0.3111	0.3111	0.3111	0.3111	0.3111
process_model_outputs_latency	0.0889	0.0811	0.1456	0.0729	0.1492
engine_step_latency	34.2448	34.1604	35.1421	33.9669	35.2271
step_postprocess_latency	0.0327	0.0116	0.2048	0.0108	0.2238
across_async_put_queue_thread_latency	0.0104	0.0101	0.0125	0.0094	0.0126
across_async_put_queue_actor_latency	0.1957	0.1905	0.2199	0.1821	0.2205
across_queue_client_latency	0.0358	0.0332	0.0551	0.0313	0.0568
queue_rpc_latency	0.2613	0.2592	0.2973	0.2379	0.2976
api_server_get_queue_latency	0.0965	0.0950	0.1098	0.0886	0.1098
across_request_streams_latency	0.0725	0.0606	0.1491	0.0540	0.1544

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.4140	1.4140	1.4140	1.4140	1.4140
across_llumlet_latency	1.1103	1.1103	1.1103	1.1103	1.1103
across_engine_latency	0.2957	0.2957	0.2957	0.2957	0.2957
process_model_outputs_latency	0.0835	0.0778	0.1278	0.0680	0.1311
engine_step_latency	34.1554	34.0655	34.9481	33.8928	35.0132
step_postprocess_latency	0.0249	0.0108	0.1375	0.0104	0.1499
across_async_put_queue_thread_latency	0.0103	0.0099	0.0140	0.0089	0.0143
across_async_put_queue_actor_latency	0.1872	0.1840	0.2024	0.1789	0.2027
across_queue_client_latency	0.0324	0.0318	0.0394	0.0279	0.0399
queue_rpc_latency	0.2782	0.2725	0.3108	0.2534	0.3114
api_server_get_queue_latency	0.1083	0.1063	0.1226	0.0996	0.1231
across_request_streams_latency	0.0774	0.0656	0.1515	0.0609	0.1577

github-actions · 2025-08-11T13:26:38Z

test_simple_benchmark[engine_BladeLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	56.39	63.64	71.07	84.51	98.14	63.97
prefill	171.08	235.45	457.97	1371.64	3193.41	437.04

test_simple_benchmark[engine_BladeLLM-True-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	35.90	36.60	37.46	90.45	130.97	41.52
prefill	46.39	50.14	97.52	576.69	1873.59	166.54

test_simple_benchmark[engine_BladeLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	56.12	65.00	74.08	87.51	105.19	67.15
prefill	191.33	267.14	540.08	1422.67	3150.83	497.44

test_simple_benchmark[engine_BladeLLM-False-zmq-True-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	35.56	36.08	36.66	37.75	38.34	36.15
prefill	139.08	154.78	206.48	684.73	1104.45	252.87

test_simple_benchmark[engine_BladeLLM-False-zmq-True-True-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	35.19	35.51	36.15	38.02	38.36	35.72
prefill	140.87	167.35	363.88	1256.62	3036.17	402.94

github-actions · 2025-08-11T13:40:10Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1113	1.1113	1.1113	1.1113	1.1113
across_llumlet_latency	0.8330	0.8330	0.8330	0.8330	0.8330
across_engine_latency	0.1055	0.1055	0.1055	0.1055	0.1055
process_model_outputs_latency	0.4241	0.4079	0.5610	0.3895	0.5749
engine_step_latency	33.9957	33.8014	35.2354	33.7339	35.3439
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0138	0.0136	0.0154	0.0128	0.0155
across_async_put_queue_actor_latency	0.0395	0.0366	0.0581	0.0349	0.0594
across_queue_client_latency	0.0308	0.0294	0.0370	0.0284	0.0373
queue_rpc_latency	0.1969	0.1897	0.2474	0.1807	0.2509
api_server_get_queue_latency	0.0991	0.0962	0.1274	0.0884	0.1294
across_request_streams_latency	0.0371	0.0273	0.1097	0.0243	0.1168

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.0711	1.0711	1.0711	1.0711	1.0711
across_llumlet_latency	0.8861	0.8861	0.8861	0.8861	0.8861
across_engine_latency	0.1018	0.1018	0.1018	0.1018	0.1018
process_model_outputs_latency	0.4466	0.4183	0.6041	0.4066	0.6154
engine_step_latency	34.0551	33.8541	35.2141	33.7864	35.3077
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0151	0.0150	0.0166	0.0140	0.0167
across_async_put_queue_actor_latency	0.0481	0.0456	0.0545	0.0453	0.0546
across_queue_client_latency	0.0396	0.0372	0.0499	0.0342	0.0501
queue_rpc_latency	0.2153	0.2029	0.2559	0.1821	0.2560
api_server_get_queue_latency	0.1081	0.1049	0.1274	0.0935	0.1275
across_request_streams_latency	0.0418	0.0299	0.1244	0.0265	0.1318

github-actions · 2025-08-11T13:54:40Z

test_simple_benchmark[engine_vLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	62.76	69.44	74.90	117.75	188.26	75.12
prefill	160.54	4031.24	23655.17	57305.10	83853.20	14884.26

test_simple_benchmark[engine_vLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	62.46	69.38	73.93	97.56	143.00	67.99
prefill	274.76	6031.64	21268.80	44187.24	55550.75	11958.72

github-actions · 2025-08-12T06:19:51Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1002	1.1002	1.1002	1.1002	1.1002
across_llumlet_latency	0.9791	0.9791	0.9791	0.9791	0.9791
across_engine_latency	0.1058	0.1058	0.1058	0.1058	0.1058
process_model_outputs_latency	0.4491	0.4273	0.5845	0.3909	0.5923
engine_step_latency	34.1733	33.9319	35.3245	33.8035	35.3882
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0140	0.0141	0.0155	0.0128	0.0156
across_async_put_queue_actor_latency	0.0373	0.0375	0.0398	0.0343	0.0399
across_queue_client_latency	0.0311	0.0316	0.0342	0.0266	0.0342
queue_rpc_latency	0.2059	0.2040	0.2338	0.1867	0.2349
api_server_get_queue_latency	0.1095	0.1055	0.1203	0.1048	0.1206
across_request_streams_latency	0.0461	0.0311	0.1137	0.0292	0.1177

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1585	1.1585	1.1585	1.1585	1.1585
across_llumlet_latency	0.8500	0.8500	0.8500	0.8500	0.8500
across_engine_latency	0.0939	0.0939	0.0939	0.0939	0.0939
process_model_outputs_latency	0.4308	0.4132	0.5651	0.4033	0.5790
engine_step_latency	34.0951	33.9133	35.2463	33.8390	35.3500
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0143	0.0141	0.0156	0.0138	0.0157
across_async_put_queue_actor_latency	0.0380	0.0388	0.0414	0.0343	0.0414
across_queue_client_latency	0.0319	0.0338	0.0366	0.0256	0.0367
queue_rpc_latency	0.2750	0.2752	0.2839	0.2635	0.2840
api_server_get_queue_latency	0.1920	0.1906	0.2030	0.1877	0.2038
across_request_streams_latency	0.0656	0.0520	0.1684	0.0506	0.1793

github-actions · 2025-08-12T06:34:15Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.4075	1.4075	1.4075	1.4075	1.4075
across_llumlet_latency	1.0299	1.0299	1.0299	1.0299	1.0299
across_engine_latency	0.3307	0.3307	0.3307	0.3307	0.3307
process_model_outputs_latency	0.1162	0.0884	0.3409	0.0780	0.3642
engine_step_latency	34.4310	34.2144	36.1673	33.9121	36.3209
step_postprocess_latency	0.0277	0.0125	0.1314	0.0115	0.1417
across_async_put_queue_thread_latency	0.0123	0.0113	0.0204	0.0107	0.0213
across_async_put_queue_actor_latency	0.2066	0.2015	0.2578	0.1927	0.2626
across_queue_client_latency	0.0275	0.0271	0.0301	0.0254	0.0302
queue_rpc_latency	0.3880	0.2936	0.6647	0.2419	0.6725
api_server_get_queue_latency	0.2105	0.1029	0.4115	0.0927	0.4136
across_request_streams_latency	0.0988	0.0713	0.2112	0.0544	0.2124

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.8880	1.8880	1.8880	1.8880	1.8880
across_llumlet_latency	1.1270	1.1270	1.1270	1.1270	1.1270
across_engine_latency	0.3371	0.3371	0.3371	0.3371	0.3371
process_model_outputs_latency	0.0857	0.0801	0.1132	0.0792	0.1144
engine_step_latency	34.2834	34.1727	34.8474	33.8397	34.8667
step_postprocess_latency	0.0219	0.0123	0.0981	0.0117	0.1064
across_async_put_queue_thread_latency	0.0107	0.0105	0.0126	0.0095	0.0126
across_async_put_queue_actor_latency	0.1795	0.1913	0.2078	0.0475	0.2081
across_queue_client_latency	0.0314	0.0315	0.0338	0.0293	0.0338
queue_rpc_latency	0.2665	0.2668	0.2897	0.2445	0.2908
api_server_get_queue_latency	0.1052	0.1009	0.1227	0.0950	0.1230
across_request_streams_latency	0.0786	0.0696	0.1671	0.0559	0.1752

github-actions · 2025-08-12T06:54:02Z

test_simple_benchmark[engine_vLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	61.71	70.42	77.71	112.68	187.54	76.67
prefill	174.70	1051.00	23081.50	53943.83	63364.15	12169.36

test_simple_benchmark[engine_vLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	57.32	68.39	75.14	101.72	173.43	68.64
prefill	289.92	5937.50	17563.34	41250.32	51479.89	11630.68

github-actions · 2025-08-12T07:43:50Z

test_simple_benchmark[engine_vLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	60.80	68.30	75.15	102.53	144.51	70.12
prefill	186.38	3423.29	15568.42	33782.62	45568.32	9609.15

test_simple_benchmark[engine_vLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	60.03	68.87	75.49	98.82	122.48	67.78
prefill	278.28	3241.55	13857.67	46511.05	51199.98	10578.35

github-actions · 2025-08-12T08:05:23Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.4857	1.4857	1.4857	1.4857	1.4857
across_llumlet_latency	1.1595	1.1595	1.1595	1.1595	1.1595
across_engine_latency	0.2957	0.2957	0.2957	0.2957	0.2957
process_model_outputs_latency	0.0837	0.0788	0.1032	0.0746	0.1035
engine_step_latency	34.1417	34.1645	34.2868	33.8736	34.2872
step_postprocess_latency	0.0224	0.0116	0.1080	0.0109	0.1172
across_async_put_queue_thread_latency	0.0115	0.0107	0.0186	0.0099	0.0193
across_async_put_queue_actor_latency	0.1839	0.1945	0.2265	0.0471	0.2286
across_queue_client_latency	0.0345	0.0346	0.0379	0.0322	0.0381
queue_rpc_latency	0.2839	0.2822	0.2985	0.2721	0.2992
api_server_get_queue_latency	0.1065	0.1049	0.1172	0.0996	0.1178
across_request_streams_latency	0.0789	0.0670	0.1675	0.0590	0.1759

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.8619	1.8619	1.8619	1.8619	1.8619
across_llumlet_latency	1.0292	1.0292	1.0292	1.0292	1.0292
across_engine_latency	0.3066	0.3066	0.3066	0.3066	0.3066
process_model_outputs_latency	0.0832	0.0785	0.1058	0.0727	0.1067
engine_step_latency	34.2022	34.1484	34.6299	33.9498	34.6485
step_postprocess_latency	0.0216	0.0117	0.1004	0.0113	0.1090
across_async_put_queue_thread_latency	0.0107	0.0106	0.0115	0.0102	0.0115
across_async_put_queue_actor_latency	0.1943	0.1923	0.2068	0.1851	0.2072
across_queue_client_latency	0.0322	0.0326	0.0360	0.0287	0.0361
queue_rpc_latency	0.2698	0.2645	0.3023	0.2563	0.3041
api_server_get_queue_latency	0.1073	0.1065	0.1199	0.0970	0.1202
across_request_streams_latency	0.0752	0.0617	0.1587	0.0587	0.1657

github-actions · 2025-08-12T08:13:16Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.4780	1.4780	1.4780	1.4780	1.4780
across_llumlet_latency	1.0017	1.0017	1.0017	1.0017	1.0017
across_engine_latency	0.1188	0.1188	0.1188	0.1188	0.1188
process_model_outputs_latency	0.4385	0.4187	0.5862	0.3985	0.5993
engine_step_latency	34.0544	33.8228	35.3215	33.7912	35.4328
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0144	0.0139	0.0172	0.0133	0.0174
across_async_put_queue_actor_latency	0.0374	0.0360	0.0424	0.0333	0.0425
across_queue_client_latency	0.0318	0.0305	0.0366	0.0275	0.0367
queue_rpc_latency	0.1953	0.1930	0.2249	0.1805	0.2268
api_server_get_queue_latency	0.1060	0.1051	0.1249	0.0927	0.1261
across_request_streams_latency	0.0402	0.0305	0.1201	0.0258	0.1283

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1436	1.1436	1.1436	1.1436	1.1436
across_llumlet_latency	0.8236	0.8236	0.8236	0.8236	0.8236
across_engine_latency	0.0966	0.0966	0.0966	0.0966	0.0966
process_model_outputs_latency	0.4100	0.3929	0.5393	0.3850	0.5527
engine_step_latency	34.1115	33.9343	35.3864	33.8912	35.5082
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0140	0.0137	0.0162	0.0132	0.0163
across_async_put_queue_actor_latency	0.0340	0.0340	0.0364	0.0322	0.0366
across_queue_client_latency	0.0267	0.0265	0.0300	0.0239	0.0301
queue_rpc_latency	0.2749	0.2711	0.2959	0.2639	0.2962
api_server_get_queue_latency	0.1915	0.1907	0.1964	0.1880	0.1964
across_request_streams_latency	0.0652	0.0525	0.1640	0.0514	0.1744

github-actions · 2025-08-12T09:08:12Z

test_request_trace[rayqueue-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.5101	1.5101	1.5101	1.5101	1.5101
across_llumlet_latency	1.0692	1.0692	1.0692	1.0692	1.0692
across_engine_latency	0.2973	0.2973	0.2973	0.2973	0.2973
process_model_outputs_latency	0.0863	0.0824	0.1092	0.0753	0.1102
engine_step_latency	34.1383	34.0815	34.4467	33.8589	34.4500
step_postprocess_latency	0.0218	0.0124	0.0996	0.0117	0.1082
across_async_put_queue_thread_latency	0.0107	0.0107	0.0113	0.0101	0.0113
across_async_put_queue_actor_latency	0.1946	0.1937	0.2025	0.1825	0.2026
across_queue_client_latency	0.0331	0.0331	0.0358	0.0312	0.0359
queue_rpc_latency	0.4168	0.3195	0.7150	0.2708	0.7213
api_server_get_queue_latency	0.1821	0.1254	0.3916	0.1099	0.3948
across_request_streams_latency	0.0856	0.0757	0.1629	0.0695	0.1704

test_request_trace[zmq-engine_vLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.6128	1.6128	1.6128	1.6128	1.6128
across_llumlet_latency	1.6992	1.6992	1.6992	1.6992	1.6992
across_engine_latency	0.3542	0.3542	0.3542	0.3542	0.3542
process_model_outputs_latency	0.0920	0.0872	0.1157	0.0797	0.1160
engine_step_latency	34.1912	34.1168	34.9151	33.8662	34.9717
step_postprocess_latency	0.0234	0.0136	0.1033	0.0123	0.1119
across_async_put_queue_thread_latency	0.0109	0.0107	0.0122	0.0101	0.0122
across_async_put_queue_actor_latency	0.2560	0.1954	0.8737	0.0457	0.9391
across_queue_client_latency	0.0292	0.0292	0.0332	0.0263	0.0334
queue_rpc_latency	0.2679	0.2690	0.2941	0.2382	0.2947
api_server_get_queue_latency	0.1049	0.1040	0.1155	0.0980	0.1162
across_request_streams_latency	0.0801	0.0665	0.1768	0.0596	0.1854

github-actions · 2025-08-12T09:17:19Z

test_simple_benchmark[engine_vLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	60.08	67.50	71.99	98.50	139.12	67.81
prefill	174.83	3770.32	21054.85	41086.27	45982.58	11097.76

test_simple_benchmark[engine_vLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	55.18	66.39	73.62	98.96	173.51	66.62
prefill	305.78	6067.97	19914.96	40052.78	43001.51	11766.64

github-actions · 2025-08-12T09:43:11Z

test_request_trace[rayqueue-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.1351	1.1351	1.1351	1.1351	1.1351
across_llumlet_latency	0.9288	0.9288	0.9288	0.9288	0.9288
across_engine_latency	0.1082	0.1082	0.1082	0.1082	0.1082
process_model_outputs_latency	0.4551	0.4443	0.5651	0.4162	0.5731
engine_step_latency	34.0249	33.8582	35.1883	33.7586	35.2965
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0149	0.0149	0.0168	0.0138	0.0169
across_async_put_queue_actor_latency	0.0387	0.0380	0.0470	0.0339	0.0476
across_queue_client_latency	0.0303	0.0294	0.0378	0.0264	0.0383
queue_rpc_latency	0.2086	0.2127	0.2273	0.1820	0.2282
api_server_get_queue_latency	0.1046	0.1041	0.1194	0.0952	0.1202
across_request_streams_latency	0.0396	0.0298	0.1173	0.0273	0.1253

test_request_trace[zmq-engine_BladeLLM-/mnt/model/Qwen2.5-7B]

latency(ms)	mean	p50	p99	min	max
across_manager_latency	1.0107	1.0107	1.0107	1.0107	1.0107
across_llumlet_latency	0.9336	0.9336	0.9336	0.9336	0.9336
across_engine_latency	0.0954	0.0954	0.0954	0.0954	0.0954
process_model_outputs_latency	0.4024	0.3885	0.5170	0.3792	0.5283
engine_step_latency	34.0577	33.9105	35.2475	33.8112	35.3615
step_postprocess_latency	0.0000	0.0000	0.0000	0.0000	0.0000
across_async_put_queue_thread_latency	0.0131	0.0131	0.0142	0.0119	0.0143
across_async_put_queue_actor_latency	0.0360	0.0342	0.0495	0.0324	0.0508
across_queue_client_latency	0.0280	0.0278	0.0340	0.0237	0.0343
queue_rpc_latency	0.1792	0.1752	0.2131	0.1590	0.2149
api_server_get_queue_latency	0.1031	0.0987	0.1233	0.0897	0.1240
across_request_streams_latency	0.0373	0.0273	0.1088	0.0246	0.1159

github-actions · 2025-08-12T10:08:59Z

test_simple_benchmark[engine_BladeLLM-False-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	56.69	64.49	72.85	96.19	125.21	67.43
prefill	180.41	261.46	514.55	1304.45	3185.11	455.90

test_simple_benchmark[engine_BladeLLM-True-zmq-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	35.88	36.76	37.32	92.03	130.72	41.36
prefill	46.61	56.23	99.96	629.90	1564.27	170.64

test_simple_benchmark[engine_BladeLLM-False-rayqueue-False-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	57.44	64.14	71.43	84.90	144.86	65.29
prefill	181.03	266.54	526.42	1414.82	4066.85	460.11

test_simple_benchmark[engine_BladeLLM-False-zmq-True-False-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	35.57	35.89	36.75	37.67	37.88	36.13
prefill	135.52	150.36	194.11	670.80	1097.55	233.12

test_simple_benchmark[engine_BladeLLM-False-zmq-True-True-/mnt/model/Qwen2.5-7B]

latency(ms)	p25	p50	p75	p95	p99	mean
decode	35.00	35.54	36.10	37.59	37.73	35.69
prefill	138.40	168.44	323.01	1201.46	3003.70	396.48

KuilongCui force-pushed the adapt_to_v1_migration branch 6 times, most recently from b367f20 to 0c3f994 Compare August 5, 2025 11:54

adapt to v1 migration interface

8a04f93

KuilongCui force-pushed the adapt_to_v1_migration branch from 93301a3 to 8a04f93 Compare August 8, 2025 08:16

fix lint

8b84d8f

fix setuptoolsscm install

b5a1cbd

wip

06e5c94

KuilongCui added 2 commits August 11, 2025 05:04

fix

f59f686

fix test

24300f5

KuilongCui changed the title ~~[WIP][Migration] Adapt to vLLM-V1 engine migration interface~~ [Migration] Adapt to vLLM-V1 engine migration interface Aug 11, 2025

fix

b156a2f

s5u13b reviewed Aug 11, 2025

View reviewed changes

Comment thread llumnix/llumlet/llumlet.py Outdated

s5u13b reviewed Aug 11, 2025

View reviewed changes

Comment thread llumnix/llumlet/llumlet.py Outdated

fix

f84ef20

fix

8dd2523

s5u13b reviewed Aug 12, 2025

View reviewed changes

Comment thread llumnix/backends/vllm_v1/core.py Outdated

s5u13b reviewed Aug 12, 2025

View reviewed changes

Comment thread llumnix/backends/vllm_v1/scheduler.py

s5u13b reviewed Aug 12, 2025

View reviewed changes

fix comment

ffac1ad

fix

3f3da05

fix

8625fc1

s5u13b approved these changes Aug 12, 2025

View reviewed changes

ZeldaHuang approved these changes Aug 12, 2025

View reviewed changes

KuilongCui merged commit 9b407d4 into main Aug 12, 2025
34 of 36 checks passed

KuilongCui deleted the adapt_to_v1_migration branch August 12, 2025 10:39

Uh oh!

Conversation

KuilongCui commented Jul 31, 2025

Uh oh!

github-actions Bot commented Aug 7, 2025

Uh oh!

github-actions Bot commented Aug 7, 2025

Uh oh!

github-actions Bot commented Aug 7, 2025

Uh oh!

github-actions Bot commented Aug 7, 2025

Uh oh!

github-actions Bot commented Aug 7, 2025

Uh oh!

github-actions Bot commented Aug 7, 2025

Uh oh!

github-actions Bot commented Aug 8, 2025

Uh oh!

github-actions Bot commented Aug 8, 2025

Uh oh!

github-actions Bot commented Aug 8, 2025

Uh oh!

github-actions Bot commented Aug 8, 2025

Uh oh!

github-actions Bot commented Aug 8, 2025

Uh oh!

github-actions Bot commented Aug 8, 2025

Uh oh!

github-actions Bot commented Aug 8, 2025

Uh oh!

github-actions Bot commented Aug 8, 2025

Uh oh!

github-actions Bot commented Aug 11, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2025

Uh oh!

github-actions Bot commented Aug 11, 2025

Uh oh!

github-actions Bot commented Aug 11, 2025

Uh oh!

github-actions Bot commented Aug 11, 2025

Uh oh!

github-actions Bot commented Aug 11, 2025

Uh oh!

github-actions Bot commented Aug 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!