Details
[2026-04-28 14:18:16.766058][UC][I] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization). [314,314][shm_broadcast.py:681,acquire_read]
[2026-04-28 14:18:17.233516][UC][I] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization). [368,368][shm_broadcast.py:681,acquire_read]
[2026-04-28 14:19:15.028990][UC][E] Dumping input data for V1 LLM engine (v0.18.0) with config: model='/data/0318-GLM-5-w4a8', speculative_config=None, tokenizer='/data/0318-GLM-5-w4a8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=2, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=ascend, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=bfloat16, device_config=npu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=1024, served_model_name=glm-5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'vllm_ascend.compilation.compiler_interface.AscendCompiler', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, [314,314][dump_input.py:72,_dump_engine_exception]
[2026-04-28 14:19:15.029363][UC][E] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-a76349d7-15d7-4c1d-9af4-7c5d6771923a-a5494758', 'chatcmpl-b164750c-dfc9-49ea-9dc5-c6d55ff2f8c3-be62f7c6'],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[None, None],num_computed_tokens=[7880, 7845],num_output_tokens=[4, 2]), num_scheduled_tokens={chatcmpl-a76349d7-15d7-4c1d-9af4-7c5d6771923a-a5494758: 1, chatcmpl-b164750c-dfc9-49ea-9dc5-c6d55ff2f8c3-be62f7c6: 1}, total_num_scheduled_tokens=2, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=UCMConnectorMetadata(request_meta={chatcmpl-b164750c-dfc9-49ea-9dc5-c6d55ff2f8c3-be62f7c6: RequestDispatchMeta(load_block_ids=[[], []], dump_block_ids=[[], []]), chatcmpl-a76349d7-15d7-4c1d-9af4-7c5d6771923a-a5494758: RequestDispatchMeta(load_block_ids=[[], []], dump_block_ids=[[], []])}), ec_connector_metadata=null, new_block_ids_to_zero=null) [314,314][dump_input.py:79,_dump_engine_exception]
[2026-04-28 14:19:15.033444][UC][E] Dumping scheduler stats: SchedulerStats(num_running_reqs=2, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.7848101265822784, encoder_cache_usage=0.0, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None) [314,314][dump_input.py:81,_dump_engine_exception]
[2026-04-28 14:19:15.036656][UC][E] EngineCore encountered a fatal error.
Traceback (most recent call last):
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 376, in get_response
status, result = mq.dequeue(timeout=dequeue_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/distributed/device_communicators/shm_broadcast.py", line 755, in dequeue
with self.acquire_read(timeout, indefinite) as buf:
File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/distributed/device_communicators/shm_broadcast.py", line 674, in acquire_read
self._spin_condition.wait(timeout_ms=read_timeout.timeout_ms())
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/distributed/device_communicators/shm_broadcast.py", line 631, in timeout_ms
raise TimeoutError
TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1092, in run_engine_core
engine_core.run_busy_loop()
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1696, in run_busy_loop
executed = self._process_engine_step()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1172, in _process_engine_step
outputs, model_executed = self.step_fn()
^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 497, in step_with_batch_queue
model_output = future.result()
^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 84, in result
return super().result()
^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 88, in wait_for_response
response = self.aggregate(get_response())
^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 378, in get_response
raise TimeoutError(f"RPC call to {method} timed out.") from e
TimeoutError: RPC call to sample_tokens timed out. [314,314][core.py:1101,run_engine_core]
[2026-04-28 14:19:15.037865][UC][I] Parent process exited, terminating worker queues [433,4246][multiproc_executor.py:759,death_pipe_monitor]
[2026-04-28 14:19:15.037944][UC][I] Parent process exited, terminating worker queues [517,4026][multiproc_executor.py:759,death_pipe_monitor]
[2026-04-28 14:19:15.037949][UC][I] Parent process exited, terminating worker queues [639,4033][multiproc_executor.py:759,death_pipe_monitor]
[2026-04-28 14:19:15.037965][UC][I] Parent process exited, terminating worker queues [745,3684][multiproc_executor.py:759,death_pipe_monitor]
[2026-04-28 14:19:15.038885][UC][E] AsyncLLM output_handler failed.
Traceback (most recent call last):
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][async_llm.py:707,output_handler]
[2026-04-28 14:19:15.038892][UC][E] AsyncLLM output_handler failed.
Traceback (most recent call last):
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][async_llm.py:707,output_handler]
[2026-04-28 14:19:15.040164][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.040222][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.041891][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.041973][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.043180][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.043279][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.044609][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.044703][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.046262][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.046377][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.048100][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.048221][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.050138][UC][E] Error in chat completion stream generator.
</details>
### 🐛 Describe the bug
配置： --kv-transfer-config \
'{
"kv_connector": "UCMConnector",
"kv_connector_module_path": "ucm.integration.vllm.ucm_connector",
"kv_role": "kv_consumer",
"kv_connector_extra_config": {"UCM_CONFIG_FILE": "/mnt/cephfs/ucmkvCache/ucm_config/ucm_config.yaml"}
}'
yaml配置：ucm_connectors:
- ucm_connector_name: "UcmPipelineStore"
ucm_connector_config:
store_pipeline: "Cache|Posix"
storage_backends: "/mnt/cephfs/ucmkvCache"
io_direct: false
cache_buffer_capacity_gb: 128
# posix_capacity_gb: 1024
测试命令： "vllm", "bench", "serve",
"--backend", "openai",
"--base-url", "http://localhost:11223",
"--model", "glm-5",
"--tokenizer", "/data/0318-GLM-5-w4a8",
"--dataset-name", "random",
"--num-prompts", "200",
"--random-input-len", "4096",
"--random-output-len", "10",
"--max-concurrency", "25",
"--request-rate", "inf",
"--seed", "12345678",
"--percentile-metrics", "ttft",
"--metric-percentiles", "90,99",
Your current environment
Details
[2026-04-28 14:18:16.766058][UC][I] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization). [314,314][shm_broadcast.py:681,acquire_read] [2026-04-28 14:18:17.233516][UC][I] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization). [368,368][shm_broadcast.py:681,acquire_read] [2026-04-28 14:19:15.028990][UC][E] Dumping input data for V1 LLM engine (v0.18.0) with config: model='/data/0318-GLM-5-w4a8', speculative_config=None, tokenizer='/data/0318-GLM-5-w4a8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=2, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=ascend, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=bfloat16, device_config=npu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=1024, served_model_name=glm-5, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'vllm_ascend.compilation.compiler_interface.AscendCompiler', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, [314,314][dump_input.py:72,_dump_engine_exception] [2026-04-28 14:19:15.029363][UC][E] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-a76349d7-15d7-4c1d-9af4-7c5d6771923a-a5494758', 'chatcmpl-b164750c-dfc9-49ea-9dc5-c6d55ff2f8c3-be62f7c6'],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[None, None],num_computed_tokens=[7880, 7845],num_output_tokens=[4, 2]), num_scheduled_tokens={chatcmpl-a76349d7-15d7-4c1d-9af4-7c5d6771923a-a5494758: 1, chatcmpl-b164750c-dfc9-49ea-9dc5-c6d55ff2f8c3-be62f7c6: 1}, total_num_scheduled_tokens=2, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=UCMConnectorMetadata(request_meta={chatcmpl-b164750c-dfc9-49ea-9dc5-c6d55ff2f8c3-be62f7c6: RequestDispatchMeta(load_block_ids=[[], []], dump_block_ids=[[], []]), chatcmpl-a76349d7-15d7-4c1d-9af4-7c5d6771923a-a5494758: RequestDispatchMeta(load_block_ids=[[], []], dump_block_ids=[[], []])}), ec_connector_metadata=null, new_block_ids_to_zero=null) [314,314][dump_input.py:79,_dump_engine_exception] [2026-04-28 14:19:15.033444][UC][E] Dumping scheduler stats: SchedulerStats(num_running_reqs=2, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.7848101265822784, encoder_cache_usage=0.0, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None) [314,314][dump_input.py:81,_dump_engine_exception] [2026-04-28 14:19:15.036656][UC][E] EngineCore encountered a fatal error. Traceback (most recent call last): File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 376, in get_response status, result = mq.dequeue(timeout=dequeue_timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vllm-workspace/vllm/vllm/distributed/device_communicators/shm_broadcast.py", line 755, in dequeue with self.acquire_read(timeout, indefinite) as buf: File "/usr/local/python3.11.14/lib/python3.11/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/vllm-workspace/vllm/vllm/distributed/device_communicators/shm_broadcast.py", line 674, in acquire_read self._spin_condition.wait(timeout_ms=read_timeout.timeout_ms()) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/vllm-workspace/vllm/vllm/distributed/device_communicators/shm_broadcast.py", line 631, in timeout_ms raise TimeoutError TimeoutErrorThe above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1092, in run_engine_core
engine_core.run_busy_loop()
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1696, in run_busy_loop
executed = self._process_engine_step()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 1172, in _process_engine_step
outputs, model_executed = self.step_fn()
^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 497, in step_with_batch_queue
model_output = future.result()
^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 84, in result
return super().result()
^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.14/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 88, in wait_for_response
response = self.aggregate(get_response())
^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/executor/multiproc_executor.py", line 378, in get_response
raise TimeoutError(f"RPC call to {method} timed out.") from e
TimeoutError: RPC call to sample_tokens timed out. [314,314][core.py:1101,run_engine_core]
[2026-04-28 14:19:15.037865][UC][I] Parent process exited, terminating worker queues [433,4246][multiproc_executor.py:759,death_pipe_monitor]
[2026-04-28 14:19:15.037944][UC][I] Parent process exited, terminating worker queues [517,4026][multiproc_executor.py:759,death_pipe_monitor]
[2026-04-28 14:19:15.037949][UC][I] Parent process exited, terminating worker queues [639,4033][multiproc_executor.py:759,death_pipe_monitor]
[2026-04-28 14:19:15.037965][UC][I] Parent process exited, terminating worker queues [745,3684][multiproc_executor.py:759,death_pipe_monitor]
[2026-04-28 14:19:15.038885][UC][E] AsyncLLM output_handler failed.
Traceback (most recent call last):
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][async_llm.py:707,output_handler]
[2026-04-28 14:19:15.038892][UC][E] AsyncLLM output_handler failed.
Traceback (most recent call last):
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][async_llm.py:707,output_handler]
[2026-04-28 14:19:15.040164][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.040222][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.041891][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.041973][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.043180][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.043279][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.044609][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.044703][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.046262][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.046377][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.048100][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [399,399][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.048221][UC][E] Error in chat completion stream generator.
Traceback (most recent call last):
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm-ascend/vllm_ascend/patch/platform/patch_glm_tool_call_parser.py", line 260, in _patched_chat_completion_stream_generator
async for res in result_generator:
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 582, in generate
out = q.get_nowait() or await q.get()
^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/output_processor.py", line 85, in get
raise output
File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 663, in output_handler
outputs = await engine_core.get_output_async()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 1022, in get_output_async
raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. [398,398][patch_glm_tool_call_parser.py:790,_patched_chat_completion_stream_generator]
[2026-04-28 14:19:15.050138][UC][E] Error in chat completion stream generator.