Skip to content

After running a session for 20 mins, I cannot start another session. #5

@ericxtang

Description

@ericxtang

Steps:

  1. Ran a session for 20 mins. The screen turned black-and-white.
  2. Tried to run another session, got this bug:
2026-01-17 14:19:03,689 - scope.server.webrtc - INFO - Removing session: Session(35ff0743-5f75-4cb1-805d-9f2f5e4e06e8, state=closed)
2026-01-17 14:19:03,689 - scope.server.pipeline_processor - ERROR - Error processing chunk for waypoint: CUDA error: device-side assert triggered
Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Traceback (most recent call last):
  File "C:\Users\user\eric-workspace\scope\src\scope\server\pipeline_processor.py", line 450, in process_chunk
    output = self.pipeline(**call_params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\eric-workspace\scope\.venv\Lib\site-packages\scope_overworld\pipeline.py", line 111, in __call__
    frame = self.engine.gen_frame(ctrl=ctrl)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\eric-workspace\scope\.venv\Lib\site-packages\torch\utils\_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\eric-workspace\scope\.venv\Lib\site-packages\world_engine\world_engine.py", line 108, in gen_frame
    x = torch.randn(self.frm_shape, device=self.device, dtype=self.dtype)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: device-side assert triggered
Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions