Hyperion: concurrent code generation permanently deadlocks the shared taskExecutor (blocking .get() in HyperionWebsocketService.send()) #13556

Describe the bug

HyperionWebsocketService.send() blocks on the websocket send future with no timeout:

// src/main/java/de/tum/cit/aet/artemis/hyperion/service/websocket/HyperionWebsocketService.java:39
websocketMessagingService.sendMessageToUser(userLogin, topic, payload).get();

That future is completed by running the actual send on the Spring taskExecutor (WebsocketMessagingService.sendMessageToUser → CompletableFuture.runAsync(..., asyncExecutor) with @Qualifier("taskExecutor")). The Hyperion code generation jobs themselves also run on that same taskExecutor (via HyperionCodeGenerationJobService.startJob → taskService.runJobAsync(...), @Async).

The pool is configured in application.yml as:

spring:
    task:
        execution:
            thread-name-prefix: artemis-task-
            pool:
                core-size: 2
                max-size: 50
                queue-capacity: 10000

Because ThreadPoolTaskExecutor only grows beyond core-size once the queue is full, and the queue holds 10,000 entries, the pool effectively never has more than 2 threads.

As soon as two Hyperion code generation jobs occupy both pool threads, each job's next progress-event send() submits its websocket send task to the same pool and then parks in CompletableFuture.get(). The send tasks sit in the queue behind the two blocked threads, so neither future can ever complete → permanent deadlock:

• Both artemis-task-* threads park forever at HyperionWebsocketService.java:39 (thread dump below).
• The running jobs never emit another event (no DONE/FAILED), so the client UI waits forever.
• Every subsequently started job returns a job id over REST but is queued behind the deadlock and never even emits STARTED.
• All other @Async work scheduled on taskExecutor silently stops executing as well.

Only a server restart recovers the instance.

Expected: sending a websocket progress event must not be able to block a code generation worker, and the shared @Async pool must not be starvable by two concurrent Hyperion jobs.

Possible fixes (any one of these breaks the cycle):

1. Don't block at all — the call is fire-and-forget in spirit; log failures via .whenComplete(...) instead of .get().
2. If ordering matters, use .get(timeout, TimeUnit) so a stuck send can't wedge a worker forever.
3. Run websocket sends on a dedicated executor instead of the same taskExecutor that hosts the @Async jobs.

(Separately, core-size: 2 with queue-capacity: 10000 means the pool never actually grows toward max-size: 50; any blocking wait on work submitted to the same pool is a deadlock waiting to happen.)

To Reproduce

1. Enable Hyperion and open a programming exercise in the code editor as instructor.
2. Start code generation so that two jobs run concurrently (e.g. trigger generation for a second repository/exercise while the first job is still running — easy to hit in practice by reloading the page mid-generation and starting another generation, since the first job keeps running server-side).
3. Once both pool threads are occupied and each job emits its next progress event, both threads park in HyperionWebsocketService.send().
4. Observe: no further websocket events for any job (not even STARTED for newly created jobs), UI shows no progress; jcmd <pid> Thread.print (or kill -3) shows both artemis-task-* threads waiting at HyperionWebsocketService.java:39. The instance stays in this state until restart.

Expected behavior

Code generation jobs emit their progress events without being able to deadlock; a slow or impossible websocket delivery (e.g. the user reloaded the page and the session is gone) must not block or wedge the worker pool. Concurrent code generation jobs should run and complete independently.

Screenshots

(none — server-side; thread dump below)

Which version of Artemis are you seeing the problem on?

10.0 (self-hosted, single-node Docker deployment of ghcr.io/ls1intum/artemis, localvc/localci profiles)

What browsers are you seeing the problem on?

Chrome

Additional context

Single-node docker compose setup with local VC/CI and Hyperion enabled. The deadlock was diagnosed after code generation jobs appeared to "hang with no progress": REST POST .../generate-code still returned 200 with a job id, but no STARTED event ever arrived. Restarting the container resolved it; the deadlock is reproducible once two jobs overlap.

Relevant log output

Thread dump (both taskExecutor threads, parked for ~40+ minutes):

"artemis-task-1" #740 [2356] prio=5 os_prio=0 waiting on condition
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@25.0.3/Native Method)
        - parking to wait for  <0x0000000702000040> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.CompletableFuture.waitingGet(java.base@25.0.3/CompletableFuture.java:1919)
        at java.util.concurrent.CompletableFuture.get(java.base@25.0.3/CompletableFuture.java:2093)
        at de.tum.cit.aet.artemis.hyperion.service.websocket.HyperionWebsocketService.send(HyperionWebsocketService.java:39)
        ...

"artemis-task-2" #743 [2358] prio=5 os_prio=0 waiting on condition
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@25.0.3/Native Method)
        - parking to wait for  <0x0000000702800060> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.CompletableFuture.waitingGet(java.base@25.0.3/CompletableFuture.java:1919)
        at java.util.concurrent.CompletableFuture.get(java.base@25.0.3/CompletableFuture.java:2093)
        at de.tum.cit.aet.artemis.hyperion.service.websocket.HyperionWebsocketService.send(HyperionWebsocketService.java:39)
        ...

Server log symptons:
- Job A (exercise 9, tests repo): STARTED 09:19:45, PROGRESS iteration=2 at 09:28:41 — last event ever; thread artemis-task-1 has no log lines after 09:28:41.
- Job B (exercise 10): STARTED 09:26:59 — last event ever; thread artemis-task-2 has no log lines after 09:26:59.
- Job C (exercise 11, started 09:33:01): REST returned 200 with jobId, client subscribed to /user/topic/hyperion/code-generation/jobs/{jobId}, but runJobAsync never entered and zero events (not even STARTED) were sent.
- No exceptions, no 5xx — the deadlock is silent.