Programming exercises: Stop cancelling build jobs during docker image pulls

[krusche]

Summary

The stale build job watchdog cancels build jobs while they are still pulling their Docker image. Users see Could not pull Docker image <image> even though the image exists and the pull is simply slow. This PR makes the watchdog aware of in-flight image pulls, fixes a case where it silently skipped its own grace period, and bounds image pulls with an explicit timeout of their own.

Checklist

General

• I tested all changes and their related features with all corresponding user types on a test server.
• Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
• I chose a title conforming to the naming conventions for pull requests.

Server

• Important: I implemented the changes with a very good performance and prevented too many (unnecessary) and too complex database calls.
• I strictly followed the server coding and design guidelines and the REST API guidelines.
• I added multiple integration tests (Spring) related to the features (with a high test coverage).
• I documented the Java code using JavaDoc style.

Motivation and Context

detectAndCleanupStaleBuildJobs() treats a running job without a Docker container as stale. A job that is pulling its image has no container yet by definition, so a slow pull is indistinguishable from a stuck job. Two things then go wrong:

1. A slow pull gets the job killed. After the grace period the job is force-cancelled and requeued. The partial pull is discarded, the retry lands on another agent with an equally cold cache, and the same failure repeats.

2. The grace period is skipped entirely in the worst case. The STALE_DETECTION_MIN_JOB_AGE_SECONDS check only ran when the job was found in the distributed processing map:

   BuildJobQueueItem job = distributedDataAccessService.getDistributedProcessingJobs().get(jobId);
   if (job != null && job.jobTimingInfo() != null && job.jobTimingInfo().buildStartDate() != null) {
       // ...grace period...
   }
   // falls through to stale counting when the job is unknown

   When the lookup returned nothing, execution fell through to the stale counter and the budget dropped from ~150 s to ~30 s. That lookup fails precisely when another agent has just force-cancelled and requeued the job, since that removes it from the map. A requeued retry is therefore more likely to be killed than the original attempt, which is what turns a single slow pull into a cancel-and-requeue loop.

Observed on the production build agents:

10:52:45  agent06: Build job ... stale for 6 consecutive checks (~30 seconds). Force-cancelling and requeuing.
10:52:45  agent19: Processing build job: ... retryCount=1
10:52:45  agent19: Pulling docker image ls1tum/artemis-maven-template:java17-21 ...
10:52:46  agent19: Stale build job detected ... (detection count: 1/6)   <- grace period never applied
10:53:11  agent19: Build job ... stale for 6 consecutive checks. Force-cancelling and requeuing.

The job was killed 26 s after it started, while the pull it was waiting for needed roughly 190 s.

Reported in #13306, #13308 and #13310. The infrastructure trigger there (a registry mirror unreachable from the build agents, adding a ~32 s dial timeout to every Docker Hub request) is being fixed separately. This PR addresses the Artemis-side behaviour, which turns slow pulls into failed builds regardless of why a pull is slow.

Description

Do not treat jobs that are pulling an image as stale. BuildAgentDockerService now tracks which build jobs are inside pullDockerImage, and exposes isImagePullInProgress(String). The registration covers the whole method, including the time a job spends waiting for the pull lock while another job downloads the same image, since those jobs have no container either. SharedQueueProcessingService skips stale detection for such jobs.

Fail safe when the job is unknown. If the build start date cannot be determined, stale detection is skipped instead of counting the job as stale. Genuinely orphaned jobs are still cleaned up by the existing cross-check further down the same method, which removes jobs that are in the distributed map but not running locally.

Bound image pulls explicitly. awaitCompletion() was called with no timeout, so a pull that stopped making progress would block the build thread forever. In practice the watchdog was the only thing bounding it, and that backstop is now removed for pulls, so the bound has to be real: pulls use awaitCompletion(timeout, SECONDS), and on expiry the callback is closed to abort the pull and a clear message is written to the build log.

Reject a non-positive pull timeout at startup. awaitCompletion(0, SECONDS) returns immediately, so a misconfigured value would report every pull as timed out and no build could run. The service fails fast in @PostConstruct with a message naming the property, matching the existing validation of build-agent.short-name in SharedQueueProcessingService.init().

Should the pull timeout be separate from the build timeout?

Yes, and this PR adds it as artemis.continuous-integration.image-pull-timeout-seconds (default 300) rather than reusing the build timeout:

• They bound different things. How long a pull takes depends on the image size and on the registry and network. How long a build may take is a property of the exercise, set per exercise by the instructor and capped by build-timeout-seconds.max (240 s by default). A cold pull of a large image legitimately exceeds that cap, so folding the two together would either fail slow pulls or force every exercise timeout to absorb infrastructure latency.
• They have different scopes. A pull is amortised: it happens once per image per agent, under a lock, and every later job on that agent reuses the result. A build timeout applies per job.
• A student's time budget should not depend on cache state. With a shared timeout, the same submission would get a different amount of build time depending on whether its image happened to be cached.

The default of 300 s is generous compared to what a pull was effectively allowed before, while still being shorter than the whole build timeout used to be (120 to 240 s for pull plus build together). A stuck pull holds a slot in the shared queue, so a much longer default would let one agent hold up the cluster.

A second property, artemis.continuous-integration.image-pull-stall-timeout-seconds (default 60), separates the two failure modes so the slow case does not need the long timeout:

• A pull that is downloading, however slowly, keeps reporting progress and is only bounded by the 300 s overall timeout.
• A pull that reports nothing at all for 60 s is aborted right away with a message naming the likely cause, which is a registry the agent cannot reach because a firewall drops the packets instead of refusing the connection. Without this, such a pull would burn the full overall timeout for a result that was never going to arrive.

Agents on a genuinely slow link can still raise image-pull-timeout-seconds explicitly.

Steps for Testing

Prerequisites:

• 1 Instructor
• 1 Programming Exercise (Java)
• A build agent with an integrated lifecycle setup (LocalVC and LocalCI)

1. On the build agent, remove the exercise's Docker image so the next build has to pull it: docker rmi ls1tum/artemis-maven-template:java17-25
2. Optionally make the pull slow, for example by adding an unreachable registry-mirrors entry to /etc/docker/daemon.json and restarting Docker, which adds a dial timeout to every Docker Hub request.
3. Trigger a build of the template or solution repository.
4. The build waits for the pull and then runs normally. It must not fail with Could not pull Docker image, and the agent log must not contain Force-cancelling and requeuing for this job.
5. Check the agent log for Job ... is currently pulling its Docker image, skipping stale detection.
6. To verify the timeout path, set artemis.continuous-integration.image-pull-timeout-seconds: 5 and repeat with a large uncached image. The build fails with Pulling docker image ... timed out after 5 seconds in the build log, rather than hanging.

Exam Mode Testing

Build jobs from exams go through the same build agent path with the same queue and the same watchdog, so exam programming exercises benefit identically. There are no UI changes.

Testserver States

You can manage test servers using Helios. Check environment statuses in the environment list. To deploy to a test server, go to the CI/CD page, find your PR or branch, and trigger the deployment.

Review Progress

Code Review

• Code review 1
• Code review 2

Manual Tests

• Test 1
• Test 2

Test Coverage

Note: Some tests in the Test job did not pass (failure). Coverage below may be partial.

Server

Class/File	Line Coverage	Lines
BuildAgentDockerService.java	not found (modified)	436
BuildJobManagementService.java	not found (modified)	307
SharedQueueProcessingService.java	not found (modified)	651

Last updated: 2026-08-11 18:35:01 UTC

Screenshots

Not applicable, server-side only.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Docker image pulls now enforce configurable overall and no-progress timeouts, with clear errors, log entries, and reliable cleanup.
  • Build timeouts exclude time spent pulling Docker images, preventing premature termination.
  • Stale build-job detection is suppressed during active image pulls and safely handles jobs without start times.
  • Invalid timeout settings are rejected at startup.
• Tests
  • Added coverage for stalled and overdue pulls, progress tracking, timeout validation, cleanup, and stale-job handling.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Docker image pulls now track per-job progress and enforce configurable overall and no-progress timeouts. Active pulls are excluded from build and stale-job timeout accounting. Missing build start times are handled without stale detection.

Changes

Docker pull lifecycle

Layer / File(s)	Summary
Pull tracking and timeout handling src/main/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerService.java	Tracks active image pulls, validates timeout settings, counts callback progress, and applies timeout-aware completion to primary and amd64 fallback pulls.
Build timeout accounting src/main/java/de/tum/cit/aet/artemis/buildagent/service/BuildJobManagementService.java	Polls build results in 250 ms slices and excludes active image-pull intervals from build timeout accounting.
Stale-build detection integration src/main/java/de/tum/cit/aet/artemis/buildagent/service/SharedQueueProcessingService.java	Skips stale detection for active image pulls and jobs without a known build start time.
Timeout and pull-state test coverage src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerServiceTest.java, src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildJobManagementServiceTest.java, src/test/java/de/tum/cit/aet/artemis/localci/service/DockerClientTestService.java	Tests pull stalls, total timeouts, state cleanup, timeout validation, build timeout exclusion, and timed callback completion.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant BuildJobManagementService
  participant BuildAgentDockerService
  participant MyPullImageResultCallback
  participant BuildResultFuture
  BuildJobManagementService->>BuildResultFuture: poll for build result
  BuildJobManagementService->>BuildAgentDockerService: query image pull status
  BuildAgentDockerService->>MyPullImageResultCallback: await pull with timeout
  MyPullImageResultCallback-->>BuildAgentDockerService: progress or completion
  BuildAgentDockerService-->>BuildJobManagementService: pull active or inactive
  BuildJobManagementService-->>BuildResultFuture: continue waiting or enforce build timeout

Loading

Possibly related PRs

• ls1intum/Artemis#13375: Changes Docker or Kubernetes runner handling and queue processing related to image pulls and stale jobs.

Suggested labels: docker

Suggested reviewers: claudia-anthropica

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: preventing build jobs from being cancelled during Docker image pulls.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch bugfix/build-agent-image-pull-watchdog

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerServiceTest.java (1)

137-150: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert that a timed-out pull closes its callback.

The test currently proves logging and failure, but not the abort operation. A regression removing callback.close() would leave Docker pulling in the background while this test still passes.

Proposed fix

+import java.io.IOException;
 
-    void testPullDockerImageFailsWhenPullExceedsTimeout() throws InterruptedException {
+    void testPullDockerImageFailsWhenPullExceedsTimeout() throws InterruptedException, IOException {
         var build = mockPendingImagePull();
         // Simulate a pull that never finishes: the timed await reports that the image did not arrive within the timeout.
         when(pullImageCallback.awaitCompletion(anyLong(), any(TimeUnit.class))).thenReturn(false);
 
         try {
             assertThatThrownBy(() -> buildAgentDockerService.pullDockerImage(build, buildLogsMap)).isInstanceOf(LocalCIException.class);
 
+            verify(pullImageCallback).close();
             assertThat(buildLogsMap.getAndTruncateBuildLogs(build.id())).anyMatch(logEntry -> logEntry.log().contains("timed out after"));

As per PR objectives, expired pull callbacks must be aborted.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerServiceTest.java`
around lines 137 - 150, Extend testPullDockerImageFailsWhenPullExceedsTimeout to
verify that pullImageCallback.close() is invoked when pullDockerImage times out.
Keep the existing exception, log, and in-progress assertions, and place the
close verification within the existing cleanup-safe test flow.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@src/main/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerService.java`:
- Around line 137-138: Update BuildAgentDockerService to constructor-inject
imagePullTimeoutSeconds instead of field injection, and validate the value
during bean construction so zero or negative values are rejected immediately.
Preserve the existing default while ensuring the constructor stores only a
strictly positive timeout for pull operations.

In
`@src/main/java/de/tum/cit/aet/artemis/buildagent/service/SharedQueueProcessingService.java`:
- Around line 498-503: Update the unknown-buildStartDate branch in
SharedQueueProcessingService to clear the job’s existing consecutive
stale-detection counter before logging and continuing. Preserve the current
behavior of skipping stale detection for jobs without a known start date, while
ensuring prior detections cannot trigger cancellation when the distributed entry
later reappears.

---

Nitpick comments:
In
`@src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerServiceTest.java`:
- Around line 137-150: Extend testPullDockerImageFailsWhenPullExceedsTimeout to
verify that pullImageCallback.close() is invoked when pullDockerImage times out.
Keep the existing exception, log, and in-progress assertions, and place the
close verification within the existing cleanup-safe test flow.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 920edfb9-0ebf-419c-b626-1d69f916915c

📥 Commits

Reviewing files that changed from the base of the PR and between 4da355e and 900d307.

⛔ Files ignored due to path filters (1)

• src/main/resources/config/application-buildagent.yml is excluded by !**/*.yml

📒 Files selected for processing (4)

• src/main/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerService.java
• src/main/java/de/tum/cit/aet/artemis/buildagent/service/SharedQueueProcessingService.java
• src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerServiceTest.java
• src/test/java/de/tum/cit/aet/artemis/localci/service/DockerClientTestService.java

[github-actions]

End-to-End Test Results

Phase	Status	Details
All Tests	❌ Failed	Tests Passed ☑️ Skipped ⚠️ Failed ❌️ Time ⏱ All E2E Tests Report (PR) 354 ran 344 passed 6 skipped 4 failed 40m 27s

❌ Failed Tests

• Channel messages › Edit channel › Instructor should be able to edit a channel (1m 42s)
• Exam assessment › Programming exercise assessment › Assess a programming exercise submission (MANUAL) (0s)
• Programming exercise practice mode › After the due date with a graded submission › Keeps the practice mode selectable when switching back to graded (7m 47s)
• Programming exercise practice mode › After the due date without a graded participation › Shows the submission state when submitting in the practice mode code editor (6m 4s)

Test Strategy: Running all tests (configuration or infrastructure changes detected)

Overall: ❌ E2E: real (non-flaky) test failure

🔗 Workflow Run · 📊 Test Report

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@src/main/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerService.java`:
- Around line 141-142: Update the imagePullTimeoutSeconds configuration default
from 300 to the specified 900 seconds, preserving the existing property key and
explicit-configuration behavior.
- Around line 465-486: Set image-pull-timeout-seconds to 900 in both
configuration locations currently defining it as 300. In awaitPullCompletion,
calculate the remaining stall and overall deadline durations before each
callback wait, and pass the shorter positive duration to
callback.awaitCompletion instead of the fixed
PULL_PROGRESS_POLL_INTERVAL_SECONDS timeout, while preserving progress tracking
and abort behavior.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 34714e31-86c6-4445-a7c8-59d3c26e88dc

📥 Commits

Reviewing files that changed from the base of the PR and between 727336f and 70fced8.

⛔ Files ignored due to path filters (1)

• src/main/resources/config/application-buildagent.yml is excluded by !**/*.yml

📒 Files selected for processing (6)

• src/main/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerService.java
• src/main/java/de/tum/cit/aet/artemis/buildagent/service/BuildJobManagementService.java
• src/main/java/de/tum/cit/aet/artemis/buildagent/service/SharedQueueProcessingService.java
• src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerServiceTest.java
• src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildJobManagementServiceTest.java
• src/test/java/de/tum/cit/aet/artemis/localci/service/DockerClientTestService.java

🚧 Files skipped from review as they are similar to previous changes (5)

• src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildJobManagementServiceTest.java
• src/main/java/de/tum/cit/aet/artemis/buildagent/service/SharedQueueProcessingService.java
• src/test/java/de/tum/cit/aet/artemis/localci/service/DockerClientTestService.java
• src/main/java/de/tum/cit/aet/artemis/buildagent/service/BuildJobManagementService.java
• src/test/java/de/tum/cit/aet/artemis/buildagent/service/BuildAgentDockerServiceTest.java

[Claudia-Anthropica]

@krusche The exact head correctly separates image-pull and build-timeout budgets, resets stale counters on both skip paths, and bounds silent pulls while cleaning up callbacks and tracking state. The snapshot contains no completed relevant CI failure and declares no required checks.

[Claudia-Anthropica]

@krusche The exact head keeps the pull and build timeout budgets separated, applies independent progress and overall pull bounds, and resets stale counters on both new skip paths. The latest test setup restores a successful callback per method, but one verification can still false-pass because the shared Docker mock retains invocation history; I left that low-severity issue inline. The snapshot declares no required checks and contains no completed relevant CI failure.

[Claudia-Anthropica]

@krusche The exact head correctly separates image-pull and build-timeout budgets, clamps each pull wait to the nearest stall or overall deadline, and reliably clears pull and stale-detection state. The triggered test-isolation fix is present, and the snapshot contains no completed relevant CI failure or required pending check.