chore: build-time quick wins — rsp DAG rebuild fix, CI queue hygiene, symbols retention, LF scripts (#9616)

* chore: commit compiler rsp files, preBuildScript writes only on drift

Every cloud build wrote Assets/csc.rsp (+4 siblings) fresh onto a clean
checkout; Bee logged 'Rebuilding DAG because FileSignature timestamp
changed: Assets/csc.rsp' and re-ran IL2CPP+Usym (~267s) on no-change
builds. Committed rsp files keep a stable checkout state; the script now
rewrites only on content drift. Their .meta files are committed too, and
Explorer/Assets/*.rsp is pinned to eol=lf so CRLF-converting Windows
builders cannot defeat the byte-compare.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: CI queue hygiene — cancel-first ordering, orphan-build cleanup

UBA allows one pending build per target and the org concurrency cap
counts queued builds, so superseded or orphaned builds inflate everyone's
queue.

- workflow: the Unity-side cancel now runs first-after-build with a 2-min
  timeout — GitHub force-kills a cancelled job 5 minutes after the
  request, and a large-log upload ahead of the DELETE could eat the
  window. Log steps gate on the log existing (in-queue cancels never
  write one).
- build.py: persist the target before the build POST and let --cancel
  fall back to the target's latest non-terminal build, so a runner death
  between POST and id write cannot leave a queued build holding a slot;
  the cancel DELETE gets a 30s timeout.

A 90s push debounce was considered and rejected: flat latency tax on
every push build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: upload debug-symbol artifacts only on release builds, 7-day retention

The *_debug_symbols artifacts (546-935 MB per platform) uploaded on every
build at default retention with zero consumers: Sentry symbolication runs
against the on-disk build/ folder in the same job, and release drafting
only downloaded them because its Decentraland_.* regex accidentally
matched before attaching just the four player zips. Gate the upload to
release builds, cap retention at 7 days, and anchor the drafting regex to
the player zips.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: force LF checkout for shell scripts

Windows cloud builders check out with CRLF conversion, so the UBA
preBuildScript (scripts/generate-ignore-warnings.sh) failed with
"$'\r': command not found" on every Windows build —
preBuildScriptFailsBuild=false hid it. All committed *.sh blobs are
already LF; this only pins the checkout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: extend stale watchdog to active build phase via log growth detection

The existing STALE_POLL_THRESHOLD watchdog (600 s) only monitors queue
statuses (created / queued / sentToBuilder).  Once a build transitions to
`started` the watchdog goes blind — a deadlocked builder keeps reporting
`started` indefinitely, leaving BUILD_TIMEOUT (3 h) as the only backstop.

This change adds a log-growth liveness signal for the active phase:

• New `get_log_byte_count(id)` probes `/builds/{id}/log` via HEAD (then a
  single-byte Range request as fallback) to return the current log size
  without downloading the full log.
• `run_poll_loop` tracks `last_log_byte_count` / `last_log_growth`.  On
  every poll tick while `status in ACTIVE_STATUSES` the log size is checked;
  if it has not grown for `LOG_STALL_THRESHOLD` seconds (default 900 s /
  15 min) the build is cancelled and `log_stall` is returned.
• `log_stall` is handled identically to `build_timeout`: build_info is
  deleted, the log downloaded for debugging, and exit code 99 triggers
  nick-fields/retry on a fresh builder VM.
• `LOG_STALL_THRESHOLD: 900` is exposed in build-unitycloud.yml alongside
  the existing `STALE_POLL_THRESHOLD`.

Closes #9628

* chore: address review — harden rsp generator, guard cancel fallback, add drift CI check

- generate-ignore-warnings.sh: temp file via mktemp outside Assets/ (an
  interrupted run no longer leaves a stray file for Unity to import),
  explicit write/copy error handling replacing the stale $? idiom, cp
  keeps the destination inode, distinct unchanged/updated messages.
- build.py: the --cancel fallback only cancels a build still in an
  explicit queue status — targets are shared (release pool, consecutive
  runs per branch), so an already-started build may belong to a
  concurrent run; a missing status is treated as not-cancellable.
  get_latest_build gains timeout=30: it now sits on the 2-minute
  cancellation critical path.
- test.yml: new rsp-drift job regenerates the rsp files and fails on
  diff, so generator drift can't silently reintroduce the per-build
  Bee DAG rebuild.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: address watchdog re-review — growth-armed stall cancel, probe hardening, drift-guard gaps

- build.py watchdog: the stall cancel now arms only after one observed
  log-size increase, so a probe reporting a constant value (proxy
  answering HEAD with Content-Length: 0, endpoint dead during builds)
  reads as inactive instead of cancelling every build at 15 min. A size
  decrease (restarted builds can truncate the log) resets the clock.
  First probe result is printed once; non-2xx probe responses are
  surfaced once instead of silently returning None.
  Threshold evidence: longest log silence across 18 preserved warm
  builds is 99 s (IL2CPP), giving the 900 s threshold a ~9x margin.
- get_log_byte_count: allow_redirects=True on HEAD (requests defaults
  HEAD to not following redirects; the endpoint may 302 to signed
  storage), timeouts lowered 30 -> 10 s on the poll critical path.
- test.yml: generator added to the pull_request paths filter (an edit to
  warnings_to_ignore alone never triggered the workflow); drift check
  uses git status --porcelain so a new untracked rsp file counts as
  drift; rsp-drift job gets timeout-minutes and contents: read.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: retry externally-canceled builds instead of hard-failing

Observed on this branch's own CI (run 31087915798): the Windows build sat
sentToBuilder for 9 minutes, UBA canceled it platform-side (builder
provisioning failure; the same window produced a transient HTTP 500 from
the API), and build.py mapped the external 'canceled' to exit 1 - which
nick-fields/retry does not retry, so an infra flake became a hard red
job with the second attempt unused.

This run's own cancellations exit through the watchdog/timeout branches,
so a 'canceled' final outcome is always external: exit 99 after deleting
build_info, letting the retry create a fresh build. Bounded by
max_attempts: 2, so a supersede race cannot loop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: don't retry a superseded build on shared targets

The unconditional canceled->retry turned a supersede on the shared
release pool into a ping-pong: the superseded run's retry re-POSTs, hits
'already a build pending', cancels the superseding run's build, and
hands it the same exit 99 - both runs burn a queue+build cycle and one
still ends red (and set_parameters clobbers the winner's env vars).

Build numbers are monotonic per target: if the target's newest build is
newer than ours we were superseded and exit 1 without retrying; if ours
is still the newest, the cancel was platform-side (builder provisioning)
and the retry stands. Also download the log best-effort on this path -
with the log steps now gated on the log existing, an externally-canceled
run would otherwise finish with no log artifact at all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: close the supersede-probe gap, fail-open on probe errors

Two residuals in the canceled-outcome guard:

- run_build cancels the pending build and only re-POSTs ~30 s later, so
  a supersede is invisible to a probe landing inside that gap (~coin
  flip with POLL_TIME=60). When the first probe does not show a newer
  build, sleep 35 s and re-probe once before deciding to retry.
- get_latest_build was the one bare call site on this path: a transient
  RequestException would traceback to exit 1 - skipping build_info
  cleanup and the log download, and converting the provisioning-flake
  retry this branch exists for into a red job. The probe now fails open
  (latest = None -> retry), matching the non-200 behaviour.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Esteban Ordano <42750+eordano@users.noreply.github.com>
Co-authored-by: decentraland-bot <44584806+decentraland-bot@users.noreply.github.com>

Author: 4 people

.gitattributes

@@ -29,3 +29,13 @@ Explorer/Assets/StreamingAssets/AssetBundles/**/*.meta -filter=lfs -diff=lfs -me
 *.asset binary
 *.controller binary
 *.asset text diff=yaml merge=yaml
+
+# Root-level compiler rsp files must check out byte-identical to what
+# scripts/generate-ignore-warnings.sh writes (LF), or its drift check rewrites
+# them on CRLF-converting builders and re-triggers a Bee DAG rebuild.
+Explorer/Assets/*.rsp text eol=lf
+
+# Shell scripts must check out LF everywhere: CRLF-converting Windows builders
+# fail them with "$'\r': command not found" (seen on every Windows cloud build
+# running the UBA preBuildScript).
+*.sh text eol=lf

.github/workflows/build-release-main-page.yml

@@ -56,7 +56,9 @@ jobs:
         uses: dawidd6/action-download-artifact@v6
         with:
           run_id: ${{ github.event.workflow_run.id }}
-          name: Decentraland_.*
+          # Anchored to the four player zips: Decentraland_.* also matched the multi-GB
+          # *_debug_symbols artifacts, downloading them only to discard them.
+          name: Decentraland_(windows64|macos)(_epic)?$
           name_is_regexp: true
           skip_unpack: true
 

.github/workflows/build-unitycloud.yml

@@ -573,6 +573,7 @@ jobs:
           POLL_TIME: 60
           QUEUE_POLL_TIME: 120
           STALE_POLL_THRESHOLD: 600
+          LOG_STALL_THRESHOLD: 900
           QUEUE_TIMEOUT: 14400
           BUILD_TIMEOUT: 10800
           TARGET: t_${{ matrix.target }}
@@ -599,6 +600,19 @@ jobs:
           PARAM_IS_RELEASE_BUILD: ${{ inputs.is_release_build }}
           PARAM_UNITY_EXTRA_PARAMS: '-disable-assembly-updater'
 
+      # Must run before any always() log/report steps: on cancellation GitHub grants a 5-minute
+      # grace window, and the DELETE that frees the Unity-side queue slot has to win that race
+      # (a large-log upload ahead of it can eat the whole window). Also runs on failure() so an
+      # outer-step timeout doesn't leave a Unity-side build holding a slot.
+      - name: Cancel Unity Cloud build
+        if: ${{ cancelled() || failure() }}
+        timeout-minutes: 2
+        env:
+          API_KEY: ${{ secrets.UNITY_CLOUD_API_KEY }}
+          ORG_ID: ${{ secrets.UNITY_CLOUD_ORG_ID }}
+          PROJECT_ID: ${{ secrets.UNITY_CLOUD_PROJECT_ID }}
+        run: python -u scripts/cloudbuild/build.py --cancel || true
+
       - name: Locate Windows main executable
         if: matrix.target == 'windows64' && (inputs.is_release_build == true || github.event.inputs.force_sign == 'true')
         run: |
@@ -877,14 +891,19 @@ jobs:
             echo "[${ARTIFACT_URL}](${ARTIFACT_URL})"
           } >> "$GITHUB_STEP_SUMMARY"
 
+      # Symbol artifacts have no downstream consumer (Sentry uploads from the on-disk build/
+      # folder below; release drafting attaches only the player zips) — keep them only where a
+      # human might need post-hoc symbolication, and never at default retention.
       - name: Upload debug symbols
+        if: ${{ inputs.is_release_build == true }}
         uses: actions/upload-artifact@v6
         with:
           name: ${{ env.artifact_name }}_debug_symbols
           path: |
             build/**/*_BackUpThisFolder_ButDontShipItWithYourGame
             build/**/*_BurstDebugInformation_DoNotShip
           if-no-files-found: error
+          retention-days: 7
 
       - name: Upload debug symbols to Sentry
         if: ${{ needs.prebuild.outputs.sentry_enabled == 'true' }}
@@ -899,22 +918,22 @@ jobs:
             --project "$SENTRY_PROJECT" \
             "build"
 
-      # Will run always (even if failing)
+      # Runs even on failure, but only once a log actually exists (a run cancelled in-queue
+      # never writes one, and a guaranteed-failing upload would burn the cancellation grace window).
       - name: Upload cloud logs
-        if: always()
+        if: ${{ always() && hashFiles('unity_cloud_log.log') != '' }}
         uses: actions/upload-artifact@v6
         with:
           name: ${{ matrix.target }}_${{ needs.prebuild.outputs.install_source }}_unity_log
           path: unity_cloud_log.log
           if-no-files-found: error
 
-      # Will run always (even if failing)
       - name: Print cloud logs
-        if: always()
+        if: ${{ always() && hashFiles('unity_cloud_log.log') != '' }}
         run: cat unity_cloud_log.log
 
       - name: Extract and display errors
-        if: always()
+        if: ${{ always() && hashFiles('unity_cloud_log.log') != '' }}
         run: |
           echo "=== Extracted Errors for ${{ matrix.target }} ${{ needs.prebuild.outputs.install_source }} ==="
           grep -iE "error c|fatal" unity_cloud_log.log | sed 's/^/\x1b[31m/' | sed 's/$/\x1b[0m/' || echo "No 'error c' or 'fatal' errors found in ${{ matrix.target }} log."
@@ -931,15 +950,6 @@ jobs:
           path: shader_compilation_report.log
           if-no-files-found: warn
 
-      # Also runs on failure() so an outer-step timeout doesn't leave a Unity-side build holding a slot.
-      - name: Cancel Unity Cloud build
-        if: ${{ cancelled() || failure() }}
-        env:
-          API_KEY: ${{ secrets.UNITY_CLOUD_API_KEY }}
-          ORG_ID: ${{ secrets.UNITY_CLOUD_ORG_ID }}
-          PROJECT_ID: ${{ secrets.UNITY_CLOUD_PROJECT_ID }}
-        run: python -u scripts/cloudbuild/build.py --cancel || true
-
   build-gate:
     name: Build Gate (Windows + macOS)
     runs-on: ubuntu-latest

.github/workflows/test.yml

@@ -5,6 +5,8 @@ on:
   pull_request:
     paths:
       - 'Explorer/**'
+      # The rsp-drift job must run when the generator changes, or drift merges unnoticed.
+      - 'scripts/generate-ignore-warnings.sh'
     types:
       - opened
       - reopened
@@ -36,6 +38,35 @@ concurrency:
 
 jobs:
 
+  # The committed Explorer/Assets/*.rsp files must stay byte-identical to what
+  # generate-ignore-warnings.sh produces: on drift the preBuildScript rewrites them
+  # on every cloud build, and the Bee DAG rebuild (~4.5 min per build) silently returns.
+  rsp-drift:
+    name: rsp files match generator
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      contents: read
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v6
+        with:
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
+
+      - name: Regenerate rsp files and fail on drift
+        run: |
+          set -euo pipefail
+          bash scripts/generate-ignore-warnings.sh
+          # git status --porcelain (not `git diff`) so a newly added, still-untracked
+          # rsp file also counts as drift.
+          drift=$(git status --porcelain -- 'Explorer/Assets/*.rsp')
+          if [[ -n "$drift" ]]; then
+            echo "Committed rsp files differ from generate-ignore-warnings.sh output:"
+            echo "$drift"
+            git diff -- 'Explorer/Assets/*.rsp'
+            exit 1
+          fi
+
   changes:
     if: (github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/dev')) || (github.event.pull_request.draft == false) || (github.event.label.name == 'force-build') || (github.event.label.name == 'clean-build') || (github.event.label.name == 'force-lint')
     name: Detect C# changes

Explorer/Assets/csc.rsp

@@ -0,0 +1,14 @@
+-nowarn:8618
+-nowarn:8625
+-nowarn:8602
+-nowarn:8604
+-nowarn:8619
+-nowarn:8620
+-nowarn:8603
+-nowarn:8600
+-nowarn:8601
+-nowarn:0649
+-nowarn:0414
+-nowarn:0168
+-nowarn:0219
+-nowarn:8632

Explorer/Assets/csc.rsp.meta

@@ -0,0 +1,14 @@
+-nowarn:8618
+-nowarn:8625
+-nowarn:8602
+-nowarn:8604
+-nowarn:8619
+-nowarn:8620
+-nowarn:8603
+-nowarn:8600
+-nowarn:8601
+-nowarn:0649
+-nowarn:0414
+-nowarn:0168
+-nowarn:0219
+-nowarn:8632

Explorer/Assets/gmcs.rsp

@@ -0,0 +1,14 @@
+-nowarn:8618
+-nowarn:8625
+-nowarn:8602
+-nowarn:8604
+-nowarn:8619
+-nowarn:8620
+-nowarn:8603
+-nowarn:8600
+-nowarn:8601
+-nowarn:0649
+-nowarn:0414
+-nowarn:0168
+-nowarn:0219
+-nowarn:8632