Skip to content

fix: harden the LOD manifest-builder lane - #3

Merged
eordano merged 2 commits into
decentraland:mainfrom
eordano:fix/lod-manifest-builder-hardening
Jul 16, 2026
Merged

fix: harden the LOD manifest-builder lane#3
eordano merged 2 commits into
decentraland:mainfrom
eordano:fix/lod-manifest-builder-hardening

Conversation

@eordano

@eordano eordano commented Jul 15, 2026

Copy link
Copy Markdown
Member

What

Two independent failure modes in the JIT LOD lane's scene-lod-entities-manifest-builder step, both found by driving a deployment end-to-end:

  1. Read-only workdir poisoning. copy_tree seeds the builder workdir with fs::copy, which preserves source permissions. When the tool dir lives in a read-only store (e.g. a Nix package, mode 444), the copied files land read-only; if a first run is interrupted, every later rebuild fails at copy package.json: Permission denied — permanently, since the copy can never overwrite its own read-only output. Every LOD request then 404s lod-build-failed.

  2. Inherited CATALYST_URL. The npm child inherits the whole service environment, and the tool resolves its content server from CATALYST_URL — a generic name that deployments commonly set for other services on the same host. When the inherited value points at something that isn't a catalyst content API, the scene run completes with zero entities and the lane emits tiny empty-scene LOD bundles, cached as success — worse than the failure case.

Fix

  • copy_tree clears the read-only bit on the destination before and after each copy, so a poisoned tree self-heals and store-sourced trees stay writable.
  • The pipeline passes its own catalyst to the npm child as --catalyst=<base> (npm exposes it as npm_config_catalyst, which the tool already prefers over CATALYST_URL), deriving the base by stripping a trailing /content from the pipeline's content URL. Inherited environment can no longer redirect scene fetches.

acquire_placements / run_manifest_builder gain a catalyst parameter; all callers updated.

Verification

  • 97 lodgen unit tests pass, plus a new catalyst_base case.
  • End-to-end: with a deliberately poisoned workdir (444 package.json) and CATALYST_URL pointing at a dead port, abgen-lod generate on a production scene builds real placements and passes the 32-check self-gate; before the fix the same setup failed instantly (case 1) or emitted empty-scene bundles (case 2).

🤖 Generated with Claude Code

eordano and others added 2 commits July 15, 2026 20:23
Overwrite read-only files when seeding the manifest-builder workdir (a
nix-store copy preserves 444 and poisons every rebuild) and pass the
pipeline catalyst explicitly to the npm child via --catalyst, so an
inherited CATALYST_URL from the service environment can no longer point
scene fetches at the wrong server and emit empty-scene LODs.
Wrap the let-else per rustfmt. Replace set_readonly(false) with an
explicit owner-write mode bit on unix (permissions_set_readonly_false
is denied by CI clippy).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@eordano
eordano merged commit b996bf3 into decentraland:main Jul 16, 2026
2 checks passed
@eordano
eordano deleted the fix/lod-manifest-builder-hardening branch July 16, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant