Commit f731a9e
* feat(lambda): shared Redis hit-cache in front of S3 existence probes
Same optimization consumer-server runs in production: Redis memoizes
positive S3 existence checks so concurrent workers skip the round-trip,
while S3 stays the source of truth. Two probe kinds are cached — the
per-file asset-reuse HEAD in space_probe_asset, and the lambda's
per-platform already-converted manifest check.
On Lambda the wall-time matters twice: invocations are billed as
memory x duration at 10 GB, and batchSize 1 with up to 20 containers
means each container re-learns the same probe answers. Redis shares
them across all invocations, like consumer-server shares across pods.
Semantics mirror consumer-server's asset-reuse cache: positives only
(never negatives — a concurrent build may upload the object any
moment), fail-open on every Redis error with a 30 s backoff so an
outage can't make probes slower than no cache, keys scoped to bucket
(+ version for entity markers) so distinct CDNs never cross-contaminate,
24 h TTL to bound the keyspace across AB_VERSION bumps. force jobs
delete the entity markers they bypass, since a reconversion can
downgrade a previously-ok manifest.
The client is a hand-rolled blocking RESP2 codec over TcpStream
(EXISTS/SET/DEL), in the same spirit as the SigV4 client in space.rs —
no async runtime, no driver crate, no new dependencies. Opt-in via
ABGEN_REDIS_URL; unset leaves behavior byte-identical to today.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(rediscache): review fixes for #56
- move the test-only Read import into the test module (deny-warnings CI)
- forget() entity markers again after a force job's publish; document the
residual re-mark race and the mutable-manifest caveat in the README
- count EXISTS/SET/DEL failures in abgen_rediscache_total{result="error"},
not just connect failures
- 2-arg AUTH when the URL carries a userinfo user (Redis ACL / ElastiCache RBAC)
- loopback fake-RESP tests pinning AUTH/SELECT order, SET..EX, fail-open on
-ERR, and the 30s backoff
* fix(sns,notify): review fixes for #57
- Sns::publish retries transient failures (3 attempts, short backoff),
re-resolving creds and re-signing per attempt; loopback test covers the
500-then-200 path
- extract finished_event() as a pure function and pin the exact JSON body
(fields per @dcl/schemas AssetBundleConversionFinishedEvent)
- the skip log now distinguishes 'ARN not set' from 'creds did not resolve'
* fix(rediscache): speak TLS for rediss:// URLs (#62)
ElastiCache clusters that mandate in-transit encryption were unusable: the
hand-rolled RESP client rejected the rediss:// scheme outright and fell back
to the S3-probe path. The codec is transport-agnostic, so the fix is a
two-variant Stream (plain TcpStream / rustls StreamOwned over the same
socket) behind the BufReader, picked by the URL scheme.
No new crate in the tree: rustls and webpki-roots are already pulled in by
ureq, and the feature set here mirrors ureq's (ring provider, no aws-lc-rs,
so nothing new needs cmake at build time). Certificates are verified against
the webpki roots that ElastiCache's Amazon-issued chain terminates in; there
is deliberately no insecure/skip-verify escape hatch.
The TLS handshake gets the connect timeout rather than the 1 s per-command
budget, since it costs round-trips a single command doesn't, and is driven to
completion in connect() so a non-TLS or untrusted peer fails there — where
the existing fail-open backoff already handles it — instead of mid-command.
Tests cover the scheme parse, a full plain-TCP AUTH/SELECT/EXISTS round-trip
against a loopback server, and that a rediss:// target puts a TLS ClientHello
with SNI on the wire and errors cleanly against a plaintext peer. Verifying a
real server certificate would need a CA fixture and a test-only trust hook,
which is left out on purpose.
* docs: note upstream names for the Redis cache env vars (#62)
The consumer-server reads REDIS_URL (components.ts) and
REDIS_CACHE_TTL_SECONDS (scenes/component.ts) for the same hit-cache;
this repo keeps its deliberate ABGEN_-prefixed convention, so record the
mapping instead of renaming.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VS5aLVBCRTe8NfffUvj1mD
* fix(lambda): partial-batch responses for SQS batches (#63)
The event-source mapping sets reportBatchItemFailures, but the handler
failed the whole invocation on any record error, redelivering every record
in the batch. SQS events now answer
{"batchItemFailures":[{"itemIdentifier":<messageId>}]}: a record that fails
to parse or to convert is reported by its own message id and redelivered
alone.
Parsing keeps per-record identity (event::Event::Sqs of event::Record,
each holding its messageId and its own parse Result) instead of failing the
batch on the first malformed body. Direct invokes stay outside the protocol
and keep the {"jobs": [...]} shape. A failing record with no usable
messageId still fails the invocation — the failure cannot be named, and
answering success would delete the message.
Moot at batchSize 1; this is what raising it needs.
* feat(lambda): Function URL POST-to-convert with shared-secret auth (#58)
An event carrying requestContext.http.method is handled as an HTTP request:
the body is the same JSON the SQS/manual paths take, and the response is the
same summary synchronously, wrapped in a payload-format-2.0 HTTP envelope.
Ad-hoc conversions no longer require hand-crafting an SQS message.
Auth is a constant-time shared-secret check on x-abgen-secret, configured
via ABGEN_HTTP_SECRET and fail-closed (503) when unset, so a Function URL
with auth NONE is not open to the internet. HTTP invocations answer in-band
(4xx/5xx) rather than failing the invocation; the SQS path is unchanged and
still fails so the queue can retry.
* feat(lambda): CloudWatch EMF metrics exporter (#64)
The crate records metrics:: counters and histograms, but the lambda
installed no recorder, so nothing left the process. Install an EMF
recorder behind ABGEN_EMF_NAMESPACE: it accumulates in-process and, at
the end of each invocation, writes Embedded Metric Format JSON to stdout
(one line per label set), which CloudWatch Logs turns into metrics with
no API calls, no agent and no new dependency. Also records the
job/invocation/convert timings, bundle counts and texture-cache hit
rates the soak dashboards need.
* fix: align metric label names with upstream asset-bundle-converter (#64)
Upstream's hit/miss counters label the axis 'outcome'
(ab_converter_glb_deps_cache_total{outcome in hit,miss} in
consumer-server/src/metrics.ts), so abgen_lambda_texencode_cache_total
uses 'outcome' too instead of the invented 'result'. Metric names stay
in the abgen_* namespace on purpose — none of the lambda-side concepts
overlaps the ab_converter_* registry one-to-one; the README now records
that mapping.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VS5aLVBCRTe8NfffUvj1mD
* feat(lambda): generate LOD bundles for lods jobs behind ENABLE_LODS (#61)
LOD deployment jobs were acked and skipped ("lods-unsupported") because the
FBX sources they carry have no importer here. Instead of transcoding them,
route the job through the lodgen chain the abcdn already runs JIT: levels 0+1
are regenerated from the scene entity, self-gated, and published to the
unversioned LOD/{level}/ and lods-unity/manifests/ keys.
Off by default (ENABLE_LODS=1 opts in) — a LOD build is a whole-scene bake
with a very different cost profile from a single-entity conversion, so the
default stays ack-and-skip, now reported as "lods-disabled".
lods::published_objects enumerates the space objects a generated LOD scene
directory publishes, so the key layout lives next to the LOD code.
* fix(space): match prod ab-cdn Content-Type/Cache-Control on uploads (#60)
Uploads set Content-Type from a hardcoded per-call-site literal and never
sent Cache-Control, so S3 metadata diverged from what the production
consumer-server writes (bundles application/wasm + immutable, manifests
application/json + no-cache).
Derive both headers from the object key in space::object_headers and drop
the content_type argument from Space::put / Proxy::space_put_key, so no
call site can diverge again.
* fix: align Cache-Control values with upstream asset-bundle-converter (#60)
Upstream writes two byte-distinct immutable spellings: bundle keys go
through @dcl/cdn-uploader whose cacheHeader() joins directives with a
bare comma ('public,max-age=31536000,immutable', .br variants adding
no-transform), while scene source files are uploaded directly by
scenes/component.ts with the comma-space string
'public, max-age=31536000, immutable'. object_headers now reproduces
both verbatim per lane instead of using one spaced spelling everywhere;
this also makes uploads agree with what abcdn::serve already sends for
bundles.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VS5aLVBCRTe8NfffUvj1mD
* fix(lambda): drop metadata-only files from conversion input (#59)
The prod converter passes -skippedHashes so Unity never sees display-only
files (creator-hub autogenerated-thumbnail.png, scene.json's
display.navmapThumbnail target); abgen converted them, so a manifest carried
extra bare-named entries prod's lacks and any tolerated failure on them
reported exitCode 12 where prod reports 0.
Scene::metadata_only_hashes mirrors asset-reuse.ts::findMetadataOnlyHashes
(filename patterns + metadata dot-paths, case-insensitive resolution) and
build_entity_into_corpus drops those hashes before naming, probing or
building. Also names the file when an image source fails to decode — that
counter fed exitCode silently, which is why an exit-12 run could not be
attributed to a file.
* style: rustfmt after merges
* fix(lod): adopt #60's key-derived upload headers in the LOD publish path
space_put_key derives Content-Type/Cache-Control from the key since the
upload-headers change (which already covers LOD/ and lods-unity/manifests/
lanes); drop the now-removed explicit content-type argument.
* fix(lambda): LOD lane parity in job metrics and finished events
Adversarial-review fixes on the LOD/metrics theme:
- lod::convert's success summary carried no top-level exitCode, so
job_outcome classified every successful LOD conversion as
outcome="failed" in abgen_lambda_jobs_total and the duration
histogram. The summary now carries exitCode: 0 (and the shape is
pinned by a test against job_outcome).
- The LOD lane never published an AssetBundleConversionFinishedEvent,
and finished_event hardcoded isLods: false — a registry consuming the
dedicated topic would see LOD conversions pending forever. The success
path now publishes one event per supported platform with isLods: true
and statusCode: 0, mirroring upstream conversion-orchestrator's
publishFinishedEvent(…, isLods: !!job.lods); the skip branches stay
silent by design and the README's terminal-branch claim is scoped to
conversion jobs.
- Records that failed to parse bypassed instrumented_job entirely, so a
poison-pill message redelivering to the DLQ appeared in no EMF metric.
run_batch now routes Result<Job> through instrumented_job, counting
parse failures as outcome="error" (test installs a local metrics
recorder to pin it).
- The two main.rs handler tests built Config::from_env(), so an ambient
ENABLE_LODS=1 sent them through the real network-touching LOD lane;
they now build an explicit test config.
* fix(lambda): honest HTTP status codes on the Function URL path
- A Records-shaped POST whose record failed to parse or convert returned
HTTP 200 wrapping {"batchItemFailures":[…]} — the itemIdentifier is
meaningless outside SQS and nothing redelivers on this path, so the
job was silently lost for any caller keying on the status code. A
non-empty failure set now maps to 500 (body stays the batch summary).
- An unrecognized event shape returned 500; the README documents 400 for
client mistakes. parse_event now runs before the handler on the HTTP
path and its errors answer 400.
- Handler-error 500 bodies echoed the full anyhow context chain (local
paths, upstream URLs, the SNS topic ARN) to anyone holding the shared
secret. The body is now a generic message; the chain goes to the log
only.
* fix(lambda,rediscache): security hardening from adversarial review
- Content-server scheme/shape validation (https only, no userinfo,
non-empty host) now runs unconditionally in handle_job; only the host
allowlist stays gated on ALLOWED_CONTENT_SERVER_HOSTS. Previously an
unset allowlist skipped every check, letting an event-supplied
contentServerUrl point the handler at plaintext/internal targets
(SSRF read via fetch_entity, exfiltration via upload_scene_sources).
- The x-abgen-secret comparison leaked the configured secret's length:
constant_time_eq returned early on a length mismatch. The check now
hashes both sides (SHA-256) and compares equal-length digests.
- A malformed ABGEN_REDIS_URL was logged verbatim, embedded AUTH
password included; parse errors now redact userinfo (scheme+host only)
and a test pins that no credential survives into any parse error.
- handle_job now runs under catch_unwind: a panic in conversion/lodgen
code became a process abort that skipped emf::flush, posted no
response, and redelivered the whole batch (re-running and re-notifying
already-succeeded records). A panic is now an Err -> batchItemFailure
/ handler error, and the flush stays on the normal path.
* fix(lambda,rediscache): close the force-path stale-marker window
- The force job's post-publish marker forget ran after notify and only
when notify succeeded: SNS retries (3 attempts/platform with sleeps)
widened the publish->forget window for no reason, and an SNS outage
after a successful S3 publish skipped the forget entirely — on an
HTTP/direct force invoke (no redelivery) a marker re-set by a
concurrent non-force job then masked the downgraded manifest for up to
the 24h TTL. The sequence is now publish -> forget -> notify, with the
ordering (including forget-despite-notify-failure) pinned by tests on
the extracted publish_forget_notify helper.
- rediscache::forget was silently fail-open: during the 30s backoff the
DEL was skipped with no log at all. State::forget now reports whether
the DEL ran (loopback-tested), the public forget logs a keyed warn
when it did not, and the README documents the fail-open-delete caveat
next to the other marker caveats.
* fix(space,lods): match each key family's production writer on uploads
- lods-unity/manifests/* was classified as a consumer-server manifest
and stamped 'private, max-age=0, no-cache' — but that key family's
production writer is lod-generator-unity's storage adapter, which
uploads with CACHE_CONTROL_ONE_YEAR = 'public, max-age=31536000' (the
file names embed the content-addressed entity id). A CDN honoring
origin Cache-Control made every ISS descriptor uncacheable. The family
now gets prod's exact spelling; only manifest/ stays no-cache.
- .br objects now carry Content-Encoding: br, matching @dcl/cdn-uploader
(uploadFile brFileConfig) and the abcdn edge's own serving contract —
clients relying on the header for transparent decode got raw brotli
bytes from abgen-written objects at prod-shared keys. The two
remaining cdn-uploader divergences (decompressed-content-length
metadata, public-read object ACL) are deliberate and now documented in
lambda/README.md.
- Removed the dead PublishedObject.content_type: unread since the LOD
publish path moved to key-derived headers, but still populated and
pinned to values ('application/octet-stream' for LOD bundles) that
contradict what actually uploads — a trap for future readers.
- The abcdn server-lane LOD write-back keeping the shared key-derived
headers is deliberate (documented at spawn_lod_writeback): origin
objects match their production writers no matter which lane wrote
them.
- lambda/README.md: documented the intentional absence of prod's
manifest/{entityId}_failed.json failure sentinel (failures here
redeliver to the DLQ instead of being terminal).
* chore: record artifact hashes for the merged tree
* fix(lint): clippy manual_clear + rustfmt drift in lz4, red on main since #76
#76's own arm lane failed on exactly these and merged anyway; every
branch merging main inherits the red. dst.truncate(0) -> dst.clear()
(same semantics), and the fmt wrap at the test pattern loop. Full
workspace clippy --all-targets -D warnings + fmt --check green locally.
* chore: record artifact hashes for the merged tree
---------
Signed-off-by: Esteban Ordano <42750+eordano@users.noreply.github.com>
Co-authored-by: Juan Molteni <juanignaciomolteni@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent c2ee474 commit f731a9e
35 files changed
Lines changed: 3755 additions & 187 deletions
File tree
- ci/artifact-hashes
- crate
- abgen-node
- abgen-wasm
- src
- abcdn/handlers
- bvwebgpu
- lambda
- examples
- src
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
145 | 153 | | |
146 | 154 | | |
147 | 155 | | |
| |||
160 | 168 | | |
161 | 169 | | |
162 | 170 | | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
163 | 177 | | |
164 | 178 | | |
165 | 179 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
| 106 | + | |
| 107 | + | |
106 | 108 | | |
107 | 109 | | |
108 | 110 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments