Feat/dat async read#2385
Merged
Merged
Conversation
Add GwDatArchive::ReadFileAsync(file_id, callback, stream_id): it queues the request on an internal list and returns immediately. The dat worker, which now wakes on a short poll interval as well as on tasks, scans the list each pass - when a file resolves it fires the callback with the decompressed bytes and drops the entry; if the file hasn't appeared within ~1 minute it fires the callback once with an empty vector and drops it. This lets callers wait for files the client streams in on demand (e.g. the Steam .snapshot archive, which the game persists to disk) without blocking: the existing throttled MaybeRefresh picks the file up on a later poll. Undelivered requests are dropped if the worker is stopped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e each) Add GwDatArchive::SetTrigger(fn): an optional hook the worker calls to ask the client to load a file id that isn't resident. ProcessPendingReads invokes it exactly once per pending request (a `triggered` flag guards against re-asking on every poll), outside the list lock. The pure reader stays free of any game-function lookup; if no trigger is wired it just waits passively. GwDatTextureModule resolves the client's request-files function (the safe FUN_0082da30 variant) by pattern and wires a trigger that issues the request on the game thread. On a client update the pattern may need re-deriving; it fails soft (logs, async reads still wait for natural streaming). Together with ReadFileAsync this closes the loop: request -> client streams the file into the .snapshot on disk -> MaybeRefresh picks it up -> callback fires with the bytes, or empty after ~1 minute. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mmit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the 38-byte prologue pattern for the request function with a string-anchored scan (ToFunctionStart(FindUseOfString(DOWNLOAD_FLAG assert))), matching the codebase convention and surviving client updates that shift the byte layout. This retargets from the byte-pattern'd wrapper to the request function it wraps - safe to call in-game (its requests-suspended assert is only toggled from the client's Main) and enqueue-only (the client pumps its download queue itself, so the wrapper's kick isn't needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Condense the doc comments and fold ProcessPendingReads' deferred-trigger vector back into the loop (fire the tickle under the lock - it's just a game-thread enqueue). No behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… files) Route the texture/greyscale/item-image decode paths and the DDS export through GwDatArchive::ReadFileAsync: dispatch one async read per image that waits for the file to be resident (tickling the client to stream it in), then decode on the worker and upload on the DX thread. ReadFileAsync owns the retry, the ~1 min give-up, and the fetch trigger, so the per-frame retry bookkeeping on GwImg (m_attempts, m_first_missed_ms and the two window constants) is gone; WantsDecode is now just 'no texture, not yet dispatched'. Gating on one stream is enough since a file's streams arrive together. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the fetch trigger into ReadFile: on a miss (after the throttled MaybeRefresh still can't find the file) ask the client to fetch it, throttled per file id so a per-frame retry loop can't spam. Now every reader benefits - the sync callers (pathfinding map loads, the decoders' re-reads) prompt a fetch so a later attempt may find the file, not just the async path. This unifies the trigger: ReadFileAsync's polling goes through ReadFile too, so its per-request 'triggered' flag is gone. m_trigger moves to its own mutex (+ a per-id throttle map) since ReadFile can now tickle while the pending-list lock is held. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The request function internally allocs a request batch and frees it itself (FUN_007cdea0 reclaims it after dispatch), and returns only an int cancel token we discard - so flags=0 is fire-and-forget with nothing to release. A cancelable flag would hand back a handle that leaks if not released. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… unchanged AcquireMappingInto already GetFileSizeEx's the fresh handle, so compare it to the size at the last parse: if the dat hasn't grown there's nothing new to index, so skip ParseFrom (the ~14MB MFT copy + 600k-entry hashmap rebuild) that otherwise ran every 2s while any read was pending. Still re-acquires the handle each time, so an atomic .snapshot rename (which resizes) is caught. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Calling the bare request enqueue left the file queued until the client pumped it on its own (next map change). Also resolve and call the client's download flush after each request - the same one-bit clear the request wrapper does for its 0x2 flag - so the fetch is sent immediately. The flush is a provably-unique 8-byte 'AND dword[flag],~1; RET'; best-effort, so if it's ever not found the request just falls back to the client's own pump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tionFromNearCall) Collapse the two scans (string-anchored enqueue + separate flush) into one: scan a unique call site (PUSH 0x10008; ...; CALL) and FunctionFromNearCall the target, which lands on the client's request+flush wrapper (FUN_0082da30). Call it with flags=0x2 so it enqueues AND flushes in one go - files fetch on the fly instead of at the next map change. Anchoring on the call site's source-derived constant is more update-resilient than a callee prologue scan, and the wrapper also no-ops on the suspend flag rather than asserting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A read miss now queues its file id (still throttled per id) instead of firing the trigger immediately; the worker flushes the accumulated ids to the client as a single FUN_0082da30(count, ids, 0x2) call after each poll. A burst of missing files (e.g. a full icon grid on a fresh streamed install) becomes one game-thread task and one request batch instead of one per file, so no per-file hitch. TriggerFn is now (ids, count); undelivered tickles are dropped on stop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tTextureModule)
Fold the GwDatArchive singleton (dat mapping, MFT parsing, decompression,
async reads, fetch tickle) into the texture module - it was the only user -
and rename the combined class GwDatTextureModule -> GwDatModule. The archive
read API stays public (the module's own decode/trigger helpers drive it via
Instance()); the internals are private. GwDatArchive.{h,cpp} are removed and
its Win32 code now builds with the module's PCH. Pure move + rename, no
behaviour change; ~20 call sites updated to GwDatModule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…he archive API ReadFile/ReadFileAsync are now static (they grab Instance() internally), so callers drop the .Instance() dance. Everything else archive-side that isn't used outside the module - EnsureLoaded, DatPath, Loaded, HandleEnumerationBlocked, SetTrigger, StartWorker, StopWorker - moves to private. The SetTrigger call moved from the anon-namespace helper into Initialize so it can be private (the helper just resolves the request fn now). Also drop the dead task queue: EnqueueTask had no callers, so it, m_tasks, and the WorkerLoop task branch are gone - the worker just polls pending async reads (m_task_mutex/cv renamed m_worker_mutex/cv). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ReadFileAsync refactor left m_pending set forever after the first attempt, so WantsDecode (!m_tex && !m_pending) never re-dispatched: an image whose file wasn't resident within ReadFileAsync's ~1 min window (e.g. an armory item icon on a streamed install) stayed blank permanently, even after a later map change streamed the file in. Clear m_pending in the upload callback (as the old code did) so a failed attempt retries; on success m_tex is set so it won't. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…preloads RE of the request call sites: FUN_008e3a40 just returns static preload lists (not a per-file transform), so our FUN_0082da30(1,&id,flags) is correct. But the client's own file requests use flags 0 / 0x10000 / 0x10008 and never the 0x2 'flush' we added - it isn't how it kicks a download and clearing that flag mid-play may interfere. Use flag 0 (matches FUN_0082dd80): a plain queued request the client downloads when it next processes its queue (map change), which the read retry then picks up. On-the-fly download during play doesn't appear achievable via this API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…the dat grows A map can be a long way off, so drop the ~1 min async-read timeout: a pending read stays queued and keeps re-tickling until the file streams in, then delivers. Callbacks now fire only on success, so the texture path dispatches once and leaves m_pending set (retry lives in the archive), which also avoids looping when a file is resident but its bytes don't decode. When MaybeRefresh sees the dat grow (files streamed in, e.g. a map load - which also clears the client's own request queue), clear the tickle throttle so still-missing pending reads re-request their files on the next pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Decompiling the client's read paths left the per-file crc ambiguous - it CRC32s the whole MFT for integrity, and the per-file check folds a 4-byte value (file id?) into the payload CRC - so gating reads on CRC32(payload) blind could reject every read. Add a temporary, non-gating probe: for the first few reads, log the stored crc vs CRC32(payload) and CRC32(payload+file_id) so we can confirm the real formula from a live dat, then flip it to reject mid-write reads. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ually fetch The on-demand fetch never worked: kTriggerFlags was 0, but in the client wrapper FUN_0082da30 only "flags & 2" calls FUN_00832f00() - the flush that kicks the download queue. With 0 the file id was enqueued (FUN_007cd160) but never flushed, so it was only fetched when the client flushed on its own; a file the game never loads (e.g. an unowned armory item icon) thus never downloaded, never persisted to the .snapshot, and stayed blank even across restarts. FUN_0082e4f0 confirms 0x2 is safe: it dodges the 0x8 state assert, the 0x10001 offline bail-out, and the 0x10000/0x1 cancel-handle leak; it enqueues at priority 1 then flushes. Also add temp diagnostics (tagged // TEMP) to confirm the chain live: the archive we bind to + file count on load, each outgoing request batch, and each dat-grew reindex. These plus the crc probe isolate where the pipeline breaks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-bug vs missing)
FUN_00832f00 ("flush") turns out to just clear a state bit, not kick a download,
so the on-demand fetch problem is independent of the request flags. To tell
whether the missed files are genuinely absent (need a download) or resident but
unfound (a lookup/stream bug), have the request log count how many of the batch
already have a base slot in the index. High => read-path bug; ~0 => not local.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… layout All requested files have a valid base slot, so this is a read bug, not a download one: ReadRaw finds the base but the stream walk (idx = candidate.id) fails for stream>0. Dump slots base..base+13 with their c/b/a/size/id/crc for the first few reads so we can see whether streams are contiguous (base+stream) or chained, and whether the 24-byte field split (id vs crc) is right. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Traced how the client loads item icons: FUN_004d3660 (CrAppearanceDoll.cpp - the character dress-up doll, the armory analog) requests a single file via FUN_0082da30(1,&id,0x10) then opens it at stream 1 (FUN_00470bf0(id,1,0)) to read the icon. Flag 0x10 is used consistently for UI item/icon/model fetches (FUN_004c8180 StoreFile, composite loaders FUN_008458e0/FUN_007fa450). We were the only caller using 0x2 (priority 1 + the no-op flush bit). Match the client: 0x10 enqueues at priority 2 (interactive/UI bucket), no tracking-handle leak, and skips the 0x8 readiness assert and 0x10001 offline bail-out. Also confirms stream 1 is the correct icon stream (the client opens it directly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… target Flag 0x10 changed nothing and the bound Gw.dat never grows, so the request flag was never the issue. Steam GW may stream content into a .snapshot (or _temp$.dat) distinct from the Gw.dat we bind to - in which case our file stays static and no read ever finds the icon. Log every disk handle whose path looks dat-like (.dat/.snapshot/temp/$) with size + archive-magic on first load, so we can see if there is a second, growing file we should be reading instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FUN_0082da30 silently no-ops the whole request while DAT_01075278 (a suspend flag Main toggles) is non-zero. Resolve that flag from the CMP [addr],0 at the top of the already-resolved wrapper, and zero it on the game thread immediately before each request (equivalent to FUN_0082d830(1); its only resume-path effect is this store). Tests whether Main-suspend is what drops our fetches. Main may re-suspend on its next tick, and the pump is still gated on a live connection (DAT_010744c8 &1), so this may not be sufficient on its own. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Terminate() joined the worker then deleted every GwImg, but decode/upload tasks the worker had already handed to Resources::dx_jobs still held raw GwImg* and wrote img->m_tex when the render thread drained them afterward - a use-after-free that corrupted the heap and hung the game. Two fixes: - Shared-own GwImg (maps hold shared_ptr; QueueDecode captures it in both the worker and DX lambdas), so an in-flight task keeps its image alive past Terminate and frees it when it completes. Terminate just clears the maps. - Latch m_terminated (set before StopWorker) so a late ReadFileAsync/render cant respawn the worker or queue new reads during/after teardown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eriments Async reads no longer retry forever. Each pending read gets kMaxReadAttempts (~20s) of misses per dat-growth epoch; a dat growth (map load) bumps the epoch and resets the budget so newly-streamed files still get picked up. When the budget runs out the read delivers an empty vector (treated as "unavailable"), so the consumer leaves m_tex null (a placeholder) instead of us re-tickling ~500 unfetchable files every 10s forever. This is the honest landing after confirming unowned-item icons cannot be fetched on-demand (no in-game download session; the pump dedups our partially-present files on stream-0 + version). Icons present in Gw.dat still load normally (all full installs, plus owned/equipped/seen gear on partial ones). Also strip the investigation scaffolding: the un-suspend SuspendFlag poke, the crc probe + Crc32, the slot-neighbourhood dump, the dat-handle enumeration, and the bound-archive / dat-grew / requesting-N temp logs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resigned to reading whatever is already in the on-disk Gw.dat, strip everything that interfaced with the client: the async ReadFileAsync API + poll worker, the FUN_0082da30 request trigger + scanner, the per-id tickle/batch, MaybeRefresh and its epoch/retry-cap bookkeeping, and the associated mutexes/threads/members. GwDatModule now maps the dat once, parses the MFT once, and reads synchronously; LoadX decode+upload happens in a single DX-upload task (GwImg stays shared_ptr so an in-flight task cant touch a freed image). A read that finds the dat but not the file id Log::Warning()s once suggesting the user run Guild Wars with -image to download all game data (this is the incomplete-Steam-install case; requesting the client to stream it was proven unworkable). ~340 fewer lines. Docs: add a "Missing images in Toolbox" troubleshooting section explaining the -image download with Steam launch-option and standalone-shortcut setup steps; the warning links to it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…te dat GwDatModule now flags when a read finds the dat mapped but not the requested file (the incomplete Steam/streaming-install case) and exposes it via MissingDatData(). The Armory window shows an amber footer warning when that is set, telling the user to run Guild Wars with -image to download all game data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… pieces Track (per render) whether any armor/weapon piece the window actually shows came back with no icon, and display the amber -image warning at the TOP of the window only when that is true AND the dat is genuinely missing data. This scopes it to the armory's own images (not any module-wide miss) and, by gating on MissingDatData(), avoids flashing during the normal first-frame decode on a complete install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oolboxpp into feat/dat-async-read
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.