Offline RetroAchievements Support#707
Open
clintonium-119 wants to merge 44 commits intoLoveRetro:mainfrom
Open
Offline RetroAchievements Support#707clintonium-119 wants to merge 44 commits intoLoveRetro:mainfrom
clintonium-119 wants to merge 44 commits intoLoveRetro:mainfrom
Conversation
…fully downloading
…fline RA ledgerGetPendingUnlocks and ledgerCompact used naive matching that cancelled ALL UNLOCKs with a matching SYNC_ACK achievement ID regardless of order, causing re-unlocks after a sync to be silently dropped. Both now use order-aware matching: a SYNC_ACK only cancels the earliest preceding unmatched UNLOCK, preserving later re-unlocks as pending. Also includes Phase 6 production hardening: - Ledger compaction after successful sync (removes acked pairs) - Resilient chain validation (no longer truncates at first break) - Diagnostic logging removed, verbose logs downgraded to DEBUG - API token sanitized in HTTP error logs
Replace blocking WiFi wait with async background probe. When a cached login exists, start in offline mode immediately and probe the RA server every 30s in the background. On success, seamlessly transition to online mode with deferred notification, sync, and hardcore re-enable on the main thread. Fall back to current blocking behavior only when no cache exists. Wire up DISCONNECTED/RECONNECTED events to flip offline state, manage the probe lifecycle, and always track pending unlocks regardless of online/offline mode.
…noise Process deferred connectivity flags (online/offline notifications, sync, hardcore re-enable, login retry) during gameplay via periodic 500ms check in RA_doFrame(), instead of only when the menu is opened in RA_idle(). Prevent false RECONNECTED events by returning retryable errors for non-cacheable requests (awardachievement, ping) in offline mode instead of synthetic success responses. Suppress transient offline/connected notification spam at startup when WiFi connects quickly. Add sync retry on failure via connectivity probe re-trigger.
… safety comment Downgrade per-response queue log from INFO to DEBUG and rcheevos internal logging from INFO to WARN. Remove unused variables (ra_probe_thread, ra_sync_needs_retry). Strip redundant "Warning:" prefix from WARN-level messages. Add comment explaining why the direct callback in the offline non-cacheable path is thread-safe.
Add progress_mutex to protect progress_state, which is written by background threads (sync engine, badge downloads) and read by the main thread during rendering. Writers lock around all updates; the render path snapshots progress_state under lock then renders from the local copy to keep critical sections sub-microsecond. Fix system indicator pill width using hardcoded 10 (internal pill padding) instead of the PADDING macro, matching the formula in GFX_blitHardwareIndicator. On Brick where PADDING=5, the old formula allocated a surface too narrow, clipping the pill on the right edge.
…idate sync responses
Suppress the 4 notifications that alert users to RetroAchievement connectivity state changes (offline mode, reconnected, connected). These transitions are handled transparently with automatic syncing, so surfacing them adds noise without actionable information.
…restart Remove ra_offline_mode guard from startsession patching to avoid race with connectivity probe. Re-apply pending offline unlocks after hardcore re-enable (both deferred and reconnect paths) since softcore-only bits get cleared. Add deferred sync-apply so the sync thread's confirmed unlocks update rcheevos state without requiring a restart. Augment game-load notification count with pending offline unlocks.
…ate ledger logic - Extract SHA-256 to common/sha256.c + sha256.h for reuse - Fix hardcore filter asymmetry: reject hardcore unlocks at write time and filter during compaction to prevent immortal unsynced records - Extract ledger_record_init() helper to deduplicate ledgerWrite* boilerplate - Fix thread-safety: replace static buffer in get_request_type with caller-provided buffer - Replace prev_hash[0] scratch flag with dedicated bool* cancelled array - Remove redundant double ledger read in patchStartsessionResponse - Consolidate 9 identical passthrough blocks into goto passthrough - Extract shared ledger_read_pending_records() to unify SYNC_ACK cancellation logic between ledgerCompact and ledgerGetPendingUnlocks
The probe thread's online transition triggered sync before rcheevos finished loading the game. The sync thread would compact the ledger before startsession patching or ra_reapply_pending_unlocks could use it, causing the just-synced achievement to appear locked until the next restart. Defer both sync start and sync_apply processing until ra_game_loaded is true.
When an achievement was unlocked while online, the event handler added it to the pending cache (write-ahead for crash safety) but the HTTP callback only wrote a SYNC_ACK to the ledger — it never removed the entry from the in-memory pending cache. The UI kept showing the [O] offline indicator for server-confirmed unlocks. Remove the pending cache entry when awardachievement succeeds.
- Add common/md5.c+h: standalone MD5 with nui_ prefix to avoid LTO symbol collision with rcheevos' internal rhash/md5.c - Add common/ra_sync.c+h: shared offline sync engine used by both settings and minarch; supports game_id filtering, configurable delays, MD5-signed award requests, SYNC_ACK ledger writes, and startsession cache invalidation after successful sync - Add "Sync Offline Unlocks" button to settings with B-button cancel, progress overlay, and RA_SYNC_CONFIG_INTERACTIVE timing - Refactor minarch ra_integration.c to use shared sync engine instead of inline rc_api award calls; per-game sync on game load - Fix overlay text ghosting in settings showOverlay force-draw path - Fix stale startsession cache causing synced achievements to show as locked on first offline-first game launch after a settings sync
…ne award When an achievement is confirmed online (either during gameplay or via settings sync), the cached startsession file is now surgically updated to include the new unlock rather than being deleted entirely. This prevents the next offline-first load from seeing 0/X unlocks and re-triggering all past achievements as new offline unlocks.
- Move ledger fsync off main thread via async write queue - Pre-decode badge surfaces on HTTP worker thread at download time - Replace O(N) response queue shift loop with O(1) ring buffer - Fix use-after-free in progress indicator icon
Replace [M] mute and [O] offline text tags with ASSET_VOLUME_MUTE and ASSET_WIFI_OFF sprite icons in achievement list rows and the detail page. Mute icon is src_rect-cropped to 12px to match text height. List X button hint now toggles MUTE/UNMUTE based on selected item.
…tered count When launching a game with offline achievements pending across multiple games, the initial "Syncing N offline achievements..." notification incorrectly showed the total count from all games rather than only the count for the loaded game. The completion notification was correct since RA_Sync_syncAll filters by game_id internally.
Three interrelated bugs in the offline→online achievement sync path: 1. waiting_for_reset never cleared: rc_client_set_hardcore_enabled() sets waiting_for_reset=1, but rc_client_reset() was never called to acknowledge it, permanently blocking do_frame achievement processing after offline→online transitions with hardcore re-enable. 2. On-device unlock timestamps overwritten: the deferred sync-apply code replaced the original ledger timestamp with time(NULL), causing the on-device display to show the sync time instead of the actual unlock time after sync completed. 3. Startsession patching only injected into softcore "Unlocks" array, not "HardcoreUnlocks". When hardcore was re-enabled, rcheevos saw pending achievements as not-yet-unlocked-in-hardcore, reset them to ACTIVE, and could re-award them via its own server call without &o=, causing the RA server to record the sync time as the unlock time. Also adds diagnostic logging: per-achievement startsession injection details, rc_client_reset confirmation, redacted POST body (to verify &o= is present), and raw server response (to distinguish "Success" from "User already has").
When a ROM hash is unknown to RetroAchievements, users previously had
no indication of the issue until they opened the achievements menu.
Now a toast notification is shown immediately ("No achievements found
for this game"), and the achievements menu provides guidance about
checking retroachievements.org for compatibility patches or supported
ROM versions.
Also removes erroneous ra_start_offline_sync(0) calls from the game
load failure paths, which were triggering sync-all behavior for
unrelated games instead of being scoped to the current game.
Reported: offline achievement unlock times being replaced by sync time on the RA server. Root cause: race between our sync engine (which sends &o=<seconds_since>) and rcheevos' own retry path (which doesn't include &o=, causing server to use current time). Fix 1: In ra_server_call(), intercept awardachievement requests and return synthetic success if the achievement is in our pending ledger. This lets our sync engine handle the submission with the correct timestamp. Fix 2: In ra_http_callback(), query ledger for original timestamp BEFORE removing from cache, use it when patching startsession cache. Previously time(NULL) was always used, corrupting on-device display. Also adds build version to startup log for easier testing.
…ent submissions When achievements are unlocked offline, rcheevos' internal retry mechanism races with our sync engine, sending awardachievement with incorrect CLOCK_MONOTONIC-based timestamps that can overwrite correct wall-clock timestamps on the RA server. - Gate awardachievement in ra_server_call() to return synthetic success when unlock is pending or sync is in progress - Fix startsession cache patch to skip update when ledger entry is already compacted, preventing time(NULL) fallback corruption - Add tagged diagnostic logging ([AWARD_GATE], [AWARD_HTTP], etc.) - Bump build version to 3
…isses
When starting offline with a game never previously played online, game
data requests (gameid, achievementsets, patch) have no cached response.
Previously these got a synthetic {"Success":true} which rcheevos
couldn't
parse, permanently failing the game load with no retry mechanism.
- Only synthesize responses for simple types (login2, startsession)
- Return retryable error for game data types on cache miss
- Add game_load_retry deferred flag to retry load on connectivity
restore
- Preserve pending load info until game load succeeds (not on attempt)
- Clear retry state on game unload to prevent stale retries
The RA server validates the sync hash using its internal `username` field, which may differ from `display_name` after an account rename. Our sync engine was using the locally-configured display name, causing a hash mismatch that made the server silently drop the `&o=` (seconds_since_unlock) offset parameter — recording unlock times as the sync time instead of the actual unlock time. Extract the internal username from the `AvatarUrl` field in the login response (which is built from the server's `username`, not `display_name`), persist it in config as `raServerUsername`, and use it for sync hash computation with fallback to the local username for first-boot compatibility.
…mismatch The RA server silently ignores the backdate offset (&o=) when the MD5 hash is computed with the wrong username. The login API returns display_name (which users can change) rather than the internal username used for hash validation. Extract the server's internal username from the AvatarUrl field (/UserPic/USERNAME.png) at every login path and persist it in config for the sync engine. Also fixes an off-by-one in config.c that corrupted raServerUsername on every load/save cycle (strncmp length 16 vs correct 17), and adds a missing rc_client_reset() call in the RECONNECTED handler.
Five-wave refactoring of the offline RA code (~1500 net lines across 12
files):
Wave 1 — Bug fixes:
- Join probe/sync threads on shutdown instead of detaching (BF-1)
- Fix JSON success-field parsing for awardachievement responses (BF-2)
- URL-encode username/token in sync HTTP requests (BF-3)
- Fix fwrite short-write checks in cache persistence (BF-4)
- Block with timeout when offline queue is full instead of dropping
(BF-5)
Wave 2/3 — Deduplication:
- Extract shared helpers to new header-only ra_util.h (JSON parsing,
URL param extraction, recursive mkdir, login POST, interruptible
sleep)
- Add filtered query variants to ra_offline API to eliminate
caller-side
loops over the full pending-unlock set
- Consolidate duplicate cache-corruption handling
Wave 4 — Concurrency hardening:
- Dedicated ra_probe_mutex for probe thread lifecycle (CH-1)
- New ra_cache_mutex serializing read-modify-write on startsession
cache updates (CH-2)
- Mark cross-thread flag volatile (CH-4, CH-3 skipped: gnu99 has no
_Atomic)
- Lock notification progress-indicator reads (CH-5)
Wave 5 — State machine refactoring:
- SM-1: State enums (RAConnState, RALoginState, RAGameState,
RASyncState)
with derivation functions replacing scattered compound flag checks
- SM-2: Mutex-protected event queue (ra_fsm.c/h) so background threads
post events instead of writing shared flags
- SM-3: FSM becomes authoritative — RADeferredState struct and its
mutex
removed; ra_process_deferred_flags() rewritten as event-driven
dispatch
- SM-4: ra_logged_in/ra_game_loaded booleans replaced with
authoritative
ra_login_state/ra_game_state enums; every transition is an explicit
assignment; fixes stale-state bug on online game-load failure (T7)
…reconnect Two complementary mechanisms to reliably detect connectivity changes: 1. Lightweight WiFi polling (Phase 3 in ra_process_deferred_flags): Checks wpa_cli association state every 5 seconds — no network traffic. Detects WiFi loss (walk out of range, router reboot, interface toggle) within seconds and switches to offline mode, starting the connectivity probe for automatic sync on return. 2. AWARD_GATE fallback: When an awardachievement HTTP request fails and rcheevos retries into a pending-cache hit, the gate now recognizes this as evidence of connectivity loss and transitions to offline mode. Catches edge cases where WiFi appears associated but the route to the internet is broken. Previously, mid-game WiFi loss was only detected after rcheevos exhausted its internal retry backoff and fired RC_CLIENT_EVENT_DISCONNECTED, which could take over a minute — or never fire at all if the AWARD_GATE's synthetic success satisfied rcheevos first. Offline achievements would be recorded in the ledger but never synced.
…m, harden offline subsystem - Fix use-after-free in startsession cache patching (read pre_patch_len before the call that may realloc the buffer) - Add missing init guard to RA_Offline_refreshPendingCache - Replace all cross-thread volatile bool flags with SDL_atomic_t in ra_offline.c (3 flags, ~27 sites), ra_integration.c (3 flags, ~22 sites), ra_sync.c (cancel parameter), and settings.cpp (cancel flag) - Update ra_interruptible_sleep and RA_Sync_syncAll signatures to accept SDL_atomic_t* instead of volatile bool* - Rename ra_fsm to ra_event_queue (files, RA_FSM_* symbols to RA_EVQ_*, internal statics fsm_* to evq_*, log tags, includes, both makefiles) - Namespace sha256 public symbols with nui_ prefix (SHA256_CTX → NUI_SHA256_CTX, sha256_init → nui_sha256_init, etc.) and add extern "C" guard to sha256.h - Fix include guard style (__RA_OFFLINE_H__ → RA_OFFLINE_H, same for ra_sync.h) - Downgrade per-item sync logging from INFO to DEBUG, remove timezone delta diagnostic logging from api.c and both platform.c files - Seed srand() only once in RA_Sync_syncAll via static guard - Document lock ordering, ledger hash chain, and two-tier connectivity probe architecture
On login success, the internal/server username is extracted from the AvatarUrl field so offline unlock signatures use a name the server still recognizes. If AvatarUrl is missing or unparseable, the prior code silently left any previously-cached value in place — so a stale name from an earlier login would keep being used even after a rename. Make CFG_setRAServerUsernameFromAvatarUrl return bool, and at every login/probe success site clear raServerUsername when extraction fails. This lets the existing ra_sync.c fallback select the user-entered username (CFG_getRAUsername) — unlock timestamps may be wrong for renamed accounts, but that's better than signing with a stale name.
…writes raServerUsername signs offline achievement unlocks. It was being re- extracted from AvatarUrl on every successful rc_client background login, every connectivity-probe success, and lazily on first sync via sync_resolve_server_username(). If the RA server ever tightened its rules around renamed accounts, these paths would keep overwriting the stored value with a stale internal name and unlock signing would keep failing with no user-visible recovery. Restrict writes to RA_authenticateSync in ra_auth.c, which is the only place the user consciously enters credentials (settings "Authenticate" menu). Remove the extraction calls from the ra_integration.c login callback and probe, delete sync_resolve_server_username in ra_sync.c, and simplify the sync-site lookup to CFG_getRAServerUsername() with a direct fallback to CFG_getRAUsername() when empty. raServerUsername now reflects the last time the user explicitly authenticated. If unlock signing breaks after a rename, re-running "Authenticate" in settings is the single, obvious recovery path.
RA API responses escape forward slashes in JSON string values, so the AvatarUrl field arrives as "http:\/\/...\/UserPic\/name.png". The parser was only looking for the unescaped "/UserPic/" marker, which worked when the string came pre-decoded from rcheevos (user->avatar_url) but silently failed on raw API response bodies — including the settings "Authenticate" path, which is now the sole writer of raServerUsername. The result was an empty raServerUsername after every settings auth. Try the unescaped marker first, then fall back to the escaped form.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Offline RetroAchievements Support
This PR adds comprehensive offline achievement support to NextUI's RetroAchievements integration. Players can now earn achievements while offline, with automatic synchronization to the RA server when connectivity is restored.
Key Features
Offline Achievement Tracking
Automatic Sync on Reconnect
Manual Sync from Settings
Architecture
Response Cache (
ra_offline.c)Caches RA server responses (login, game data) to disk so games can load offline. Automatically patches cached
startsessionresponses to include pending offline unlocks, ensuring rcheevos shows them as unlocked.Unlock Ledger (
ra_offline.c)Append-only binary log with SHA-256 hash chain integrity:
UNLOCKrecords written when achievements are earned offlineSYNC_ACKrecords written when the server confirms receiptSESSION_START/ENDrecords for session trackingSync Engine (
ra_sync.c)Standalone sync module shared by minarch (background per-game sync) and settings (interactive all-games sync):
SYNC_ACKto ledger on success, preserving unlock timestampsEvent Queue (
ra_event_queue.c)Thread-safe circular buffer for background→main thread communication:
RA_EV_PROBE_ONLINEwhen connectivity is restoredRA_EV_SYNC_DONEwith synced achievement IDsConnectivity Detection (Two-Tier)
PLAT_wifiConnected()to detect link-layer changes without network I/OState Machines
Four coordinated state machines manage the subsystem:
Additional Improvements
volatile boolflags withSDL_AtomicIntthroughout the offline subsystem