Skip to content

feat: _ANR detector and Sentry reporting Minidump files - #8811

Merged
NickKhalow merged 47 commits into
devfrom
feat/anr-detection
Jun 1, 2026
Merged

feat: _ANR detector and Sentry reporting Minidump files#8811
NickKhalow merged 47 commits into
devfrom
feat/anr-detection

Conversation

@NickKhalow

@NickKhalow NickKhalow commented May 19, 2026

Copy link
Copy Markdown
Contributor

Pull Request Description

What does this PR change?

Features

Key components

  • DclAnrIntegration — Sentry ISdkIntegration that creates and registers the watchdog
  • DclAnrWatchDog (abstract) — Base watchdog with pause/resume awareness, configurable timeout, and minidump collection on report
  • DclAnrWatchDogMultiThreaded — Desktop implementation: a background thread monitors main-thread heartbeats via a coroutine tick counter
  • DclAnrWatchDogSingleThreaded — WebGL fallback: coroutine-based elapsed-time check (single-threaded environment)
  • DclApplicationNotRespondingException — Custom exception type with Sentry mechanism metadata; on Windows, carries the dump file path for attachment
  • ThreadsDumpUtility — Windows-only utility that:
    • Calls the native MiniDumpWriteDump Win32 API (via MiniDumpNative) to generate a .dmp file — no external binaries required
    • Opens a process handle with PROCESS_QUERY_INFORMATION | PROCESS_VM_READ via ProcessInfoNative, with proper IDisposable handle cleanup
    • Polls for file readiness with a 5s timeout to handle async write completion
    • Compresses the dump into a ZIP archive for smaller Sentry attachment payloads
    • Caches Application.persistentDataPath and Application.streamingAssetsPath on the main thread ([RuntimeInitializeOnLoadMethod]) since Unity APIs are unavailable from background threads
    • Provides a Unity Editor menu item (Tools/ProcDump/Dump Current) for quick testing during development
  • MiniDumpNative — Static class wrapping the Dbghelp.dll MiniDumpWriteDump P/Invoke with thread-info, handle-data, and unloaded-modules dump flags
  • ProcessInfoNative / ProcessHandle — Win32 process handle management with IDisposable for safe cleanup
  • DclProcesses.ExecuteBlocking / dcl_start_process_blocking — New native C function for blocking process execution (waits for child exit and returns exit code), available on both Windows (_spawnvp _P_WAIT) and macOS (posix_spawnp + waitpid)

Debug chat commands (registered in DynamicWorldContainer)

  • /anr-simulate [ms] — Freezes the main thread for the specified duration (default 10s) to trigger ANR detection
  • /anr-dump — Manually collects and archives a process dump to the app directory (Windows only); runs on a background thread via ExecuteOnThreadPoolScope

Sentry diagnostics & CI

  • Sentry status loggingDiagnosticInfoUtils now logs whether Sentry is enabled, plus its environment, release, and DSN status at startup
  • enable-sentry PR label — The build-unitycloud.yml workflow now checks for an enable-sentry label on PR builds, allowing Sentry to be activated for specific PRs without requiring a manual workflow input
  • script-debugging PR label / script_debugging workflow input — Enables AllowDebugging build option (forces a Development build). Can be toggled via the script-debugging label on PR builds or the script_debugging input for workflow_dispatch / workflow_call triggers
  • Application arguments formatting — Log output for app args now uses separator lines for better readability
  • S3 upload URL reporting — New CI step prints the artifact upload URL to the job summary for easy access
  • Command error logging — Chat command execution errors are now reported via ReportHub.LogError in addition to the user-facing error message

Native code changes

  • dcl_processes.c / .h — Added dcl_start_process_blocking() for synchronous child process execution; also fixed a CloseHandle leak in get_process_name on Windows (handle was closed after the buffer was freed)
  • DCLProcesses.dll / .exp / .lib — Rebuilt native binaries with the new blocking export
  • DclProcessesNativeMethods.cs — Added managed P/Invoke binding for dcl_start_process_blocking

The built-in Sentry ANR integration is explicitly disabled (DisableAnrIntegration()) to avoid duplicate detection.


Test Instructions

Steps (standard run):

metaforge explorer run 8811

Expected result:

  • Explorer launches and runs normally without false-positive ANR reports

Steps (fresh account):

metaforge account create --clear
metaforge explorer run 8811

Expected result:

  • Same as above — no spurious ANR reports during normal operation

Prerequisites

  • Windows build for minidump verification (macOS dump collection not yet supported)
  • Sentry dashboard access to verify ANR event format and dump attachment

Test Steps

  1. Launch the explorer and navigate normally — no ANR events should fire
  2. Use the /anr-simulate chat command (or /anr-simulate 8000 for 8s) to freeze the main thread and trigger ANR detection
  3. Verify Sentry receives a DclApplicationNotRespondingException event with a .dmp file attachment (ZIP-compressed)
  4. Use the /anr-dump chat command to manually collect a dump and confirm the file paths are printed
  5. In Unity Editor (Windows), use Tools > ProcDump > Dump Current to verify the editor integration works
  6. Confirm the built-in Sentry ANR integration does not fire duplicate events
  7. Add the enable-sentry label to a test PR and verify Sentry is enabled in the resulting build
  8. Add the script-debugging label to a test PR and verify the build includes AllowDebugging and Development flags
  9. Check startup logs for the new Sentry status section (Enabled, Environment, Release, DSN)

Additional Testing Notes

  • macOS does not yet support dump collection — ANR events will report "Dump is not available on macOS yet"
  • WebGL uses a single-threaded fallback (coroutine-based timing) — verify no false positives on that platform
  • The watchdog pauses during app background/focus-loss to avoid false reports
  • Dump file readiness is polled with a 5s timeout — if MiniDumpWriteDump takes longer, the dump will fail gracefully
  • Minidump uses MiniDumpNormal | WithThreadInfo | WithHandleData | WithUnloadedModules flags (equivalent to procdump -mt)

Quality Checklist

  • Changes have been tested locally
  • Documentation has been updated (if required)
  • Performance impact has been considered
  • For SDK features: Test scene is included

Code Review Reference

Please review our Branch & PR Standards before submitting. It explains the automated review flow, QA/DEV approval requirements, and what each label does — especially useful for first-time contributors.

@NickKhalow NickKhalow self-assigned this May 19, 2026
@github-actions

github-actions Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

badge

New build in progress, come back later!

@NickKhalow NickKhalow added force-build Used to trigger a build on draft PR and removed force-build Used to trigger a build on draft PR labels May 19, 2026
@NickKhalow NickKhalow added force-build Used to trigger a build on draft PR clean-build Used to trigger clean build on PR labels May 19, 2026
@NickKhalow NickKhalow added the enable-sentry Enables Sentry integration label May 19, 2026
NickKhalow and others added 2 commits May 20, 2026 19:26
@NickKhalow NickKhalow changed the title feat: anr detection feat: ANR detector and Sentry reporting Minidump files May 21, 2026
@m3taphysics

Copy link
Copy Markdown
Contributor

PR #8811, run #26238775740

Builds: Windows change, Windows baseline, macOS change, macOS baseline

Framework 13 i7

Metric Change Baseline Delta Improvement
Samples 2700 2699
CPU average 33.3 ms 33.3 ms -0.0 ms 0.0%
CPU 1% worst 33.5 ms 33.6 ms -0.1 ms 0.3%
CPU 0.1% worst 33.7 ms 33.8 ms -0.1 ms 0.4%
GPU average 7.2 ms 7.4 ms -0.1 ms 2.0%
GPU 1% worst 18.6 ms 20.4 ms -1.9 ms 9.1% 🟢
GPU 0.1% worst 24.7 ms 26.6 ms -1.8 ms 6.9% 🟢

@claude

claude Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

Claude PR Review — security re-review after minidump collect via native api

Re-reviewing the latest changes on top of the previous review.

  • Read CLAUDE.md and project docs
  • Diff current branch against origin/dev
  • Check the new native minidump path (replaces procdump?)
  • Re-evaluate dependency surface (procdump still bundled?)
  • Re-evaluate workflow changes
  • Post updated review

View job run · branch feat/anr-detection

@NickKhalow NickKhalow changed the title feat: ANR detector and Sentry reporting Minidump files feat: _ANR detector and Sentry reporting Minidump files May 21, 2026
@claude

claude Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

Claude PR Review — security re-review after minidump collect via native api

Re-reviewing the latest changes on top of the previous review.

  • Read CLAUDE.md and project docs
  • Diff current branch against origin/dev
  • Check the new native minidump path (replaces procdump?)
  • Re-evaluate dependency surface (procdump still bundled?)
  • Re-evaluate workflow changes
  • Post updated review

View job run · branch feat/anr-detection

@claude

This comment has been minimized.

@m3taphysics

Copy link
Copy Markdown
Contributor

PR #8811, run #26248729154

Builds: Windows change, Windows baseline, macOS change, macOS baseline

Framework 13 i7

Metric Change Baseline Delta Improvement
Samples 2700 2700
CPU average 33.3 ms 33.3 ms -0.0 ms 0.0%
CPU 1% worst 33.5 ms 34.0 ms -0.5 ms 1.5%
CPU 0.1% worst 34.2 ms 37.0 ms -2.8 ms 7.7% 🟢
GPU average 8.1 ms 8.3 ms -0.3 ms 3.5% 🟢
GPU 1% worst 17.8 ms 19.3 ms -1.6 ms 8.1% 🟢
GPU 0.1% worst 21.4 ms 23.8 ms -2.5 ms 10.4% 🟢

@NickKhalow
NickKhalow changed the base branch from dev to main May 22, 2026 12:31
@balducciv

Copy link
Copy Markdown

Build: v0.149.0-alpha-feat/anr-detection-4b3a988
Tested on: Windows 11 + macOS (Apple M3 Pro)

Passed: ✅

  • Normal operation (old + new account) — no false-positive ANR events on either platform

  • /anr-simulate freezes the main thread and the app recovers normally

  • Sentry status section appears in startup logs (Enabled: True, DSN set)

  • App args formatting with separator lines working

  • WatchDog lifecycle (init → run → clean shutdown) confirmed on both platforms

  • Mac and Windows smoke test passed

Everything I can validate from the client side looks good on both platforms.

Sentry was verified here

Windows with dump
https://decentraland.sentry.io/issues/7497666819/?project=4510719707250688&query=&referrer=issue-stream

MacOS without dump
https://decentraland.sentry.io/issues/7206982691/?project=4510719707250688&query=&referrer=issue-stream
Player-prev tests 2-3-4-5-6.log

Logs:
Player Mac.log
Player old account test 1.log
Player-prev - new account test 1.log

@NickKhalow
NickKhalow changed the base branch from main to dev May 22, 2026 13:12
@NickKhalow
NickKhalow enabled auto-merge (squash) June 1, 2026 11:19
@claude

This comment has been minimized.

@m3taphysics

Copy link
Copy Markdown
Contributor

PR #8811, run #26754316705

Builds: Windows change, Windows baseline, macOS change, macOS baseline

Framework 13 i7

Metric Change Baseline Delta Improvement
Samples 2701 2700
CPU average 33.3 ms 33.3 ms -0.0 ms 0.0%
CPU 1% worst 33.5 ms 33.6 ms -0.1 ms 0.2%
CPU 0.1% worst 34.7 ms 35.0 ms -0.3 ms 0.9%
GPU average 8.3 ms 8.3 ms -0.0 ms 0.2%
GPU 1% worst 17.8 ms 17.9 ms -0.1 ms 0.8%
GPU 0.1% worst 25.7 ms 23.6 ms 2.1 ms -8.8% 🔴

@Ludmilafantaniella
Ludmilafantaniella self-requested a review June 1, 2026 12:57

@Ludmilafantaniella Ludmilafantaniella left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA Re-test Pass ✅

Build: v0.149.0-alpha-feat/anr-detection-hotfix — Windows 11 + macOS (Apple M3 Pro)

Sentry verified:

  • DclApplicationNotRespondingException events arriving correctly in unity-test-environment
  • 12 attachments present (minidump ZIP) ✅
  • Environment tag showing development — no more <REPLACE_ENVIRONMENT> placeholder ✅
  • Legacy Sentry.Unity.Integrations.ApplicationNotRespondingException not firing — no duplicates ✅

Client-side (both platforms):

  • Normal operation, no false-positive ANR events ✅
  • /anr-simulate freezes main thread, app recovers normally ✅
  • Sentry status section present in startup logs ✅

Good to merge 🚀

Image Image Image
8811-evi2.mp4
8811-evi1.mp4

✅Smoke test performed:

  • ✔️ Log In/Log Out
  • ✔️ Backpack and wearables in world
  • ✔️ Emotes in world and in backpack
  • ✔️ Teleport with map/coordinates/Jump In
  • ✔️ Chat and multiplayer
  • ✔️ Profile card
  • ✔️ Camera
  • ✔️ Skybox

@claude

This comment has been minimized.

@claude

claude Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Claude finished @NickKhalow's task in 1m 36s —— View job


Claude PR Review — re-review after pull_request synchronize

  • Diff current branch against the previously-reviewed tip
  • Re-check dependency / native-binary surface
  • Re-check workflow / prompt file changes
  • Check whether previously-flagged carry-over nits were addressed
  • Post updated review

What actually changed since the previous Claude review

Exactly one commit on top of the previously-reviewed tip: 67f94ef0e address chat command. It touches one file — Explorer/Assets/DCL/Chat/Commands/AnrSimulateChatCommand.cs (+3 / -1). The change wraps Thread.Sleep(freezeMs) (line 26) in a #if !UNITY_WEBGL guard with the project's IGNORE_LINE_WEBGL_THREAD_SAFETY_FLAG marker. On WebGL, /anr-simulate now no-ops the sleep but still returns the success string. Reasonable — Thread.Sleep would stall the WebGL single-threaded main loop in a way that's not the simulation people actually want.

git diff f396f4274..HEAD -- '.github/' '**/*.dll' '**/*.exp' '**/*.lib' '**/*.dylib' '**/*.so' 'Packages/manifest.json' 'Packages/packages-lock.json' '**/*.asmdef' is empty. No workflow files, no binaries, no manifests, no asmdefs changed.


Dependency / workflow surface delta

Nothing to add to the previous review. The dependency-supply-chain verdict (DEPENDENCY_REVIEW: PASS) still holds:

Concern Status
External dependencies / native binaries No change since previous review
Workflow / prompt files git diff origin/dev...HEAD -- '.github/' only contains pre-existing changes already reviewed
Asmdef visibility / asmref changes None new

Carry-over nits — re-checked, still standing

The address chat command commit didn't touch any of these:

  • Explorer/Assets/DCL/Chat/Commands/AnrDumpChatCommand.cs:2,6using Cysharp.Threading.Tasks; is still duplicated. Fix this →

  • Explorer/Assets/DCL/Chat/Commands/AnrSimulateChatCommand.cs:15-26ValidateParameters (line 15) only checks parseability; on line 23 the int.TryParse return is ignored. The new WebGL guard mitigates WebGL behavior but does not address the input validation issue: /anr-simulate -1 still freezes for negative values on desktop (some Thread.Sleep overloads throw ArgumentOutOfRangeException), and /anr-simulate 2147483647 is still a ~25-day freeze. Clamp to e.g. [0, 60000] ms and reject invalid input in ValidateParameters. Fix this →

  • Explorer/Assets/DCL/PerformanceAndDiagnostics/Diagnostics/ReportsHandling/Sentry/DclAnrIntegration.cs:58-73Watchdog.OnApplicationNotResponding += is still outside the if (Watchdog is null) guard. If Register runs twice (editor reload, or two integrations registered) the same ANR is reported once per Register call. Move the subscription inside the if (Watchdog is null) block. Fix this →

  • macOS libDCLProcesses.dylib drift / dcl_start_process_blocking dead code — unchanged; ExecuteBlocking still has zero C# callers in this PR. Either rebuild the dylib or remove the export from dcl_processes.{c,h} and DclProcessesNativeMethods.cs if it's not going to be used.

  • DclAnrIntegration.cs (NewDumpAttachment) — on dump failure the original .dmp is not cleaned up; the only cleanup happens in CollectDumpInfoBase64. ANR-path attachments live in Application.persistentDataPath and may accumulate for users that experience repeated ANRs.

  • WebGL fallback still dead code#if !UNITY_WEBGL at the top of DclAnrIntegration.cs excludes the whole file from WebGL builds, so DclAnrWatchDogSingleThreaded (and the inner #else branch at lines 50-54) are never compiled. The "WebGL fallback (coroutine-based)" wording in the PR description no longer matches behavior. Either drop the dead class & branch, or move the file-level guard to only wrap the parts that truly need it.


Verdict

The single commit since the previous review is a one-line WebGL safety guard for the debug /anr-simulate command. No new dependency / workflow / binary risk surface. The non-blocking nits from the previous review are unchanged.

DEPENDENCY_REVIEW: PASS
· branch feat/anr-detection

@NickKhalow
NickKhalow merged commit f95a91b into dev Jun 1, 2026
17 of 18 checks passed
@NickKhalow
NickKhalow deleted the feat/anr-detection branch June 1, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enable-sentry Enables Sentry integration force-build Used to trigger a build on draft PR new-dependency

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants