Skip to content

Fix dark cluster URI rewrite deopt storm consuming 80% CPU #1160

Open
jcleezer wants to merge 1 commit intomasterfrom
jleezer/dc-deop-storm
Open

Fix dark cluster URI rewrite deopt storm consuming 80% CPU #1160
jcleezer wants to merge 1 commit intomasterfrom
jleezer/dc-deop-storm

Conversation

@jcleezer
Copy link
Copy Markdown
Contributor

@jcleezer jcleezer commented Apr 7, 2026

Summary

  • Adds a skipReEncoding fast path to D2URIRewriter that constructs rewritten URIs directly from raw (already percent-encoded) components via StringBuilder + URI.create, bypassing the expensive character-by-character
    UriComponent._encode loop.
  • Opts in from DarkClusterManagerImpl only (new D2URIRewriter(configuredURI, true)). All other callers use the default constructor and are completely unaffected.
  • Existing UriBuilder-based path is preserved unchanged behind skipReEncoding=false (the default).

Problem

Production profiling shows DarkClusterFilter.onRestRequestDarkClusterManagerImpl.rewriteRequestD2URIRewriter.rewriteURIUriBuilder.replaceQueryUriComponent._encode consuming ~80% of total CPU.

The root cause is a JIT deoptimization storm on _encode:

  • The method has bimodal branching (StringBuilder sb = null fast path vs slow path) that C2 speculatively compiles for one mode
  • Varying query strings across requests invalidate the speculation → uncommon trap → deopt to interpreter
  • Running the character-by-character loop interpreted over large query strings is 10-100x slower
  • Recompilation uses contaminated profiles → the cycle repeats

The re-encoding is also unnecessary: getRawQuery() returns already percent-encoded strings, so _encode scans every character only to leave them unchanged.

Fix

rewriteURIFromRaw eliminates the problem entirely:

  • No character-by-character iteration — StringBuilder.append(String) is a JIT intrinsic (System.arraycopy)
  • No bimodal branching — no speculative optimization to get wrong
  • Monomorphic call sites — stable JIT targets with no deopt triggers

Test plan

  • Added skipReEncoding tests to TestD2URIRewriter (simple rewrite, encoded query params, no query, with fragment)
  • Added skipReEncoding tests to TestDarkClusterUrlRewrite (basic, query params, encoded params, parity with default)
  • Tests explicitly assert fast path produces identical results to the default UriBuilder path
  • ./gradlew :d2:test --tests TestD2URIRewriter :darkcluster:test --tests TestDarkClusterUrlRewrite passes

…append(String) is an intrinsic, a single System.arraycopy

  - No bimodal branching — no speculative optimization to get wrong
  - Monomorphic call sites — StringBuilder.append and URI.create are stable JIT targets with no polymorphic dispatch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant