Debugging Doc: Slow Android Lint Execution Time

[Sandesh282]

Context

Issue: #5886
PR: #6098
Debugging doc: https://docs.google.com/document/d/1RGcf2j_c8Nfh_emBfQOE7ZrlVQyhAuxb-mbPQeaJ_gI/edit?tab=t.0

Issue Details

Describe the problem

The Android Lint script (bazel run //scripts:android_lint_check) takes ~4.5 minutes locally (M2 Mac) and >8 minutes on CI (GitHub Actions Ubuntu 2-core x86_64). The original issue reported 10–20 minute execution times. The goal is to optimize execution without sacrificing correctness.

Screenshots + videos

Not Applicable (script-only, no UI)

Repro steps

1. Check out a fresh branch from develop
2. Run:

bazel run //scripts:android_lint_check -- $(pwd)

3. Observe execution time using --timer flag (measures script-only time, excluding Bazel build phase)

Observed behaviour:
Full lint check takes ~4m 33s (script-only, M2 Mac) or >8m on CI. Individual checks each take ~2 minutes due to fixed project model construction overhead.

Expected behaviour:
Faster execution, ideally under 2 minutes for common workflows.

Error logs

Not applicable (no errors, this is a performance optimization issue)

Where did the problem occur?

• My local machine
• CI checks for PR Fix #5886: Optimize lint execution #6098 on GitHub

How often does it happen?

Every time I perform the repro steps.

Link to GitHub issue

Feature Request: Optimize execution time for Android Lint Script

---

Your Initial Investigation

Online resources

"android lint slow performance large project"

• Android Lint's dominant cost is UAST initialization (Kotlin/Java source file parsing via TopDownAnalyzerFacadeForJVM). No built-in cross-run cache exists for standalone invocations.
• https://googlesamples.github.io/android-custom-lint-rules/usage/performance.md.html
• https://googlesamples.github.io/android-custom-lint-rules/checks/index.md.html

"android lint incremental analysis bazel"

• Gradle's incremental lint (AGP 7.0+) re-analyzes only changed modules, but it's Gradle-specific. rules_android_lint supports persistent JVM workers for Bazel.
• https://github.com/AdrianAndroid/rules_android_lint
• https://bazel.build/remote/persistent

"android-lint-performance-probe"

• Google-provided tool for profiling per-detector costs using bytecode instrumentation. Key to 2x perf improvements in AS 3.3.
• https://github.com/nicholasgasior/android-lint-performance-probe

"android lint --analyze-only --report-only"

• Lint supports splitting analysis and reporting phases, designed for Gradle multi-module partial analysis.
• https://googlesamples.github.io/android-custom-lint-rules/usage/flags.md.html

"large android projects lint optimization"

• Most large projects: (a) accept 5–10 min CI-only, (b) use Gradle incremental module-level lint, or (c) limit local runs to specific checks.
• https://developer.android.com/studio/write/lint

---

Code analysis

https://github.com/oppia/oppia-android/blob/develop/scripts/src/java/org/oppia/android/scripts/lint/AndroidLintRunner.kt#L253-L278

runAnalysis() generates a project description XML and then runs lint against it:

val projectDescriptionFile = generateProjectDescription()
val cliArgs = lintRunner.prepareLintArguments(
  jdkHome = javaConfig.getJdkHome(),
  suppressLintIssues = suppressLintIssues
)
lintRunner.runLint(cliArgs)

The project description is built by LintProjectDescription, which walks every .kt/.java file in all 5 layers (app, domain, testing, utility, data) and writes them as <src> entries:

https://github.com/oppia/oppia-android/blob/develop/scripts/src/java/org/oppia/android/scripts/lint/LintProjectDescription.kt#L491-L527

private val SOURCE_EXTENSIONS = setOf("kt", "java")
fun collectSourceFiles(): List<String> = collectFilesFromDirectory(sourceDir)

config.srcFiles.forEach { srcFile ->
  appendLine("""  <src file="$srcFile"/>""")
}

Android Lint then has to parse ALL of these files via UAST (TopDownAnalyzerFacadeForJVM) before any checks run. This is the ~3.5 minute bottleneck — it happens even for checks that only look at XML/resources.

The lint invocation uses -Wall (all ~400+ checks). 10 checks are suppressed:
https://github.com/oppia/oppia-android/blob/develop/scripts/src/java/org/oppia/android/scripts/lint/AndroidLintRunner.kt#L230-L247

Also noticed: the script creates a per-run cache directory, but the Lint CLI may still internally access ~/.android/cache, which violates Bazel hermeticity.
https://github.com/oppia/oppia-android/blob/develop/scripts/src/java/org/oppia/android/scripts/lint/LintProjectDescription.kt#L252

---

Hypothesis Testing

UAST initialization is the bottleneck — source file parsing dominates even for XML-only checks

Confirmed

Ran --checks=HardcodedText (an XML-only check) with and without .kt/.java source files in the project description.

• Without sources: 43s
• With sources: 9m 43s

Same pattern for IconDensities:

• 11s vs 9m 22s

Re-ran with --timer to isolate script time from Bazel build overhead.

• Full check: 4m 33s
• Without sources: 0m 58s

That's a ~4.7x speedup; ~92% of runtime is spent on source parsing for checks that don't need it.

The UAST module (TopDownAnalyzerFacadeForJVM) parses every .kt/.java file before any detectors run, regardless of which checks are enabled.

This only helps when running source-free checks though; checks like NewApi need sources.

---

Some individual detectors are disproportionately expensive

Ruled out

Profiled 22 representative checks individually using --checks=X --timer (mix of source-only, XML, icon, manifest checks).

• All ranged from 1m 49s to 2m 23s (~34s variance)

Running all 150 checks together:

• 4m 38s

So ~2 minutes is fixed overhead (project model construction) and all 150 detectors combined add only ~2.5 more minutes.

No outlier. Disabling individual checks won't help.

---

Running too many irrelevant checks via -Wall

Under investigation

Enumerated all 183 active checks using --list.

Breakdown:

• ~91 don't need sources (52 XML/resource, 15 icon, 24 manifest)
• ~60 need sources
• ~25 are Gradle-only (irrelevant for Bazel)
• ~7 misc

Of the 17 checks currently firing on Oppia:

• 13/17 are file-scoped
• 16/17 get no value from third-party dependency scanning
• 16/17 don't need generated code

Ben asked (Mar 19) for the full grouped list of all ~400 checks (useful vs not-useful), then incremental vs holistic breakdown. Also wants incremental perf for a single expensive check. Still working on this.

---

~/.android/cache hermeticity violation

Untested

Ben flagged that the Lint CLI may access ~/.android/cache outside our per-run cache dir, violating Bazel hermeticity.

Potential fixes:

• ANDROID_LINT_CACHE env var
• Override $HOME
• Two-phase Bazel integration

---

Solution

(Not yet solved. Next steps: full check categorization, incremental benchmarking)

[BenHenning]

I don't have many specific suggestions yet, but the write-up so far is excellent and I like the direction this is going.

Note that the environment variable override will be harder than it seems because we directly call to the linter rather than running it in a separate process. I tried doing this earlier in the week by copying what we do in BundleToolClient since that actually copies the current classpath to run a Java application using CommandExecutor (which also simplifies some of the lint runner due to auto-managing timeouts), but it didn't work and I didn't have time to dig into why.

I think broadly moving toward a very fast incremental run model for local runs and a full run model on CI would be a massive improvement over the current situation. Assuming that incremental rarely misses anything, the CI run is a good backup in case incremental fails but incremental encourages checking changes before uploading (which is broadly a good practice since CI can take a long time to run these days). Not sure if this will actually be feasible, but I'm keen to see the investigation results once more info is uncovered.

[Sandesh282]

Thanks @BenHenning, glad the direction resonates!

On the env var / BundleToolClient approach

Good to know about the subprocess limitation. Since we're calling LintCli().run() directly in the same process, env vars set through a ProcessBuilder wouldn't even apply. I'll dig into whether lint reads any system properties or has some internal config hook for redirecting cache paths when called programmatically. If that doesn't pan out, the two-phase Bazel approach you mentioned earlier (cache as a Bazel artifact, then feed it into the analysis rule) is probably the right long-term answer. That's a bigger change though, so I'll tackle it as a separate follow-up.

Next steps on the investigation

I'm planning to work through these in order:

1. Complete the full check categorization. I'll extend the current 183-check breakdown to cover all ~400 checks, grouped into: useful+file-scoped, useful+project-scoped, Gradle-only, Compose/Navigation-only and already-suppressed. I'll use the lint checks index + detector source to classify each systematically rather than running them individually.

2. Benchmark incremental mode. I'll pick a source-requiring check (likely NewApi) and compare full project vs. 5–10 changed-files-only runs. This should give concrete numbers on the incremental speedup potential.

3. Sketch the two-mode design. Based on the categorization + benchmarks, I'll draft a concrete proposal for:

   • --incremental (local): run file-scoped checks on changed files only (target: <30s)
   • Default (CI): full analysis, but with clearly-irrelevant checks (Gradle/Compose) disabled

I'll update the debugging doc and post results here as I go. Aiming to have (1) and (2) done within the next few days.

[Sandesh282]

Update: Full check categorization + incremental benchmark

I extended the 183-check breakdown to cover all ~400 built-in checks and ran the incremental benchmark. Here are the results.

Active checks breakdown (183 from --list)

Category	Count	Needs sources?	Incremental?
XML / Resource	~52	❌	✅ File-scoped
Icon / Drawable	~15	❌	✅ File-scoped
Manifest	~24	❌	✅ File-scoped
Source code (file-scoped)	~45	✅	✅ Changed files only
Source code (project-scoped)	~15	✅	❌ Needs full project
Gradle-specific (irrelevant)	~25	N/A	N/A
Misc (irrelevant)	~7	N/A	N/A

~91 checks don't need sources at all -- confirmed by earlier benchmarks (58s without sources vs 4m33s with).

Source-requiring checks: file-scoped vs project-scoped

File-scoped (~45) -- incremental candidates (can run on changed files only):

NewApi, InlinedApi, ObsoleteSdkInt, SimpleDateFormat, DefaultLocale, Wakelock, WakelockTimeout, HandlerLeak, StaticFieldLeak, DrawAllocation, ViewHolder, Recycle, CommitPrefEdits, CommitTransaction, ShowToast, MissingSuperCall, CheckResult, ClickableViewAccessibility, SetTextI18n, CustomSplashScreen, VisibleForTests, Range, WrongConstant, WrongThread, WrongCall, ResourceType, ServiceCast, ParcelCreator, ViewConstructor, StopShip, GetInstance, TrustAllX509TrustManager, SetJavaScriptEnabled, SecureRandom, HardwareIds, SdCardPath, UnspecifiedImmutableFlag, UnspecifiedRegisterReceiverFlag, InflateParams, MergeMarker, PrivateApi, DefaultEncoding, BidiSpoofing, AuthLeak, CustomX509TrustManager

Project-scoped (~15) -- need full project context:

Check	Why project-scoped
OnClick	XML→source cross-ref (verifies handler method exists)
RestrictedApi	Needs @RestrictTo annotations from library classpath
MissingPermission	Source→manifest cross-ref for permissions
Registered / Instantiatable / MissingClass	Manifest↔source cross-ref
MissingInflatedId / WrongViewCast	Layout↔source cross-ref
SwitchIntDef	Type resolution across files
Overdraw	View hierarchy analysis
InvalidPackage	Full classpath resolution
WrongThreadInterprocedural	Cross-method analysis
UnknownId / DuplicateIncludedIds / ResourceCycle	Cross-file references

Irrelevant checks (~32)

~25 Gradle-specific (GradleCompatible, GradleDependency, GradleDynamicVersion, etc.) + ~7 misc. These are active but never produce findings in a Bazel project. Explicitly disabling via --disable is a free optimization.

Remaining ~200+ built-in checks (not in --list)

The full index has ~700+ entries, but most beyond our 183 are library-bundled (Compose ~40, Moshi ~30, Navigation ~10, Timber ~8, Wear/TV/ChromeOS ~25, etc.) — they only activate when the library's lint JAR is on the classpath. Since Oppia doesn't use those libraries, they're already inactive and add zero cost.

Incremental benchmark results

Tested NewApi (representative source-requiring, file-scoped check) with limited source files vs the full project:

Scenario	Source files	Checks	Time
Full lint (baseline)	1,713	All (~183)	4m 38s
Incremental (NewApi, limited files)	~50 (10/layer)	1	54s

5.1x speedup. The 54s breaks down as:

• ~30s: project model construction (Bazel queries, AAR extraction, dependency resolution)
• ~24s: UAST parsing + NewApi analysis on ~50 files

With project model caching (dependencies don't change between runs), the incremental run could be under 30s.

Proposed two-mode architecture

Local / --incremental (target: <30s):

1. Run ~91 source-free checks on all XML/resource/manifest/icon files (~1 min today, cacheable)
2. Run ~45 file-scoped source checks on changed files only (seconds, validated by benchmark above)
3. Skip ~15 project-scoped checks (caught on CI)
4. Disable ~32 irrelevant checks (Gradle + misc)

CI / default (modest improvement):

1. Full analysis as today
2. Disable ~32 irrelevant checks (free speedup)

On the ~/.android/cache hermeticity issue

Since LintCli().run() is in-process, env var overrides via ProcessBuilder don't apply. I'll investigate whether lint respects system properties or internal config hooks for redirecting cache paths programmatically. If not, the two-phase Bazel approach is the right long-term fix; will tackle as a separate follow-up.

@BenHenning Would love your thoughts on the categorization and the two-mode approach.

[BenHenning]

Thanks @Sandesh282. This looks promising. I think you can proceed with this approach and we can evaluate further in the PR itself. A few additional thoughts:

• We do use Moshi and Compose so the bit about "Since Oppia doesn't use those libraries..." seems incorrect, at least in part.
• It may be possible to run the LintCli tool as its own process with CommandExecutor by cloning the current process's classpath like we do for bundletool but as I mentioned earlier this may have some additional work needed since my attempt to try it didn't "just work" (the lint tool seemed to hang but I didn't investigate it in depth).