Skip to content

Commit 64b3081

Browse files
chore: generate BuildInfo version, remove dead mirror tree, guard broadcast default
- Generate BuildInfo.version from version.value (dynver) so the JSON report's spark_lens_version can never drift from the published artifact; removes the hand-edited constant that was already stale (1.6.4 vs 1.6.5). - Delete the dead mirrored src/ tree at the repo root: a full uncompiled duplicate of core/ + listener/ (excluded via unmanagedSourceDirectories := Nil) that had already drifted out of sync (Reporter.scala differed). - Audited every config-reading analyzer for the AQE-class absent-key bug; none found. Added a JoinAnalyzer regression test pinning that an absent autoBroadcastJoinThreshold reads as Spark's 10 MB default, not disabled.
1 parent 58e7894 commit 64b3081

90 files changed

Lines changed: 39 additions & 11291 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CHANGELOG.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,19 @@ Versioning follows [Semantic Versioning](https://semver.org/).
2121
(mis-fired under default AQE) and `config-aqe-skew-disabled` (was silently suppressed
2222
when AQE was on by default). Added `minorVersion` helper to the `Analyzer` trait.
2323

24+
### Internal
25+
- **`BuildInfo.version` is now generated from `version.value`** (dynver) at compile time
26+
instead of being a hand-edited constant, so the `spark_lens_version` field in the JSON
27+
report can no longer drift from the published artifact version.
28+
- **Removed the dead mirrored `src/` source tree** at the repo root. It was a full,
29+
uncompiled duplicate of `core/` + `listener/` (excluded from the build via
30+
`unmanagedSourceDirectories := Nil`) that had already drifted out of sync and was pure
31+
maintenance risk. All compiled sources live in `core/`, `listener/`, and `testing/`.
32+
- Audited every config-reading analyzer for the same absent-key-misread class of bug the
33+
AQE checks had; no further instances found (other checks use Spark's real defaults,
34+
warn only on explicit misconfiguration, or require runtime evidence). Added a
35+
regression test pinning `JoinAnalyzer`'s absent-`autoBroadcastJoinThreshold` behaviour.
36+
2437
---
2538

2639
## [1.6.5] — 2026-06-11

build.sbt

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,22 @@ lazy val core = (project in file("core"))
7575
"org.slf4j" % "slf4j-api" % "1.7.36" % "provided",
7676
"org.scalatest" %% "scalatest" % "3.2.18" % Test,
7777
),
78+
// BuildInfo.version is generated from version.value (dynver) so the version
79+
// string in the JSON report can never drift from the published artifact.
80+
// Do not check in a hand-written BuildInfo.scala — it will collide with this.
81+
Compile / sourceGenerators += Def.task {
82+
val file = (Compile / sourceManaged).value /
83+
"com" / "github" / "saadaouini" / "sparklens" / "BuildInfo.scala"
84+
IO.write(file,
85+
s"""package com.github.saadaouini.sparklens
86+
|
87+
|/** Build-time constants — generated from build.sbt. Do not edit. */
88+
|object BuildInfo {
89+
| val version = "${version.value}"
90+
|}
91+
|""".stripMargin)
92+
Seq(file)
93+
}.taskValue,
7894
)
7995

8096
// ── listener: SparkLensListener + SparkAppModelBuilder ───────────────────────

core/src/main/scala/com/github/saadaouini/sparklens/BuildInfo.scala

Lines changed: 0 additions & 6 deletions
This file was deleted.

core/src/test/scala/com/github/saadaouini/sparklens/analyzers/JoinAnalyzerSpec.scala

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,16 @@ class JoinAnalyzerSpec extends AnyFlatSpec with Matchers {
2424
issues.exists(_.id.startsWith("join-broadcast-disabled")) shouldBe false
2525
}
2626

27+
// Regression guard: sparkProperties only carries explicitly-set keys, so an absent
28+
// autoBroadcastJoinThreshold must be read as Spark's real default (10 MB, broadcast
29+
// ON), never as "disabled". Mirrors the AQE absent-config fix in ConfigAnalyzer.
30+
it should "not flag broadcast disabled when the threshold is absent (Spark default 10 MB)" in {
31+
val issues = JoinAnalyzer.analyze(app(
32+
sqlExecs = Map(0L -> sqlExec(plan = SMJ_PLAN)),
33+
))
34+
issues.exists(_.id.startsWith("join-broadcast-disabled")) shouldBe false
35+
}
36+
2737
it should "flag SortMergeJoin as Info when broadcast threshold is -1" in {
2838
val issues = JoinAnalyzer.analyze(app(
2939
sqlExecs = Map(0L -> sqlExec(plan = SMJ_PLAN)),
-90.9 KB
Binary file not shown.

src/main/scala/com/github/saadaouini/sparklens/Analyzers.scala

Lines changed: 0 additions & 108 deletions
This file was deleted.

src/main/scala/com/github/saadaouini/sparklens/BuildInfo.scala

Lines changed: 0 additions & 6 deletions
This file was deleted.

0 commit comments

Comments
 (0)