Ch. 4: Research Design and Methodology — Notes

[jathavaan]

Purpose

Working notes for Chapter 4: Research Design and Methodology of the master thesis
"Benchmarking Cloud-Native and Traditional Geospatial Technologies."

Use this thread to draft, discuss, and refine content for the following sections:

Sections covered

• 4.1 Overview and experimental design (factors, levels, replication, randomization)
• 4.2 Workloads (taxonomy, definitions, iteration counts)
• 4.3 Metrics and cost model (primary/auxiliary metrics, distributed phase metrics, cost model)
• 4.4 Statistical analysis (measurement-quality terminology, summary statistics, uncertainty quantification, pairwise comparison and effect size, answering RQ1–RQ3)
• 4.5 Threats to validity

Key design decisions to document

• Two-axis factorial design: cloud-native vs. traditional × single-node vs. distributed
• Three methodological commitments: end-to-end network transfer as primary outcome, identical workloads across stacks, single-node and distributed engines on shared axes
• Sequential stopping rule (bootstrapped CI) vs. fixed iteration count for long-running benchmarks
• Choice of Vargha–Delaney Â₁₂ effect size over parametric alternatives
• Threat-to-validity analysis scope

How to use this thread

Post notes, draft paragraphs, open questions, and revision requests as comments.

[jathavaan]

From issue #292 — Sequential stopping rule

§3.2 Benchmarking methodology for data systems

• Add the sequential stopping rule as a methodological contribution: bootstrapped 95% CI on the mean elapsed time (Efron & Tibshirani 1993), target 5% relative half-width, 10-iteration floor, 60-second timed-window floor (Azure Monitor granularity constraint), soft ceiling at the BenchmarkIteration value, hard timeout at 1 hour.
• Cite Georges, Buytaert & Eeckhout (2007) "Statistically Rigorous Java Performance Evaluation" (OOPSLA) as the methodological precedent for relative-precision sequential stopping in performance benchmarks.
• Distinguish the two stopping branches: sequential (data-driven, for high-frequency single-machine queries) vs fixed-count (for long-running low-variance national-scale joins).

§4.1.2 Replication and randomization

• Note that the bootstrap RNG is seeded deterministically from (run_id, query_id) via blake2b, so the stopping point is reproducible across reruns.

§4.2.3 Iteration counts

• Replace the fixed-iteration-count description with the sequential stopping rule. Iteration count is now an outcome of measured variance, not an input.
• Report achieved_iterations from the metadata as the actual sample size per benchmark cell, not the BenchmarkIteration ceiling.
• Document that national-scale spatial joins retain a fixed count of 5 with warmup_iterations=1 (reduced from 5 warmups to 1).

§4.5 Threats to validity

• The 60-second floor is a compromise between Azure Monitor metric granularity and stopping efficiency. Acknowledge that very fast queries (sub-ms) may spend most of their wall-clock budget in the floor window rather than precision-seeking.
• The bootstrap CI on <30 samples is noisy; the 10-iteration MIN_ITERATIONS floor mitigates but does not eliminate this.

bibliography.bib

• Add Efron & Tibshirani (1993), An Introduction to the Bootstrap, Chapman & Hall.
• Add Georges, Buytaert & Eeckhout (2007), Statistically Rigorous Java Performance Evaluation, OOPSLA.

[jathavaan]

From issue #310 — Stop reason fix on warmup failure

Minor. The fix ensures benchmark_metadata.parquet correctly tags warmup failures as stop_reason="failed" rather than "fixed". If the thesis reports metadata filtering logic (e.g. "we excluded runs with stop_reason='failed'"), this fix makes that filter accurate for the non-sequential branch (Databricks, DuckDB, PostGIS national-scale joins). No new section needed — just ensure the analysis pipeline uses the corrected metadata from post-#311 runs.

[jathavaan]

From issue #312 — Transient failure tolerance (Ch.4 sections)

§4.2.3 Iteration counts

• Document the transient-failure tolerance: failed iterations are recorded as status="failed" per-iteration samples but do not abort the run. Only 3 consecutive failures abort (BENCHMARK_MAX_CONSECUTIVE_FAILURES). Runs with mixed results are tagged stop_reason="partial".
• Clarify that mean, median, and bootstrapped CI are computed over successful samples only. Failed iterations contribute to failed_iterations count but never to the elapsed-time distribution.
• The MIN_ITERATIONS floor for the sequential stopping rule now gates on successful sample count, not raw iteration count. This means that if 2 of 12 iterations fail, the CI check waits until 10 successful samples exist.

§4.5 Threats to validity

• Add a paragraph on the distinction between deterministic resource-exhaustion failures (e.g. Spark OOM on default-2-nodes-large, PostGIS OOM on 100M-row joins) and transient cluster flakiness (e.g. Spark ExecutorLostFailure on partitioned-12-nodes-large that succeeds on retry). The framework now tolerates the latter but cannot distinguish the two at runtime — classification is left to downstream analysis via failed_iterations count and the per-iteration failure_reason string.
• Document that stop_reason="partial" runs are included in the analysis with the caveat that their sample is smaller than intended and may be drawn from a non-representative subset (e.g. if the first iteration failed and the remaining 4 succeeded, the distribution excludes whatever the first iteration would have measured).

[jathavaan]

From issue #306 — GiST index + ANALYZE after seed

§4.5 Threats to validity

Acknowledge planner-statistics staleness as a former confound. Suggested row for Table 4.5.1:

Threat	Type	Mitigation
PostgreSQL planner statistics stale after bulk insert	Construct	Explicit ANALYZE after every seed; GiST index on both sides of the spatial join

[jathavaan]

From issue #309

Section 4.1.1 — Factors and levels (p. 41, Table 4.1.1)

Table 4.1.1 lists RQ2 worker count levels as 2, 4, 8, 12, 16. After #309 the default strategy is tested only at 2, 8, 16 (large tier). broadcast and partitioned are unchanged.
Action: add a footnote or note that the default baseline covers a 3-level subset (2 / 8 / 16) of the full 5-level grid.

Section 4.2.1 — Workload taxonomy (p. 42)

Text says the spatial join "resolves through either broadcast or partitioned execution." The default strategy (CBO-chosen plan) is implicitly omitted from this sentence but is still tested. Consider clarifying that default is a third variant retained at a reduced matrix as a within-Sedona illustration.

Section 4.4.6 — Answering RQ2: Scaling against single-node (p. 51)

Text reads: "Speedup and parallel efficiency are computed as functions of worker count across the levels 2, 4, 8, 12, and 16 nodes. Each Sedona execution strategy is treated separately."
After #309, the default strategy only has data at 2, 8, and 16 nodes. The speedup/efficiency curve for default is drawn from 3 points instead of 5. Action: note that the default scaling curve is sampled at 3 levels; the full 5-level grid applies to broadcast and partitioned only.

[jathavaan]

From PR #326

§4.5 Threats to validity

The internal-validity paragraph (p 53) lists four measures that limit the shared-infrastructure confound: concurrent pairing, seeded order randomization, warmup, and a fresh cluster per distributed run. It does not address what happens when a batch member fails mid-run (e.g. Databricks quota exhaustion). PR #326 adds orchestrator-level detection for this: when one or more members of a batch crash, the surviving peers complete without their matched counterpart, weakening the concurrent-pairing assumption. The orchestrator now logs whether the failure was partial (some peers survived) or total (entire batch lost).

Suggested addition to Table 4.5.1 or the internal-validity paragraph: a note that partial batch failure is detected at the orchestrator level and recorded, so affected batches can be identified during analysis. The threat is that surviving peers ran under the same wall-clock window but without the contention their paired counterpart would have introduced, which could bias their measurements favorably.

[jathavaan]

From PR #328

§4.5 Threats to validity — Table 4.5.1 (p54)

Add a new construct validity row:

Validity	Threat	Mitigation
Construct	Databricks Photon engine silently bypasses Sedona's custom Catalyst strategies (JoinQueryDetector), causing spatial joins to fall back to BroadcastNestedLoopJoin — a brute-force cross product with no spatial index. The physical plan changes without any error or warning, so the benchmark measures a fundamentally different operation than intended.	Photon disabled on all Databricks clusters (spark.databricks.photon.enabled=false). Physical plan verified via result.explain(True) to confirm RangeJoin (partitioned) or BroadcastIndexJoin (broadcast) appears in the plan.

This threat was discovered during development: a partitioned join on the small dataset with 4 workers ran for 1.3+ hours on a single task instead of minutes across all executors.

§4.2.3 Iteration counts (p44)

The section states that the distributed configuration runs seven iterations. Add a note that a per-iteration hard timeout (3600 s) stops the benchmark if any single iteration exceeds the threshold, preventing runaway iterations from consuming cluster resources indefinitely.

[jathavaan]

From issue #254

Section 4.1.1 — Factors and levels

Table 4.1.1 lists "Client compute" as a constant factor ("Same instance sizing and process"). The concrete sizing is now 4 vCPU / 16 GB for all ACI containers (changed from 3 vCPU / 8 GB). The table itself is abstract enough that no text change is needed, but when cross-referencing Ch. 5 (where concrete values are stated), ensure alignment with the new spec.

Section 4.1.2 — Replication and randomization

Text mentions "identically provisioned client instances" and defers "container sizing" to Ch. 5. No change needed here — the statement remains true, and the new uniform 4/16 spec actually strengthens the "identically provisioned" claim by also matching the Databricks worker class.

[jathavaan]

From issue #254

§4 — Benchmark selection: Sedona default join strategy dropped

The three Sedona default-strategy national-scale spatial join benchmarks (national-scale-spatial-join-databricks-default-{2,8,16}-nodes-large) have been removed from the experiment suite.

Reason: These benchmarks were intended to serve as a baseline for Sedona's spatial join performance without an explicit broadcast or repartition hint. In practice they:

1. Took too long — single iterations routinely exceeded the 60-minute hard timeout, making data collection impractical.
2. Cost too much — each timed-out run still consumed full Databricks cluster hours with no usable sample.
3. Failed too often — transient Spark executor losses compounded with the long execution times, producing repeated consecutive-failure aborts.

The thesis should acknowledge this in the methodology chapter: the default join strategy was considered as a baseline but excluded because the per-iteration cost and failure rate made it infeasible to collect reliable measurements within the budget and time constraints of the study. The broadcast and partitioned strategies remain and provide the Sedona comparison points against DuckDB and PostGIS.

[jathavaan]

From issue #254 — Hard timeout increased to 450 minutes

§4.2.3 Iteration counts

The cumulative hard timeout (BENCHMARK_MAX_TIMED_WINDOW_SECONDS) has been increased from 5400 s (90 min) to 27 000 s (450 min / 7.5 h). This same constant also serves as the per-iteration timeout.

Supersedes prior notes: the #292 comment describes the hard timeout as "1 hour" and the #328 comment cites "3600 s" — both now outdated. Update the stopping-rule description to state the 450-minute hard timeout for both the per-iteration and cumulative checks.

Why: 90 minutes was too tight for long-running national-scale spatial joins on larger dataset tiers, especially on Databricks where cluster startup and executor provisioning consume part of the window. The 5× increase gives enough headroom for slow iterations to complete rather than being killed as timeouts, producing usable samples instead of discarded TIMEOUT stops.

[jathavaan]

From issue #297 — *_local.py uses GeoPandas, not DuckDB+Shapefile

§4.1.1 Factors and levels (Table 4.1.1)

The "Configuration" factor lists DuckDB + Shapefile as the fourth RQ1 level. The code's *_local.py entrypoints use GeoPandas (gpd.read_file), not DuckDB's ST_Read GDAL bridge. Rename to GeoPandas + Shapefile in the table.

§4.2.1 Workload taxonomy / Figure 4.1.1

Any reference to "DuckDB + Shapefile" as a configuration should read "GeoPandas + Shapefile".

§4.4.5 Analysis framing

The interpretation assumes the same engine (DuckDB) across both format paths. Since the Shapefile path uses GeoPandas, acknowledge that the comparison conflates engine and format differences: a slower *_local result could be either the Shapefile format or the GeoPandas engine. Add this as a construct-validity caveat.

§4.5 Threats to validity

Add a row to Table 4.5.1:

Validity	Threat	Mitigation
Construct	RQ1 Shapefile configuration uses GeoPandas engine instead of DuckDB, conflating engine and format differences	Acknowledged as a limitation; results for this configuration reflect the GeoPandas+Shapefile stack rather than DuckDB+Shapefile

[jathavaan]

From issue #298 — National-scale join iteration count: thesis says 7, code says 5

§4.2.3 Iteration counts

Text states "the national-scale join runs for seven iterations." Code has NATIONAL_SCALE_SPATIAL_JOIN = 5. Update to five iterations and note that the count was reduced from 7 to 5 after the section was written, due to per-iteration cost (cluster time) outweighing precision gains at the additional two sample points.

[jathavaan]

From issue #301 — BBox workload has undocumented area predicate

§4.2.2 Workload definitions — Bounding-box filtering

Text says the workload "returns every building whose geometry intersects [the window]." Code adds a compound predicate: ST_Intersects plus ST_Area(ST_Transform(geometry, 25832)) > 10 (projected area > 10 m²). Update the description to acknowledge the area filter and explain its motivation (excluding degenerate micro-polygons).

Also note that the ST_Transform to EPSG:25832 introduces a projection cost that varies between engines (DuckDB / PostGIS / GeoPandas implementations differ). The "fixed selectivity" claim needs re-justification since the second predicate's selectivity depends on the building-size distribution per dataset tier.

§4.5 Threats to validity

Add a row to Table 4.5.1:

Validity	Threat	Mitigation
Construct	BBox workload includes an undocumented area-filter predicate (> 10 m²) beyond ST_Intersects, changing selectivity and adding projection cost	Documented in workload definition; area cutoff is constant across configurations so relative comparison is preserved

[jathavaan]

From issue #299 — Distributed benchmarks run warmup despite §4.1.2 saying they don't

§4.1.2 Replication and randomization

Text states: "The distributed configuration is the one exception to the warmup discipline. [...] They are therefore omitted." Code does the opposite — _databricks_benchmark_runner.py runs with skip_warmup=False, executing BENCHMARK_WARMUP_ITERATIONS warmup iterations before timed iterations against the same persistent cluster.

Update §4.1.2 to acknowledge that warmup is run against the reused cluster, and remove the "first-run variance carried into analysis" framing.

Also update the cost rationale: §4.1.2 says "a discarded warmup run would provision an entire cluster only to throw the result away," but the code provisions one cluster and reuses it for warmup + timed iterations. The provisioning-cost argument does not apply.

§4.4.5 / §4.4.6 — Analysis

Table 4.5.1 references the warmup asymmetry ("Warmup is omitted for [distributed]... its first-run variance is carried into the analysis"). Since warmup is actually run, this row should be removed or revised — the timed iterations measure a warm engine, not a cold cluster.

§4.5 Threats to validity

Remove or revise any row in Table 4.5.1 that treats distributed-benchmark cold-start variance as an acknowledged threat, since warmup eliminates it in practice.

[jathavaan]

From issue #303 — PiP workload uses 10 mixed probes, not a single fixed point

§4.2.2 Workload definitions — Point-in-polygon

Replace "tests a fixed query point" with:

The point-in-polygon workload tests a fixed set of 10 probe points against the building polygons. Three points (30%) are guaranteed-inside (sampled from the dataset's own building centroids), and seven (70%) are uniform-random within the Trondheim bounding box. The probe set is seeded (POINT_IN_POLYGON_PROBE_SEED = 42) for reproducibility. The reported time per iteration is the sum of all 10 probes.

§4.4.5 Analysis framing

Note that per-probe latency is averaged across the 10-point set. The 30/70 inside/outside mix exercises both fast-reject and confirmed-hit paths of the spatial index, which is a deliberate design choice not captured in the original "fixed query point" description.

[jathavaan]

From issue #304 — PiP queries return COUNT(*), not containing polygons

§4.2.2 Workload definitions — Point-in-polygon

Replace "returns the polygon or polygons containing it" with:

...and returns the number of polygons containing each probe point.

§4.3.2 Metrics — Result cardinality

Add a note that for PiP, result_cardinality equals the number of probes (10), not the number of matching polygons. The COUNT(*) aggregation collapses the polygon payload to a scalar per probe.

§4.4.5 Analysis framing — Bytes received

Note that PiP's bytes_received reflects only the scalar count (4–8 bytes per probe), not polygon-payload transfer. The §4.4.5 interpretation ("formats that a client can read in part through range requests... are expected to separate first on bytes received") does not apply to PiP — the result-set transfer term is degenerate for this workload.

[jathavaan]

From issue #305 — BENCHMARK_RUNS = 1, reproducibility analysis has no data

§4.4.1 Repeatability vs reproducibility (Table 4.4.1)

The reproducibility row in Table 4.4.1 cannot be filled — there is no second run on a separately-provisioned host. Drop the reproducibility row or mark it as "not exercised in this study."

§4.1.2 Replication and randomization

Remove or revise the sentence: "the seed is reused across runs to ensure reproducibility." With BENCHMARK_RUNS = 1, there are no seeded re-runs. The seed still ensures deterministic iteration order within the single run.

§4.4.3 Reproducibility references

Remove the claim "the seeded re-runs probe reproducibility instead" — no re-runs exist. Frame the design as repeatability-only: within-run iterations provide the variance estimate, but cross-run reproducibility was not exercised due to cost constraints.

§4.4.6 Multi-run variance

Remove the multi-run variance treatment ("warm-cluster measurements do not satisfy the independence that the single-node samples approximate"). With one run, there is no multi-run variance to treat.

§4.5 Threats to validity

Add a row:

Validity	Threat	Mitigation
External	Single benchmark run; reproducibility across independent provisioning events not quantified	Acknowledged as a limitation; within-run repeatability (bootstrap CI on timed iterations) is the sole variance estimate

[jathavaan]

Correction — issues #303, #304, #305

The earlier comments for these three issues are superseded. The thesis is the source of truth — code will be updated to match:

• PiP workload uses 10 mixed probes; methodology says "fixed query point" #303: Code will use a single fixed query point, not 10 mixed probes. Config constants POINT_IN_POLYGON_TOTAL_POINTS, INSIDE_RATIO, PROBE_SEED will be removed.
• PiP queries return COUNT(*) instead of containing polygons #304: Queries will return actual polygons (SELECT *) instead of COUNT(*). result_cardinality and bytes_received will then carry the meaning §4.3.2 / §4.4.5 assign them.
• BENCHMARK_RUNS = 1 prevents reproducibility quantification per §4.4.1 #305: BENCHMARK_RUNS will be bumped to enable the reproducibility analysis described in §4.4.1 / §4.4.3.

No thesis text changes needed for these three issues.

[jathavaan]

From PR #343

§4.3.4 Cost model (p. 46–47)

The text correctly states: "The egress term is treated as near-zero under one modeling assumption: that transfer remaining within a single region and tenant is not billed as outbound traffic. A charge therefore arises only when the distributed engine moves data across a region boundary."

PR #343 now implements this charge. Consider adding a sentence after the quoted passage, e.g.:

In the present design, the Databricks/Sedona configuration is the only one that crosses a region boundary: executors in Sweden Central read GeoParquet from Blob Storage in Norway East, incurring intra-Europe cross-region egress at $0.02/GB. All other configurations remain intra-region and intra-tenant, so their egress term stays zero.

Also: Equation 4.1 uses c_compute · t with t as a scalar duration. For Databricks, c_compute must now be understood as the per-node rate multiplied by the total node count (workers + 1 driver), not just the worker count. Consider a footnote clarifying that the driver node is included in the cluster size when applying Equation 4.1 to Databricks.

§4.4.6 Answering RQ2: Scaling against single-node (p. 51–52)

The crossover analysis compares distributed cost against single-node cost. The driver node fix increases Databricks cost asymmetrically by cluster size (33% for 2-worker, 20% for 4-worker, 6% for 16-worker). Smaller clusters are relatively more expensive than previously reported, which could shift crossover points rightward. If results are re-collected with the corrected cost model, note that the crossover may occur at a higher worker count than earlier drafts suggested.

§4.5 Threats to validity (p. 52–54)

Table 4.5.1 lists under External validity: "The distributed configuration alone may cross a region boundary and so incur an egress charge the single-node configurations avoid (Section 4.3.4)." This was listed as an accepted asymmetry.

PR #343 now corrects this: cross-region egress is measured and priced at $0.02/GB for Databricks blob reads. Update Table 4.5.1 to reflect that this is no longer an unmeasured asymmetry — it is now an explicit cost term. The residual threat is that the $0.02/GB rate is a hardcoded constant that could become stale if Azure pricing changes.

[jathavaan]

From issue #341

Section 4.1.1 — Factors and levels (p. 40–41)

• Table 4.1.1, Worker count row: Levels currently read "2, 4, 8, 12, 16". The 12-node level has been removed from the codebase (issue Reduce databricks runs #341). Update to 2, 4, 8, 16.
• Body text (p. 40): "RQ2 varies the worker count of the distributed engine across five levels" → change to four levels.

Section 4.2.2 — Workload definitions (p. 44)

• Table 4.2.1, Point-in-polygon / k-nearest-neighbor / Bounding-box filter rows: The Configurations column lists "Sedona + GeoParquet" alongside the single-machine engines. No Sedona single-machine entrypoints exist in the codebase (only multi-node national-scale-spatial-join). Remove "Sedona + GeoParquet" from these three RQ1 rows.
• Table 4.2.1, Spatial join row: Configurations column reads "partitioned; 2–16 workers". The range 2–16 is still correct (2, 4, 8, 16), but if the table enumerates individual levels anywhere, remove 12.

Section 4.2.3 — Iteration counts (p. 44)

• Text states the national-scale join "runs for seven iterations". Code uses BenchmarkIteration.NATIONAL_SCALE_SPATIAL_JOIN = 3 (3 timed iterations) + 1 warmup = 4 total. Update the thesis to match.
• A new cumulative wall-clock ceiling (BENCHMARK_MAX_FIXED_WINDOW_SECONDS = 4500, 75 minutes) now caps the fixed-iteration branch so a pathological iteration cannot run away into unbilled node-hours. This safety mechanism should be mentioned alongside the iteration count.

Section 4.4.6 — Answering RQ2: Scaling against single-node (p. 51)

• "Speedup and parallel efficiency are computed as functions of worker count across the levels 2, 4, 8, 12, and 16 nodes" → remove 12.
• "with the default strategy as the baseline against which broadcast and partitioned execution are read" → the default strategy has been fully removed from the codebase (issues Increase resources for ACI containers and PostgreSQL #254, Prune Sedona default strategy matrix to large-tier only #309). Revise to describe how broadcast and partitioned are compared (e.g., against each other and against single-node baselines).

[jathavaan]

From PR #347

Section 4.5 — Threats to validity

Consider adding a construct validity entry to Table 4.5.1:

Validity	Threat	Mitigation
Construct	Spark's AQE or Photon can silently replace Sedona's BroadcastIndexJoin with a brute-force BroadcastNestedLoopJoin, changing the algorithm the benchmark actually measures.	Photon disabled at cluster creation; AQE disabled in both notebook variants before the timed section (PR #347 for broadcast, earlier for partitioned).

This is distinct from the existing internal validity threat about shared cloud infrastructure — it concerns the identity of the algorithm under test, not its variance across runs.

[jathavaan]

From issue #348

§4.1.2 — Replication and randomization (p. 41)

The thesis describes a single replication count applied uniformly to all experiments. The framework now differentiates: each experiment in benchmarks.yml declares its own runs field. RQ1 experiments (point-in-polygon, knn-search, bbox-filtering) specify runs: 30; RQ2 experiments (national-scale-spatial-join) specify runs: 3. The orchestrator checks benchmark_run > experiment["runs"] and skips the experiment once its limit is reached, so all experiments still execute within the same loop — the ordering and pairing protocol is unchanged.

Consider adding a sentence noting that the replication count is set per workload family (30 for RQ1, 3 for RQ2) rather than globally, with the rationale that RQ1's cheap iterations benefit from more samples while RQ2's expensive cluster-backed iterations do not.

§4.2.3 — Iteration counts (p. 44–45)

Three timeout constants changed:

Parameter	Old value	New value	Thesis text to update
Per-iteration hard timeout	4 500 s (75 min)	3 600 s (60 min)	"a hard timeout of 450 minutes" should read "360 minutes" for the cumulative sequential-stopping window, and the per-iteration timeout is now 60 min, not 75 min
Fixed-iteration ceiling	75 min (4 500 s)	300 min (18 000 s)	"A cumulative wall-clock ceiling of 75 minutes caps this fixed-iteration branch" → 300 minutes
Sequential cumulative window	6 × 4 500 = 27 000 s (450 min)	6 × 3 600 = 21 600 s (360 min)	Follows from the per-iteration change

The fixed-iteration ceiling increase (75 → 300 min) is the key fix: the old 75-minute ceiling was capping RQ2 experiments (e.g., duckdb-medium at ~25 min/iter × 4 total ≈ 100 min) before they could complete all timed iterations.

§4.5 — Threats to validity (p. 52–56)

Table 4.5.1 lists the threat: "A bootstrap confidence interval computed from fewer than thirty samples is noisy." With 30 RQ1 benchmark runs (each producing many sequential-stopping iterations), the sample pool for RQ1 experiments is now substantially larger. Consider noting that this threat is largely mitigated for RQ1 but remains for RQ2 (3 runs × 3 timed iterations = 9 samples per experiment).

[jathavaan]

From issue #357

Adding 12-node clusters to the Sedona benchmark matrix changes the worker-count factor from four levels to five. Three places in Ch. 4 need updating:

§4.1.1 Factors and levels (p40)

"RQ2 varies the worker count of the distributed engine across four levels, from 2 to 16 nodes"

→ Change to five levels: 2, 4, 8, 12, 16.

Table 4.1.1 (p41)

Worker count row currently lists levels as 2, 4, 8, 16.

→ Update to 2, 4, 8, 12, 16.

Table 4.2.1 (p44)

Spatial join configurations column says "partitioned; 2–16 workers".

→ Update to explicitly list 2, 4, 8, 12, 16 workers or note the five levels, since the range "2–16" no longer implies uniform doubling.

Rationale

The 12-node level fills the gap between 8 and 16 to give a finer-grained scaling curve. Partitioned-join entries at medium and large remain skip: failed (executor OOM).

[jathavaan]

From issue/PR #357

Section 4.1.1 — Factors and levels (p40)

Text says: "RQ2 varies the worker count of the distributed engine across four levels, from 2 to 16 nodes"

Should be five levels now that 12-node clusters are added: "…across five levels, from 2 to 16 nodes, tracing how the spatial join scales with increasing parallelism."

Table 4.1.1 (p41)

Worker count row currently lists levels as 2, 4, 8, 16. Update to 2, 4, 8, 12, 16.

Table 4.2.1 (p44)

Spatial join configurations column says "partitioned; 2–16 workers". The range notation still covers 12, but for consistency with the explicit level count change in Table 4.1.1, consider updating to "partitioned; 2–16 workers, five levels" or listing them explicitly.