fix(scaling): correlate semantic frames across trial artifacts instead of artifact-local frame IDs

[morluto]

Severity

High analysis correctness and feature failure — Perfetto-based scaling experiments can never correlate the same operator across ordinary trial traces because each artifact produces a different frame_id; strong monotonic hotspots silently disappear

Validated against main at 0a60b288342cd96e2608302fa9fd2917e59d7e58 by composing the actual Perfetto frame identity with _correlated_hotspots().

Audit only; no implementation change is included.

Summary

The scaling recipe is designed to identify frame/operator costs that correlate with the experiment's numeric input.

It groups hotspot observations across trials by exact frame_id:

(variant, frame_id, function, file, line, metric, unit)

and requires at least three samples in one group.

Perfetto extraction defines frame_id with the source artifact ID included:

frame_id = digest_model({
    "artifact_id": registration.artifact_id,
    "function": name,
    "category": category,
    "file": filename,
    "line": line,
    "symbolization": "partial",
})

Every separately captured trial normally has a different trace artifact. Therefore the same operator/source location in three trials receives three different frame IDs, producing three one-sample groups. _correlated_hotspots() drops all of them because len(samples) < 3.

A perfect correlation such as:

input 1 -> reverse_scan 10 ns
input 2 -> reverse_scan 20 ns
input 3 -> reverse_scan 30 ns

returns no ScalingCorrelatedHotspot when those values come from real Perfetto trial artifacts.

The repository regression test passes only because it manually assigns the same synthetic frame_id="reverse-scan-frame" and the same artifact ID to every run, a state that the production Perfetto extractor does not generate for independent traces.

Direct code evidence

Perfetto frame identity is artifact-local

frame_id = digest_model(
    {
        "artifact_id": registration.artifact_id,
        "function": name,
        "category": category,
        "file": filename,
        "line": line,
        "symbolization": "partial",
    }
)

https://github.com/morluto/flameox/blob/0a60b288342cd96e2608302fa9fd2917e59d7e58/src/flameox/adapters/perfetto.py

The frame row also stores the artifact ID separately, but no cross-artifact semantic frame identity is published.

Scaling correlation groups by exact frame ID

The hotspot query retrieves fm.frame_id for each trial.

_correlated_hotspots() first creates per-trial values keyed by:

(
    trial_id,
    variant,
    frame_id,
    function,
    file,
    line,
    metric,
    unit,
    input_value,
)

It then groups across trials by:

(variant, frame_id, function, file, line, metric, unit)

and rejects groups with fewer than three samples:

if len(samples) < 3 or len({sample[0] for sample in samples}) < 2:
    continue

flameox/src/flameox/analysis/recipe_scaling.py

Lines 360 to 500 in 0a60b28

	def _median_interval(
	values: np.ndarray,
	*,
	confidence_level: float,
	) -> tuple[float | None, float | None]:
	if values.size < 2:
	return None, None
	median = float(np.median(values))
	if np.allclose(values, median):
	return median, median
	try:
	with warnings.catch_warnings():
	warnings.simplefilter("ignore", RuntimeWarning)
	result = bootstrap(
	(values,),
	np.median,
	vectorized=False,
	confidence_level=confidence_level,
	n_resamples=1_999,
	method="BCa",
	rng=np.random.default_rng(0),
	)
	except ValueError:
	return None, None
	low = float(result.confidence_interval.low)
	high = float(result.confidence_interval.high)
	if not math.isfinite(low) or not math.isfinite(high):
	return None, None
	return low, high
	def _correlated_hotspots(
	self,
	rows: list[tuple[object, ...]],
	) -> tuple[ScalingCorrelatedHotspot, ...]:
	per_trial: dict[
	tuple[
	str,
	str,
	str,
	str | None,
	str | None,
	int | None,
	str,
	str,
	float,
	],
	float,
	] = {}
	for row in rows:
	input_value = (
	float(cast(Any, row[2]))
	if row[2] is not None
	else float(cast(Any, row[3]))
	if row[3] is not None
	else None
	)
	if input_value is None or not math.isfinite(input_value):
	continue
	key = (
	str(row[0]),
	str(row[1]),
	str(row[4]),
	str(row[5]) if row[5] is not None else None,
	str(row[6]) if row[6] is not None else None,
	int(cast(Any, row[7])) if row[7] is not None else None,
	str(row[8]),
	str(row[9]),
	input_value,
	)
	per_trial[key] = per_trial.get(key, 0.0) + float(cast(Any, row[10]))
	groups: dict[
	tuple[str, str, str | None, str | None, int | None, str, str],
	list[tuple[float, float]],
	] = {}
	for (
	_trial_id,
	variant,
	frame_id,
	function,
	file,
	line,
	metric,
	unit,
	input_value,
	), value in per_trial.items():
	groups.setdefault(
	(variant, frame_id, function, file, line, metric, unit),
	[],
	).append((input_value, value))
	results: list[ScalingCorrelatedHotspot] = []
	for (
	variant,
	frame_id,
	function,
	file,
	line,
	metric,
	unit,
	), samples in groups.items():
	if len(samples) < 3 or len({sample[0] for sample in samples}) < 2:
	continue
	x = np.asarray([sample[0] for sample in samples], dtype=float)
	y = np.asarray([sample[1] for sample in samples], dtype=float)
	with warnings.catch_warnings():
	warnings.simplefilter("ignore", RuntimeWarning)
	correlation = spearmanr(x, y)
	rho = float(correlation.statistic)
	p_value = float(correlation.pvalue)
	if not math.isfinite(rho) or not math.isfinite(p_value):
	continue
	results.append(
	ScalingCorrelatedHotspot(
	variant=variant,
	frame_id=frame_id,
	function=function,
	file=file,
	line=line,
	metric=metric,
	unit=unit,
	spearman_rho=rho,
	p_value=p_value,
	adjusted_p_value=p_value,
	multiplicity_method="benjamini-hochberg-fdr",
	independent_trial_count=len(samples),
	supported_min=float(np.min(x)),
	supported_max=float(np.max(x)),
	)
	)
	if results:
	adjusted = multipletests(
	[item.p_value for item in results],
	method="fdr_bh",
	)[1]
	tested = len(results)
	results = [
	item.model_copy(
	update={
	"adjusted_p_value": float(adjusted[index]),
	"tested_hypothesis_count": tested,
	}
	)

Including both frame_id and its descriptive fields does not help: the unique artifact-local frame_id partitions otherwise equal descriptions before correlation.

Deterministic proof

Run the same instrumented operator in three scaling trials.

Trial artifacts

T1 artifact_id = A1
T2 artifact_id = A2
T3 artifact_id = A3

Ordinary traces differ in timestamps/metadata, so:

A1 != A2 != A3

Each contains the same operator description:

function = reverse_scan
category = cpu_op
file = scan.py
line = 10
symbolization = partial

with values:

x=1, inclusive=10
x=2, inclusive=20
x=3, inclusive=30

Production frame identities are:

F1 = H(A1, reverse_scan, cpu_op, scan.py, 10, partial)
F2 = H(A2, reverse_scan, cpu_op, scan.py, 10, partial)
F3 = H(A3, reverse_scan, cpu_op, scan.py, 10, partial)

Absent a SHA-256 collision:

F1 != F2 != F3

_correlated_hotspots() creates:

group F1 -> [(1, 10)]
group F2 -> [(2, 20)]
group F3 -> [(3, 30)]

Each group has one sample and is discarded.

Current result:

correlated_hotspots = ()

Correct semantic grouping would produce:

samples = [(1,10), (2,20), (3,30)]
Spearman rho = 1.0

The current test is not production-representative

test_scaling_reports_dispersion_models_and_supported_range manually publishes every trial with:

artifact_id = "sha256:" + "a" * 64
frame_id = "reverse-scan-frame"

and one shared frame row.

It then asserts a strong positive correlation.

https://github.com/morluto/flameox/blob/0a60b288342cd96e2608302fa9fd2917e59d7e58/tests/analysis/test_scaling_recipes.py

This proves the statistical reducer when a stable semantic frame ID is supplied. It does not prove the production extraction→correlation composition. Differential coverage through PerfettoExtractor would fail.

Inconsistent adapter semantics

Memray creates frame IDs from language/function/file/line without artifact ID, while Perfetto includes artifact ID. Thus the meaning of the shared frames.frame_id column changes by producer:

Memray: semantic-ish source frame identity
Perfetto: artifact-local frame occurrence identity

Generic analyses cannot safely assume one meaning.

Removing artifact ID from the hash globally is not automatically sufficient. Native frames may require build ID/module-relative address/inline chain/source state to prevent false matches across binaries or code revisions.

The model needs two explicit concepts:

1. artifact-local frame record/occurrence identity;
2. versioned semantic frame identity used for qualified cross-run matching.

Impact

• Perfetto/PyTorch scaling hotspots are systematically empty across normal trial traces.
• Agents lose the main discriminating evidence connecting input growth to operator cost.
• The absence looks like “no correlated hotspot,” not “identity prevented matching.”
• Provider behavior differs silently: the same recipe may work for Memray and fail for Perfetto.
• Tests give false confidence because they bypass extractor-generated identities.
• Any future cross-run call-edge/stack comparison using frame_id inherits the same mismatch.

Required invariant

Cross-trial correlation may group records only by a semantic identity that is:

stable across compatible artifacts
and distinct across incompatible source/build/symbol states

Artifact-local uniqueness and semantic equality must not be represented by one overloaded ID.

Proposed direction

1. Split frame identities

For example:

frame_record_id / occurrence_id
semantic_frame_id
artifact_id

The record ID remains unique in one artifact. The semantic ID is derived from a provider-qualified projection such as:

• Python/source: language, module/function, normalized source path/line, source-state compatibility;
• native: build ID, module-relative address, inline-chain identity, symbolization profile;
• operator trace: operator/category/source identity plus qualified provider semantics.

2. Make cross-run matching explicit

Scaling should group by semantic_frame_id only after source/build/provider compatibility is established. Unknown/partial identities should remain unmatched with a coverage limitation.

3. Preserve artifact-local details

Per-trial aggregation still uses artifact-local IDs to avoid merging distinct occurrences accidentally. The semantic join is a separate step.

4. Add extractor-composition tests

Generate/import three actual Perfetto-shaped artifacts, run PerfettoExtractor, then run scaling correlation. Do not inject shared frame IDs directly.

Acceptance criteria

• The same compatible Perfetto operator across trial artifacts receives one stable semantic frame identity.
• Artifact-local frame records remain uniquely attributable to their artifact.
• The 1→10, 2→20, 3→30 counterexample yields Spearman rho 1.0.
• Different source states/build IDs/inline chains do not match accidentally.
• Unknown/partial symbol identity yields explicit unmatched coverage rather than a false no-correlation result.
• Memray and Perfetto expose the same documented frame-ID semantics to generic analyses.
• Scaling tests execute the real extractor→publication→recipe path.
• Existing artifact-local frame IDs have an explicit migration/compatibility policy.

Relationship to existing issues

• fix(evidence): reserve common provenance columns and remove duplicate extractor-version field #207 governs common provenance/schema fields needed for semantic identity.
• fix(storage): bind run projections and revisions to their directory identity #245 binds exact source run revisions.
• fix(scaling): prevent environment changes from masquerading as input-size scaling #262 prevents environment confounding after frames are matched.
• fix(accelerator): scope trace timing and correlation to one artifact/session #275 defines artifact/session boundaries for trace events.
• fix(comparisons): reject ambiguous measurement dimensions instead of pooling them #249/fix(scaling): bind one measurement series and unit before aggregation or model fitting #256 define exact measurement-series populations.
• [Architecture] Add an explicit migration contract for every persisted schema #157/[Architecture][Simplification] Replace custom canonical JSON hashing with a versioned RFC 8785 implementation #180 govern persisted and canonical semantic identities.