fix(accelerator): scope trace timing and correlation to one artifact/session

[morluto]

Severity

Critical scientific correctness — independent trace artifacts on one run can be fused into one timing/correlation region, producing host-to-device matches and gaps that never existed in any trace

Validated against main at 0a60b288342cd96e2608302fa9fd2917e59d7e58.

Audit only; no implementation change is included.

Summary

AcceleratorRecipes resolves a run to all of its trace artifact IDs and queries normalized trace.event rows across the whole scope.

The query does not select or retain artifact_id. Region grouping uses only:

cycle-like artifact role, when present
phase

and ignores every other artifact role.

Within a region, runtime→kernel correlation uses only the stringified correlation_id:

runtime_correlation_ids = {str(value["correlation_id"]) ...}
...
str(kernel["correlation_id"]) in runtime_correlation_ids

Timing summaries likewise combine start_ns/duration_ns from all selected artifacts and treat them as one clock domain.

This is invalid for independent trace files. Correlation IDs are meaningful within a captured activity stream; numeric equality across two independent artifacts does not establish a relationship. Trace timestamps also need one clock/session mapping before subtraction.

A concrete production path creates exactly this state: inference profiling preserves each trace file as a separate import run with role inference_profile, then copies all registrations into one parent profiling run. The accelerator recipe ignores that role, so stage-separated or multiple profiler outputs can merge under the same phase or <unscoped> region.

Direct code evidence

The recipe drops artifact identity at the query boundary

rows = snapshot.execute(
    "SELECT name, value_json, context FROM observations WHERE ("
    + where
    + ") AND kind = 'trace.event' ORDER BY observation_id",
    parameters,
).fetchall()

flameox/src/flameox/analysis/recipe_accelerator.py

Lines 160 to 185 in 0a60b28

	status="partial",
	reason="runtime_or_accelerator_tracks_missing",
	)
	if not (
	coverage["runtime_launches"]
	and coverage["accelerator_kernels"]
	and (
	comparison_coverage is None
	or (
	comparison_coverage["runtime_launches"]
	and comparison_coverage["accelerator_kernels"]
	)
	)
	)
	else available_availability()
	)
	),
	)
	def _accelerator_launch_regions(
	self,
	snapshot: Snapshot,
	input_id: str,
	*,
	phase: str | None,
	limit: int,

resolve_evidence_scope() may select several artifact IDs for a run, but the row consumer cannot tell which artifact produced an event.

Only cycle-prefixed roles survive grouping

artifact_role = str(value.get("artifact_role") or "")
cycle = (
    artifact_role
    if artifact_role.startswith(("cycle_", "partial_cycle_"))
    else None
)
region = f"{cycle}/{phase}" if cycle is not None else phase

Every role such as:

primary
inference_profile
prefill_profile
decode_profile

collapses into the same phase bucket unless the provider separately injects a distinct phase.

Correlation is a bare-ID set membership test

runtime_correlation_ids = {
    str(value["correlation_id"])
    for value in (*direct, *graph)
    if value.get("correlation_id") not in {None, ""}
}
...
correlated_kernel_count=sum(
    str(item["correlation_id"]) in runtime_correlation_ids
    for _, item in kernels
    if item.get("correlation_id") not in {None, ""}
)

https://github.com/morluto/flameox/blob/0a60b288342cd96e2608302fa9fd2917e59d7e58/src/flameox/analysis/recipe_accelerator.py#L380-L460

Artifact, process, context, device, and provider/session identity do not participate.

Timing also crosses the erased boundary

The same merged event list is used for:

region_start_ns
region_end_ns
region_duration_ns
runtime launch gaps
per-stream idle gaps

Events are grouped into runtime tracks and accelerator streams only after artifact/session identity has been lost.

Inference profiling creates multiple independent trace artifacts on one parent run

InferenceProfilingService._preserve() imports every discovered trace candidate separately and assigns all of them:

kind=ArtifactKind.EXECUTION_TRACE
role="inference_profile"

It returns multiple artifact run IDs.

_canonical_registrations() then copies every source registration onto the parent profiling run:

for artifact_run_id in artifact_run_ids:
    source = self.runs.read(artifact_run_id)
    registrations.extend(
        registration.model_copy(
            update={"registration_id": new_id(), "run_id": parent_run_id}
        )
        for registration in source.artifacts
    )

https://github.com/morluto/flameox/blob/0a60b288342cd96e2608302fa9fd2917e59d7e58/src/flameox/application/inference_profiling.py

The parent run's evidence scope therefore includes every independently extracted trace.

The SGLang plan explicitly says it captures separate prefill/decode traces, and generic directory discovery can preserve several .json, .json.gz, .pftrace, or .sqlite trace candidates.

Deterministic false-correlation proof

Let parent run R register two independently captured traces, A and B. Both registrations have role inference_profile, and both normalized events use phase decode.

Trace A

{
  "artifact_id": "A",
  "name": "cudaLaunchKernel",
  "category": "cuda_runtime",
  "start_ns": 1000,
  "duration_ns": 10,
  "process": "P1",
  "context": "C1",
  "correlation_id": "7"
}

A contains no kernel with correlation 7.

Trace B

{
  "artifact_id": "B",
  "name": "projection_kernel",
  "category": "kernel",
  "start_ns": 0,
  "duration_ns": 5,
  "process": "P2",
  "context": "C2",
  "device": "1",
  "stream": "9",
  "correlation_id": "7"
}

B contains no runtime launch with correlation 7.

Current analysis over R:

region = decode for both events
runtime_correlation_ids = {"7"}
kernel B correlation "7" is in that set
correlated_kernel_count = 1
coverage.host_to_device_correlation = True

No host-to-device correlation exists. The launch and kernel came from different artifacts, processes, contexts, and capture sessions.

CUPTI's correlation ID contract links API/activity records inside the collected activity stream; it does not make integer 7 a global cross-file identifier. Official CUPTI documentation describes the ID as the value carried by the API call and the activity records generated by that call:
https://docs.nvidia.com/cupti/main/main.html

Deterministic false-timing proof

Using the same two traces:

A clock origin: event at 1000..1010
B clock origin: event at 0..5

Current region summary returns:

region_start_ns = 0
region_end_ns = 1010
region_duration_ns = 1010

There was no 1,010 ns region in either trace. The value is the span between two unrelated clock origins.

If A and B each contain kernels with the same (device, context, stream) strings, _positive_gaps() sorts them together and invents cross-file idle intervals.

Same-artifact scope still needs more than a bare integer

Even inside one report, multi-process/system-wide traces and provider-specific correlation namespaces require the exact documented key. Normalized rows already carry fields such as process, thread, context, device, and track. The recipe discards them for correlation matching.

The correct key may differ by provider/profile, for example:

artifact/session + process + correlation ID
artifact/session + context + correlation ID
provider-owned explicit flow/link identity

It must be qualified rather than guessed universally.

Additional consequences

• Trace-event counts from separate captures are added as if they were one execution.
• Kernel-name counts can double-count repeated stage traces.
• Phase annotations with identical names do not prove common clock or repeated-work semantics.
• correlation_ids=True and host_to_device_correlation=True can overstate coverage.
• Comparisons can attribute multi-artifact composition differences to launch behavior.
• A user cannot reconstruct which artifact contributed a reported region because artifact identity is absent from AcceleratorLaunchRegion.

Violated invariant

Every timing subtraction, gap computation, stream grouping, and correlation join must be performed only within one qualified trace/session namespace unless an explicit provider mapping proves cross-session alignment.

At minimum:

artifact/session identity
provider/profile identity
clock domain
process/context scope
correlation namespace

must participate in the grouping/join contract.

Proposed direction

1. Preserve artifact/session identity in normalized rows and query models

Select and retain artifact_id, registration role, extractor profile, provider, and clock identity for every event.

2. Produce summaries per trace session by default

Region identity should include the trace artifact/session, or the API should require one artifact input. Multi-artifact aggregation must be an explicit higher-level operation with a declared alignment rule.

3. Use provider-qualified correlation keys

Define exact join keys for Perfetto, Nsight Systems, rocprofv3, and other providers based on maintained native semantics. Unknown/incomplete keys remain uncorrelated.

4. Never subtract unaligned clocks

Gap and region-duration calculations must require one clock domain. Cross-trace comparison should compare durations/aggregates, not absolute timestamps, unless synchronization evidence exists.

5. Preserve inference stages explicitly

Stage-separated prefill/decode traces need distinct typed roles/stage identities in registration and analysis. A generic inference_profile label is insufficient.

Acceptance criteria

• Accelerator queries retain exact artifact/session identity for every event.
• Events from different artifacts are not merged into one timing region by default.
• Correlation keys are provider/profile-qualified and include the required process/context/session scope.
• The A-runtime/B-kernel counterexample has correlated_kernel_count=0.
• Independent clock origins cannot produce a cross-artifact region duration or idle gap.
• SGLang prefill/decode and other multi-trace inference outputs remain separate.
• Coverage distinguishes IDs present from correlations actually proven in a complete namespace.
• Result provenance identifies every contributing artifact and clock/profile.
• Tests cover same numeric correlation ID across artifacts/processes/contexts, same stream labels across artifacts, stage-separated traces, and one valid same-session correlation.
• Historical rows without session scope remain explicitly unqualified and cannot claim host-to-device correlation.

Relationship to existing issues

• [Bug][Adapter][Simplification] Version-gate Nsight Systems SQLite extraction and reuse NVIDIA report scripts #190 requires exact Nsight Systems export/query profiles.
• Capture rocprofv3 PFTrace output through the existing Perfetto pipeline #136 requires genuine rocprofv3 PFTrace qualification.
• fix(evidence): reserve common provenance columns and remove duplicate extractor-version field #207 governs authoritative trace-event schema fields.
• fix(comparisons): reject ambiguous measurement dimensions instead of pooling them #249 requires exact measurement/configuration dimensions; this is the corresponding trace namespace invariant.
• fix(analysis): preserve observation sequence and compare repeated SDK events without hash-order collapse #260 preserves ordered SDK event streams; this issue preserves trace-session identity.
• [Architecture] Add an explicit migration contract for every persisted schema #157 owns migration of existing trace rows/results without artifact-scoped correlation semantics.