Skip to content

Commit 924e16c

Browse files
committed
Add a SymInt-free tensor specialization key for exact torch.Tensor args
_tensor_key wraps every size and stride element in _hashable_dim to normalize torch.SymInt into a hashable (shape_env_id, expr) pair. That wrap exists only for symbolic shapes, which appear on FakeTensors during tracing -- yet concrete tensors paid for it on every kernel call: two Python-level loops (sizes + strides) with an isinstance check per dimension, plus rebuilding each as a fresh tuple. For the default static_shapes=True path this was the bulk of per-tensor key extraction (~0.6us each, x number of tensor args, every call). Specialization-extractor dispatch is by exact type (_specialization_extractors.get(type(obj))), so the torch.Tensor and torch.nn.Parameter entries are only ever hit by objects whose sizes/strides are guaranteed concrete ints. Route those to a new _concrete_tensor_key that uses obj.size() (a torch.Size) and obj.stride() directly as key components. Both are tuple subclasses: they hash and compare identically to plain int tuples, so keys produced by the fast path and the SymInt path for the same concrete shape are interchangeable -- no cache invalidation, and the specialization axes (dtype, shape, stride, _dynamo_static_indices, int32/int64 index width, dynamic-shape buckets) are bit-identical. Anything that can carry SymInts keeps the safe path: * FakeTensor has its own dispatch entry -> _tensor_key (unchanged) * torch.Tensor *subclasses* (e.g. the JAX-export adapter) miss the exact-type dict and hit the isinstance fallback in _specialization_key, which now routes to _tensor_key explicitly (This is the same optimization as the Python-only part of upstream PR #2611, independently arrived at from profiling.) Verified: test_misc, test_specialize, test_torch_compile (306 passed) -- the torch.compile suite exercises the FakeTensor/SymInt path. Benchmark (B200, end-to-end wall time per call, add-style kernel, N=4096 fp32, HELION_AUTOTUNE_EFFORT=none, steady state, 20k iters): ``` n_args | baseline | prev commit | this commit -------+----------+-------------+------------ 2 | 24.14 us | 16.39 us | 15.53 us 8 | 33.05 us | 24.53 us | 22.12 us 16 | 43.56 us | 34.82 us | 30.62 us ``` Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> stack-info: PR: #2748, branch: yushangdi/stack/30
1 parent d739597 commit 924e16c

1 file changed

Lines changed: 40 additions & 4 deletions

File tree

helion/runtime/kernel.py

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -352,8 +352,10 @@ def _specialization_key(self, obj: object) -> Hashable:
352352
extractor = _specialization_extractors[torch.fx.GraphModule]
353353
elif isinstance(obj, torch.Tensor):
354354
# torch.Tensor subclasses (e.g. the JAX-export adapter)
355-
# share the standard tensor specialization key.
356-
extractor = _specialization_extractors[torch.Tensor]
355+
# share the standard tensor specialization key. Use the
356+
# SymInt-safe extractor: unlike exact ``torch.Tensor``,
357+
# subclasses may carry symbolic sizes/strides.
358+
extractor = _tensor_key
357359
elif isinstance(obj, tuple) and hasattr(obj, "_fields"):
358360
# this is a namedtuple
359361
extractor = _specialization_extractors["namedtuple"]
@@ -1329,6 +1331,36 @@ def _hashable_dims(dims: Sequence[int | torch.SymInt]) -> tuple[Hashable, ...]:
13291331
return tuple(_hashable_dim(s) for s in dims)
13301332

13311333

1334+
def _concrete_tensor_key(fn: Kernel, obj: torch.Tensor) -> Hashable:
1335+
# Fast extractor for plain ``torch.Tensor`` / ``torch.nn.Parameter``:
1336+
# exact-type dispatch guarantees concrete int sizes/strides, so
1337+
# ``torch.Size`` and the stride tuple can be used directly (both are
1338+
# tuple subclasses that hash/compare identically to plain int tuples).
1339+
# The ``_hashable_dims`` wrap in ``_tensor_key`` exists only to
1340+
# normalize SymInts, which appear on FakeTensors during tracing.
1341+
si = getattr(obj, "_dynamo_static_indices", None)
1342+
static_indices = frozenset(si) if si is not None else _EMPTY_FROZENSET
1343+
if fn.settings.static_shapes:
1344+
return (obj.dtype, obj.size(), obj.stride(), static_indices)
1345+
bucketed = _bucketed_size(obj)
1346+
if fn.settings.index_dtype is None:
1347+
try:
1348+
needs_int64 = bool(obj.numel() > _INT32_INDEX_LIMIT)
1349+
except RuntimeError:
1350+
needs_int64 = True # unbacked SymInt
1351+
return (
1352+
obj.dtype,
1353+
bucketed,
1354+
needs_int64,
1355+
static_indices,
1356+
)
1357+
return (
1358+
obj.dtype,
1359+
bucketed,
1360+
static_indices,
1361+
)
1362+
1363+
13321364
def _tensor_key(fn: Kernel, obj: torch.Tensor) -> Hashable:
13331365
si = getattr(obj, "_dynamo_static_indices", None)
13341366
static_indices = frozenset(si) if si is not None else _EMPTY_FROZENSET
@@ -1408,8 +1440,12 @@ def _graph_module_key(fn: Kernel, obj: torch.fx.GraphModule) -> Hashable:
14081440
type[object] | str, Callable[[Kernel, object], Hashable]
14091441
# pyrefly: ignore [bad-assignment]
14101442
] = {
1411-
torch.Tensor: _tensor_key,
1412-
torch.nn.Parameter: _tensor_key,
1443+
# Exact-type dispatch (see ``_specialization_key``): plain tensors and
1444+
# Parameters always have concrete int sizes/strides and take the fast
1445+
# extractor. Subclasses (FakeTensor below, or anything hitting the
1446+
# ``isinstance`` fallback) go through SymInt-safe ``_tensor_key``.
1447+
torch.Tensor: _concrete_tensor_key,
1448+
torch.nn.Parameter: _concrete_tensor_key,
14131449
FakeTensor: _tensor_key,
14141450
torch.dtype: lambda fn, x: x,
14151451
torch.device: lambda fn, x: x,

0 commit comments

Comments
 (0)