Skip to content

Commit 182b5b0

Browse files
factnnautomerge-botclaude
authored
feat: S-Level anti-cheat sandbox with 7-layer defense system
* feat: add complete sandbox system with 6 defense layers Implements full competition-grade anti-cheat sandbox from anti-cheat.md: 1. CacheIsolator (cache_isolator.py) — File system isolation: - Isolated HOME directory per test - Disabled triton/torch/cuda caches - Auto cleanup via context manager 2. ImportHookSandbox (import_hook.py) — Runtime import enforcement: - sys.meta_path hook for live import interception - Auto-patches triton.autotune/heuristics/Config at import time - Auto-patches torch.compile/CUDA Graph at import time - Blocks multiprocessing.shared_memory/posix_ipc/mmap - Secure exec/eval wrapper with keyword scanning 3. CUDALayerProtector (cuda_protector.py) — CUDA protection: - Disables CUDA Graph capture/replay - Resets CUDA state between tests - Disables TF32 for consistent precision 4. BucketedShapeGenerator (shape_generator.py) — Shape randomization: - GPU-alignment-friendly random shapes - GEMM/Attention/Conv specialized generators - TensorLayoutRandomizer for stride randomization 5. ProcessIsolatedEvaluator (process_isolator.py) — Process isolation: - Each test in fresh subprocess (mp.spawn) - All sandbox layers auto-applied in worker - Batch evaluation support 6. StatisticalTimingValidator (timing_validator.py) — Statistical checks: - CV/IQR/convergence scoring - Outlier detection (1.5*IQR rule) - Retest consistency check FullSandbox (full_sandbox.py) ties all layers together into a single API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: implement complete 7-layer S-Level anti-cheat sandbox Strictly follows triton_competition_anti_cheat_guide.md: Layer 1 - cache_isolator.py (Section 3): CacheIsolator with HOME isolation, triton/torch cache dirs, cleanup Layer 2 - import_hook.py (Section 4): ForbiddenModuleLoader, DisabledCUDAGraph, ImportHookSandbox, RuntimeSandbox, SecureBuiltins, SecurityError, enable_competition_sandbox Layer 3 - cuda_protector.py (Section 5): CUDALayerProtector, DisabledCUDAGraphContext, CUDA Graph/TF32/profiler disable, CUDA state reset Layer 4 - shape_generator.py (Section 6): ShapeBucket, BucketedShapeGenerator (STANDARD_BUCKETS, GEMM_BUCKETS, generate_gemm/conv/attention_shape), TensorLayoutRandomizer (randomize_layout/contiguity/strides) Layer 5 - process_isolator.py (Section 7): TestConfig, isolated_test_worker (7-step isolation), ProcessIsolatedEvaluator (evaluate_single/batch with mp.spawn) Layer 6 - timing_validator.py (Section 8): TimingAnomalyType, TimingValidationResult, StatisticalTimingValidator (CV/IQR/convergence/outliers/retest), AdvancedTimingValidator Layer 7 - competition_evaluator.py (Section 9): CompetitionConfig, TestCase, TestCaseGenerator, isolated_worker_main, CompetitionEvaluator, main() Plus full_sandbox.py convenience wrapper Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: block code-level hack vectors (input sniffing, caching, memory access) Three new defenses against real-world competition attacks: 1. print() blocked: - AST scan: 'print()' calls flagged as hack - Runtime: builtins.print replaced with no-op in SecureBuiltins 2. data_ptr() / storage() blocked: - AST scan: .data_ptr, .untyped_storage, .storage, .storage_offset all flagged as forbidden memory access 3. Per-iteration random seeds: - Each timed run uses a different seed -> different input values - Kills hardcoded lookup tables and inter-iteration caching - Fresh clone() per iteration prevents pointer equality checks - Separate warmup seed to isolate warmup from scoring These close the "200x speedup" hacks: input sniffing via print(), inter-iteration result caching, and raw memory pointer reading. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: detect module-level mutable state as anti-hack vector Adds scope tracking to AST HackDetector: - visit_FunctionDef/AsyncFunctionDef/ClassDef -> increment scope depth - visit_Assign/visit_AnnAssign -> at depth 0, detect: - name = {} / name = [] / name = {...} (literals) - name = dict() / name = list() / name = set() (constructor calls) Module-level dict/list/set declarations are banned in competition mode because they enable inter-iteration result caching attacks. Local mutable state inside functions is NOT flagged (legitimate use). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * perf: optimize dual-execution check by reusing accuracy test output Add dual_execution_check_with_ref() that accepts already-computed reference output from the accuracy test, only running the triton.jit-disabled execution and comparing with the saved result. Saves 50% of Layer 2 execution time. Also enables anti_hack (Layer 2 + Layer 3) by default in VerifyConfig. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add lightweight sandbox protections to Verifier __init__ Applied automatically to both LLM track and Agent track: 1. Env vars: TRITON_DISABLE_AUTOTUNE, TRITON_CACHE_DIR, TORCHINDUCTOR_DISABLE, CUDA_CACHE_DISABLE 2. CUDA layer protection: disable CUDA Graph, TF32, reset CUDA state 3. Runtime import hook: patch triton.autotune/torch.compile at import time All set via Verifier._setup_sandbox() called in __init__. Failures are non-fatal (try/except). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: wire SecureBuiltins into RuntimeSandbox, remove import hook from daily verify 1. RuntimeSandbox.enable() now auto-enables SecureBuiltins (print noop) 2. Verifier._setup_sandbox: removed import hook (too heavy for daily use, only env vars + CUDA protector for lightweight path) 3. Both fixes verified: 8 modules all pass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: automerge-bot <devnull@local> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent e039e53 commit 182b5b0

10 files changed

Lines changed: 1564 additions & 15 deletions

src/sandbox/anti_hack.py

Lines changed: 106 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,52 @@ def __init__(self, blacklist: List[str] = None):
8383
self.blacklist = blacklist or []
8484
# Track import aliases: {"tr": "torch", "ts": "torch.sum", ...}
8585
self._aliases: dict = {}
86+
# Scope depth for detecting module-level mutable state
87+
self._scope_depth: int = 0
88+
89+
# ---- Scope tracking ----
90+
def visit_FunctionDef(self, node):
91+
self._scope_depth += 1
92+
self.generic_visit(node)
93+
self._scope_depth -= 1
94+
95+
def visit_AsyncFunctionDef(self, node):
96+
self._scope_depth += 1
97+
self.generic_visit(node)
98+
self._scope_depth -= 1
99+
100+
def visit_ClassDef(self, node):
101+
self._scope_depth += 1
102+
self.generic_visit(node)
103+
self._scope_depth -= 1
104+
105+
# ---- Module-level mutable state detection ----
106+
def visit_Assign(self, node):
107+
if self._scope_depth == 0:
108+
for target in node.targets:
109+
if isinstance(target, ast.Name):
110+
if isinstance(node.value, (ast.Dict, ast.List, ast.Set)):
111+
self.violations.append(
112+
f"Forbidden module-level mutable state: '{target.id} = {{...}}' "
113+
f"is banned in competition mode (line {node.lineno})"
114+
)
115+
elif isinstance(node.value, ast.Call):
116+
call_name = self._get_attr_chain(node.value.func)
117+
if call_name in ("dict", "list", "set"):
118+
self.violations.append(
119+
f"Forbidden module-level mutable state: '{target.id} = {call_name}()' "
120+
f"is banned in competition mode (line {node.lineno})"
121+
)
122+
self.generic_visit(node)
123+
124+
def visit_AnnAssign(self, node):
125+
if self._scope_depth == 0 and isinstance(node.target, ast.Name):
126+
if isinstance(node.value, (ast.Dict, ast.List, ast.Set)):
127+
self.violations.append(
128+
f"Forbidden module-level mutable state: '{node.target.id}: ... = {{...}}' "
129+
f"is banned in competition mode (line {node.lineno})"
130+
)
131+
self.generic_visit(node)
86132

87133
# ---- Hard blacklist + alias tracking: imports ----
88134
def visit_Import(self, node: ast.Import):
@@ -127,6 +173,12 @@ def visit_Call(self, node: ast.Call):
127173
f"Forbidden torch API: '{call_chain}()' not in allowed whitelist (line {node.lineno})"
128174
)
129175

176+
# Detect print() — input sniffing
177+
if isinstance(node.func, ast.Name) and node.func.id == "print":
178+
self.violations.append(
179+
f"Forbidden call: 'print()' is banned in competition mode (line {node.lineno})"
180+
)
181+
130182
# Detect getattr(torch, "sum") — dynamic attribute access
131183
if (
132184
isinstance(node.func, ast.Name) and node.func.id == "getattr"
@@ -168,13 +220,22 @@ def visit_Call(self, node: ast.Call):
168220

169221
self.generic_visit(node)
170222

171-
# ---- Hard blacklist: attribute access ----
223+
# ---- Hard blacklist: attribute access + memory pointer detection ----
224+
_FORBIDDEN_ATTRS = {
225+
"data_ptr", "untyped_storage", "storage", "storage_offset",
226+
}
227+
172228
def visit_Attribute(self, node: ast.Attribute):
173229
attr_chain = self._get_attr_chain(node)
174230
if attr_chain and self._is_blacklisted(attr_chain):
175231
self.violations.append(
176232
f"Forbidden attribute access: '{attr_chain}' (line {node.lineno})"
177233
)
234+
# Detect memory pointer access: x.data_ptr(), x.storage(), etc.
235+
if node.attr in self._FORBIDDEN_ATTRS:
236+
self.violations.append(
237+
f"Forbidden memory access: '.{node.attr}' is banned in competition mode (line {node.lineno})"
238+
)
178239
self.generic_visit(node)
179240

180241
# ---- Helpers ----
@@ -319,22 +380,16 @@ def dual_execution_check(
319380
try:
320381
out_normal = func(**kwargs)
321382
except Exception:
322-
# If normal run fails, can't do comparison
323383
return False, ""
324384

325385
# Run 2: with triton.jit disabled
326386
try:
327387
with disable_triton_jit():
328-
# Re-import the module to pick up the patched triton.jit
329-
# The kernel functions become plain python functions
330388
out_disabled = func(**kwargs)
331389
except Exception:
332-
# If disabled run crashes, triton kernel was actually needed -> not hack
333390
return False, ""
334391

335-
# Compare results
336392
if out_normal is None and out_disabled is None:
337-
# Both None - check in-place outputs via kwargs
338393
return False, ""
339394

340395
if _results_match(out_normal, out_disabled, rtol, atol):
@@ -346,6 +401,50 @@ def dual_execution_check(
346401
return False, ""
347402

348403

404+
def dual_execution_check_with_ref(
405+
func: Callable,
406+
kwargs: dict,
407+
ref_output: Any,
408+
rtol: float = 1e-3,
409+
atol: float = 1e-3,
410+
) -> Tuple[bool, str]:
411+
"""
412+
Optimized dual-execution: compare against an already-computed reference output.
413+
414+
Only runs ONCE (with triton.jit disabled), then compares with ref_output
415+
from the accuracy test's normal execution. Saves 50% of execution time.
416+
417+
Args:
418+
func: the registered triton function to test
419+
kwargs: input parameters for the function
420+
ref_output: already-computed output from normal execution
421+
rtol/atol: tolerance for "same result" comparison
422+
423+
Returns:
424+
(is_hack, reason): True if hack detected.
425+
"""
426+
import torch
427+
428+
# Run: with triton.jit disabled only (normal execution already done in verify)
429+
try:
430+
with disable_triton_jit():
431+
out_disabled = func(**kwargs)
432+
except Exception:
433+
# If disabled run crashes, triton kernel was actually needed -> not hack
434+
return False, ""
435+
436+
if ref_output is None and out_disabled is None:
437+
return False, ""
438+
439+
if _results_match(ref_output, out_disabled, rtol, atol):
440+
return True, (
441+
"Dual-execution hack detected: output is identical "
442+
"with triton.jit disabled, indicating no real triton kernel is used."
443+
)
444+
445+
return False, ""
446+
447+
349448
def _results_match(a: Any, b: Any, rtol: float, atol: float) -> bool:
350449
"""Check if two results are effectively identical."""
351450
import torch

src/sandbox/cache_isolator.py

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
"""
2+
Layer 1: File System Isolation.
3+
From: triton_competition_anti_cheat_guide.md - Section 3
4+
"""
5+
import os
6+
import shutil
7+
import tempfile
8+
9+
10+
class CacheIsolator:
11+
"""缓存目录隔离器"""
12+
13+
CACHE_PATHS = [
14+
'~/.triton',
15+
'~/.cache/triton',
16+
'~/.torch',
17+
'~/.cache/torch',
18+
'~/.nv',
19+
]
20+
21+
def __init__(self):
22+
self.original_home = os.environ.get('HOME', '/root')
23+
self.isolated_home = None
24+
25+
def isolate(self):
26+
"""创建隔离环境"""
27+
# 创建隔离HOME
28+
self.isolated_home = tempfile.mkdtemp(prefix='isolated_home_')
29+
os.environ['HOME'] = self.isolated_home
30+
31+
# 设置所有缓存变量指向临时目录或禁用
32+
os.environ['TRITON_CACHE_DIR'] = os.path.join(self.isolated_home, '.triton')
33+
os.environ['TORCHINDUCTOR_CACHE_DIR'] = os.path.join(self.isolated_home, '.torch')
34+
os.environ['CUDA_CACHE_DISABLE'] = '1'
35+
os.environ['XDG_CACHE_HOME'] = os.path.join(self.isolated_home, '.cache')
36+
37+
# 创建必要的目录结构
38+
os.makedirs(os.environ['TRITON_CACHE_DIR'], exist_ok=True)
39+
os.makedirs(os.environ['TORCHINDUCTOR_CACHE_DIR'], exist_ok=True)
40+
41+
return self.isolated_home
42+
43+
def cleanup(self):
44+
"""清理隔离环境"""
45+
if self.isolated_home and os.path.exists(self.isolated_home):
46+
shutil.rmtree(self.isolated_home, ignore_errors=True)
47+
os.environ['HOME'] = self.original_home
48+
49+
def __enter__(self):
50+
return self.isolate()
51+
52+
def __exit__(self, *args):
53+
self.cleanup()

0 commit comments

Comments
 (0)