Skip to content

Commit 8ac3b8f

Browse files
author
crispasr integration
committed
kaggle(rebake): hide the GPU from torch — 8 backends died to the P100 sm_60 drop
v2 got past the source_model blocker (only the 4 deliberately-blank entries still report it) and hit a new uniform failure. The log names it: torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device x8 That is kaggle_usage.md gotcha #23 verbatim — Kaggle's preinstalled torch has dropped sm_60, and gotcha #21 says P100 is effectively the only draw. nemotron, canary-1b-v2, parakeet-tdt-0.6b-ja, granite-4.1-plus/-nar, voxtral4b-realtime and vibevoice-asr each downloaded their model, loaded it, then died on the first kernel launch — ~100 s of wasted download apiece. MY FAULT, and the fix is to undo my own choice: I set enable_gpu:"true" in this kernel's metadata. The canonical regression kernel is deliberately enable_gpu:"false" because reference dumps do not need a GPU. Enabling one bought nothing and exposed every torch dumper to the lottery. Not simply flipping the flag, though: a CPU-only Kaggle worker loses internet (gotcha #3) and this job must pull models from HF. So the kernel keeps its GPU allocation and hides it from torch with CUDA_VISIBLE_DEVICES="". Same effect, network retained. REMAINING, separate cause, not addressed here: a dependency tail — funasr, mimo_audio_tokenizer, whisper, qwen_asr, fireredasr all ModuleNotFoundError, and transformers too old for VoxtralRealtimeForConditionalGeneration. Those need per-backend pip installs in the canonical suite, which is a wider change. Log saved to /mnt/volume1/tmp-overflow/rebake-v2.log BEFORE re-pushing this time: kernels_logs serves only the current version, so v1's log was destroyed by the v2 push along with the tracebacks I needed.
1 parent f871121 commit 8ac3b8f

1 file changed

Lines changed: 16 additions & 1 deletion

File tree

tools/kaggle/regression-rebake/rebake.py

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,22 @@
2525
os.environ["CRISPASR_REGRESSION_UPLOAD"] = "1"
2626
os.environ["CRISPASR_REGRESSION_BUILD"] = os.environ.get("CRISPASR_REGRESSION_BUILD", "cpu")
2727

28-
SCRIPT_VERSION = "2026-09-07-rebake-1"
28+
# HIDE THE GPU FROM TORCH. v2 lost 8 backends to
29+
# torch.AcceleratorError: CUDA error: no kernel image is available
30+
# which is kaggle_usage.md gotcha #23: Kaggle's preinstalled torch has dropped
31+
# sm_60, so a P100 draw is fatal to any torch code — and P100 is effectively the
32+
# only draw right now (#21). nemotron, canary-1b-v2, parakeet-tdt-0.6b-ja,
33+
# granite-4.1-*, voxtral4b and vibevoice each downloaded their model, loaded it,
34+
# and died on the first kernel launch, ~100 s each.
35+
#
36+
# Reference dumps do not need a GPU — the canonical regression kernel is
37+
# deliberately enable_gpu:false. But a CPU-only Kaggle worker loses internet
38+
# (gotcha #3) and this job must pull from HF, so the kernel keeps its GPU and
39+
# simply does not show it to torch. Same effect, without giving up the network.
40+
os.environ["CUDA_VISIBLE_DEVICES"] = ""
41+
os.environ["CRISPASR_REF_DEVICE"] = "cpu"
42+
43+
SCRIPT_VERSION = "2026-09-07-rebake-3-cpu-torch"
2944
WORK = pathlib.Path("/kaggle/working")
3045

3146
# Clone into a SEPARATE bootstrap dir: the canonical script manages its own

0 commit comments

Comments
 (0)