Skip to content

Commit d6f8c4f

Browse files
ellisbrownellisbrown
committed
update
1 parent 618da95 commit d6f8c4f

3 files changed

Lines changed: 56 additions & 9 deletions

File tree

‎site/index.html‎

Lines changed: 56 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
<html lang="en">
33
<head>
44
<title>TsT: Test-Set Stress-Test</title>
5-
<link rel="icon" type="image/x-icon" href="figs/header.png">
5+
<link rel="icon" type="image/x-icon" href="static/img/header.png">
66

77
<meta charset="utf-8">
88
<meta name="viewport" content="width=device-width, initial-scale=1">
@@ -150,7 +150,7 @@ <h2>Benchmark Designers Should “Train on the Test Set”</h2>
150150
</div>
151151
</div>
152152
<div class="button-container">
153-
<a href="#abstract" class="button">
153+
<a href="https://arxiv.org/abs/2511.04655" class="button">
154154
<span class="icon is-small">
155155
<i class="ai ai-arxiv"></i>
156156
</span>
@@ -233,7 +233,7 @@ <h1 class="text">Overview</h1>
233233
<p class="text">
234234
We focus on <em>non-visual shortcuts</em>:
235235
cases where questions can be answered correctly without using the visual input at all.
236-
These shortcuts can come from natural world knowledge (e.g., “fridges are usually around 170–180cm tall”),
236+
These shortcuts can come from natural world knowledge (e.g., "fridges are usually around 170–180cm tall"),
237237
or from statistical quirks of the benchmark (e.g., certain answers appearing disproportionately often,
238238
or specific templates almost always mapping to the same label).
239239
Either way, when the goal is to measure <em>visual</em> understanding, such patterns undermine evaluation.
@@ -245,13 +245,33 @@ <h1 class="text">Overview</h1>
245245
<img src="static/img/teaser.png" alt="TsT overview teaser" class="pdf-figure" style="width: auto; height: auto; border: none;" data-zoomable="" draggable="false">
246246
<figcaption>
247247
<strong>Figure:</strong>
248-
From “blind” shortcuts to stress-tested benchmarks.
248+
From "blind" shortcuts to stress-tested benchmarks.
249249
TsT trains on the test set text in a \(k\)-fold manner to expose how much of a benchmark can be solved without vision,
250250
and uses those insights to build more robust versions that truly require visual reasoning.
251251
</figcaption>
252252
</figure>
253253
</d-figure>
254254
</section>
255+
256+
<section id="bias-examples" style="margin-top: 2rem; margin-bottom: 4rem;">
257+
<h2 class="text">Statistical Biases Create Non-Visual Shortcuts</h2>
258+
<p class="text">
259+
To make this concrete, here are four types of statistical biases we discovered across real benchmarks.
260+
These patterns enable models to achieve high accuracy without visual reasoning:
261+
</p>
262+
<d-figure>
263+
<figure>
264+
<img src="static/img/bias_examples.png" alt="Statistical bias examples" class="pdf-figure" style="width: 100%; height: auto; border: none;" data-zoomable="" draggable="false">
265+
<figcaption>
266+
<strong>Figure:</strong>
267+
<strong>(a) Counting:</strong> VSI-Bench shows severe answer skew—over 50% of questions have ground truth ≤3, enabling high accuracy by always guessing "2".
268+
<strong>(b) Spatial Relation:</strong> In CV-Bench depth, certain categories like "keyboard" and "clothes" appear as the correct answer 100% of the time.
269+
<strong>(c) Appearance Order:</strong> "Clock" appears in the 4th position in 100% of VSI-Bench questions where it appears.
270+
<strong>(d) Size Estimation:</strong> Room sizes cluster around typical dimensions (log μ ≈ 17m²), making them predictable without seeing the room.
271+
</figcaption>
272+
</figure>
273+
</d-figure>
274+
</section>
255275

256276
<ul class="text">
257277
<li>
@@ -283,6 +303,31 @@ <h1 class="text">Overview</h1>
283303

284304
<section id="shortcuts">
285305
<h1 class="text">Non-Visual Shortcuts Undermine Multimodal Evaluation</h1>
306+
307+
<h3 class="text" style="margin-top: 2rem;">Knowledge-Based Shortcuts</h3>
308+
<p class="text">
309+
The first category of shortcuts comes from world knowledge embedded in LLMs during pretraining.
310+
As shown below, benchmarks like MMMU benefit more from scaling the LLM backbone than from enabling vision,
311+
suggesting they rely heavily on linguistic knowledge. In contrast, VSI-Bench shows negligible gains from
312+
LLM scaling in blind settings but substantial improvements when vision is enabled—demonstrating greater
313+
robustness to knowledge-based shortcuts.
314+
</p>
315+
316+
<d-figure style="margin: 2rem 0;">
317+
<figure>
318+
<img src="static/img/blind_ov.png" alt="Knowledge-based shortcuts" class="pdf-figure" style="width: 95%; height: auto; border: none;" data-zoomable="" draggable="false">
319+
<figcaption>
320+
<strong>Figure:</strong>
321+
<strong>Knowledge-based shortcuts in multimodal benchmarks.</strong>
322+
Blind (red squares) vs. vision-enabled (blue circles) performance across LLaVA-OneVision model scales.
323+
MMMU shows substantial gains from scaling the LLM backbone (x-axis) but minimal improvement from enabling vision (y-axis),
324+
indicating reliance on linguistic knowledge. VSI-Bench demonstrates the opposite pattern—large vision gains with negligible
325+
blind scaling—confirming robustness to knowledge-based shortcuts.
326+
</figcaption>
327+
</figure>
328+
</d-figure>
329+
330+
<h3 class="text" style="margin-top: 2rem;">Statistical Shortcuts</h3>
286331
<p class="text">
287332
Not every pattern in a dataset is a shortcut.
288333
The key question is not where a pattern comes from (world statistics vs. annotation artifacts),
@@ -291,7 +336,7 @@ <h1 class="text">Non-Visual Shortcuts Undermine Multimodal Evaluation</h1>
291336
then for a vision-centric benchmark, that pattern is a problem.
292337
</p>
293338
<p class="text">
294-
For example, consider questions like “Which item is closest to the bed?” where “lamp”
339+
For example, consider questions like "Which item is closest to the bed?" where "lamp"
295340
happens to be the correct answer far more often than chance.
296341
Even if this reflects some real-world regularity, in a benchmark that is supposed to probe spatial reasoning,
297342
it lets models answer correctly by leaning on text-only priors rather than the actual image.
@@ -338,12 +383,14 @@ <h1 class="text">The Test-set Stress-Test (TsT) Framework</h1>
338383

339384
<d-figure>
340385
<figure>
341-
<img src="static/img/TsT_overview_bias_space.png" alt="Bias Space" class="pdf-figure" style="width: 50%; height: auto; border: none;" data-zoomable="" draggable="false">
342-
<img src="static/img/TsT_overview.png" alt="TsT framework overview" class="pdf-figure" style="width: 50%; height: auto; border: none;" data-zoomable="" draggable="false">
386+
<div style="display: flex; gap: 1rem; align-items: center; justify-content: center; flex-wrap: wrap;">
387+
<img src="static/img/TsT_overview_bias_space.png" alt="Bias Space" class="pdf-figure" style="width: 38%; height: auto; border: none;" data-zoomable="" draggable="false">
388+
<img src="static/img/TsT_overview.png" alt="TsT framework overview" class="pdf-figure" style="width: 58%; height: auto; border: none;" data-zoomable="" draggable="false">
389+
</div>
343390
<figcaption>
344391
<strong>Figure:</strong>
345-
The TsT pipeline.
346-
The test set is partitioned into \(k\) folds, a blind diagnostic model is trained on \(k{-}1\) folds and evaluated on the held-out fold,
392+
<strong>(Left)</strong> TsT directly probes biases intrinsic to the specific test set (pink region), rather than approximating them via external training data.
393+
<strong>(Right)</strong> The test set is partitioned into \(k\) folds, a blind diagnostic model is trained on \(k{-}1\) folds and evaluated on the held-out fold,
347394
and this is repeated until all samples are covered.
348395
Aggregating across folds yields both a global non-visual solvability estimate and per-sample bias scores \(s(x)\).
349396
</figcaption>

‎site/static/img/bias_examples.png‎

344 KB
Loading

‎site/static/img/blind_ov.png‎

89.6 KB
Loading

0 commit comments

Comments
 (0)