-
Notifications
You must be signed in to change notification settings - Fork 58
Expand file tree
/
Copy pathindex.html
More file actions
563 lines (562 loc) · 29.1 KB
/
Copy pathindex.html
File metadata and controls
563 lines (562 loc) · 29.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Schur Covariance Evaluation</title>
<meta name="description" content="A Principled Pseudo-Likelihood in
High Dimensions">
<meta property="og:title" content="Schur Covariance Evaluation">
<meta property="og:type" content="article">
<link rel="stylesheet" href="../../academic.css">
<script defer=""
src="https://cdn.jsdelivr.net/npm/katex@latest/dist/katex.min.js"></script>
<script>document.addEventListener("DOMContentLoaded", function () {
var mathElements = document.getElementsByClassName("math");
var macros = [];
for (var i = 0; i < mathElements.length; i++) {
var texText = mathElements[i].firstChild;
if (mathElements[i].tagName == "SPAN") {
katex.render(texText.data, mathElements[i], {
displayMode: mathElements[i].classList.contains('display'),
throwOnError: false,
macros: macros,
fleqn: false
});
}}});
</script>
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/npm/katex@latest/dist/katex.min.css" />
<style>
.site-nav{position:sticky;top:0;z-index:1000;background:#34495e;color:#fff;
padding:12px 24px;font-family:'Times New Roman',Times,serif;}
.site-nav-inner{max-width:820px;margin:0 auto;display:flex;
justify-content:space-between;gap:16px;flex-wrap:wrap;}
.site-nav a{color:#ecf0f1;text-decoration:none;}
.site-nav a:hover{color:#fff;text-decoration:underline;}
.container{max-width:820px;} .paper-meta{text-align:center;color:var(--muted);margin-bottom:8px;}
.katex-display{overflow-x:auto;overflow-y:hidden;}
</style>
</head>
<body>
<nav class="site-nav"><div class="site-nav-inner">
<a href="../../">← precise</a><a href="../">papers</a>
</div></nav>
<div class="header"><div class="container">
<h1>Schur Covariance Evaluation</h1><div class="subtitle">A
Principled Pseudo-Likelihood in High Dimensions</div>
</div></div>
<div class="container">
<p class="paper-meta">Peter Cotton · Draft —
<code>microprediction/precise</code> · <a href="https://github.com/microprediction/precise/blob/main/papers/schur_likelihood_paper.pdf">PDF</a></p>
<div class="abstract"><p>We introduce the <em>Schur
pseudo-likelihood</em>, a one-parameter family that damps the
cross-block coupling of the Gaussian likelihood through its Schur
complements, for <em>scoring</em> and regularizing covariance and
correlation estimates in high dimensions. The Gaussian log-likelihood is
the standard criterion, but when the dimension rivals the sample size it
is governed by the smallest, least-identifiable eigenvalues of the
estimate, and as a criterion for ranking or selecting estimates it
becomes unreliable; damping the coupling restores it. The optimal
damping has a closed form—the reliability of the coupling, a James–Stein
shrinkage. On real crypto returns, choosing a shrinkage estimate by the
Schur pseudo-likelihood rather than by the full likelihood yields
several-fold lower out-of-sample portfolio variance once the dimension
rivals the sample size.</p></div>
<h2 id="the-high-dimensional-failure-of-the-gaussian-likelihood">The
high-dimensional failure of the Gaussian likelihood</h2>
<p>A correlation matrix is the hard part of a covariance matrix:
variances are a one-dimensional, per-series problem, while the
high-dimensional difficulty—the near-singular spectrum and the unstable
inverse—lives in the correlation. The standard way to judge a
correlation (or covariance) estimate <span
class="math inline">\widehat{R}</span> is the log-likelihood it assigns
to held-out data, <span class="math inline">\ell(\widehat{R}) =
-\tfrac12(\log\det\widehat{R}+
\operatorname{tr}(\widehat{R}^{-1}S))</span> for a test covariance <span
class="math inline">S</span>. Both terms are governed by the smallest
eigenvalues of <span class="math inline">\widehat{R}</span>: <span
class="math inline">\log\det\widehat{R}=\sum_i\log\lambda_i</span>
diverges as any <span class="math inline">\lambda_i\to0</span>, and
<span class="math inline">\widehat{R}^{-1}</span> diverges with it. When
<span class="math inline">p</span> rivals <span
class="math inline">n</span> those eigenvalues are unidentifiable (the
Marčenko–Pastur regime), so the likelihood is dominated by the least
reliable part of the spectrum <span class="citation"
data-cites="ledoit2004 bun2017">(Ledoit and Wolf 2004; Bun et al.
2017)</span>. As a <em>criterion</em>, it then fails: in a controlled
experiment in which the truth is known, the held-out likelihood ranks
the better of two estimates below chance, whereas inversion-free
criteria such as Frobenius distance and the variogram score <span
class="citation" data-cites="scheuerer2015">(Scheuerer and Hamill
2015)</span> exceed <span class="math inline">80\%</span>. This is the
evaluation-side analogue of the instability of the minimum-variance
portfolio <span
class="math inline">w\propto\widehat{R}^{-1}\mathbf1</span>, which fails
through the same inverse. The remedy is to discount that
inverse—continuously, through one interpretable parameter, rather than
by discarding the cross-block structure outright.</p>
<h2 id="the-schur-pseudo-likelihood">The Schur pseudo-likelihood</h2>
<p>Partition the variables into <span class="math inline">K</span>
ordered blocks. The Gaussian density factors exactly into block
conditionals, <span class="math inline">p(x)=\prod_k p(x_k\mid
x_{<k})</span>, each Gaussian with covariance the Schur complement
<span class="math display">S_k \;=\; R_{kk} -
R_{k,<k}\,R_{<k,<k}^{-1}\,R_{<k,k},
\qquad
\mu_{k\mid<k} = R_{k,<k}R_{<k,<k}^{-1}x_{<k},</span> so
<span class="math inline">\log\det R=\sum_k\log\det S_k</span> and the
likelihood is a sum of block-conditional terms, each inverting only the
conditioning block <span class="citation"
data-cites="vecchia1988 katzfuss2021">(Vecchia 1988; Katzfuss and
Guinness 2021)</span>. We damp the coupling by <span
class="math inline">\gamma\in[0,1]</span>, <span
class="math display">S_k(\gamma) = (1-\gamma)\,R_{kk} + \gamma\,S_k,
\qquad
\mu_{k\mid<k}(\gamma) =
\gamma\,R_{k,<k}R_{<k,<k}^{-1}x_{<k},</span> and score <span
class="math inline">\ell_\gamma(R;x)=\sum_k\log\mathcal
N(x_k;\mu_{k\mid<k}(\gamma),S_k(\gamma))</span>. At <span
class="math inline">\gamma=1</span> this is the exact joint likelihood;
at <span class="math inline">\gamma=0</span> it is the block-diagonal
composite likelihood <span class="citation"
data-cites="besag1975 lindsay1988 varin2011">(Besag 1975; Lindsay 1988;
Varin et al. 2011)</span>; intermediate <span
class="math inline">\gamma</span> inverts no block larger than a
conditioning set and is better conditioned. The same <span
class="math inline">\gamma</span> interpolates an allocation rule, from
hierarchical risk parity at <span class="math inline">\gamma=0</span>
<span class="citation" data-cites="lopezdeprado2016">(López de Prado
2016)</span> to minimum variance at <span
class="math inline">\gamma=1</span> <span class="citation"
data-cites="cotton2024">(Cotton 2024)</span>.</p>
<h2 id="the-optimal-damping-is-the-coupling-reliability">The optimal
damping is the coupling reliability</h2>
<p>In a tractable two-block model—<span
class="math inline">x_2=bx_1+\varepsilon</span> with conditional <span
class="math inline">R^2</span> equal to <span
class="math inline">\rho^2</span>—minimising the expected predictive
residual of the damped score gives a closed form, <span
class="math display">\gamma^\star \;=\;
\frac{b^2}{b^2+\operatorname{Var}(\hat b)}
\;=\; \frac{(n-2)\,\rho^2}{(n-2)\,\rho^2+(1-\rho^2)},</span>
the <em>reliability</em> of the estimated coupling: a Wiener/James–Stein
shrinkage <span class="citation"
data-cites="james1961 ledoit2012">(James and Stein 1961; Ledoit and Wolf
2012)</span>. It tends to <span class="math inline">1</span> as <span
class="math inline">n\to\infty</span> or <span
class="math inline">\rho^2\to1</span> and to <span
class="math inline">0</span> as <span
class="math inline">\rho^2\to0</span>, and is interior when the coupling
is partially reliable. The empirical optimum tracks it: sweeping a
synthetic block correlation’s coupling strength <span
class="math inline">v</span> against the aspect ratio <span
class="math inline">n/p</span> (Figure <a href="#fig:gamma"
data-reference-type="ref" data-reference="fig:gamma">1</a>), the <span
class="math inline">\gamma</span> minimising recovery error rises
monotonically with <span class="math inline">v</span>, from <span
class="math inline">0</span> (no coupling) to <span
class="math inline">1</span> (strong coupling), and dips where samples
are scarcest—the reliability law, recovered across the plane.</p>
<figure id="fig:gamma" data-latex-placement="h">
<img src="figures/corr_gamma_star.png" style="width:84.0%" />
<figcaption>Optimal Schur <span class="math inline">\gamma^\star</span>
over coupling strength <span class="math inline">v</span> and aspect
ratio <span class="math inline">n/p</span> (<span
class="math inline">p=128</span>, synthetic). It rises monotonically
with <span class="math inline">v</span>—the reliability law—and dips
where samples are scarcest.</figcaption>
</figure>
<h2 id="selecting-estimates-in-high-dimensions-real-data">Selecting
estimates in high dimensions: real data</h2>
<p>The decisive test of a <em>scoring rule</em> is whether the estimate
it prefers generalizes. We test it on real daily crypto returns (<span
class="math inline">72</span> liquid assets) with a same-frequency,
disjoint-split design that avoids both look-ahead and the
cross-frequency (Epps) attenuation that confounds naive
hourly-versus-daily comparisons: days are partitioned at random into a
training set, a validation set (on which each scoring rule chooses among
a <em>structurally diverse</em> pool of candidate estimates—empirical,
identity, OAS, Ledoit–Wolf, factor models of several ranks, and
block-Schur estimates), and a held-out test set. The external criterion
is the realized variance of the minimum-variance portfolio formed from
the selected estimate—neither scoring rule, so the comparison is not
circular. We use a structural pool, not a one-dimensional shrinkage
ladder, precisely so that “a good scoring rule” cannot reduce to “select
more shrinkage.”</p>
<p>A clear split emerges (Figure <a href="#fig:select"
data-reference-type="ref" data-reference="fig:select">2</a>). When <span
class="math inline">n/p\le1</span> the <em>inverse-heavy</em> scores
select fragile estimates: the full likelihood and Stein loss give median
out-of-sample variance roughly an order of magnitude higher than the
rest, and tend to pick the raw empirical estimate. The
<em>inverse-discounting</em> scores—the Schur pseudo-likelihood, the
composite likelihood, and the matched GMV criterion—instead select a
structurally sensible estimate (most often a low-rank factor model
capturing the market factor) and cluster near the safe optimum. The
honest qualifier is the baseline: against a parameter-free <em>batch
Ledoit–Wolf</em> that uses no held-out selection at all, Schur-selection
is a <em>tie</em> (within a few percent across the range). So the win is
over the full likelihood—which fails as a high-dimensional
criterion—rather than over good shrinkage; selecting by <span
class="math inline">\ell_\gamma</span> does not beat simply shrinking
well. What it shares with good shrinkage is exactly “discount the
inverse,” and the Schur dial is the principled, closed-form way to do
so. Whether that dial can be turned to <em>beat</em> structure-agnostic
shrinkage is the subject of the next two sections.</p>
<figure id="fig:select" data-latex-placement="h">
<img src="figures/scoring_selection_crypto.png" style="width:78.0%" />
<figcaption>Real crypto. Out-of-sample minimum-variance-portfolio
variance (median, log scale) of the estimate <em>selected</em> by each
scoring rule, versus aspect ratio. Selecting by the Schur
pseudo-likelihood is several-fold safer than by the full likelihood when
undersampled; they converge once well sampled.</figcaption>
</figure>
<h2 id="the-same-selector-across-domains">The same selector across
domains</h2>
<p>The minimum-variance portfolio is one instance of <span
class="math inline">w\propto\Sigma^{-1}\mathbf1</span>; forecast
combination <span class="citation" data-cites="markowitz1952">(Markowitz
1952)</span>, Gaussian-process regression, and Kalman filtering are
others, and all are tuned by the same log-det-plus-quadratic
likelihood—indeed scalable GPs already use the block-conditional
(Vecchia) factorization <span class="citation"
data-cites="vecchia1988 katzfuss2021">(Vecchia 1988; Katzfuss and
Guinness 2021)</span> on which <span
class="math inline">\ell_\gamma</span> is built. We therefore run the
<em>identical</em> selection experiment on residual panels from five
domains: daily crypto returns; real numerical-weather forecast residuals
(archived forecast minus ERA5 reanalysis) on a <span
class="math inline">323</span>-point CONUS grid; draws from a known
block correlation and from a spatial Gaussian-process kernel; and the
innovations of a stable vector autoregression (Kalman-style residuals).
In every domain, when <span class="math inline">n/p\le1</span>,
selecting by the full likelihood gives higher out-of-sample
combined-residual variance than selecting by the Schur pseudo-likelihood
(Table <a href="#tab:domains" data-reference-type="ref"
data-reference="tab:domains">1</a>, Figure <a href="#fig:domains"
data-reference-type="ref" data-reference="fig:domains">3</a>)—by roughly
an order of magnitude in every domain—while the inverse-discounting
scores cluster near the safe optimum. As on crypto, the honest qualifier
is that Schur-selection only <em>ties</em> a parameter-free batch
Ledoit–Wolf (last column <span class="math inline">\approx1</span>): the
win is over the full likelihood, not over good shrinkage. The pathology
and its remedy are thus not financial: wherever a high-dimensional
covariance is inverted to combine correlated signals, the full
likelihood is a fragile selection criterion and any inverse-discounting
choice—Schur or plain shrinkage—avoids it.</p>
<div id="tab:domains">
<table>
<caption>Out-of-sample combined-residual-variance ratios, <span
class="math inline">n/p\le1</span> (diverse candidate pool). Column 1:
full-likelihood selection over Schur selection (<span
class="math inline">\gg1</span>: the full likelihood fails). Column 2:
parameter-free batch Ledoit–Wolf over Schur selection (<span
class="math inline">\approx1</span>: Schur ties good
shrinkage).</caption>
<thead>
<tr>
<th style="text-align: left;">domain</th>
<th style="text-align: center;">full-likelihood <span
class="math inline">/</span> Schur</th>
<th style="text-align: center;">batch Ledoit–Wolf <span
class="math inline">/</span> Schur</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">crypto (portfolio)</td>
<td style="text-align: center;">12.7</td>
<td style="text-align: center;">0.97</td>
</tr>
<tr>
<td style="text-align: left;">Gaussian-process field</td>
<td style="text-align: center;">13.1</td>
<td style="text-align: center;">0.95</td>
</tr>
<tr>
<td style="text-align: left;">VAR innovations (Kalman-style)</td>
<td style="text-align: center;">10.2</td>
<td style="text-align: center;">1.02</td>
</tr>
<tr>
<td style="text-align: left;">synthetic block correlation</td>
<td style="text-align: center;">10.2</td>
<td style="text-align: center;">1.01</td>
</tr>
<tr>
<td style="text-align: left;">weather forecast residuals (CONUS)</td>
<td style="text-align: center;">12.3</td>
<td style="text-align: center;">0.86</td>
</tr>
</tbody>
</table>
</div>
<figure id="fig:domains" data-latex-placement="h">
<img src="figures/generalize_domains.png" />
<figcaption>The same selection experiment across five domains:
out-of-sample combined-residual variance (median, log) of the estimate
selected by each scoring rule, versus aspect ratio. The full likelihood
is fragile when undersampled in every domain; inverse-discounting scores
are not.</figcaption>
</figure>
<h2
id="schurledoitwolf-an-analytic-optimal-coupling-damping">Schur–Ledoit–Wolf:
an analytic optimal coupling damping</h2>
<p>That Schur ties generic shrinkage but does not beat it suggests the
dial has not been turned to its optimum—we used a fixed <span
class="math inline">\gamma</span>. The fix is to estimate <span
class="math inline">\gamma</span> the Ledoit–Wolf way. Shrinking the
cross-block entries toward zero, the Frobenius-optimal keep fraction is,
in closed form, <span class="math display">\gamma^\star \;=\;
\frac{\sum_{\text{cross}}\sigma_{ij}^2}
{\sum_{\text{cross}}\bigl(\sigma_{ij}^2+\operatorname{Var}(s_{ij})\bigr)}
\;=\; 1-\delta^{\mathrm{LW}}_{\text{cross}},</span>
i.e. <em>Ledoit–Wolf shrinkage applied to the cross-block block
alone</em>, whose keep fraction is exactly the coupling reliability of
Section 3—now data-estimable, hence adaptive in <span
class="math inline">n</span>. Estimating <span
class="math inline">\operatorname{Var}(s_{ij})</span> by the empirical
fourth moment gives <strong>Schur–LW</strong>; by the Gaussian
finite-sample variance <span
class="math inline">\operatorname{Var}(r_{ij})\approx(1+r_{ij}^2)/n</span>
(less noisy at small <span class="math inline">n</span>, in the spirit
of OAS <span class="citation" data-cites="bun2017">(Bun et al.
2017)</span>) gives <strong>Schur–OAS</strong>.</p>
<p>We test recovery on a structured correlation in the spirit of options
grouped by underlying—<span class="math inline">G=10</span> groups of
<span class="math inline">m=6</span> members, strong within-group and
weak cross-group coupling—against a fair <em>block-aware</em>
Ledoit–Wolf (shrinkage toward a block-constant target) as well as
structure-agnostic shrinkage. The blocks are not assumed known: they are
recovered by clustering the (shrunk) training correlation, the premise
of hierarchical risk parity <span class="citation"
data-cites="lopezdeprado2016">(López de Prado 2016)</span>, which
succeeds almost perfectly here (adjusted Rand index <span
class="math inline">\approx0.99</span> even at <span
class="math inline">n/p=0.5</span>, so clustered blocks match the
oracle). Table <a href="#tab:schurlw" data-reference-type="ref"
data-reference="tab:schurlw">2</a> reports off-diagonal recovery
error.</p>
<div id="tab:schurlw">
<table>
<caption>Off-diagonal correlation-recovery RMSE (lower is better),
grouped (options-by-underlying) structure. <span
class="math inline">\widehat\gamma</span> is the data-estimated
cross-block keep fraction; it rises with <span
class="math inline">n</span>—the reliability law, learned from
data.</caption>
<thead>
<tr>
<th style="text-align: center;"><span
class="math inline">n/p</span></th>
<th style="text-align: center;"><span
class="math inline">\widehat\gamma</span></th>
<th style="text-align: center;">Schur–OAS</th>
<th style="text-align: center;">Schur (fixed <span
class="math inline">\gamma{=}\tfrac12</span>)</th>
<th style="text-align: center;">block-aware LW</th>
<th style="text-align: center;">batch LW</th>
<th style="text-align: center;">empirical</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center;">0.5</td>
<td style="text-align: center;">0.59</td>
<td style="text-align: center;">0.118</td>
<td style="text-align: center;"><strong>0.112</strong></td>
<td style="text-align: center;">0.119</td>
<td style="text-align: center;">0.145</td>
<td style="text-align: center;">0.176</td>
</tr>
<tr>
<td style="text-align: center;">1.0</td>
<td style="text-align: center;">0.66</td>
<td style="text-align: center;"><strong>0.095</strong></td>
<td style="text-align: center;">0.106</td>
<td style="text-align: center;">0.097</td>
<td style="text-align: center;">0.111</td>
<td style="text-align: center;">0.121</td>
</tr>
<tr>
<td style="text-align: center;">2.0</td>
<td style="text-align: center;">0.76</td>
<td style="text-align: center;"><strong>0.073</strong></td>
<td style="text-align: center;">0.103</td>
<td style="text-align: center;">0.075</td>
<td style="text-align: center;">0.081</td>
<td style="text-align: center;">0.085</td>
</tr>
<tr>
<td style="text-align: center;">4.0</td>
<td style="text-align: center;">0.84</td>
<td style="text-align: center;"><strong>0.056</strong></td>
<td style="text-align: center;">0.105</td>
<td style="text-align: center;">0.057</td>
<td style="text-align: center;">0.059</td>
<td style="text-align: center;">0.060</td>
</tr>
</tbody>
</table>
</div>
<p>The data-driven <span class="math inline">\widehat\gamma</span>
climbs from <span class="math inline">0.59</span> to <span
class="math inline">0.84</span> as <span class="math inline">n</span>
grows—the reliability law, estimated rather than assumed—so Schur–OAS
tracks the best estimator across the whole range, fixing the
over-damping of a fixed <span class="math inline">\gamma</span> (stuck
near <span class="math inline">0.10</span> once well sampled) while
keeping its undersampled edge. It beats structure-agnostic shrinkage
throughout and edges block-aware Ledoit–Wolf once adequately sampled,
with clustered blocks performing as well as the oracle. Two honest
caveats: the margin over block-aware Ledoit–Wolf is small (a few
percent), so the result is “a principled, parameter-free analytic
estimator competitive with the best block-aware shrinkage,” not a rout;
and at the most undersampled corner (<span
class="math inline">n/p=0.5</span>) the analytic <span
class="math inline">\widehat\gamma</span> is slightly high, where a
fixed smaller <span class="math inline">\gamma</span> still wins
narrowly. The substance is that the coupling reliability—the paper’s
<span class="math inline">\gamma^\star</span>—is a Ledoit–Wolf-style
estimable quantity, turning the Schur dial into a closed-form,
block-aware shrinkage estimator with no tuning.</p>
<figure id="fig:schurlw" data-latex-placement="h">
<img src="figures/known_blocks.png" style="width:82.0%" />
<figcaption>Grouped (options-by-underlying) recovery. The analytic
Schur–OAS/LW estimate (data-driven <span
class="math inline">\widehat\gamma</span>) tracks the best estimator
across aspect ratios, fixing the over-damping of a fixed <span
class="math inline">\gamma</span>, and ties or edges block-aware
Ledoit–Wolf; clustered blocks match the oracle.</figcaption>
</figure>
<h2 id="regularization-and-conditioning">Regularization and
conditioning</h2>
<p>The damping is a structured shrinkage: <span
class="math inline">S_k(\gamma)=(1-\gamma)R_{kk}+\gamma S_k</span>, a
convex combination of positive-definite matrices, pulls each conditional
covariance toward its unconditional block. This raises the smallest
eigenvalues and lowers the condition number relative to the full
likelihood, and yields a usable inverse even when the empirical
correlation is singular (<span class="math inline">n<p</span>); the
block-diagonal endpoint is positive definite whenever the blocks are.
Where the linear interpolation can itself approach singularity—near a
vanishing Schur complement under strong coupling—the affine-invariant
SPD-geodesic interpolation lifts the small eigenvalues multiplicatively
and stays well conditioned by construction. This conditioning is the
method’s general high-dimensional benefit: the same <span
class="math inline">\gamma</span> that maximises the pseudo-likelihood
regularizes any downstream use of the inverse, the evaluation-side
analogue of eigenvalue shrinkage <span class="citation"
data-cites="ledoit2004 ledoit2012 bun2017">(Ledoit and Wolf 2004, 2012;
Bun et al. 2017)</span>.</p>
<h2 id="scope-and-honest-limits">Scope and honest limits</h2>
<p>Three honest boundaries. First, as a <em>scoring rule</em> the
contribution is sharp and robust: the full Gaussian likelihood fails as
a high-dimensional selection criterion (an order of magnitude worse
out-of-sample, across five domains), and <span
class="math inline">\ell_\gamma</span>—like any inverse-discounting
score—does not. But <span
class="math inline">\ell_\gamma</span>-selection only <em>ties</em> a
parameter-free batch Ledoit–Wolf; it does not beat good shrinkage.
Second, as an <em>estimator</em>, fixed-<span
class="math inline">\gamma</span> Schur is regime-specific, but the
analytic <em>Schur–Ledoit–Wolf</em> of Section 6—closed-form <span
class="math inline">\gamma^\star=</span> reliability—is a competitive,
tuning-free block-aware shrinkage when the blocks are known or
(equivalently here) discovered by clustering; on dense, unstructured
data it simply reverts toward plain shrinkage. It is competitive with,
not dominant over, the best block-aware shrinkage. Third, the optimal
damping inverts with the target—recovering the matrix wants large <span
class="math inline">\gamma</span>, recovering its inverse (partial
correlations) wants small <span class="math inline">\gamma</span>—so
“best” is undefined until the target is named. The durable contributions
are the diagnosis, the closed-form reliability that unifies scoring,
regularization, and allocation, and the resulting parameter-free
Schur–Ledoit–Wolf estimator.</p>
<h2 id="conclusion">Conclusion</h2>
<p>The Schur pseudo-likelihood is a one-knob interpolation between
decoupled and full Gaussian scoring whose optimal setting is the
closed-form reliability of the cross-block coupling. In high dimensions
the Gaussian likelihood is dominated by unidentifiable small eigenvalues
and fails as a selection criterion—by an order of magnitude
out-of-sample, across portfolios, weather forecast residuals,
Gaussian-process fields, and Kalman-style innovations alike—while any
inverse-discounting score avoids the failure. The same reliability that
governs the score has a Ledoit–Wolf form, yielding a parameter-free,
block-aware shrinkage estimator (Schur–Ledoit–Wolf) that adapts its
coupling trust to the data and is competitive with the best block-aware
shrinkage when structure is present or discovered. The honest summary is
modest but unified: discount the inverse in high dimensions, do it by
the coupling reliability, and the same closed form serves scoring,
regularization, and allocation.</p>
<div id="refs" class="references csl-bib-body hanging-indent"
role="list">
<div id="ref-besag1975" class="csl-entry" role="listitem">
Besag, Julian. 1975. <span>“Statistical Analysis of Non-Lattice
Data.”</span> <em>Journal of the Royal Statistical Society: Series D
(The Statistician)</em> 24 (3): 179–95.
</div>
<div id="ref-bun2017" class="csl-entry" role="listitem">
Bun, Joël, Jean-Philippe Bouchaud, and Marc Potters. 2017.
<span>“Cleaning Large Correlation Matrices: Tools from Random Matrix
Theory.”</span> <em>Physics Reports</em> 666: 1–109.
</div>
<div id="ref-cotton2024" class="csl-entry" role="listitem">
Cotton, Peter. 2024. <span>“Schur Complementary Portfolios.”</span>
<em>arXiv Preprint arXiv:2411.05807</em>.
</div>
<div id="ref-james1961" class="csl-entry" role="listitem">
James, W., and Charles Stein. 1961. <span>“Estimation with Quadratic
Loss.”</span> <em>Proceedings of the Fourth Berkeley Symposium on
Mathematical Statistics and Probability</em> 1: 361–79.
</div>
<div id="ref-katzfuss2021" class="csl-entry" role="listitem">
Katzfuss, Matthias, and Joseph Guinness. 2021. <span>“A General
Framework for Vecchia Approximations of Gaussian Processes.”</span>
<em>Statistical Science</em> 36 (1): 124–41.
</div>
<div id="ref-ledoit2004" class="csl-entry" role="listitem">
Ledoit, Olivier, and Michael Wolf. 2004. <span>“A Well-Conditioned
Estimator for Large-Dimensional Covariance Matrices.”</span> <em>Journal
of Multivariate Analysis</em> 88 (2): 365–411.
</div>
<div id="ref-ledoit2012" class="csl-entry" role="listitem">
Ledoit, Olivier, and Michael Wolf. 2012. <span>“Nonlinear Shrinkage
Estimation of Large-Dimensional Covariance Matrices.”</span> <em>The
Annals of Statistics</em> 40 (2): 1024–60.
</div>
<div id="ref-lindsay1988" class="csl-entry" role="listitem">
Lindsay, Bruce G. 1988. <span>“Composite Likelihood Methods.”</span>
<em>Contemporary Mathematics</em> 80: 221–39.
</div>
<div id="ref-lopezdeprado2016" class="csl-entry" role="listitem">
López de Prado, Marcos. 2016. <span>“Building Diversified Portfolios
That Outperform Out of Sample.”</span> <em>The Journal of Portfolio
Management</em> 42 (4): 59–69.
</div>
<div id="ref-markowitz1952" class="csl-entry" role="listitem">
Markowitz, Harry. 1952. <span>“Portfolio Selection.”</span> <em>The
Journal of Finance</em> 7 (1): 77–91.
</div>
<div id="ref-scheuerer2015" class="csl-entry" role="listitem">
Scheuerer, Michael, and Thomas M. Hamill. 2015. <span>“Variogram-Based
Proper Scoring Rules for Probabilistic Forecasts of Multivariate
Quantities.”</span> <em>Monthly Weather Review</em> 143 (4): 1321–34.
</div>
<div id="ref-varin2011" class="csl-entry" role="listitem">
Varin, Cristiano, Nancy Reid, and David Firth. 2011. <span>“An Overview
of Composite Likelihood Methods.”</span> <em>Statistica Sinica</em> 21
(1): 5–42.
</div>
<div id="ref-vecchia1988" class="csl-entry" role="listitem">
Vecchia, Aldo V. 1988. <span>“Estimation and Model Identification for
Continuous Spatial Processes.”</span> <em>Journal of the Royal
Statistical Society: Series B</em> 50 (2): 297–312.
</div>
</div>
</div>
<footer><div class="container"><p>precise · MIT licensed ·
<a href="https://github.com/microprediction/precise">microprediction/precise</a></p></div></footer>
</body>
</html>