Papers: Schur Pseudo-Likelihood + Two Sides of Schur Damping - #50
Merged
Conversation
Empirical additions backing the Schur-likelihood claims, generated from the precise-lab experiment harness: - schur_likelihood_paper.tex: regime map (gamma* is regime-dependent; Schur(1/2) wins low-n/p high-p cells); two-block gamma*_eval == gamma*_alloc == reliability, with allocation the more conservative corner (HRP = reliability->0); the relation survives a multi-level hierarchy with no compounding gap; judge-power-by-gold table (Schur is the robust all-rounder, best for matrix recovery). - covariance_evaluation.md: 'match the judge to the objective' meta-evaluation with the shipped assessors; likelihood is KL-parochial; two distinct power-limiting axes (inverse-fragility vs probe-rank). Numbers tagged [prelim] refresh when the full 28,800-cell grid and judge-power runs complete. Figures in papers/figures/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Title -> 'Schur Pseudo-Likelihood / A Principled Way to Evaluate Covariance in High Dimensions'. Add booktabs and graphicx to the preamble, required by the empirical tables and figures (paper now compiles clean under tectonic). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Judge-power experiment complete (all golds, all c). Corrections to the first draft: - 'likelihood collapses to chance' -> it DECLINES with concentration c on the practical golds (Frobenius 0.75->0.60, GMV 0.65->0.56 as c:0.5->2); not literally chance in this range. - drop 'Schur is the robust all-rounder': by worst-case across golds the likelihood family is most balanced. Accurate story: inversion-light judges (VariogramScore 0.79, SchurLikelihood 0.77) win matrix recovery and are flat in c; GMVVariance is the allocation specialist (0.96, rank-1 probe); no gold-free best judge. - final power table; judge_power_by_gold figure refreshed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Final synthetic grid (200 reps) corrects the preliminary estimator claim: - SchurCovariance wins NO cell as a point estimator on synthetic data; shrinkage (OAS/Ledoit-Wolf) wins n/p<=1, Empirical wins n/p>=4. This matches the abstract: l_gamma is a scoring/regularization device, not an estimation objective. - Real data: the block-diagonal Schur corner (gamma=0) wins a plurality of realized -GMV cells on crypto and ff49 (the reliability->0 regime of noisy returns); OAS/LW win ff100; Diagonal wins Polymarket. - Reconcile the judge-power numbers with the opening section: the broad-pool ranking (likelihood ~0.6) is an easier task than the near-optimal-pair discrimination (~0.45 below chance), a conservative reading of the same effect. - Verified recursion paragraph (bisection != GMV; level-Schur exact; gap non-monotone in depth). Stability: Tyler 100% non-PD, GeoEWA ~10%. - Figures refreshed with LaTeX-math labels. Paper compiles clean under tectonic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Branch correlation-paper. Retarget the contribution to correlation (the hard, high-dimensional half of covariance), where the Schur dial earns its keep: - title -> 'Estimating and Evaluating Correlation in High Dimensions' - abstract: add the correlation results (gamma* = reliability surface; significant- but-modest interior-gamma wins; matrix-vs-inverse inversion; structure-dependent precision recovery) - new section 'Correlation estimation: the natural home' with the gamma* reliability surface and (v,n/p) win-region figures, paired-bootstrap significance, and the matrix-vs-inverse / AR(1)-Schur(0.75) precision result. Covariance results retained as the general frame. Real-data devol (DCC/EWMA) panels still to add. Compiles clean under tectonic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the patched covariance+correlation hybrid with a single clean correlation paper (150 lines): 3-sentence abstract, correlation throughout, no covariance-first framing, no developmental/provenance comments, no [prelim]. Sections: intro (high-dim correlation, inverse-fragility) -> Schur pseudo-likelihood -> gamma*=reliability closed form -> correlation recovery (gamma* surface, win region) -> matrix-vs-inverse. Figures: bigger fonts, muted colourblind-friendly family palette (dropped tab20/orange). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rypto) Broadened, subset-based study: 61 weekly+monthly windows, 51 crypto assets, random sub-universes, n/p in [0.15,3.8]. Hourly-realized correlation as proxy truth; daily returns as the undersampled sample. Interior-gamma Schur (~0.5) recovers best at every n/p; window-clustered bootstrap shows it significantly beats OAS (+0.027) and both endpoints; best estimator in 57% of windows. Honest scope: recovery metric (not OOS likelihood, under which it loses), Epps caveat, no trading claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the correct framing: l_gamma is a scoring rule + regularizer, not a recovery estimator. Rewrite around that. - Lead claim: in high-D the Gaussian likelihood fails as a CRITERION (dominated by unidentifiable small eigenvalues); damping restores it. - Core empirical (real crypto, same-frequency disjoint split, external = OOS GMV variance, no circularity): as a model SELECTOR among shrinkage candidates, the inverse-heavy scores (full likelihood, Stein) under-shrink and are ~6x more fragile OOS when n/p<=1; the inverse-discounting scores (Schur, composite, matched-GMV) select near-optimally. Schur ties the composite endpoint on dense crypto (low reliability -> small gamma*), beats full likelihood/Stein/variogram (window-clustered CIs). Honest: Schur does not uniquely dominate; it's the principled closed-form dial. - Keep gamma*=reliability + surface, regularization/conditioning. - Demote recovery to a scope section (regime-specific; crypto-recovery confound removed). - Drop stale figures; figs now gamma_star + scoring_selection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cross-domain selector testbed finalized with the genuine weather panel (945 days x 323 CONUS grid points; archived forecast minus ERA5 actual), replacing the earlier deseasonalized-anomaly proxy. Full-likelihood/Schur OOS-variance ratios (n/p<=1): GP field 18.1, VAR/Kalman 4.0, crypto 2.7, synthetic block 2.7, weather 1.8 (real forecast residuals are noisier/less correlated => mildest, the honest number). Inverse-discounting scores (Schur ~ composite ~ matched-GMV) safe in all five domains; inverse-heavy (full likelihood, Stein) fragile. Figure refreshed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t scoping - Selection sections redone on a structurally DIVERSE candidate pool (not a 1-D shrinkage ladder): full likelihood fails ~10-13x in every domain; but Schur-selection only TIES a fair batch Ledoit-Wolf baseline -- the win is over the likelihood, not over good shrinkage (honest). - NEW section: Schur-Ledoit-Wolf. gamma* = cross-block reliability = LW shrinkage on the cross-block block (Schur-OAS variant via Gaussian variance for small n). Data-driven gamma_hat rises 0.59->0.84 with n (reliability law, estimated); tracks best estimator across n/p, ties/edges block-aware LW, beats structure-agnostic; blocks discoverable by clustering (ARI~1, HRP premise). Honest caveats (few-percent margin; small-n corner). - Scope/conclusion rewritten to the unified honest story. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r identity) Companion to the Schur Pseudo-Likelihood note. The same Schur complement and coupling-reliability gamma* underlie both the high-dimensional pseudo-likelihood (Vecchia conditioning) and portfolio allocation (HRP<->minimum-variance): S = 1 - rho^2 is at once a conditional variance and a hedged residual risk. Adds the cross-domain identity, a two-variable worked example, a base-free transfer experiment, and credits the contemporaneous ShrinkTM (Chakraborty & Katzfuss 2025). Leaves paper 1 unchanged as the historical record. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s docs-up-to-date CI) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two companion notes under
papers/:🤖 Generated with Claude Code