-
Notifications
You must be signed in to change notification settings - Fork 58
Expand file tree
/
Copy pathindex.html
More file actions
197 lines (182 loc) · 12.3 KB
/
Copy pathindex.html
File metadata and controls
197 lines (182 loc) · 12.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>precise — online covariance & correlation estimation</title>
<meta name="description" content="precise: sklearn-style online (incremental) covariance and correlation estimators with a single partial_fit contract — the online complement to sklearn.covariance.">
<meta property="og:title" content="precise — online covariance & correlation estimation">
<meta property="og:description" content="Online (incremental) covariance/correlation estimators with a partial_fit contract; assessment and recommendation; the Schur pseudo-likelihood.">
<meta property="og:url" content="https://precise.microprediction.org/">
<meta property="og:type" content="website">
<meta property="og:site_name" content="precise">
<meta name="twitter:card" content="summary">
<link rel="stylesheet" href="academic.css">
<style>
.site-nav { position: sticky; top: 0; z-index: 1000; background: #34495e; color: white;
padding: 12px 24px; font-family: 'Times New Roman', Times, serif;
box-shadow: 0 2px 4px rgba(0,0,0,0.1); }
.site-nav-inner { max-width: 900px; margin: 0 auto; display: flex;
justify-content: space-between; align-items: center; gap: 16px; flex-wrap: wrap; }
.site-nav-brand { color: white; text-decoration: none; font-weight: 700; font-size: 1.25em;
letter-spacing: 0.02em; }
.site-nav-links { display: flex; gap: 24px; }
.site-nav-links a { color: #ecf0f1; text-decoration: none; font-size: 1em; }
.site-nav-links a:hover { color: white; text-decoration: underline; }
</style>
</head>
<body>
<nav class="site-nav">
<div class="site-nav-inner">
<a class="site-nav-brand" href="./">precise</a>
<div class="site-nav-links">
<a href="#install">Install</a>
<a href="#quickstart">Quick start</a>
<a href="#estimators">Estimators</a>
<a href="papers/">Papers</a>
<a href="https://github.com/microprediction/precise">GitHub</a>
<a href="https://pypi.org/project/precise/">PyPI</a>
</div>
</div>
</nav>
<div class="header">
<div class="container">
<h1>precise</h1>
<div class="subtitle">Online (incremental) covariance & correlation estimation —
the online complement to <code>sklearn.covariance</code></div>
</div>
</div>
<div class="container">
<div class="abstract">
<strong>precise</strong> is a small, numpy-only library of <em>online</em> covariance and
correlation estimators behind a single sklearn-style <code>partial_fit</code> contract.
Where <code>sklearn.covariance</code> estimators are batch-only, every estimator here
updates one observation at a time and exposes <code>covariance_</code>,
<code>correlation_</code>, <code>precision_</code> and <code>location_</code>. On top of
the estimators it adds a registry, river-style adapters for variables that enter and leave
a universe, a panel of <em>assessors</em> for judging an estimate, a recommender that
picks an estimator from observable data features, and the <em>Schur pseudo-likelihood</em>
— a one-parameter bridge between the full and block-diagonal Gaussian likelihoods.
</div>
<section class="section" id="install">
<h2>Install</h2>
<pre>pip install precise</pre>
<p class="text-muted">numpy is the only required dependency (tested on numpy 1.26 and
2.x, Python 3.9–3.13). Optional extras: <code>[pandas]</code> (DataFrame output for
the keyed adapters), <code>[research]</code>, <code>[dev]</code>.</p>
</section>
<section class="section" id="quickstart">
<h2>Quick start</h2>
<pre>import numpy as np
from precise import EwaCovariance
est = EwaCovariance(r=0.05) # exponentially weighted, online
for y in stream_of_vectors(): # y is a 1-D array, one observation
est.partial_fit(y)
est.covariance_ # (d, d) ndarray, always symmetric PSD
est.correlation_ # unit-diagonal correlation
est.precision_ # inverse (when well-conditioned)
est.location_ # running mean</pre>
<p>Every estimator follows the same contract, so a test harness (or your code) only needs
to look for <code>.fit</code> / <code>.partial_fit</code>. Discover them programmatically:</p>
<pre>from precise import all_estimators, estimator_from_name
all_estimators() # list of estimator classes
estimator_from_name("LedoitWolfCovariance")</pre>
</section>
<section class="section" id="estimators">
<h2>Estimators</h2>
<p>Fourteen truly-online estimators across families (no buffered recompute):</p>
<table>
<thead><tr><th>Class</th><th>Family</th><th>Notes</th></tr></thead>
<tbody>
<tr><td><code>EmpiricalCovariance</code></td><td>sample</td><td>Welford running covariance</td></tr>
<tr><td><code>DiagonalCovariance</code></td><td>sample</td><td>variances only</td></tr>
<tr><td><code>EwaCovariance</code></td><td>weighted</td><td>exponentially weighted</td></tr>
<tr><td><code>AdaptiveEwaCovariance</code></td><td>weighted</td><td>surprise-smoothed forgetting</td></tr>
<tr><td><code>LedoitWolfCovariance</code></td><td>shrinkage</td><td>online linear shrinkage</td></tr>
<tr><td><code>OASCovariance</code></td><td>shrinkage</td><td>oracle approximating shrinkage</td></tr>
<tr><td><code>ShrunkCovariance</code></td><td>shrinkage</td><td>fixed-intensity target</td></tr>
<tr><td><code>SchurCovariance</code></td><td>shrinkage</td><td>cross-block coupling damping (linear/geodesic)</td></tr>
<tr><td><code>PartialMomentsCovariance</code></td><td>semi</td><td>downside / semi-covariance</td></tr>
<tr><td><code>HuberCovariance</code></td><td>robust</td><td>online Huber M-estimator</td></tr>
<tr><td><code>TylerCovariance</code></td><td>robust</td><td>recursive Tyler M-estimator</td></tr>
<tr><td><code>GeodesicEwaCovariance</code></td><td>geometric</td><td>affine-invariant SPD geodesic step</td></tr>
<tr><td><code>DCCCovariance</code></td><td>dynamic</td><td>decoupled vol & correlation</td></tr>
<tr><td><code>FactorCovariance</code></td><td>factor</td><td>low-rank + diagonal, O(d·k)/step</td></tr>
</tbody>
</table>
</section>
<section class="section" id="dynamic">
<h2>Dynamic universes</h2>
<p>Finance rarely offers a fixed set of variables. The keyed adapters wrap <em>any</em>
positional estimator into a river-style one over named variables that enter and leave:</p>
<pre>from precise import keyed, EwaCovariance
est = keyed(EwaCovariance(r=0.05))
est.partial_fit({"AAPL": 0.01, "MSFT": -0.02}) # dict in, dict-of-dicts out
est.covariance_["AAPL"]["MSFT"]</pre>
<p class="text-muted"><code>FixedUniverse</code> imputes missing keys;
<code>DynamicUniverse</code> handles staleness/longevity eviction and pairwise assembly
across changing key sets.</p>
</section>
<section class="section" id="assess">
<h2>Assess & recommend</h2>
<p>Because no estimator dominates every regime, <code>precise</code> treats
<em>choosing</em> and <em>judging</em> an estimate as first-class:</p>
<pre>from precise import all_assessors, suggest
all_assessors() # LogLikelihood, BlockPseudoLikelihood,
# SchurLikelihood, SteinLoss, ... (higher = better)
suggest(X, top=3) # recommend estimator classes from data features</pre>
<p><code>suggest</code> maps observable, truth-free features (p/n, effective rank,
sphericity, condition number, off-diagonal mass, excess kurtosis) to an estimator via a
frozen, numpy-only decision tree.</p>
</section>
<section class="section" id="schur">
<h2>The Schur pseudo-likelihood</h2>
<p>The full Gaussian likelihood factorizes exactly, through Schur complements, into
block-conditional terms. A single <em>coupling-strength</em> parameter
<code>γ ∈ [0,1]</code> damps that conditioning, giving the <strong>Schur
pseudo-likelihood</strong> <code>ℓ<sub>γ</sub></code>: <code>γ=1</code> is the full
likelihood (fragile in high dimensions), <code>γ=0</code> is the block-diagonal /
composite likelihood (robust), and the interior is the bridge they lack. The same
<code>γ</code> damping of the same Schur complement is Schur complementary portfolio
allocation — interpolating hierarchical risk parity and minimum variance.</p>
<p>For the estimator/predictive use, the optimal trust has a closed form — the
<em>reliability</em> of the coupling (a Wiener / James–Stein shrinkage):</p>
<pre>γ* = (n−2) ρ² / [ (n−2) ρ² + (1 − ρ²) ] (ρ² = block coupling, the conditional R²)</pre>
<p>rising to 1 as the sample size or coupling grows, and to 0 as the coupling becomes
noise. See the working paper,
<a href="papers/schur-likelihood/"><em>Schur Covariance Evaluation: A Principled
Pseudo-Likelihood in High Dimensions</em></a> (<a href="papers/schur-likelihood/">read online</a> ·
<a href="https://github.com/microprediction/precise/blob/main/papers/schur_likelihood_paper.pdf">PDF</a>).</p>
</section>
<section class="section" id="links">
<h2>Links</h2>
<ul>
<li><a href="https://pypi.org/project/precise/">PyPI</a> — <code>pip install precise</code></li>
<li><a href="https://github.com/microprediction/precise">GitHub repository</a></li>
<li><a href="https://github.com/microprediction/precise/blob/main/MIGRATING.md">Migrating from the 0.16.x skater API</a></li>
<li><a href="papers/">Papers</a> — working papers on covariance/correlation estimation and the Schur likelihood</li>
<li><a href="https://allocation.microprediction.org">allocation.microprediction.org</a> — portfolio / allocation (Schur-complementary; moved out of precise)</li>
</ul>
</section>
<section class="section" id="skill">
<h2>Save the planet, one update at a time</h2>
<p>Online estimation does a constant amount of work per observation instead of recomputing
over a growing window — orders of magnitude less compute, and energy, over a long stream.
As a bonus, the <code>partial_fit</code> contract makes lookahead bias structurally hard: an
estimator only ever sees data up to the current observation, never the future. Drop the
<code>precise</code> skill into your own agent:</p>
<pre class="code-block" id="precise-skill-prompt"><code>Read https://raw.githubusercontent.com/microprediction/precise/main/SKILL.md
and create a project skill from it.</code></pre>
<p style="text-align: center;">
<button class="btn" onclick="(function(){const t=document.getElementById('precise-skill-prompt').innerText;navigator.clipboard.writeText(t).then(()=>{event.target.textContent='Copied!';setTimeout(()=>event.target.textContent='Copy',1500);});})()">Copy</button>
<a class="btn btn-secondary" href="https://github.com/microprediction/precise/blob/main/SKILL.md" target="_blank" rel="noopener">SKILL.md</a>
</p>
</section>
</div>
<footer>
<div class="container">
<p>precise · MIT licensed · <a href="https://github.com/microprediction/precise">microprediction/precise</a></p>
</div>
</footer>
</body>
</html>