Exact and sequential-importance-sampling fixed-margin samplers#6
Open
mkborregaard wants to merge 7 commits into
Open
Exact and sequential-importance-sampling fixed-margin samplers#6mkborregaard wants to merge 7 commits into
mkborregaard wants to merge 7 commits into
Conversation
Add two independent-draw alternatives to the curveball sampler, both from Miller & Harrison, selected via the `method` keyword on `randomize_matrix!` and `matrixrandomizer`: - `exact`: exact uniform sampling via memoized counting of fixed-margin binary matrices (Miller & Harrison 2013). The generator counts once and reuses the result across draws. - `sis`: sequential importance sampling, a fast near-uniform proposal (Harrison & Miller 2013), suited to large matrices. Implements the uniform case; weighted sampling is left as a future extension. The dynamic programs are expressed as small dispatch-based functions over bespoke state types (ExactProblem/ExactCounts/Cursor, SISResidual/ SISWorkspace/ColumnScorer). Verified against brute-force counts and a uniformity check; the SIS proposal probability is validated by recovering the matrix count via importance sampling. Also bump StatsBase compat to 0.34 and the package version to 0.2.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the per-column full re-sort with an O(ones-placed) repair of the descending row order (each placed row drops by one, so it only slides past the block holding its old value), tracked via an added inverse-permutation field. Also clear only the touched band of the DP buffer each row instead of the whole vector, and normalise with a reciprocal multiply. ~20% faster per draw with identical output; importance-weight count estimates unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The curveball generator now warm-starts its chain from one independent SIS draw instead of the observed matrix, so the first sample is decorrelated from the input. A new `trades` keyword on randomize_matrix! and matrixrandomizer sets how many trades each draw performs (default 5*ncols), controlling thinning between successive samples. Refactor the generator to dispatch a single draw on a per-method sampler type (CurveballSampler/SISSampler/ExactCounts) rather than branching on the method enum, keeping each method's draw logic in its own file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bare `show` defined a new RandomBooleanMatrices.show instead of extending Base.show, so the custom display never fired. Qualify it and use print (no trailing newline) so it shows as intended. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalise the SIS sampler to the Harrison & Miller (2013) weighted target P*(z) ∝ ∏ w[i,j]^z[i,j], of which the uniform sampler is the w≡1 special case. A WeightModel (UniformWeights / WeightMatrix) multiplies each row's odds of a one by a factor v_i built from the elementary symmetric polynomials of the row's remaining weights (carried in logs), leaving the dynamic program otherwise unchanged. Each draw now carries its log importance weight log(∏w^z / Q*), and the public importance_sampler(m, w) returns (matrix, logweight) pairs whose mean weight estimates the weighted count κ. Verified: the mean importance weight recovers brute-force κ within noise, the uniform case reduces to the matrix count, and cv² stays low. Scope: strictly positive weights; structural zeros, weight canonicalisation, and variance-based column ordering are deferred (they affect efficiency, not correctness). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Allow nonnegative weights. A zero weight is a structural zero forbidding a one at that position: the log-space symmetric polynomials and the v_i factor drop those positions automatically, and a draw the zeros leave no way to complete is returned with logweight = -Inf (importance weight zero) rather than erroring. The v_i binomial term now uses the count of available (positive) positions in the row's remaining columns. Build the proposal from the Sinkhorn-canonical w̄ (eq. 16) by default (canonicalize=false to opt out); P* is invariant to it, so the importance weights are unchanged. Verified on zero-diagonal targets (κ recovered, 0% rejections) and that canonicalize on/off give the same estimate. Also make the exact m3!=m4 test deterministic and robust (it relied on an unseeded generator and a large enough support). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6 +/- ##
==========================================
+ Coverage 90.78% 97.31% +6.52%
==========================================
Files 2 6 +4
Lines 76 447 +371
==========================================
+ Hits 69 435 +366
- Misses 7 12 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two new fixed-margin matrix samplers alongside
curveball, plus a weighted importance sampler, and updates dependencies. The public API (randomize_matrix!,matrixrandomizer) is extended with amethodkeyword; weighted sampling gets its ownimportance_samplerentry point.New methods
exact— exact uniform sampling and counting (Miller & Harrison 2013). Ported from the authors' referenceexact_pure.py; counts withBigInt, samples row-by-row from the cached count. Feasible for small matrices (rows + cols ≲ 100, or very sparse); the generator counts once and reuses it.sis— the sequential importance sampling heuristic (Harrison & Miller 2013), uniform case. Builds a matrix column-by-column via a banded dynamic program over per-row Canfield odds and Gale–Ryser feasibility. Independent draws, scales to large matrices.Both are exposed through the existing generator, which now dispatches a single draw on a per-method sampler type (
CurveballSampler/SISSampler/ExactCounts) rather than branching on the enum.Weighted importance sampling
importance_sampler(m, w)targets P*(z) ∝ ∏ w[i,j]^z[i,j] over fixed-margin binary matrices (uniformsisis thew ≡ 1case). Eachrandreturns(matrix, logweight), so estimates reweight byexp(logweight)—mean(exp(logweight))estimates the weighted count κ, and a weight-weighted average estimatesE[h(Z)].vᵢbuilt from the elementary symmetric polynomials of the row's remaining weights (carried in logs).w[i,j]=0, e.g. a zero diagonal for directed graphs) forbid a one there; an uncompletable draw returnslogweight = -Infand is discarded.canonicalize=falseto opt out);P*is invariant to it.Curveball improvements
The curveball generator now warm-starts its chain from an independent SIS draw, so the first sample is decorrelated from the (often atypical) input matrix, and a
tradeskeyword controls how many trades separate successive draws (thinning). Empirically the running chain samples uniformly (χ² ≈ df).Other
StatsBase0.34); resolves to the latest releases.MatrixGenerator'sshowdefined a shadowing local instead of extendingBase.show.curveball(9),sis(6),exact(11),weighted sis(10) — all green, with brute-force κ/count cross-checks.Not included (efficiency-only)
The paper's variance-based column ordering for the weighted sampler; the importance weights are already correct and low-variance without it.
🤖 Generated with Claude Code