feat(math): add finite stochastic processes domain with sigma algebras, conditional expectation, filtrations, and Doob martingales (#1806) - #2036
Conversation
…s, conditional expectation, filtrations, and Doob martingales Add the finite_stochastic_processes domain implementing exact, bounded, deterministic finite stochastic process operations over immutable finite probability spaces: - probability.finite_sigma_algebra.from_observation.compute: sigma algebra generated by an observation map, with equal-value fibers as blocks - probability.finite_sigma_algebra.join.compute: least sigma algebra containing both inputs, as the finest common refinement - probability.conditional_expectation.finite.compute: E[X | G] as a block-constant random variable with probability-weighted block averages - probability.filtration.natural.compute: natural filtration F_t = sigma(Y_0, ..., Y_t) with monotone refinement - probability.process.doob_martingale.compute: Doob martingale M_t = E[payoff | F_t] with rational-valued conditional expectations The FiniteProbabilitySpace value parses only well-formed spaces with unique labels, positive rational masses summing to exactly 1. The FiniteSigmaAlgebra value parses only valid partitions (disjoint, nonempty, union = entire space). Closes #1806
…es, tests - Import TOOLS and ADMISSIONS in catalog/builtins.py. - Update frozen admission baselines and add the finite_stochastic_processes schema-snapshot fragment (5 operations). - Add 12 focused tests (sigma algebra, join, conditional expectation, filtration, Doob martingale, validation).
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| class FromObservationRequest(StrictModel): | ||
| """Construct a sigma algebra from an observation map.""" | ||
|
|
||
| space: FiniteProbabilitySpace | ||
| observation: tuple[str, ...] = Field(min_length=1) | ||
|
|
||
|
|
||
| class JoinRequest(StrictModel): | ||
| """Compute the join of two sigma algebras.""" | ||
|
|
||
| sigma1: FiniteSigmaAlgebra | ||
| sigma2: FiniteSigmaAlgebra | ||
|
|
||
|
|
||
| class ConditionalExpectationRequest(StrictModel): | ||
| """Compute E[X | G].""" | ||
|
|
||
| rv: FiniteRandomVariable | ||
| sigma: FiniteSigmaAlgebra | ||
|
|
||
|
|
||
| class FiltrationRequest(StrictModel): | ||
| """Compute the natural filtration of observations.""" | ||
|
|
||
| space: FiniteProbabilitySpace | ||
| observations: tuple[tuple[str, ...], ...] = Field(default=()) | ||
|
|
||
|
|
||
| class DoobMartingaleRequest(StrictModel): | ||
| """Compute the Doob martingale of a payoff process.""" | ||
|
|
||
| space: FiniteProbabilitySpace | ||
| observations: tuple[tuple[str, ...], ...] = Field(default=()) | ||
| payoff: tuple[str, ...] = Field(min_length=1) |
There was a problem hiding this comment.
🔴 Valid-looking probability requests crash the server instead of returning a result
Requests that pass validation but describe mismatched data are rejected deep inside the computation by raising a plain error (raise ValueError("observation must have one entry per sample") at src/jacobian/math/finite_stochastic_processes/operations.py:35) instead of being refused up front, so the call surfaces as a host crash rather than a typed answer.
Impact: A caller sending an observation list, payoff list, or pair of sigma algebras that do not line up with the sample space gets an unhandled server error instead of a clear rejection.
Missing cross-field validation on the request models lets kernel ValueErrors escape through dispatch
Verified by running the adapters: FromObservationRequest(space={'samples':['H','T'],...}, observation=('a',)) parses fine (the model only enforces min_length=1 at src/jacobian/math/finite_stochastic_processes/_models.py:19) and then compute_sigma_from_observation raises ValueError: observation must have one entry per sample. Likewise JoinRequest with two sigma algebras over different spaces raises ValueError: sigma algebras must share the same probability space (operations.py:68), ConditionalExpectationRequest with mismatched spaces raises at operations.py:90, and DoobMartingaleRequest with a mis-sized payoff or mis-sized observation entries raises at operations.py:143 / operations.py:35.
invoke_operation in src/jacobian/dispatch.py:47 calls operation.run(parsed) with no exception handling, so these become host exceptions. AGENTS.md requires: "The request must encode the advertised mathematical domain—bounds, positivity, completeness, non-degeneracy—not only the JSON shape. A request the model accepts must return a typed domain result; mathematical inapplicability belongs in the request validator or the result, not in a raised backend exception."
Fix by adding model_validator(mode="after") cross-field checks to FromObservationRequest, JoinRequest, ConditionalExpectationRequest, FiltrationRequest, and DoobMartingaleRequest.
Prompt for agents
The finite stochastic process request models only validate JSON shape, not the mathematical domain. Requests that parse successfully can still hit raise ValueError inside operations.py (observation length vs samples, payoff length vs samples, sigma algebras sharing a space, random variable and sigma algebra sharing a space), and jacobian/dispatch.py invoke_operation does not catch those, so an accepted request becomes a host exception. Add after-model validators on FromObservationRequest, JoinRequest, ConditionalExpectationRequest, FiltrationRequest (each observation entry) and DoobMartingaleRequest so every accepted request is guaranteed to produce a typed domain result, and add tests covering each rejection path.
Was this helpful? React with 👍 or 👎 to provide feedback.
| def sigma_algebra_join_correct( | ||
| sigma1: FiniteSigmaAlgebra, | ||
| sigma2: FiniteSigmaAlgebra, | ||
| ) -> FiniteSigmaAlgebra: | ||
| """Return the least sigma algebra containing both sigma algebras.""" | ||
| if sigma1.space != sigma2.space: | ||
| raise ValueError("sigma algebras must share the same probability space") | ||
| blocks: list[tuple[str, ...]] = [] | ||
| for b1 in sigma1.blocks: | ||
| for b2 in sigma2.blocks: | ||
| intersection = frozenset(b1) & frozenset(b2) | ||
| if intersection: | ||
| blocks.append(tuple(sorted(intersection))) | ||
| return FiniteSigmaAlgebra( | ||
| space=sigma1.space, blocks=tuple(b for b in blocks) | ||
| ) |
There was a problem hiding this comment.
🟡 Two identical copies of the sigma-algebra join exist, and the published one is not the one actually used
A second, byte-identical copy of the join computation is defined and used by the tools (sigma_algebra_join_correct at src/jacobian/math/finite_stochastic_processes/operations.py:62) while the copy advertised in the module's public list is never called, so the documented public function and the executed function can silently drift apart.
Impact: Maintainers fixing the published join function would not change the behaviour agents actually get.
Duplicate kernel plus __all__ / public API mismatch
operations.py:43-59 defines sigma_algebra_join, which is listed in operations.__all__ (operations.py:9-15) and re-exported from the package (src/jacobian/math/finite_stochastic_processes/__init__.py:8). operations.py:62-77 defines sigma_algebra_join_correct with an identical body; it is absent from __all__ yet is imported (via a function-local import) by the catalog adapter compute_join in src/jacobian/math/finite_stochastic_processes/_operations.py:38-42 and by the test at tests/math/finite_stochastic_processes/test_finite_stochastic_processes.py:81.
AGENTS.md: "Native public functions belong under jacobian.math, have explicit __all__" — the function backing the public probability.finite_sigma_algebra.join.compute operation is not part of any declared surface. The duplicate should be deleted and the adapter should call sigma_algebra_join with a normal module-level import.
Prompt for agents
operations.py contains two identical join implementations: sigma_algebra_join (exported in __all__ and from the package __init__) and sigma_algebra_join_correct (not exported, but imported lazily inside compute_join in _operations.py and by the domain test). Remove the duplicate, have compute_join import sigma_algebra_join at module level like the other adapter functions, and update the test accordingly.
Was this helpful? React with 👍 or 👎 to provide feedback.
| class FiltrationResult(StrictModel): | ||
| """The natural filtration as a tuple of sigma algebra dicts.""" | ||
|
|
||
| sigmas: tuple[dict[str, object], ...] = Field(default=()) |
There was a problem hiding this comment.
🟡 Filtration results are returned as untyped dictionaries that drop the probability space
Each step of the filtration result is emitted as a loose key/value bag containing only the blocks ({"blocks": s.blocks} at src/jacobian/math/finite_stochastic_processes/_operations.py:55) rather than the typed sigma-algebra value, so the published output schema places no constraints on the data and the sample space with its probabilities is lost.
Impact: Callers receive an unconstrained, weakly described filtration output that cannot be fed straight back into the other sigma-algebra operations.
FiltrationResult uses tuple[dict[str, object], ...]
FiltrationResult.sigmas is declared as tuple[dict[str, object], ...] (src/jacobian/math/finite_stochastic_processes/_models.py:54), so the generated JSON schema for probability.filtration.natural.compute allows arbitrary objects. The kernel already returns tuple[FiniteSigmaAlgebra, ...] (src/jacobian/math/finite_stochastic_processes/operations.py:113-129), which is a StrictModel usable directly as the element type; the projection to a dict discards space and prevents round-tripping the output into JoinRequest / ConditionalExpectationRequest.
AGENTS.md "Types and transport": request/result models are authoritative at operation and wire boundaries and operations should compose through their typed mathematical values.
| class FiltrationResult(StrictModel): | |
| """The natural filtration as a tuple of sigma algebra dicts.""" | |
| sigmas: tuple[dict[str, object], ...] = Field(default=()) | |
| class FiltrationResult(StrictModel): | |
| """The natural filtration as a tuple of sigma algebras.""" | |
| sigmas: tuple[FiniteSigmaAlgebra, ...] = Field(default=()) |
Was this helpful? React with 👍 or 👎 to provide feedback.
| from jacobian.math.finite_stochastic_processes._admission import ( | ||
| ADMISSIONS as FINITE_STOCHASTIC_PROCESSES_ADMISSIONS, | ||
| ) | ||
| from jacobian.math.finite_stochastic_processes._tools import ( | ||
| TOOLS as FINITE_STOCHASTIC_PROCESSES_TOOLS, | ||
| ) |
There was a problem hiding this comment.
🔍 Import block ordering breaks ruff check despite the claimed clean lint
The new imports for finite_stochastic_processes are inserted before finite_state_transducers, but isort ordering puts finite_state_... first (a < o). Running uv run ruff check src/jacobian/catalog/builtins.py on this branch reports I001 Import block is un-sorted or un-formatted (1 error, fixable with --fix). The PR description claims make check lint is clean, so this would fail CI as-is.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if not observations: | ||
| return () |
There was a problem hiding this comment.
🔍 Doob martingale with no observations silently returns an empty result
observations defaults to () on both FiltrationRequest and DoobMartingaleRequest (src/jacobian/math/finite_stochastic_processes/_models.py:40,47), and filtration_natural short-circuits to () for empty input (operations.py:118-119). I confirmed by execution that compute_doob_martingale(DoobMartingaleRequest(space=..., payoff=("1","0"))) returns martingale=(). That is defensible (zero time steps), but an agent supplying a payoff and no observations gets an empty answer with no signal; the conventional convention would be to include the trivial F_0 (giving E[payoff]) or to require at least one observation. Worth deciding explicitly and testing.
Was this helpful? React with 👍 or 👎 to provide feedback.
morluto
left a comment
morluto
left a comment
There was a problem hiding this comment.
Review verdict: no mathematical correctness blocker found; tighten the finite contract
I checked the core constructions independently:
- equal-observation fibers are exactly the atoms of
sigma(Y); - nonempty intersections of two atom partitions give the atom partition of their sigma-algebra join;
- the blockwise probability-weighted average is the exact finite conditional expectation because every atom has positive mass;
- repeated joins produce the natural filtration
F_t = sigma(Y_0,...,Y_t); M_t = E[X | F_t]satisfies the expected tower/martingale identities. I exhaustively checked the partition/join/tower identity over every set partition through five atoms with nonuniform rational masses.
I did not find a formula or logic bug in those kernels. The main issues are at the boundary:
observationshas no maximum number of time steps, so filtration and Doob outputs are unbounded despite the domain's otherwise bounded value model. Add a time-horizon bound and cross-field validation that every observation and payoff has exactly one entry per sample.FromObservationRequestlikewise accepts the wrong observation length and fails only inside the kernel. Make this a request invariant.FiltrationResulterases the typedFiniteSigmaAlgebravalues intodict[str, object]records containing only blocks. Returningtuple[FiniteSigmaAlgebra, ...]would preserve the exact space/partition invariant and make monotone refinement replayable.sigma_algebra_join_correct()is byte-for-byte the same construction as the publicsigma_algebra_join(), while the catalog adapter imports the private-sounding duplicate. Keep one primitive and test it directly.
Useful property tests would assert E[E[X|H]|G] = E[X|G] whenever G ⊆ H, preservation of expectation, block constancy, and that every successive filtration partition refines the previous one. The current mathematics appears sound.
Deep review summaryVerdict: REQUEST CHANGES — the probability identities are correct, but accepted requests can escape as host exceptions and the executed join kernel is not the published one. I independently checked the core constructions:
The blocker is the request boundary. Models accept mismatched observation lengths, payoff lengths, probability spaces, and observation rows, then kernels raise plain There is also an avoidable split-brain implementation: Finally, canonicalize the partition/block ordering and rational serialization so equivalent probability spaces and sigma algebras have deterministic wire identity. |
1 participant
Closes #1806.
Summary
Add the
finite_stochastic_processesdomain implementing exact, bounded, deterministic finite stochastic process operations over immutable finite probability spaces. This is the missing layer above Jacobian's exact finite distributions and below its Markov-chain operations.Design and library choices
The domain uses an exact rational kernel over immutable
FiniteProbabilitySpacevalues. No martingale verifier, optimal-stopping solver, trading simulator, or general measure-theory framework is introduced. Per the issue, sample identity survives random-variable pushforwards, and conditional expectation preserves sample identity, combines atoms into measurable blocks, and returns a random variable constant on those blocks.Representation (
values.py).FiniteProbabilitySpacebinds unique sample labels with positive rational masses summing to exactly 1.FiniteRandomVariablebinds a complete value per sample.FiniteSigmaAlgebrarepresents a sigma algebra by its atom partition (blocks of sample labels). The values parse only well-formed inputs: unique labels, positive masses, valid partitions (disjoint, nonempty, union = entire space).Operations (
operations.py). All functions are deterministic and complete for accepted values.from_observation.compute— sigma algebra generated by an observation mapY: Omega -> labels, with equal-value fibers as blocks.join.compute— least sigma algebra containing both inputs, as the finest common refinement.conditional_expectation.finite.compute—E[X | G]as a block-constant random variable. On each block, the value is the probability-weighted average of X over the samples in that block.filtration.natural.compute— natural filtrationF_t = sigma(Y_0, ..., Y_t)with monotone refinement.process.doob_martingale.compute— Doob martingaleM_t = E[payoff | F_t]with rational-valued conditional expectations.Invariances verified by tests
E[X | trivial] = E[X](e.g.,1/2for a fair coin).E[X | {H}] = X(H)andE[X | {T}] = X(T).1onHand0onTunder the full-information filtration.Validation
make checklint + typecheck clean; 12 focused domain tests; all 1134tests/math,tests/catalog,tests/dispatchtests pass.finite_stochastic_processesschema-snapshot fragment added (5 operations).Continue this on Linzumi