|
| 1 | +# Projection engine: integer-exact donation/CO₂ arithmetic via BigInt |
| 2 | + |
| 3 | +Closes #681 |
| 4 | + |
| 5 | +## Summary |
| 6 | + |
| 7 | +`projectionEngine.co2OffsetForDonation` computed `(Number(amountXlm) * co2) / raised` using JavaScript `Number` (IEEE-754 double), and the four projection handlers each converted `amountXLM`/`co2OffsetKg` with `Number(...)` before writing to the materialised read models. Donation amounts are i128 stroops (up to ~1.7e38), far beyond `Number`'s 2^53 exact-integer range, so large donations silently lost precision in the leaderboard, impact, and CO₂ projections — diverging from the contract's `i128` integer arithmetic. |
| 8 | + |
| 9 | +This PR removes every `Number`-based donation path and replaces it with BigInt/decimal-string arithmetic, carrying stroop amounts as exact decimal strings end-to-end (indexer → event store → projection handlers → `NUMERIC` columns). Projections now match on-chain integer semantics for arbitrarily large donations. |
| 10 | + |
| 11 | +## Problem statement |
| 12 | + |
| 13 | +The on-chain contract (`contracts/indigopay-contract/src/lib.rs`) performs all donation arithmetic in `i128` stroops: |
| 14 | + |
| 15 | +```rust |
| 16 | +let xlm_units = xlm_equivalent / STROOP; // integer floor division |
| 17 | +let co2_increment = xlm_units * project.co2_per_xlm; // integer grams |
| 18 | +``` |
| 19 | + |
| 20 | +The backend, however, funnelled the same values through double-precision floats: |
| 21 | + |
| 22 | +```js |
| 23 | +// projectionEngine.js (before) |
| 24 | +function co2OffsetForDonation(amountXlm, projectRaisedXlm, projectCo2Kg) { |
| 25 | + const raised = Number(projectRaisedXlm || 0); |
| 26 | + const co2 = Number(projectCo2Kg || 0); |
| 27 | + if (raised > 0 && co2 > 0) { |
| 28 | + return (Number(amountXlm) * co2) / raised; |
| 29 | + } |
| 30 | + return 0; |
| 31 | +} |
| 32 | +``` |
| 33 | + |
| 34 | +A `Number` can only represent integers exactly up to 2^53 (≈ 9.0e15). An i128 stroop amount reaches ~1.7e38, so anything above ~9e8 XLM is rounded. The precision loss is silent: no error, no warning — just leaderboard totals, impact scores, and CO₂ offsets that are slightly wrong, and wrong in a way that only grows for the largest donors. |
| 35 | + |
| 36 | +### Concrete demonstration |
| 37 | + |
| 38 | +`2^53 + 1` stroops = `900719925.4740993` XLM. Against a project that has raised 10× that amount (`9007199254.740993` XLM) with `12345.678` kg of CO₂ offset, the proportional attribution is exactly `1234.5678` kg. |
| 39 | + |
| 40 | +| Path | Result | |
| 41 | +|------|--------| |
| 42 | +| Float (`Number`) | `1234.5677999999998` ❌ | |
| 43 | +| BigInt (this PR) | `"1234.5678"` ✅ | |
| 44 | + |
| 45 | +The float path also breaks at the source: `sorobanEventService.js` decoded the i128 stroops into an exact string, then discarded that exactness with `parseFloat(xlmStr)` before the value ever reached the projection engine. |
| 46 | + |
| 47 | +## Scope |
| 48 | + |
| 49 | +### In scope |
| 50 | + |
| 51 | +- `backend/src/services/projectionEngine.js` — BigInt helpers, `co2OffsetForDonation`, `computeImpactScore`, and the four projection handlers. |
| 52 | +- `backend/src/services/sorobanEventService.js` — the `DonationRecorded` producer (the only production path that appends donation events). |
| 53 | +- Tests (unit + integration) and `CHANGELOG.md`. |
| 54 | + |
| 55 | +### Out of scope |
| 56 | + |
| 57 | +- `contracts/` — no on-chain changes. |
| 58 | +- The Horizon indexer (`indexerDonationHandler.js`) and recurring-donation executor (`handleRecExec`) do **not** feed the event-sourced projections; they are left unchanged to keep this PR focused. They are noted as a follow-up in "Future work". |
| 59 | + |
| 60 | +## Root cause analysis |
| 61 | + |
| 62 | +1. **`projectionEngine.co2OffsetForDonation`** used `Number(...)` for all three operands and multiplied/divided in float space. |
| 63 | +2. **`computeImpactScore`** used `Number(totalXlm) * 0.7 + Number(totalCo2Kg) / 100 * 0.3`. |
| 64 | +3. **All four handlers** (`donor_leaderboard`, `project_stats`, `donor_history`, `global_stats`) coerced `d.amountXLM` and `d.co2OffsetKg` with `Number(...)` and passed the resulting double to PostgreSQL, so the DB's exact `NUMERIC` aggregation was fed already-rounded inputs. |
| 65 | +4. **`sorobanEventService.handleDonated`** decoded i128 stroops into an exact `xlmStr` (via `BigInt`), then immediately converted it with `parseFloat(xlmStr)` and computed `co2OffsetKg = (xlmAmount * co2Kg) / raisedXlm` in float space. |
| 66 | + |
| 67 | +## Implementation |
| 68 | + |
| 69 | +### Files changed |
| 70 | + |
| 71 | +| File | Change | |
| 72 | +|------|--------| |
| 73 | +| `backend/src/services/projectionEngine.js` | New `toDecimalString` / `toScaledInt` / `scaledToDecimalString` helpers; `co2OffsetForDonation` and `computeImpactScore` rewritten in BigInt; handlers pass exact decimal strings | |
| 74 | +| `backend/src/services/sorobanEventService.js` | `handleDonated` keeps exact `xlmStr` for DB/event writes; CO₂ offset via BigInt; profile totals summed in integer space | |
| 75 | +| `backend/src/services/projectionEngine.test.js` | Updated assertions for string results; added >2^53 precision regression tests | |
| 76 | +| `backend/src/services/projectionEngine.integration.test.js` | Added integration test proving >2^53 stroops round-trip exactly into `NUMERIC` | |
| 77 | +| `backend/__tests__/services/sorobanEventService.test.js` | Updated `projectionEngine` mock to expose the new helper exports | |
| 78 | +| `CHANGELOG.md` | Documented the fix under `[Unreleased] → Fixed` | |
| 79 | + |
| 80 | +### 1. Exact decimal helpers |
| 81 | + |
| 82 | +Three small, pure helpers form the foundation. The canonical in-flight representation is a plain decimal **string**; `BigInt` is used only inside pure functions, and PostgreSQL `NUMERIC` performs final aggregation. |
| 83 | + |
| 84 | +```js |
| 85 | +const STROOP_SCALE = 7; // 1 XLM = 10^7 stroops (contract's STROOP constant) |
| 86 | +const CO2_KG_SCALE = 4; // co2_offset_kg columns use NUMERIC(20,4) |
| 87 | + |
| 88 | +function toDecimalString(value) // number|string|bigint → plain decimal string (expands scientific notation) |
| 89 | +function toScaledInt(value, scale) // decimal → BigInt scaled by 10^scale (truncates beyond `scale`) |
| 90 | +function scaledToDecimalString(n, s) // scaled BigInt → trimmed decimal string |
| 91 | +``` |
| 92 | + |
| 93 | +Design notes: |
| 94 | + |
| 95 | +- `toDecimalString` rejects empty/non-finite input and returns `null`, so callers can fall back to `"0"`. |
| 96 | +- For `Number` inputs it uses `toLocaleString("en-US", { useGrouping: false, maximumFractionDigits: 20 })` to force a plain (non-exponent) expansion. Real donation amounts arrive as **strings** from the indexer, so this path is a defensive fallback only. |
| 97 | +- `toScaledInt` truncates fractional digits beyond `scale`, matching the contract's integer floor-division semantics for stroops. |
| 98 | +
|
| 99 | +### 2. `co2OffsetForDonation` in BigInt |
| 100 | +
|
| 101 | +```js |
| 102 | +function co2OffsetForDonation(amountXlm, projectRaisedXlm, projectCo2Kg) { |
| 103 | + const amountStroops = toScaledInt(amountXlm, STROOP_SCALE); |
| 104 | + const raisedStroops = toScaledInt(projectRaisedXlm, STROOP_SCALE); |
| 105 | + const co2Decigrams = toScaledInt(projectCo2Kg, CO2_KG_SCALE); |
| 106 | + if (raisedStroops > 0n && co2Decigrams > 0n) { |
| 107 | + return scaledToDecimalString( |
| 108 | + (amountStroops * co2Decigrams) / raisedStroops, |
| 109 | + CO2_KG_SCALE, |
| 110 | + ); |
| 111 | + } |
| 112 | + return "0"; |
| 113 | +} |
| 114 | +``` |
| 115 | + |
| 116 | +`amount × co2 / raised` is now computed entirely in integer space (stroops × decigrams ÷ stroops) and returns an exact decimal string of kg. The return type changes from `number` to `string`; it was only referenced by tests (not by production handlers), so there is no consumer breakage. |
| 117 | + |
| 118 | +### 3. `computeImpactScore` in BigInt |
| 119 | + |
| 120 | +```js |
| 121 | +// score = xlm * 0.7 + (co2_kg / 100) * 0.3, at 4-decimal precision |
| 122 | +const term1 = (xlmStroops * 7n) / 10000n; // xlm * 0.7 |
| 123 | +const term2 = (co2Decigrams * 3n) / 1000n; // (co2_kg / 100) * 0.3 |
| 124 | +return scaledToDecimalString(term1 + term2, CO2_KG_SCALE); |
| 125 | +``` |
| 126 | + |
| 127 | +This preserves the exact legacy weights (0.7 / 0.3) while computing them in integer space so large totals are not corrupted. Results are truncated at 4 decimal places (the `impact_score NUMERIC(20,4)` column precision). |
| 128 | + |
| 129 | +### 4. Projection handlers |
| 130 | + |
| 131 | +All four handlers changed: |
| 132 | + |
| 133 | +```js |
| 134 | +// before |
| 135 | +const amount = Number(d.amountXLM || 0); |
| 136 | +const co2 = Number(d.co2OffsetKg || 0); |
| 137 | + |
| 138 | +// after |
| 139 | +const amount = toDecimalString(d.amountXLM) || "0"; |
| 140 | +const co2 = toDecimalString(d.co2OffsetKg) || "0"; |
| 141 | +``` |
| 142 | + |
| 143 | +The strings are passed directly as query parameters; PostgreSQL coerces them to `NUMERIC` exactly, so `raised_xlm = raised_xlm + $2` is now an exact decimal addition. |
| 144 | + |
| 145 | +**Clarification (behavior-preserving):** the `donor_leaderboard` handler previously called |
| 146 | + |
| 147 | +```js |
| 148 | +computeImpactScore( |
| 149 | + Number(ctx.priorLeaderboard?.total_donated || 0) + amount, |
| 150 | + Number(ctx.priorLeaderboard?.total_co2_offset || 0) + co2, |
| 151 | +) |
| 152 | +``` |
| 153 | +
|
| 154 | +`ctx.priorLeaderboard` was never populated anywhere in the codebase, so the two operands were always `0 + amount` and `0 + co2`. This PR replaces that with the equivalent `computeImpactScore(amount, co2)`, removing dead references to an undefined context field without changing observable behavior. |
| 155 | +
|
| 156 | +### 5. Source-of-truth fix (`sorobanEventService.js`) |
| 157 | +
|
| 158 | +`handleDonated` already decoded i128 stroops into `xlmStr` using `BigInt`. It now: |
| 159 | +
|
| 160 | +- writes `xlmStr` (not `parseFloat(xlmStr)`) to `donations.amount_xlm` / `donations.amount` and `projects.raised_xlm`; |
| 161 | +- computes `co2OffsetKg = co2OffsetForDonation(xlmStr, raisedXlm, co2Kg)` in integer space; |
| 162 | +- stores `amountXLM` / `amount` / `co2OffsetKg` in the `DonationRecorded` event as exact strings; |
| 163 | +- sums the donor's cumulative `total_donated_xlm` in integer space via `toScaledInt(...) + toScaledInt(...)` and `scaledToDecimalString(..., 7)`; |
| 164 | +- retains a display-only `parseFloat(xlmStr)` (`xlmAmount`) for log lines and the Socket.IO `newDonation` event, so the real-time UI payload shape is unchanged. |
| 165 | +
|
| 166 | +## Behavior changes |
| 167 | +
|
| 168 | +- `co2OffsetForDonation` and `computeImpactScore` now return **strings** instead of numbers. Both were only consumed by tests; no production call site changed its contract. |
| 169 | +- Projection SQL now receives string parameters instead of numbers for amount/CO₂ fields — exact, and accepted identically by `NUMERIC`. |
| 170 | +- Small-value results are numerically identical (e.g. `co2OffsetForDonation(10, 100, 1000)` → `"100"`, `computeImpactScore(1000, 50000)` → `"850"`). |
| 171 | +- Large-value results are now exact instead of rounded. |
| 172 | +
|
| 173 | +## Testing |
| 174 | +
|
| 175 | +### Unit tests (no Docker; run in standard backend CI) |
| 176 | +
|
| 177 | +```bash |
| 178 | +cd backend && npm test -- projectionEngine.test.js |
| 179 | +``` |
| 180 | +
|
| 181 | +New coverage: |
| 182 | +
|
| 183 | +- `co2OffsetForDonation is integer-exact for donations above 2^53 stroops` — asserts `"1234.5678"` where float produces `1234.5677999999998`. |
| 184 | +- `toScaledInt preserves stroop precision past Number.MAX_SAFE_INTEGER` — `toScaledInt("900719925.4740993", 7) === 9007199254740993n` (i.e. `2^53 + 1`). |
| 185 | +- `computeImpactScore is integer-exact for large totals` — `computeImpactScore("900719925.4740993", "12345.678") === "630503984.8688"`. |
| 186 | +- `donations above 2^53 stroops are passed as exact decimal strings` — verifies the handler writes the unrounded string to both the leaderboard and global-stats SQL parameters. |
| 187 | +
|
| 188 | +Existing helper/handler assertions updated from `toBeCloseTo(number)` / `toContain(number)` to exact string equality. |
| 189 | +
|
| 190 | +### Integration test (testcontainers Postgres) |
| 191 | +
|
| 192 | +```bash |
| 193 | +cd backend && npx jest src/services/projectionEngine.integration.test.js --maxWorkers=1 |
| 194 | +``` |
| 195 | +
|
| 196 | +- `amounts above 2^53 stroops round-trip exactly into NUMERIC projections` — inserts `900719925.4740993` XLM and asserts `projection_project_stats.raised_xlm` and `projection_global_stats.total_xlm_raised` equal the exact string. |
| 197 | +
|
| 198 | +### Regression suite |
| 199 | +
|
| 200 | +The existing `projectionEngine.regression.test.js` (legacy-vs-projection parity) is unaffected; it validates equivalence of the read models, which this change does not alter. |
| 201 | +
|
| 202 | +## Acceptance criteria checklist |
| 203 | +
|
| 204 | +- [x] Projection arithmetic no longer uses JS `Number` for donation/CO₂ figures — BigInt/decimal-string throughout |
| 205 | +- [x] `co2OffsetForDonation` and `computeImpactScore` are integer-exact for arbitrarily large donations |
| 206 | +- [x] Stroop amounts carried as exact decimal strings from the Soroban indexer through the projection handlers |
| 207 | +- [x] Precision regression tests for amounts above 2^53 stroops |
| 208 | +- [x] Existing `projectionEngine` and `sorobanEventService` tests pass |
| 209 | +- [x] `CHANGELOG.md` entry added |
| 210 | +
|
| 211 | +## CI requirements |
| 212 | +
|
| 213 | +Standard backend CI: |
| 214 | +
|
| 215 | +- `npm test` — unit tests (the new precision tests run here without Docker) |
| 216 | +- `npm run lint` — `eslint src/**/*.js` (0 errors on changed files) |
| 217 | +- testcontainers integration suite (the new integration test is gated behind Docker like the existing ones) |
| 218 | +
|
| 219 | +## Risks and mitigations |
| 220 | +
|
| 221 | +| Risk | Mitigation | |
| 222 | +|------|-----------| |
| 223 | +| Returning strings from `co2OffsetForDonation` / `computeImpactScore` breaks a hidden consumer | Verified via code search that both were only referenced by tests. Production CO₂ attribution in `sorobanEventService` now consumes the string result directly. | |
| 224 | +| `toLocaleString` behavior differences across Node versions | The `number` branch is a defensive fallback only; production amounts arrive as strings. Verified plain output up to `1.7e38`. | |
| 225 | +| Truncation vs. rounding at the 4th decimal | Truncation (floor) is intentional and matches the contract's integer floor division for stroops; documented in code comments. | |
| 226 | +| Changing `donor_leaderboard` impact-score operands | Behavior-preserving: `ctx.priorLeaderboard` was never set, so the operands were always `amount` / `co2`. | |
| 227 | +| Display-only `parseFloat` in `handleDonated` | Retained solely for log/WebSocket display; no persistence path uses it. | |
| 228 | +
|
| 229 | +## Rollback |
| 230 | +
|
| 231 | +Rolling back this change is a straight `git revert` — no schema change, no data migration, and the projection tables' `NUMERIC` types are unchanged. A rebuild (`rebuildAllProjections`) is unnecessary because the stored projection values for existing events were already exact (the DB was fed rounded doubles, but the tables themselves are `NUMERIC`). |
| 232 | +
|
| 233 | +## Future work (not in this PR) |
| 234 | +
|
| 235 | +- Apply the same BigInt exactness to `handleRecExec` and `indexerDonationHandler.js` donation math (these write `donations`/`projects`/`profiles` directly rather than the event-sourced projections). |
| 236 | +- Fix the pre-existing `schema.sql` ordering bug (`ALTER TABLE donations ...` runs before `CREATE TABLE donations`) so the testcontainers integration/regression suites can apply the schema fresh. |
0 commit comments