Skip to content

Commit 3807cdf

Browse files
authored
PAYG: process tracking + shadow charging engine (PR B-1) (Stirling-Tools#6477)
> 📌 **Stacked on [Stirling-Tools#6464](Stirling-Tools#6464 (lineage primitives, still in review). Stirling-Tools#6469 has merged so its commits are no longer in this PR's diff. Once Stirling-Tools#6464 merges, a final rebase collapses the lineage-primitives commits out of this diff too — leaving only the B-1 work. ## What this is Process tracking + shadow charging engine. Bundles PR-I7 service half with the non-filter piece of PR-I7a so the pieces ship together — none of them is useful in isolation. **Review focus:** the new files in: - \`app/saas/src/main/java/stirling/software/saas/payg/job/\` (\`JobService\`, \`JobContext\`, \`JoinOrOpenResult\`, \`StaleJobCloser\`) - \`app/saas/src/main/java/stirling/software/saas/payg/charge/\` (\`JobChargeService\`, \`ChargeContext\`, \`ChargeOutcome\`, \`JobInput\`) - \`app/saas/src/main/java/stirling/software/saas/payg/lineage/LineagePruneScheduler.java\` - their tests The 8 files inherited from Stirling-Tools#6464 (lineage primitives) are unchanged from there — they ride along in this diff until Stirling-Tools#6464 lands. The remaining work for shadow-in-staging is the ingress/egress filter that wires controllers into this engine — that's PR B-2. ## Scope ### \`JobService\` — persistence + lineage policy - **\`joinOrOpen\`** — the multi-input "any-match-joins, newest wins" rule. Hash every input via the lineage detector; if any matches an open process in the workflow window, attach to the one with the freshest \`lastStepAt\`. Step-limit overflow on the matched job spawns a fresh process; the new job's input signatures are still recorded so \`mostRecentMatchWins\` routes future calls forward. - **\`recordOutput\`** — post-tool-success path. Records OUTPUT signatures so the next call that takes this file as input lineage-matches into the same process. - **\`appendStep\`** — audit-trail step row written after a tool completes. - **\`close\`** — idempotent; safe to call from multiple paths (explicit, FE on-unload, scheduler). Returns the same row on re-close, no state mutation. - **\`findStale\` / \`closeStale\`** — workflow-window-based stale closure used by the scheduler. ### \`JobChargeService\` — the orchestrator (shadow variant) \`openProcess\` resolves the effective policy via \`PricingPolicyService\` (now in main via Stirling-Tools#6469), derives the step-limit for the current \`JobSource\` (with a defensive fallback if the policy is missing an entry), delegates to \`JobService.joinOrOpen\`, and on OPENED runs the \`DocumentClassifier\` + writes a \`payg_shadow_charge\` row. Applies the policy-level \`minChargeUnits\` floor per design § 3.4. Shadow variant only — never debits the ledger, never posts a Stripe meter event. The real-charging follow-up reuses the same orchestration and swaps the side-effect. \`legacyCreditsCharged\` on the shadow row stays \`0\` until the legacy \`CreditService\` is wired in (PR B-2), where the comparison becomes meaningful. ### Schedulers (both plain \`@Scheduled\`) - **\`StaleJobCloser\`** — fixed-rate 60 s. Closes \`OPEN\` jobs idle past the workflow window. API users never have to call close explicitly — this is the safety net. - **\`LineagePruneScheduler\`** — hourly cron, retention 1 h. Deletes \`job_artifact_hash\` rows older than the retention window. - **No \`@SchedulerLock\` / no \`shedlock\` table** — consistent with the 5 existing unguarded \`@Scheduled\` tasks in \`:saas\` (\`CreditResetScheduler\` and friends, none of which are guarded today). Cluster-correctness across all 7 saas schedulers is tracked in design § 9 as a separate focused cleanup. Underlying operations are idempotent — duplicate firings on multi-pod would be wasted DB load, not data corruption. ### Records (call-shape glue for PR B-2's filter) - \`JobContext\` / \`JoinOrOpenResult\` — input/output for \`JobService\`. - \`ChargeContext\` / \`ChargeOutcome\` — input/output for \`JobChargeService\`. - \`JobInput\` — paired \`(MultipartFile, materialised Path)\` so the upcoming ingress filter can pass both views without re-materialising. ## Tests **26 new, all green.** - 14 × \`JobServiceTest\` — no-match → opens new, single-match → joins existing, multi-input any-match-joins, multi-match newest-wins (older job never even looked up), step-limit hit spawns fresh job (and original \`stepCount\` is NOT mutated), empty inputs reject, stale-signature handling, recordOutput delegation, close idempotency, closeStale, appendStep persistence. - 7 × \`JobChargeServiceTest\` — JOINED skips classifier + shadow write entirely, OPENED writes shadow row + classifies (single + multi file paths), \`minChargeUnits\` floor applied, step-limit resolved per-\`JobSource\` from policy, missing source entry falls back to conservative default of 10. - 2 × \`StaleJobCloserTest\`, 3 × \`LineagePruneSchedulerTest\` — scheduler-wiring smoke + constructor-validation tests. \`ENABLE_SAAS=true ./gradlew :saas:test\` — BUILD SUCCESSFUL. ## What's not in this PR (lands in PR B-2) - **Tool ingress/egress servlet filter.** The highest-risk piece — materialises the request body into a \`JobInput\`, calls \`JobChargeService.openProcess\` from every \`@AutoJobPostMapping\`, records OUTPUT after success. Edge cases to validate: multipart parts, async controllers, streaming responses, errored 5xx paths, very large files. Design decisions for the filter are being worked through in \`notes/PAYG_FILTER_DESIGN.md\` before any code is written. - **Wire shadow path into legacy \`CreditService\`.** Every legacy debit writes a comparison row carrying both the PAYG would-be units and the legacy actual credits, populating \`diffPct\`. ## Design doc \`notes/PAYG_DESIGN.md\` — PR-I7 + PR-I7a status updated to reflect this bundle. § 9 carries the cluster-correctness deferral note alongside the existing LISTEN/NOTIFY trade-off note. § 7.5.1 readiness summary shows the path now needs just **1 more PR** (the filter half + CreditService wire-in, both bundled into PR B-2).
1 parent 35a712a commit 3807cdf

13 files changed

Lines changed: 1297 additions & 0 deletions

File tree

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
package stirling.software.saas.payg.charge;
2+
3+
import stirling.software.saas.payg.model.JobSource;
4+
import stirling.software.saas.payg.model.ProcessType;
5+
6+
/**
7+
* Per-call context for {@link JobChargeService#openProcess}. Carries the caller's identity and what
8+
* kind of process this is. Does NOT carry policy fields — the charge service resolves the effective
9+
* policy from {@code PricingPolicyService} so a stale snapshot from the caller can't desync from
10+
* the live policy.
11+
*/
12+
public record ChargeContext(
13+
Long ownerUserId, Long ownerTeamId, JobSource source, ProcessType processType) {
14+
15+
public ChargeContext {
16+
if (ownerUserId == null) {
17+
throw new IllegalArgumentException("ownerUserId is required");
18+
}
19+
if (source == null) {
20+
throw new IllegalArgumentException("source is required");
21+
}
22+
if (processType == null) {
23+
throw new IllegalArgumentException("processType is required");
24+
}
25+
}
26+
}
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
package stirling.software.saas.payg.charge;
2+
3+
import java.util.UUID;
4+
5+
/**
6+
* Result of {@link JobChargeService#openProcess}. {@code processId} is the job the caller's tool
7+
* call belongs to (newly opened or joined). {@code units} is what would have been debited had the
8+
* call been in PAYG (live) mode — for OPENED, the actual classification × policy units; for JOINED,
9+
* always 0 since the parent process already paid.
10+
*/
11+
public record ChargeOutcome(UUID processId, int units, Disposition disposition) {
12+
13+
public enum Disposition {
14+
/** Started a new process; {@code units} reflects the would-be charge. */
15+
OPENED,
16+
/** Joined an existing process within window; no incremental charge. */
17+
JOINED
18+
}
19+
}
Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
package stirling.software.saas.payg.charge;
2+
3+
import java.io.IOException;
4+
import java.nio.file.Path;
5+
import java.util.List;
6+
import java.util.Objects;
7+
8+
import org.springframework.context.annotation.Profile;
9+
import org.springframework.stereotype.Service;
10+
import org.springframework.transaction.annotation.Transactional;
11+
import org.springframework.web.multipart.MultipartFile;
12+
13+
import lombok.extern.slf4j.Slf4j;
14+
15+
import stirling.software.saas.payg.docs.DocumentClassifier;
16+
import stirling.software.saas.payg.docs.DocumentMetrics;
17+
import stirling.software.saas.payg.job.JobContext;
18+
import stirling.software.saas.payg.job.JobService;
19+
import stirling.software.saas.payg.job.JoinOrOpenResult;
20+
import stirling.software.saas.payg.model.JobSource;
21+
import stirling.software.saas.payg.policy.PricingPolicy;
22+
import stirling.software.saas.payg.policy.PricingPolicyService;
23+
import stirling.software.saas.payg.repository.PaygShadowChargeRepository;
24+
import stirling.software.saas.payg.shadow.PaygShadowCharge;
25+
26+
/**
27+
* Orchestrates a tool call's open-process decision: look up the team's effective policy, resolve
28+
* the step-limit ceiling for this caller surface, delegate the join-or-open decision to {@link
29+
* JobService}, and — when a new process opens — compute the would-be charge and record a {@link
30+
* PaygShadowCharge} row for comparison against the legacy engine.
31+
*
32+
* <p>Shadow mode only: this service never debits the wallet ledger or posts a Stripe meter event.
33+
* The real-charging path lives in a separate follow-up and reuses the same orchestration — only the
34+
* side-effect (shadow row vs ledger entry + Stripe call) differs.
35+
*
36+
* <p>The {@code legacyCreditsCharged} field on the shadow row is set to {@code 0} here. When the
37+
* legacy {@code CreditService} is wired to call this service (separate PR), the legacy debit amount
38+
* becomes available and {@code diffPct} can be computed against it; until then the shadow row
39+
* captures the PAYG units only.
40+
*/
41+
@Service
42+
@Profile("saas")
43+
@Slf4j
44+
public class JobChargeService {
45+
46+
private final JobService jobService;
47+
private final PricingPolicyService policyService;
48+
private final DocumentClassifier classifier;
49+
private final PaygShadowChargeRepository shadowRepository;
50+
51+
public JobChargeService(
52+
JobService jobService,
53+
PricingPolicyService policyService,
54+
DocumentClassifier classifier,
55+
PaygShadowChargeRepository shadowRepository) {
56+
this.jobService = Objects.requireNonNull(jobService, "jobService");
57+
this.policyService = Objects.requireNonNull(policyService, "policyService");
58+
this.classifier = Objects.requireNonNull(classifier, "classifier");
59+
this.shadowRepository = Objects.requireNonNull(shadowRepository, "shadowRepository");
60+
}
61+
62+
/**
63+
* Open a process (or join an existing one) for this tool call. Side effects: persists a {@code
64+
* ProcessingJob} row plus input signatures, and — on OPENED — writes a {@code
65+
* payg_shadow_charge} row carrying the would-be PAYG units.
66+
*/
67+
@Transactional
68+
public ChargeOutcome openProcess(ChargeContext ctx, List<JobInput> inputs) throws IOException {
69+
Objects.requireNonNull(ctx, "ctx");
70+
Objects.requireNonNull(inputs, "inputs");
71+
if (inputs.isEmpty()) {
72+
throw new IllegalArgumentException("inputs must not be empty");
73+
}
74+
75+
PricingPolicy policy = policyService.getEffectivePolicy(ctx.ownerTeamId());
76+
int stepLimit = resolveStepLimit(policy, ctx.source());
77+
78+
JobContext jobCtx =
79+
new JobContext(
80+
ctx.ownerUserId(),
81+
ctx.ownerTeamId(),
82+
ctx.source(),
83+
ctx.processType(),
84+
policy.getId(),
85+
stepLimit);
86+
87+
List<Path> paths = inputs.stream().map(JobInput::path).toList();
88+
JoinOrOpenResult result = jobService.joinOrOpen(jobCtx, paths);
89+
90+
if (result.disposition() == JoinOrOpenResult.Disposition.JOINED) {
91+
return new ChargeOutcome(result.job().getId(), 0, ChargeOutcome.Disposition.JOINED);
92+
}
93+
94+
int units = computeUnits(inputs, policy);
95+
result.job().setDocUnits(units);
96+
97+
recordShadowRow(ctx, result.job().getId(), policy.getId(), units);
98+
99+
return new ChargeOutcome(result.job().getId(), units, ChargeOutcome.Disposition.OPENED);
100+
}
101+
102+
private int resolveStepLimit(PricingPolicy policy, JobSource source) {
103+
Integer fromPolicy =
104+
policy.getStepLimits() == null ? null : policy.getStepLimits().get(source);
105+
if (fromPolicy != null && fromPolicy > 0) {
106+
return fromPolicy;
107+
}
108+
// Defensive default — every JobSource should have an entry per the V12 seed, but a
109+
// hand-edited policy could be missing one. Fall back to the smallest documented limit
110+
// (10 — WEB/API/DESKTOP_APP default) so an admin slip-up never spawns unbounded chains.
111+
log.debug(
112+
"PricingPolicy {} missing stepLimit for source={}; using fallback of 10.",
113+
policy.getId(),
114+
source);
115+
return 10;
116+
}
117+
118+
private int computeUnits(List<JobInput> inputs, PricingPolicy policy) {
119+
List<MultipartFile> multiparts = inputs.stream().map(JobInput::multipart).toList();
120+
DocumentMetrics metrics =
121+
multiparts.size() == 1
122+
? classifier.classify(multiparts.get(0), policy)
123+
: classifier.classify(multiparts, policy);
124+
// Apply the policy-level minChargeUnits floor per design § 3.4. The classifier returns
125+
// raw docUnits with a "non-empty input → ≥1" floor; the charge formula's
126+
// max(min_charge_units, docUnits) layers on top.
127+
return Math.max(policy.getMinChargeUnits(), metrics.docUnits());
128+
}
129+
130+
private void recordShadowRow(
131+
ChargeContext ctx, java.util.UUID jobId, Long policyId, int units) {
132+
PaygShadowCharge row = new PaygShadowCharge();
133+
row.setTeamId(ctx.ownerTeamId());
134+
row.setJobId(jobId);
135+
row.setPolicyId(policyId);
136+
row.setPaygUnits(units);
137+
// No legacy comparison yet — wired when the shadow path is connected to the legacy
138+
// CreditService in the follow-up PR. Until then, diff stays at 0.
139+
row.setLegacyCreditsCharged(0);
140+
row.setDiffPct(0);
141+
shadowRepository.save(row);
142+
}
143+
}
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
package stirling.software.saas.payg.charge;
2+
3+
import java.nio.file.Path;
4+
5+
import org.springframework.web.multipart.MultipartFile;
6+
7+
/**
8+
* One input file to a tool call, materialised. {@code path} is the on-disk copy the lineage
9+
* detector hashes; {@code multipart} carries the size + content-type metadata the classifier needs.
10+
* The pair is constructed by the ingress filter (PR-I7a filter) which materialises the request body
11+
* exactly once. Tests construct it from a fixture file plus a {@code MockMultipartFile} wrapping
12+
* the same bytes.
13+
*/
14+
public record JobInput(MultipartFile multipart, Path path) {
15+
16+
public JobInput {
17+
if (multipart == null) {
18+
throw new IllegalArgumentException("multipart is required");
19+
}
20+
if (path == null) {
21+
throw new IllegalArgumentException("path is required");
22+
}
23+
}
24+
}
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
package stirling.software.saas.payg.job;
2+
3+
import stirling.software.saas.payg.model.JobSource;
4+
import stirling.software.saas.payg.model.ProcessType;
5+
6+
/**
7+
* Input to {@link JobService#joinOrOpen}: who owns the request, what process shape it is, which
8+
* policy version applies, and the step-limit ceiling derived from that policy for this caller
9+
* surface. {@code stepLimit} is resolved by the caller (typically {@code JobChargeService}) rather
10+
* than re-fetched here so the service stays free of policy-lookup concerns.
11+
*/
12+
public record JobContext(
13+
Long ownerUserId,
14+
Long ownerTeamId,
15+
JobSource source,
16+
ProcessType processType,
17+
Long policyId,
18+
int stepLimit) {
19+
20+
public JobContext {
21+
if (ownerUserId == null) {
22+
throw new IllegalArgumentException("ownerUserId is required");
23+
}
24+
if (source == null) {
25+
throw new IllegalArgumentException("source is required");
26+
}
27+
if (processType == null) {
28+
throw new IllegalArgumentException("processType is required");
29+
}
30+
if (policyId == null) {
31+
throw new IllegalArgumentException("policyId is required");
32+
}
33+
if (stepLimit <= 0) {
34+
throw new IllegalArgumentException("stepLimit must be > 0");
35+
}
36+
}
37+
}

0 commit comments

Comments
 (0)