subscription.cancelled sync fails with duplicate unique id across persisted lines and charges when credits.enable_credit_then_invoice: true

[pratham-ez]

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

OpenMeter Version

10f948f17

Expected Behavior

Expected vs actual

Step | Expected | Actual -- | -- | -- Subscribe + first sync | Charges + gathering mirrors created | ✓ Works Cancel immediate (OM API only) | Worker trims charge periods to activeTo, may cut partial invoice if billable | Worker fails on load; gathering stale; no invoice Cancel + ops force invoice (progressiveBillingOverride) | Immediate issued invoice | Works (separate code path; not used in this repro)

---

What we are NOT doing

• Not using Synqed ops cancel (which calls ForceInvoiceCustomer with progressiveBillingOverride: true)
• Not manually calling POST /api/v1/billing/invoices/invoice or /advance
• No custom renewal-grant code deployed on staging

This reproduces on stock OpenMeter with the documented credit_then_invoice config.

---

Impact

• Any subscription with enable_credit_then_invoice: true breaks on the second billing-worker sync (cancel, renewal, cron subscription-sync, plan edit).
• Cancel sync never completes → gathering invoices stay stale → native partial-period invoicing at cancel does not work.
• DLQ fills with subscription.cancelled events.
• Workaround today: ops force invoice on cancel (bypasses sync) — not acceptable for production customer flows.

Actual Behavior

Summary

With enable_credit_then_invoice: true, the first subscription sync succeeds (charges + gathering mirror lines). Any second sync (cancel, cron, plan change) fails in persistedstate.Loader.LoadForSubscription because the same unique_reference_id exists in both billing_invoice_lines and charges. Cancel events retry 9× → DLQ; gathering invoice never trims to activeTo.

Error

synchronize subscription: duplicate unique id across persisted lines and charges:

01KSA6NP21H4EYBP2KB4QNZKC6/default/synqed_feature_premium/v[0]/period[0]

Event: io.openmeter.subscription.v1.subscription.cancelled

Repro

1. Enable both credit flags above.
2. Customer + $10 grant + credit_then_invoice subscription (2 usage rate cards).
3. Wait for initial billing-worker sync (succeeds).
4. Optional: ingest usage via POST /api/v1/events (TOOL_CALL_STANDARD / TOOL_CALL_PREMIUM).
5. Cancel OpenMeter only (no force invoice):
   POST /api/v1/subscriptions/{id}/cancel

   {"timing": "immediate"}
6. Check billing-worker logs.

Example (staging)

  |   -- | -- Subscription | 01KSA6NP21H4EYBP2KB4QNZKC6 Customer | 01KSA61A2R80VXA83SB658GPNQ Subject | 9520ea79-2aaa-4310-8b53-6e24d46e743c Cancel | 2026-05-23T10:46:11Z

After cancel (OM-only): sub → inactive ✓; gathering invoice still gathering, period unchanged, total: 0; no issued invoice; worker fails with error above.

Also reproduced on sub 01KSA3MVZSVPT731AD8H95CH5T at 2026-05-23T09:51:53Z.

Root cause (our analysis)

1. With enable_credit_then_invoice: true, sync creates charges (usageBasedChargeCollection), not classic invoice lines.
2. Charge Create also writes a gathering mirror line with the same ChildUniqueReferenceID as Intent.UniqueReferenceID and Engine: charge_usagebased (openmeter/billing/charges/usagebased/service/create.go, charges/service/create.go).
3. On the next sync, loader.go loads invoice lines and charges, then errors at lines 84–87 when the same ID appears in both maps.
4. First sync OK (empty persisted state). Every later sync fails.

Charge-backed gathering lines look like mirrors, not separate sync artifacts. Loader should skip them when loading charge state (e.g. ChargeID != nil or Engine.IsCharge()).

Steps To Reproduce

Reproduction
Enable credits.enabled: true and credits.enable_credit_then_invoice: true.

Create a customer with subjectKeys = org UUID.

Issue a credit grant (e.g. $10).

Create a subscription on a credit_then_invoice plan with two usage-based rate cards (e.g. synqed_feature_standard @ $2/unit, synqed_feature_premium @ $1/unit).

Wait for billing-worker initial sync (~seconds) — succeeds; gathering invoice + charges are created.

Optionally ingest usage events via POST /api/v1/events (CloudEvents with TOOL_CALL_STANDARD / TOOL_CALL_PREMIUM).

Cancel immediately via OpenMeter only (no force invoice):

POST /api/v1/subscriptions/{subscriptionId}/cancel
{"timing": "immediate"}
Observe billing-worker logs.

Observed error
synchronize subscription: duplicate unique id across persisted lines and charges:
01KSA6NP21H4EYBP2KB4QNZKC6/default/synqed_feature_premium/v[0]/period[0]
Full log context:

{
"ts": "2026-05-23T10:46:11Z",
"level": "WARN",
"caller": "openmeter/watermill/router/logger.go:40",
"msg": "Error occurred, retrying",
"service.name": "openmeter-billing-worker",
"service.version": "10f948f17",
"retry_no": 6,
"max_retries": 9,
"error.message": "synchronize subscription: duplicate unique id across persisted lines and charges: 01KSA6NP21H4EYBP2KB4QNZKC6/default/synqed_feature_premium/v[0]/period[0]"
}
After 9 retries → Failed to process message, message is going to DLQ on topic io.openmeter.subscription.v1.subscription.cancelled.

Additional Information

No response

[turip]

Hey @pratham-ez sorry for the long radio silence, but we have validated that the credit then invoice works as expected with latest master including this scenario (this was fixed last week I think).

Please use the latest master version. Please also note, that I had to rename the feature flag to credits.enableCreditThenInvoice: true.

[pratham-ez]

do we have threshold webhook for customer level credit balance
for now i can only see webhooks for entitlements @turip