[BUG](fn-consumer): Replay from checkpoints

[tanujnay112]

Description of changes

• Remove the requirement that an async function completion offset align with a collection snapshot.
• Fetch logs beginning after the latest live snapshot at or before the function checkpoint.
• Reconstruct materialized state through the function checkpoint, then expose only later mutations to the attached function.
• Preserve overwrite, delete/re-add, metadata, document, embedding, and completed-backfill semantics across the hidden replay prefix.
• Use the queued work frontier as the execution target, capped to one configured chunk, without requiring the target to be a compaction boundary.
• Leave partially completed work queued so large gaps drain incrementally across fn-consumer runs.

Test plan

• cargo test -p chroma-segment materialization_cutoff --no-default-features -- --nocapture
• cargo test -p chroma-segment checkpoint_preserves_overwrite_as_log_backed_baseline --no-default-features -- --nocapture
• cargo test -p worker async_function_boundary -- --nocapture
• cargo test -p worker test_async_attached_function_boundaries -- --nocapture
• cargo fmt --all -- --check
• cargo clippy -p worker -p chroma-segment --all-targets -- -D warnings

Migration plan

No data migration or API change is required. Existing aligned checkpoints continue through the same path; unaligned checkpoints now replay from the preceding live snapshot and may finish at any queued log offset.

Observability plan

The existing async-function replay log includes the selected snapshot position, function completion offset, and target log offset so replay windows can be inspected in production.

Documentation Changes

None. This changes internal fn-consumer execution semantics and does not alter user-facing APIs.

[tanujnay112]

• [BUG](fn-consumer): Replay from checkpoints #7654 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-08-29T16:02:02.820722Z	8a88306	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8a88306f33

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Exclude checkpointed metadata from later deltas

When the hidden replay prefix updates an existing record's metadata and a visible suffix operation later updates that record, resetting only final_operation leaves the prefix's metadata_to_be_merged/metadata_to_be_deleted entries in the record. The suffix is therefore materialized with both old and new changes, and compute_metadata_delta() compares all of them against the physical snapshot. StatisticsFunctionExecutor applies that delta to output state that already includes the checkpointed changes, so values changed before the completion offset are counted a second time. Preserve the checkpoint's metadata as a separate logical baseline and expose only suffix changes in the delta.

Useful? React with 👍 / 👎.