Skip to content

[ENH](wal3): Trim body metadata - #7645

Open
domenkozar wants to merge 1 commit into
chroma-core:mainfrom
domenkozar:wal3-disable-body-statistics
Open

[ENH](wal3): Trim body metadata#7645
domenkozar wants to merge 1 commit into
chroma-core:mainfrom
domenkozar:wal3-disable-body-statistics

Conversation

@domenkozar

@domenkozar domenkozar commented Aug 27, 2026

Copy link
Copy Markdown

Summary

  • disable Parquet statistics for the opaque wal3 body column
  • retain statistics for offsets and timestamps
  • add a large incompressible-record regression test covering fragment size, round-trip decoding, and setsum verification

Parquet's page-statistics default copies a single large binary value into min/max metadata multiple times. A focused probe reduced a roughly 2 MB one-record fragment from 10.24 MB to 2.05 MB. Existing fragments remain readable because this changes writer properties only.

S3 request count is unchanged, while fragment storage and transfer bytes drop substantially for large records.

Tests

  • cargo test --manifest-path rust/wal3/Cargo.toml --lib writer::tests
  • cargo test --manifest-path rust/wal3/Cargo.toml --lib interfaces::tests::checksum_parquet
  • cargo clippy --manifest-path rust/wal3/Cargo.toml --lib -- -D warnings
  • cargo fmt --manifest-path rust/wal3/Cargo.toml -- --check

Tilt-backed integration tests were not validated locally because the required services were not active.

Disable Parquet statistics for opaque log bodies. Large records no
longer get copied repeatedly into min/max metadata.

Keep offset and timestamp statistics. Cover fragment size, round-trip
decoding, and setsum verification with a regression test.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

Copy link
Copy Markdown

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

@domenkozar

Copy link
Copy Markdown
Author

cc @rescrv

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant