feat: chunk data store#17296
Open
ematipico wants to merge 8 commits into
Open
Conversation
🦋 Changeset detectedLatest commit: ca9a758 The changes in this PR will be included in the next version bump. This PR includes changesets to release 393 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
github-actions
Bot
added
semver: minor
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
yanthomasdev
reviewed
Jul 7, 2026
yanthomasdev
left a comment
yanthomasdev
left a comment
Member
There was a problem hiding this comment.
A couple bikeshedding questions from me.
| 'astro': minor | ||
| --- | ||
|
|
||
| Adds a new experimental `dataStore` option for controlling how the content layer persists its data store |
Member
There was a problem hiding this comment.
dataStore feels a bit too broad, I wonder if that might feel misleading and indicate this also controls other data like images or assets? Since this is only content collections related, I am wondering if maybe this feature would be better named something like "Collections storage" (there's probably a better name than this suggestion)
Member
Author
There was a problem hiding this comment.
What about collectionStorageKind or just collectionStorage? I am not attached to the name, so if there's a better name, it's more than welcome
Member
Author
There was a problem hiding this comment.
I chose collectionStorage. Happy to change again if we find a better name
ArmandPhilippot
approved these changes
Jul 8, 2026
ArmandPhilippot
left a comment
ArmandPhilippot
left a comment
Member
There was a problem hiding this comment.
A few file that haven't been updated to single-file, otherwise LGTM docs-wise. And, I don't have a better name to suggest so I think it's fine. Thank you both!
Co-authored-by: Armand Philippot <git@armand.philippot.eu>
matthewp
reviewed
Jul 9, 2026
| for (const [collectionName, entries] of sortCollections(collections)) { | ||
| const chunks: string[][] = []; | ||
| // Bound the size of any single serialized string. | ||
| for (const chunkedEntries of chunkMap(entries, CHUNK_ENTRY_LIMIT)) { |
Contributor
There was a problem hiding this comment.
The problem I see is that it splits on number of entries and then a size. That means that you could have one single entry that exceeds the size limit, which means you exceed the platform limits anyways.
Member
Author
There was a problem hiding this comment.
Done, I removed the number of entries as threshold ca9a758 (this PR)
Contributor
|
I don't think it should split based on number of entries and file size. I think it should only be on file size. In addition to the reason given in the inline comment, having many very small entries doesn't have any downsides I can think of. Doing it purely based on file size is both simpler for implementation and simpler for the user to reason about. I also would expect the file size to be configurable, as different platforms might have different limits. |
matthewp
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
This PR adds a new experimental option called
dataStore. It allows to split the data into multiple chunks.I also did some refactor, so that we can prepare different data sources e.g. sqlite. For this reason, the functions of the interface are also async.
Chunks are created when:
Testing
Added various tests for:
Docs
withastro/docs#14210
/cc @withastro/maintainers-docs for feedback!