Auto-prune documents on search

[ellnix]

Currently, the meilisearch database and the ORM commonly get out of sync if records are deleted while Meilisearch is disabled, or offline for whatever reason. Since Model#reindex! does not delete any invalid meili documents, a user would have to be diligent enough to manually delete the index and reindex from the start.

This makes it really hard to guarantee that every page has the same number of records (or that it even has any real records at all).

If we decided to solve this by looking ahead to the next page to get the correct number of documents, we would end up repeating the same documents on multiple pages.

I propose a system like this:

• When we search for page n
• we keep track of a prune_list: the IDs for which there is no record
• if we find fewer records than hits_per_page specifies
• we run another search on page n + 1, repeat the process until we have enough records
• we delete the documents in prune_list
  This ensures that the results we get comprise all of page n, when the request is made for a search on n + 1, the first document follows right after the results of page n.
• we return the correct number of results

Here's a rough diagram that doesn't follow any particular diagramming language:

Edit: Of course with this I would also propose that a we add an auto_prune option to the meilisearch function, so that users can disable this behavior.

[ellnix]

@pozelli let me know if you would consider this a good solution for correct pagination.

@brunoocasali let me know what you think about this idea.

[pozelli]

@ellnix

I really like this idea! Your approach ensures that each page has the correct number of results while gradually cleaning up invalid documents. Instead of requiring manual intervention (deleting or fully reindexing), it allows the system to self-correct over time. Also, we always get the expected number of results per page.

You could introduce overfetching by requesting slightly more records than hits_per_page (e.g., hits_per_page * 1.5), reducing the need for additional searches to fill a page.

[brunoocasali]

You could introduce overfetching by requesting slightly more records than hits_per_page (e.g., hits_per_page * 1.5), reducing the need for additional searches to fill a page.

I think this arbitrary number would never be enough if your index contains a lot of deleted docs.