Initial investigation on how to handle the 1Tb limit on OpenSearch Serverless index

[tloubrieu-jpl]

Are you sure this is not a new requirement or bug?

Yes

Task Type

Sub-task

💡 Description

Understand:

• when the partition of the index is needed
• how to partition them
• make the necessary upgrade in the registry app (create sub-tickets first)

Engineering Details

Questions to address:

Is it as simple as “write to index n according to some deterministic function of some timestamp property” and “read from an alias which amalgamates all n indices for a node”?

Are there any unforeseen performance implications?

How would the alias be managed? All indices created upfront, or created dynamically and the alias updated? What would be responsible for the dynamic updates?

How would we determine a reasonable splitting interval?

Does the hard limit of 1000 indices per collection pose a showstopping problem? It seems like a lot, but we’re already using what, 30 indices already? What future increase in scaling do we need to account for?

Are the proportional data volumes of the various nodes predictable/consistent, or might they change significantly in future?

Started with AWS CASE 174958255400010

Impacted Requirements

• 🦄 As a user, I want to search by a full/unique hierarchical path for a specific attribute registry-api#611
• double indexation keyword/text
• embedding indexation