Develop re-indexing strategy for migrating all indices to structured metadata

[jordanpadams]

Are you sure this is not a new requirement or bug?

Yes

Task Type

Sub-task

💡 Description

Develop a re-indexing strategy for migrating all existing registry OpenSearch indices to the new structured metadata format. Strategy should cover:

• Approach for re-indexing without data loss or downtime
• Handling of legacy flat metadata alongside structured metadata during transition
• Validation approach to confirm data integrity after re-indexing
• Rollback plan if re-indexing fails

Related:

• Complete Transition to Full Structured XML Metadata in Registry registry-api#768 (B18 structured XML theme)
• Add Initial Support for Searching Full PDS4 Structured Metadata registry-api#632 (initial structured metadata support)

[jordanpadams]

Additional Requirements for Re-indexing Strategy

Key Constraint: No Access to Data Files

During re-indexing, we will not have access to the original data files. All metadata must be derived from what is already stored in the existing OpenSearch documents. This is a critical constraint that must be accounted for in the strategy.

Required ops: Metadata Preservation

The following ops: namespace metadata must be carried forward unchanged into the new structured format. See the ops namespace documentation for reference.

Class	Attributes	Description	Example
Data_File_Info	creation_date_time		2021-09-10T15:58:03Z
	file_name		collection_document_hp3rad.csv
	file_ref	link
	file_size	in bytes	137
	md5_checksum		cd24cbc46c45ed023f039b3e2beb6606
	mime_type		text/plain
Harvest_Info	harvest_date_time		2021-11-16T06:03:30.952311900Z
	node_name		PDS_GEO
Tracking_Meta	archive_status		archived
Provenance	(not formally documented)	Tracks the origin/lineage of a product in the registry

The re-indexing strategy must ensure all of the following are preserved as-is:

• All Data_File_Info fields
• All Harvest_Info fields
• All Tracking_Meta fields
• All Provenance metadata

Since these cannot be regenerated from source files, any re-indexing approach that cannot read and re-emit these fields verbatim should be considered a blocking issue.

🤖 Generated with Claude Code

[al-niessner]

flat2structured1.pptx

Generate with Al

[al-niessner]

Updates:

Add field to documents indicating a successful transfer to structured/managed.

Harvest changes in registry-loader:

1. Make FileMetadataExtractor an interface then have the ExtractFromFilesystem (current implementation) and ExtractFromManifest(new for migration and more).
2. harvest does track the lidvids that failed the batch upload. Double checked and if return json object, which is what harvest is checking, says it is good then it will eventually make it to the visible/searchable part of opensearch.
3. track good lidvids as well as bad. set a flag in good ones to indicated transferred.
4. update harvest to fix as many sweeper has to repeatedly run to fix data problems as possible. the expensive and open ended bit.