(re)implement handling of partially-uploaded collections, missing products, etc

[alexdunnjpl]

Are you sure this is not a new requirement or bug?

Yes

Task Type

Theme

💡 Description

Corner-cases, which may require any/all of

• auxiliary index per-node to track products which were logged as missing/whatever during sweep (ancestry, specifically)
• orphan detection (implementation imminent TODO)

Cases are:

• collection is mid-harvest (i.e. some portion of nonaggs not uploaded yet) when it is picked up by ancestry sweeper
• product is just missing, and requires upload (as overwrite of enclosing collection, since that's currently the only way to stale it and trigger reprocessing while avoiding redundant processing.

Jordan likes the idea of a selective low-cadence retry for collections with "metadata ancestry-incomplete": true based on time of partial processing - other candidate designs exists but need to be assessed.

@jordanpadams @tloubrieu-jpl add anything you want, otherwise I'll flesh it out as I get to/through it.