Hi,
I would like to suggest an addition to the collection.Collection.update() method. This method currently supports updating partitions, meaning the defined callback will get a dataset matching the storage. However, if our data of interest is just one part of the partition, we need to add some filtering logic before the call ('filters' argument) and some segmentation logic in the callback (processing of the dataset.Dataset to retrieve the interesting part. Both of these logic can be generalized if we rely on the Indexer interface.
To illustrate this, let's say we work with altimetry data. Then some algorithms will need a segment of data matching an half_orbit (cycle_number, pass_number), and it can be troubling when the chosen partitioning is per day (or per month). This example in the documentation shows how to build an indexer to retrieve the half orbits of interest.
If we stay in the context of the example (in particular the same indexer), we would ideally have the following call
## Using the index for updating part of the collection
collection.update(
callback, # Function that really needs to have one half orbit, and not one day or one month of data
indexer=indexer.query(dict(pass_number=[1, 2])),
delayed=False,
)This example showed a use case with an HalfOrbitIndexer. Another good example is one using a GeohashIndexer, allowing us to update only the requested areas.
Hi,
I would like to suggest an addition to the collection.Collection.update() method. This method currently supports updating partitions, meaning the defined callback will get a dataset matching the storage. However, if our data of interest is just one part of the partition, we need to add some filtering logic before the call ('filters' argument) and some segmentation logic in the callback (processing of the dataset.Dataset to retrieve the interesting part. Both of these logic can be generalized if we rely on the Indexer interface.
To illustrate this, let's say we work with altimetry data. Then some algorithms will need a segment of data matching an half_orbit (cycle_number, pass_number), and it can be troubling when the chosen partitioning is per day (or per month). This example in the documentation shows how to build an indexer to retrieve the half orbits of interest.
If we stay in the context of the example (in particular the same indexer), we would ideally have the following call
This example showed a use case with an HalfOrbitIndexer. Another good example is one using a GeohashIndexer, allowing us to update only the requested areas.