Mechanism for pipelines to access the iFDO "image-set-header" section

[GermanHydrogen]

Currently, there is no interface for pipelines to write to or read from the iFDO "image-set-header" section. However, this access is necessary to create complete and valid iFDOs with marimba.

One solution would be to automatically pull duplicate/common information in the "image-set-items" objects in the header. This would require the pipeline to write the same information into the image information of each image in order to write to the header. While this could be easily be implemented in a non-breaking way, I think this solution may not be very intuitive for users, even if documented, as its behavior is not obvious when building pipelines. A sample implementation can be found here.

Therefore, I would prefer a solution that allows to directly change the header section from the pipeline. While this could result in conflicting information being written to the header by different pipelines, this could be easily detected and then simply force the user to choose a lower "metadata-level" configuration. So if two pipelines want to write different data to the same field in the header, the user would simply need to use a "metadata-level" config equal to pipeline or collection to avoid the collision. However, I currently have no idea, how to implement this solution.

[GermanHydrogen]

Addendum:
The first solution would not allow for setting metadata in the header, which varies in the image-data (for example "image-datetime", "image-longitude", etc.), but which is required to be set in the header.

[GermanHydrogen]

The second solution could be implemented, by adding MetadataHeader instances to the output of the BasePipeline._package method:

def _package(
        self,
        data_dir: Path,
        config: dict[str, Any],
        **kwargs: dict[str, Any],
    ) -> tuple[dict[Path, tuple[Path, list[BaseMetadata] | None, dict[str, Any] | None]], dict[type[BaseMetadataHeader], BaseMetadataHeader]]]:

This would allow for different instances of MetadataHeader objects to be output; e.g. iFDOMetadataHeader or BasicMetadataHeader.

If the metadata-output-level is not set to collection, the MetadataHeader from different collections and pipelines would be merged. However this is only possible, if there is no contradiction information between the MetadataHeader of the same kind. In this case this would restrict the metadata-output-level to either pipeline or collection level.

The MetadataHeader for a dataset is then inserted into the BaseMetadata.create_dataset_metadata function, which then can use the data wrapped by the MetadataHeader object to create a metadata file with the header information.

[cjackett]

Hi @GermanHydrogen,

This feature has been implemented in PR #32.

The implementation takes a different architectural approach than the one proposed in PR #31 - please see the comments there for details.

Feel free to review PR #32 and provide feedback on the approach and the structure of the output iFDO files.