Clarify meaning of "mandatory" status of DSD attributes & measures

[dosse]

The information model doesn't precise the meaning of the "mandatory" status of DSD attributes/measures, other than through the expression "if it must be reported or not". In which precise reporting situations mandatory attributes/measures need to be present or not?

Some implementers (especially of SDMX message validators) interpret the "mandatory" status as the necessity of the systematic presence of mandatory attribute/measure values in every data message that contain related observations. This interpretation conflicts with messages created using the updateAfter SDMX rest parameter, because those are supposed to contain only those values that have actually been changed since the given point in time. Non-changed mandatory attribute/measure values are not included in those messages.

Other implementers (especially of SDMX data storage systems) interpret the "mandatory" status as the necessity of the systematic presence of mandatory attribute/measure values in every data store that contain related (reported) observations. It means that individual messages do not need to systematically carry mandatory attribute/measure values, but whenever those messages are loaded into a data store, the data store must always be in a fully valid state, thus contain all mandatory attribute/measure values for all stored observations. If those values are already present in the data store, they do not need to be re-uploaded with observation updates.

This clarification is needed to solve the mentioned current conflicts.

[buerokrata]

This dilemma boils down to how we interpret the minimum unit of change.

In my interpretation this was always observation level (because that is the smallest thing I can pin down with a combination of dimensions, i.e. a key; and it is the level where we can most granularly package information in a message). In that spirit the 'what changed' from the UpdateAfter description text means 'what observation changed'. If an observation changed then everything associated with that observation (measures and attributes, no matter whether their values have been modified individually) have to be sent. The way affected observations and changes are conveyed of course depend a lot on the chosen packaging, in sdmx-csv this means sending all affected observations (with their current valid state - obs-value, manadatory attributes and non-mandatory attributes). In SDMX-ML it depends whether the packaging is at observation or a series level, although the attributes should be provided nonetheless, they just appear at different locations in the package.