RDF serialisation needs a specification

[gouttegd]

If the RDF format produced/consumed by sssom-py is supposed to be usable at the same level as the SSSOM/TSV format, it needs to be specified.

Having to infer the serialisation rules by looking at the output of the “reference implementation“ (which is what I had to do to implement RDF support in SSSOM-Java) is not acceptable for something that pretends to be a standard.

Same thing for the OWL/RDF serialisation, which is only specified (poorly) for the individual mappings, not for mapping set objects.

[matentzn]

I think this is critical for some of our prospective user base. for me, the main problem is how to deal with #359 when the pressure mounts (I expect it to mount the latest in the summer). will we have "mapping_id" (as an "optional" property) hang off the reified RDF statement as sssom:mapping_id, or will we use it as the resource identifier for the mapping (which is more correct, but will create an entirely different RDF serialisation).

I expect people to favour the latter, but I find it hard to reconcile this with the "optionality" of the mapping_id.

[gouttegd]

we use [the mapping_id] as the resource identifier for the mapping (which is more correct, but will create an entirely different RDF serialisation). I expect people to favour the latter

As you note, this would result in a RDF serialisation entirely different than the one currently being supported by SSSOM-Py (and SSSOM-Java, since in the absence of a spec SSSOM-Java basically did the same thing as SSSOM-Py).

That is not necessarily a blocker (RDF serialisation is not specified as of SSSOM 1.0, so it’s fine to radically change it), but still something to be carefully considered.

If people have already started using the current RDF serialisation, we might end up having to specify two different RDF serialisations: one that corresponds to the current output produced by SSSOM-Py (where individual mappings are blank nodes, without a resource identifier), and one that uses the to-be-introduced mapping_id slot as the resource identifier.

I find it hard to reconcile this with the "optionality" of the mapping_id.

(A) The new RDF serialisation format is only available for mapping sets that do use the new mapping_id slot. The slot itself remains generally optional, but the new RDF serialisation format requires it.

(B) Consequently, if you have a mapping set without mapping_id slots and you absolutely want to serialize it into the new RDF format, you must generate mapping_id slots on the fly.

[gouttegd]

I find it hard to reconcile this with the "optionality" of the mapping_id.

In fact, thinking back about this, I don’t see how is that a problem, given that RDF has the concept of blank nodes?

When serialising to RDF:

• If a mapping has a mapping_id slot, then serialise the mapping as a named node with the mapping_id as the resource identifier.
• Otherwise, then serialise the mapping as a blank node.

When deserialising from RDF:

• If a mapping has a resource identifier, store it into the mapping_id slot.
• Otherwise (blank node), leave the mapping_id slot empty.

So, for example, with identifiers:

<https://example.org/myset> a sssom:MappingSet ;
   terms:title "My mapping set" ;
   sssom:mappings <https://example.org/mymapping1>, <https://example.org/mymapping2> .

<https://example.org/mymapping1> a owl:Axiom ;
   owl:annotatedSource UBERON:0000001 ;
   owl:annotatedProperty semapv:crossSpeciesExactMatch ;
   owl:annotatedTarget FBbt:00000001 ;
   sssom:mapping_justification semapv:ManualMappingCuration .
   
<https://example.org/mymapping2> a owl:Axiom ;
   owl:annotatedSource UBERON:0000002 ;
   owl:annotatedProperty semapv:crossSpeciesExactMatch ;
   owl:annotatedTarget FBbt:00000002 ;
   sssom:mapping_justification semapv:ManualMappingCuration .

The same set, without identifiers:

[] a sssom:MappingSet ;
   terms:title "My mapping set" ;
   sssom:mappings [ a  owl:Axiom ;
       owl:annotatedSource UBERON:0000001 ;
       owl:annotatedProperty semapv:crossSpeciesExactMatch ;
       owl:annotatedTarget FBbt:00000001 ;
       sssom:mapping_justification semapv:ManualMappingCuration
   ], [ a owl:Axiom ;
       owl:annotatedSource UBERON:0000002 ;
       owl:annotatedProperty semapv:crossSpeciesExactMatch ;
       owl:annotatedTarget FBbt:00000002 ;
       sssom:mapping_justification semapv:ManualMappingCuration
   ] .

[matentzn]

I agree in that this is an ok compromise, but, there are strict limitations. If you stick with blank nodes, for example, you cannot easily build an API on top of the sssom data model (without extending it for API purposes) which requires one to be able to not only go from a mapping set to a mapping, but also back; say you have a search, then you find all mappings related to your search, how to you retain which mapping set a mapping comes from without it being a named node in the graph?

I understand this is a bit detail, but at the very least we should contemplate this as we tripped all over this when trying to build a massive RDF graph with all mappings in the world and wanting to build a super thin API layer on top..

[gouttegd]

Well then, just explicitly state that your API can only work on mapping sets whose mappings have a mapping_id slot. And if it is provided with mappings without such a slot, have it automatically inject auto-generated IDs (not the same thing as auto-generated blank node IDs!).

[gouttegd]

And if it is provided with mappings without such a slot, have it automatically inject auto-generated IDs

If we do end up introducing an optional mapping_id slot in SSSOM 1.1, it is almost certain that SSSOM tools like SSSOM-Py or SSSOM-Java will have to provide a feature allowing to inject auto-generated IDs to mappings that do not have one.

Otherwise, how would you, for example, merge a mapping set that contains mapping IDs with a mapping set that does not? As I recall, it was kind of agreed, last time we discussed about a potential mapping_id slot, that it would be optional but with the constraint that within a set, either all mappings have a mapping_id, or none of them have one (it would not make sense to have some mappings with an identifier and some mappings without one).

[matentzn]

If we do end up introducing an optional mapping_id slot in SSSOM 1.1, it is almost certain that SSSOM tools like SSSOM-Py or SSSOM-Java will have to provide a feature allowing to inject auto-generated IDs to mappings that do not have one.

So what would be the behaviour if you merge two mapping sets without mapping_id? Using blank nodes syntax? You maintain then that in these cases, we do not auto-inject? Or wouldn't it better for rdf to always autoinject (for example by hashing the entire mapping object)?

[gouttegd]

So what would be the behaviour if you merge two mapping sets without mapping_id?

If the two mapping sets are both without mapping_id slots, then there’s nothing special to do. The resulting set will also be without mapping_id slots. (And so, if the set is then exported to RDF, its mappings will be serialised as blank nodes.)

If one set has mapping_id slots and the other does not, then the user will have to decide between

a) drop the mapping_id slots from the first set;
b) fabricate mapping_id slots for the second set;
c) do nothing and proceed with the production of a set with a mix of mappings with a mapping_id slot and mappings without one – but in that case, warn the user than the resulting set will not be compliant with the spec, and therefore may not be handled correctly by all tools).

This is completely orthogonal with the question of what happens if/when the set is later exported to RDF. If the user chooses (a) and then writes to RDF, then the RDF model will contain mappings serialised as blank nodes (since they don’t have mapping_id slots); if she chooses (b) and then writes to RDF, then the RDF model will contain mappings serialised as named nodes (since they will have mapping_id slots); if she chooses (c) and then writes to RDF, then the RDF model will contain some mappings serialised as named nodes (those coming from the first set), and some mappings serialised as blank nodes (those coming from the second set) – arguably this would probably not be a very good idea, which is precisely why the user should be warned against doing that.

You maintain then that in these cases, we do not auto-inject?

I tend to think that auto-injection of mapping IDs should only ever happen upon the explicit request of the user, or at the very least with her informed consent. (“This program can only work if mappings have a unique identifier; if you load a mapping sets that do not contain such identifiers, auto-generated identifiers will be injected. Proceed? Yes/Cancel.”)

Or wouldn't it better for rdf to always autoinject (for example by hashing the entire mapping object)?

I don’t see why that would be better. I get that you are thinking about use cases where mapping identifiers would be needed anyway, but that may not always be the case. Given that neither SSSOM/TSV nor SSSOM/JSON currently mandate that mappings should have identifiers, why would RDF, specifically, mandate that? Knowing that RDF’s blank node syntax gives us a perfectly normal way to represent ID-less mappings?

(And I also think that auto-generated IDs should not be derived from the mapping itself by hashing its contents.)

[matentzn]

From @gouttegd on slack:

One thing in particular is absolutely not obvious (and would, in my opinion, deserve discussion): that all multivalued elements (including the list of mappings in a set) are simply represented by a list of triples associating the slot holding the list with the different values of the list.
For example, the list of mappings M1, M2, ..., Mn in a set S is represented by a list of triples like:
S sssom:mappings M1
S sssom:mappings M2
...
S sssom:mappings Mn
OK, this makes sense, and it is the easiest way to represent a list.
But we could have decided to use RDF containers or RDF collections instead (in fact when I started working on adding RDF support for SSSOM-Java I had assumed that RDF collections would be used – I had to look at the output of SSSOM-Py to realise that a much simpler approach had been chosen instead).

One reason I like

S sssom:mappings M1
S sssom:mappings M2
...
S sssom:mappings Mn

I am not a big fan of RDF containers, e.g:

S sssom:mappings (
  M1
  M2
  Mn
) .

or even worse, explicitly:

S sssom:mappings _:list1 .
_:list1 rdf:first M1 ;
        rdf:rest _:list2 .
_:list2 rdf:first M2 ;
        rdf:rest rdf:nil .

because it makes it very cumbersome to query with SPARQL. I am so far nearly convinced we should use the simple first option.

[gouttegd]

I am so far nearly convinced we should use the simple first option.

No objection to that, but there’s at least two things that are at least worth discussing with the simple option.

(A) It does not guarantee the order of items. Maybe it’s a big deal, maybe not. I personally don’t think it is, but this may run afoul of some people’s expectations.

For example, some people may expect that there is a correlation between the list of author_id and the list of author_label – that is, when both author_id and author_label are present, the first author_label is the label of the first author_id, the second author_label is the label of the second author_id, and so on. This is only possible if we can guarantee that the order of items in lists is always preserved.

(I personally think they should not assume that, and in fact the spec should explicitly and unambiguously state that the two lists are completely unrelated.)

People may also assume things such as, the first author is the most important one, the author that did most of the work. This again assumes that there is an order in lists. (Again, I think people should not assume things like that, but I also recognise that this is not, at first sight, an unreasonable assumption to make.)

(B) RDF lists are closed. There is an explicit list terminator (rdf:nil), so even under an open world assumption you can safely assume that the list you have is the complete list, there can’t be no other items that you are not aware of (in fact you don’t have to assume that, you know it). With a simple unstructured “list” of triples, the list is open-ended.

Again, maybe this is a big deal, maybe not, but I think this is an important semantic difference that warrants at least a discussion.

[matentzn]

All good points. I am a bit concerned if we can impose a sort order at all if we rely on standard serialisers? Like if we take SSSOMs:

MUST sort the mappings in lexicographical order on all their slots, in the order the slots appear in the “Slots” table.

Can something like this be done with an out of the box RDF writer like say Jenas or rdflib, or do we have to implement our own serialisation procedure?

[gouttegd]

I am a bit concerned if we can impose a sort order at all if we rely on standard serialisers?

Not if we stick to the “simple” representation of lists. If we want to prescribe an order, then we kinda must use RDF collections.

If we stick to the simple representation, then even if I wrote my own Turtle serialiser in SSSOM-Java to force the list items to be written in a given order, there would be no guarantee that whichever program reads the set would preserve the order. Unless you are somehow suggesting that we invent our own variant of Turtle that is almost identical to standard Turtle, except that the orders in which the triples are listed is significant and must be preserved. (I know you are not suggesting that! right?)

Note, however, that this:

MUST sort the mappings in lexicographical order on all their slots, in the order the slots appear in the “Slots” table.

comes from the spec of the SSSOM/TSV format. This is not relevant for the SSSOM/RDF serialisation.

And more precisely, it comes from the definition of the canonical SSSOM/TSV format, whose only purpose is to reduce spurious diffs due to serialisation differences. It does not mean that the order of mappings is meaningful in any way.

[gouttegd]

To clarify: I am not suggesting that we should absolutely revise the way lists are represented in the (existing, undocumented) RDF output. I am perfectly happy with the “simple” representation we currently use, and in fact I would be quite in favour of keeping it if only for the simple reason that this is what SSSOM-Py has always produced – I see no reason for changing that now, unless we decide that the shortcomings of that representation (absence of inherent order, “open-endedness”) are somehow problematic.

I personally don’t think those shortcomings are problematic, I just want to be sure that the different options have been considered – and then that the representation that is ultimately chosen is properly documented.