[superceded] Define a standard SSSOM hashing function. #529
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| # Hashing a SSSOM mapping record | ||
|
|
||
| SSSOM implementations SHOULD provide a function to compute a hash on a | ||
| SSSOM mapping record. That function is hereafter called “the SSSOM | ||
| hashing function” and defined below. | ||
|
|
||
| ## Rationale and purpose | ||
| The SSSOM hashing function defined here is intended to allow mapping | ||
| records to be quickly compared and, in particular, to quickly determine | ||
| whether two records are identical. | ||
|
|
||
| The function is intended for **interoperability** between SSSOM | ||
| implementations. Its point is to ensure that one can always compute the | ||
| same hash for the same mapping record regardless of which SSSOM | ||
| implementation is used. | ||
|
|
||
| When an implementation needs to assert whether two records are identical | ||
| **for its own internal purposes** (for example, to store records into a | ||
| hash table), it may use whatever method is best suited without regard | ||
| for the SSSOM hashing function. | ||
|
|
||
| ## Hashing procedure | ||
|
|
||
| The general principle of the SSSOM hashing function is to compute a | ||
| ZBase32-encoded SHA2-256 hash of a canonical S-expression representing | ||
| the mapping record. | ||
|
|
||
| ### Step 0: Propagate all condensed slots | ||
| If the mapping set the mapping record to hash belongs to contains | ||
| condensed slots, they MUST be propagated to the mapping record [as per | ||
| the standard rules](spec-model.md#propagation). | ||
|
|
||
| ### Step 1: Turn the mapping record into a canonical S-expression | ||
| This step creates a representation of the mapping record into a | ||
| canonical S-expression as per [RFC 9804](https://www.rfc-editor.org/rfc/rfc9804#name-canonical-representation). | ||
|
|
||
| The S-expression MUST be assembled as follows: | ||
|
|
||
| 1. Start with `(7:mapping(`, | ||
| 2. Iterate over all slots of the `Mapping` class, in the order in which | ||
| they are [listed](../Mapping/#slots) in the LinkML model. Exclude the | ||
| `record_id` slot and the `mapping_cardinality` slot. For all other | ||
| slots: | ||
| 1. If the slot has no value for the mapping record to hash, skip to | ||
| next slot. | ||
| 2. Append to the S-expression `(N:SLOTNAME`, where _SLOTNAME_ is the | ||
| LinkML name for the slot and _N_ is the length of the slot name (so, | ||
| for rxample, `(10:subject_id`, `(9:author_id`, `(10:confidence`, | ||
| etc.). | ||
| 3. If the slot is defined as a multi-valued slot (and even if it has | ||
| only one value in the mapping record to hash): | ||
| 1. Append `(`. | ||
| 2. Sort the list of values in lexicographical order and iterate | ||
| over the sorted values. For each value _V_, append `N:V`, where | ||
| _N_ is the length of _V_. | ||
| c. Append `)`. | ||
| 4. If the slot is typed as a floating point number (e.g. | ||
| `confidence`), convert the value to a string _V_ by truncating the | ||
| floating point number to up to 3 digits after the decimal point, | ||
| rounding the last digit to the nearest neighbour (rounding up if | ||
| both neighbours are equidistant). Then append `N:V`, where _N_ is | ||
| the length of _V_. | ||
| 5. If the slot is typed as an enumeration (e.g. `subject_type`), | ||
| append `N:ENUMVALUE`, where _ENUMVALUE_ is the allowed value in the | ||
| enumeration as specified in the LinkML model, and _N_ is the length | ||
| of _ENUMVALUE_ (e.g. `9:owl class` for a possible value for the | ||
| `subject_type` slot). | ||
| 6. If the slot is typed as a date, append `10:YYYY-MM-DD`, where | ||
| _YYYY-MM-DD_ is the representation of the value in ISO-8601 format. | ||
| 7. If the slot is typed as an entity reference, ensure the value is | ||
| expanded according to the mapping set’s prefix map and append `N:V`, | ||
| where _V_ is the expanded reference and _N_ is the length of the | ||
| expanded reference. | ||
| 8. If the slot is of any other type, append `N:V`, where _V_ is the | ||
| string value of the slot and _N_ is the length of the string value. | ||
| 9. Append `)`. | ||
| 3. If the implementation support [extension slots](spec-model.md#non-standard-slots) | ||
| and the mapping record does have such slots: | ||
| 1. Append `(10:extensions(`. | ||
| 2. Sort extension values by their properties in lexicographical | ||
| order. | ||
| 3. For each extension value: | ||
| 1. Append `(N:PROP`, where _PROP_ is the property identifying | ||
| the extension and _N_ is the length of the property. | ||
| 2. Append `N:V`, where _V_ is the extension value proper and _N_ | ||
| its length. | ||
| 4. Append `))`. | ||
| 4. Append `))`. | ||
|
|
||
| ### Step 2: Compute the SHA2-256 hash of the S-expression | ||
| Encore the S-expression assembled in step 1 into UTF-8 (if it was not | ||
| already assembled directly in UTF-8). Then hash the array of bytes | ||
| containing the UTF-8 representation of the S-expression using the | ||
| standard SHA2-256 hash function as defined in | ||
| [NIST FIPS-180-4](https://doi.org/10.6028/NIST.FIPS.180-4). | ||
|
|
||
| ### Step 3: Encode the hash into a ZBase32 string | ||
| Encode the hash computed in step 2 into its representation in the | ||
| ZBase32 encoding ([RFC 6189](https://tools.ietf.org/html/rfc6189#section-5.1.6)). | ||
|
|
||
| ## Example | ||
|
|
||
| > This section is not normative. It provides a step-by-step example of | ||
| how to apply the above procedure. | ||
|
|
||
| Given the following mapping record (in JSON format): | ||
|
|
||
| ```json | ||
|
gouttegd marked this conversation as resolved.
Outdated
|
||
| {"subject_id": "http://purl.obolibrary.org/obo/FBbt_00001234", | ||
| "predicate_id": "http://www.w3.org/2004/02/skos/core#exactMatch", | ||
| "object_id": "http://purl.obolibrary.org/obo/UBERON_0005678", | ||
| "mapping_justification": "https://w3id.org/semapv/vocab/ManualMappingCuration", | ||
| "creator_id": [ | ||
| "https://orcid.org/0000-0000-5678-1234", | ||
| "https://orcid.org/0000-0000-1234-5678" | ||
| ]} | ||
| ``` | ||
|
|
||
| Applying step 1 of the above procedure would yield the following | ||
| canonical S-expression (**whitespaces added for clarity**, they MUST NOT | ||
| appear in the actual S-expression): | ||
|
|
||
| ``` | ||
| (7:mapping( | ||
| (10:subject_id44:http://purl.obolibrary.org/obo/FBbt_00001234) | ||
| (12:predicate_id46:http://www.w3.org/2004/02/skos/core#exactMatch) | ||
| (9:object_id45:http://purl.obolibrary.org/obo/UBERON_0005678) | ||
| (21:mapping_justification51:https://w3id.org/semapv/vocab/ManualMappingCuration) | ||
| (10:creator_id( | ||
| 37:https://orcid.org/0000-0000-1234-5678 | ||
|
cthoyt marked this conversation as resolved.
|
||
| 37:https://orcid.org/0000-0000-5678-1234 | ||
| )) | ||
| )) | ||
| ``` | ||
|
|
||
| Applying the SHA2-256 hash function to the above S-expression would | ||
| yield the following hash (in hexadecimal): | ||
| `e3bc1b4b586c6e86d0caf369d49c161163e255c4a779821f448a8e4fbd616522`. | ||
|
|
||
| Finally, encoding the binary SHA2-256 hash in ZBase32 would yield the | ||
|
Member
There was a problem hiding this comment. is the brevity of the hash worth the complexity of encoding/unencoding in yet another step?
Contributor
Author
There was a problem hiding this comment. The output of the SHA2-256 hash function is a binary blob that cannot be used directly, so we need a post-hash encoding step anyway. We could use hexadecimal encoding instead of ZBase-32 encoding, but that’s still “yet another step” that must be performed after the hashing. As for which encoding to use, I raised that question here with some of the available options. Nobody weighed in on that issue.
Member
There was a problem hiding this comment. I would prefer using hexadecimal digest of the SHA2-256 hash function over zbase-32 since this is more common, and it doesn't appear that there is either built-in support for zbase-32 nor an obviously well-maintained/authoritative implementation of it in Python
Member
There was a problem hiding this comment. rationale behind worrying about not having an authoritative implementation - this is becoming even more of a security risk if we have to include packages like this in our infrastructure given the current state of open source
Contributor
Author
There was a problem hiding this comment. I still personally favour zbase-32, because it gives hashes that are more human-readable than all other possible encodings. I am not convinced about the availability of “an obviously well-maintained/authoritative implementation of it in Python” being an issue. On what grounds do you say that the several available implementations are not “obviously well-maintained“? If it’s because they have not been updated in years, I don’t believe this is a valid criterion for a function like I wrote a C implementation of it 9 years ago, I am still using it in some C projects to this day and I never had to retouch it since the first write. Worst-case scenario, in case a Python developers really does not want to have to depend on an external library, implementing zbase-32 is trivial. Here’s for example my aforementioned C version: static char zb32[] = "ybndrfg8ejkmcpqxot1uwisza345h769";
/**
* Encodes a buffer in Z-Base32 encoding.
* This function encodes data from the provided buffer and writes the
* encoded data to the specified buffer.
*
* @param[in] src The input buffer.
* @param[in] slen The size of the @a src buffer.
* @param[out] dst The output buffer.
* @param[in] dlen The size of the @dst buffer.
*
* @return The number of bytes written to the output buffer.
*/
size_t
zbase32_encode(const char *src, size_t slen, char *dst, size_t dlen)
{
size_t i, j, k;
for ( i = j = 0; i < slen && j + 7 < dlen ; i += 5 ) {
int64_t val;
for ( val = k = 0; k < 5; k++ )
if ( i + k < slen )
val |= (int64_t)((unsigned char)src[i+k]) << ((4 - k) * 8);
dst[j++] = zb32[(val & 0xF800000000) >> 35];
dst[j++] = zb32[(val & 0x7C0000000) >> 30];
if ( i + 1 < slen ) {
dst[j++] = zb32[(val & 0x3E000000) >> 25];
dst[j++] = zb32[(val & 0x1F00000) >> 20];
}
if ( i + 2 < slen )
dst[j++] = zb32[(val & 0xF8000) >> 15];
if ( i + 3 < slen ) {
dst[j++] = zb32[(val & 0x7C00) >> 10];
dst[j++] = zb32[(val & 0x3E0) >> 5];
}
if ( i + 4 < slen )
dst[j++] = zb32[val & 0x1F];
}
return j;
}Porting it to Python is left as an exercise to the reader. ;) |
||
| following final value: | ||
| `hq6bs14aptzepwgk6pw7j8ysnft6riqrw7har84rtk8r9xmbcwty`. | ||
|
|
||
| ## Test vectors | ||
|
|
||
| > This section is not normative. It provides examples of SSSOM mapping | ||
| records along with the corresponding hash value. | ||
|
|
||
| ```json | ||
| {"subject_id": "https://kewl-foodie.inc/food/F001", | ||
| "subject_label": "apple", | ||
| "predicate_id": "http://www.w3.org/2004/02/skos/core#exactMatch", | ||
| "object_id": "http://purl.obolibrary.org/obo/FOODON_00002473", | ||
| "object_label": "apple (whole)", | ||
| "mapping_justification": "https://w3id.org/semapv/vocab/ManualMappingCuration", | ||
| "author_id": [ | ||
| "https://orcid.org/0000-0002-7356-1779" | ||
| ], | ||
| "subject_source": "https://kewl-foodie.inc/food/DB", | ||
| "object_source": "https://www.wikidata.org/wiki/Q55118395", | ||
| "object_source_version": "http://purl.obolibrary.org/obo/foodon/releases/2022-02-01/foodon.owl", | ||
| "mapping_date": "2022-05-02", | ||
| "confidence": 0.95 | ||
| } | ||
| ``` | ||
|
|
||
| Hash value: `x4m9kcj8yjrrxh8ozwt83bkxcequb3fjqsamu9yyejyqft1gowao` | ||
|
|
||
| ```json | ||
| {"record_id": "https://example.org/sets/record-id#0000001", | ||
| "subject_id": "http://purl.obolibrary.org/obo/FBbt_0009124", | ||
| "predicate_id": "http://www.w3.org/2004/02/skos/core#exactMatch", | ||
| "object_id": "http://purl.obolibrary.org/obo/UBERON_0000003", | ||
| "mapping_justification": "https://w3id.org/semapv/vocab/LexicalMatching" | ||
| } | ||
| ``` | ||
|
|
||
| Hash value: `qn1bra45hjtazt664husfgah5ewzo3oamh4swj5gomuka88rrqgo` | ||
|
|
||
| ```json | ||
| {"subject_id": "http://purl.obolibrary.org/obo/HP_0009124", | ||
| "predicate_id": "http://www.w3.org/2004/02/skos/core#exactMatch", | ||
| "object_id": "http://purl.obolibrary.org/obo/MP_0000003", | ||
| "mapping_justification": "https://w3id.org/semapv/vocab/LexicalSimilarityThresholdMatching", | ||
| "mapping_provider": "https://w3id.org/sssom/core_team", | ||
| "similarity_score": 0.8, | ||
| "similarity_measure": "wikidata:Q865360" | ||
| } | ||
| ``` | ||
|
|
||
| Hash value: `is395b9nwm1rnz3nwkm89nmf563uw48sjspsx7ua8snjqzwz15ty` | ||
|
gouttegd marked this conversation as resolved.
Outdated
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # SSSOM supporting functions | ||
|
|
||
| This section defines functions and behaviours that SSSOM implementations | ||
| should support to help users manipulate SSSOM mapping sets. | ||
|
|
||
| * [Hashing mapping records](spec-support-hashing.md) | ||
| * [Chaining rules](chaining-rules.md) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.