GET /classes and /classes/:cls/children return 500 for ontologies containing invalid IRIs (ELD, AUTISM)

[mdorf]

Summary

GET /ontologies/:ontology/classes and GET /ontologies/:ontology/classes/:cls/children return HTTP 500 for any page whose id slice contains a class IRI that is not a valid IRI per RFC 3987. AllegroGraph accepts such an IRI at load time but its SPARQL parser rejects it inside any <...> token, so the batch attribute-loading query for the whole page fails with SPARQL::Client::MalformedQuery.

Currently affected on production: ELD and AUTISM.

Observed in New Relic (BioPortal REST API, ncbo-prd-api-01, GET /ontologies/:ontology/classes):

SPARQL::Client::MalformedQuery
MALFORMED QUERY: Line 1, Found '<'. Was expecting one of: DECIMAL, DOUBLE, FALSE, INTEGER,
Q_IRI_REF, QNAME, QNAME_NS, STRING_LITERAL1, ... or punctuation ')', '<<'.
Processing query SELECT DISTINCT ?id ?attributeProperty ?attributeObject
FROM <http://data.bioontology.org/ontologies/ELD/submissions/5> WHERE { ... VALUES (?id) { ( <...> ) ... } ... }

The client-visible response is a bare text/html <h1>Internal Server Error</h1> with no indication of the cause.

Reproduction

# 500: whole page fails
curl -si "https://data.bioontology.org/ontologies/ELD/classes?apikey=$API_KEY" | head -1

# 500: 14 of ELD's 22 root branches
curl -si "https://data.bioontology.org/ontologies/ELD/classes/https%3A%2F%2Fgithub.qkg1.top%2FVODANA%2FControlled-vocabularyDigestiveSystem/children?apikey=$API_KEY" | head -1

# 400 with a clear message: the same IRI requested individually
curl -s "https://data.bioontology.org/ontologies/ELD/classes/https%3A%2F%2Fgithub.qkg1.top%2FVODANA%2FControlled-vocabularySevereacuterespiratorysyndrome%5BSARS%5D?apikey=$API_KEY"
# {"errors":["The input class id '...[SARS]' is not a valid IRI"],"status":400}

# 200: sibling concepts whose IRIs contain parentheses or commas (those are legal)
curl -s "https://data.bioontology.org/ontologies/ELD/classes/https%3A%2F%2Fgithub.qkg1.top%2FVODANA%2FControlled-vocabularyDermatitis%28Allergiccontactdermatitis%29?apikey=$API_KEY"

Impact

Endpoint	ELD	AUTISM
/classes	500	500
/classes/roots	200	500
/classes/:cls/children	500 for 14 of 22 root branches	500

• ELD submission 5 has 53 of 2,077 concept IRIs containing square brackets (counted from the RDF/XML the API serves at /ontologies/ELD/download). Nothing else in that file is unusual: no whitespace, no other character illegal in a SPARQL IRIREF.
• Class browsing for these ontologies is effectively unusable through the API, and the whole page fails, not just the offending concept.
• The web UI degrades visibly: on bioportal.bioontology.org/ontologies/ELD, narrower links for the affected concepts render as raw IRIs instead of preferred labels, because their attributes never load.
• The 500 is opaque. Nothing in the response tells a caller that the ontology contains invalid IRIs.

Root cause

The offending IRIs contain characters that are illegal in an RFC 3987 IRI. Square brackets are reserved for IPv6 literals in the authority component and are invalid in a path, so RDF::URI#valid? returns false. Parentheses and commas, which look equally suspicious in these IRIs, are legal and work fine:

RDF::URI.new('https://github.com/VODANA/Controlled-vocabularySevereacuterespiratorysyndrome[SARS]').valid?
# => false
RDF::URI.new('https://github.com/VODANA/Controlled-vocabularyDermatitis(Allergiccontactdermatitis)').valid?
# => true

AllegroGraph is asymmetric about these IRIs: it stores them, and returns them when they are bound to a variable, but rejects them when a query names them in a <...> token. Verified against a local AllegroGraph 8.4.3, which returns the same parser message as production:

What was sent	Result
Load a triple whose subject is <https://example.org/test[BRACKET]>	200, statement stored
SELECT ?s ?o FROM <graph> WHERE { ?s ?p ?o }	200, returns https://example.org/test[BRACKET]
VALUES (?id) { ( <...test[BRACKET]> ) }	400 MALFORMED QUERY: ... Found '<'
FILTER(?id = <...test[BRACKET]>)	400 MALFORMED QUERY: ... Found '<'
<...test\u005BBRACKET\u005D> (SPARQL codepoint escapes)	400, same error
<...test%5BBRACKET%5D> (percent-encoded)	200, but 0 bindings

So the data reaches goo through a variable-bound id query, and then goo names those same ids back to the store to load their attributes (Goo::SPARQL::QueryBuilder#ids_filter, reached via model_load_sliced). The second query is unparseable, and the request 500s.

Escaping in the query layer is not a fix. Codepoint escapes are still rejected, and percent-encoding parses but changes the identity of the IRI, so it silently matches nothing. The stored IRI cannot be named in SPARQL at all.

This is not a regression from the recent ids_filter change

goo recently replaced the disjunctive FILTER with a VALUES clause (ncbo/goo#191, commit cb0b87b), deployed today, which changed the error text and therefore created fresh New Relic error groups. The underlying failure predates it: the old FILTER(?id = <...>) form is rejected by AllegroGraph identically (row 4 above), because RDF::URI#to_ntriples does not escape square brackets either.

The same root cause is already reported from a different entry point in ncbo/goo#147 (open since 2023-08-29, ncbo_ontology_report_rebuild failing for ANC, AUTISM, CMECS2, ELD, ETH_ANC). That issue's examples show the wider family of invalid IRIs in the corpus:

• spaces: <http://vocab.vodana.org/gdmt/Dateofvisit (DD/MM/YY)> (ANC)
• Windows file paths plus brackets: <C:/Users/maryk_000/2020/7/untitled-AutismOn#[OWL]_2aaf4433_...> (AUTISM)

The space case appears to be handled now by the gsub(' ', '%20') workaround in ids_filter, which is presumably why ETH_ANC currently returns 200. Brackets were never covered. ANC and CMECS2 are no longer in BioPortal (404).

Where the fix belongs

Three layers are involved. My suggestion is to do the first two and treat the third as the durable fix.

1. ontologies_api: stop returning an opaque 500. The single-class route already validates with RDF::URI#valid? and returns a clear 400 (helpers/classes_helper.rb:47). The list and children routes have no equivalent guard, so they inherit a raw MalformedQuery. At minimum, surface an actionable error that names the ontology and the invalid IRI.

2. goo: do not let one unusable id fail an entire page. ids_filter currently emits every id verbatim. Options: partition the id list, skip ids failing RDF::URI#valid? and log them, or fail the slice rather than the request. Silently dropping concepts is not great, but it is better than a 500 for the whole page, and it needs a decision.

3. Ingestion (ontologies_linked_data / ncbo_cron): reject or normalize invalid IRIs at parse time, so unreferenceable IRIs never enter the triple store. Today the parse pipeline accepts them, the store accepts them, and only the query path fails. Existing submissions would need reprocessing, and normalizing changes concept IRIs, so this needs a call on whether such submissions should fail with a clear parse error instead.

Finding other affected ontologies

Invalid IRIs can be located with a variable-bound scan, which parses fine:

SELECT DISTINCT ?s WHERE { ?s ?p ?o . FILTER(REGEX(STR(?s), "[\\[\\]]")) } LIMIT 100

Worth running per graph, widened to the other characters illegal in an IRIREF (space, <, >, ", {, }, |, ^, backtick, backslash) plus non-http schemes, to size the problem beyond ELD and AUTISM before choosing a fix.