GO hierarchy children capped at rows=10000 — truncates large terms (was unbounded under SPARQL); confirm latency-vs-completeness intent

[kltm]

Summary

GET /api/go/{id}/hierarchy silently truncates its children list at rows=10000. For GO terms with more than ~10k descendants (the ontology roots and high-level terms), results are dropped with no error or indication.

Example — GO:0008150 (the biological_process root), v0.4.0 build vs current production 0.3.9:

build	hierarchy items
production 0.3.9 (api.geneontology.org)	24,450
0.4.0 (Blazegraph-removed)	10,001

This is an accidental regression, not an intentional cap

• The old implementation was a SPARQL query with no LIMIT — unbounded, returned all rows.
• The new GOlr children query introduced the cap: app/routers/ontology.py:422 → optionals = "&rows=10000", in commit f776148 ("replace GO hierarchy SPARQL with golr query and add hierarchy tests") under PR Remove blazegraph: blazegraph dependency and replace with index gocam files #136 (umbrella Remove blazegraph dependencies in GO API #137).
• The value is arbitrary and inconsistent within the same file: line 121 uses rows=100000; ontology_utils.py uses rows=1000. No comment justifies 10000.
• It only manifests for terms with >10k descendants (the roots), so it is invisible in ordinary use.

No test catches it

The hierarchy tests added in the same commit only exercise small terms and assert structure, never completeness:

• tests/unit/test_ontology_endpoints.py::test_go_hierarchy uses GO:0003677 ("DNA binding") and asserts only status == 200, isinstance(list), len > 0, and the parent/query/child keys.
• tests/unit/test_ontology_utils.py::test_get_go_hierarchy_go_id loops small ids with len > 0 / status == 200.

Neither approaches the 10,000 boundary, so the truncation passes all tests.

Precedent

Row-default truncation has bitten this API before: #77 ("unexpected results after refactored code switchover" — a rows=100 default silently changed counts) and #138 (deliberately uses rows=-1 to avoid a cap). The isoform fix (#135) similarly used facet.limit=-1.

Proposed fix

1. Remove the cap on the hierarchy children query — use rows=-1 (or rows=100000, matching ontology.py:121).
2. Add a large-term regression test that would have caught this — e.g. assert the GO:0008150 hierarchy returns > 10,000 items.

Reproduction

# 0.3.9 (current production): 24450
curl -s "https://api.geneontology.org/api/go/GO:0008150/hierarchy" | python3 -c "import sys,json;print(len(json.load(sys.stdin)))"
# 0.4.0 build: 10001
curl -s "https://<0.4.0-host>/api/go/GO:0008150/hierarchy"        | python3 -c "import sys,json;print(len(json.load(sys.stdin)))"

— Posted by Claude Code agent on behalf of @kltm.

[kltm]

Update — performance data; reframing from "accidental" toward a latency-vs-completeness tradeoff.

I measured the hierarchy latency origin-to-origin (bypassing Cloudflare) for GO:0008150, which changes the picture:

	old 0.3.9 (SPARQL/Blazegraph)	new 0.4.0 (GOlr, rows=10000)
total	~5.3–5.8 s	~1.6–2.6 s
ttfb	~3.8–5.1 s	~0.8–1.0 s
payload	3.0 MB / 24,450 items	1.24 MB / 10,001 items

So the new GOlr hierarchy is ~3× faster than the old SPARQL one (which was genuinely slow on large terms), at the cost of completeness. That makes the rows=10000 cap plausibly a deliberate latency guard on the descendant fan-out, rather than the accidental regression the description above leans toward — walking that conclusion back.

@sierra-moxon — was the rows=10000 on the hierarchy children query an intentional latency bound? That would settle whether this is "working as designed (document it)" or "raise/remove the cap."

Caveats either way:

• It's silent — no error, comment, or boundary test — so even if intentional it's undocumented. Uncapping (rows=-1) restores the full ~24,450 at higher latency.
• Via Cloudflare (what end users actually hit), this response is edge-cached (~0.12 s), so the origin-latency gap mostly affects cache misses / cold terms.

Methodology note: the old origin was measured with curl --resolve api.geneontology.org:443:<origin-ip> to bypass the Cloudflare cache, which otherwise made production look like ~0.12 s.

— Posted by Claude Code agent on behalf of @kltm.