Skip to content

Commit afeae73

Browse files
skearnesclaude
andauthored
Add natural-language query interface (/ask) (#203)
* Add natural-language query interface (/ask) Translate a chemist's free-text question (e.g. "reactions using benzene as an input with yield greater than 70%") into the existing structured QueryParams via a single forced Claude tool call, resolve compound names to SMILES, and dispatch through the existing index-accelerated search path. The model emits a structured query, never SQL or invented SMILES; name resolution is grounded in PubChem/OPSIN. Backend (ord_interface/api/nl_query.py): - translate(): forced build_query tool call -> NLQuery; RateLimitError -> 429, other anthropic.APIError -> 503, no tool call -> 502. - Async, cached name resolution (SMARTS passthrough, verbatim SMILES, else resolve_name); blocking lookups run in a thread, failures are not cached. - Redis caches (best-effort): identical questions (1h) and name->SMILES (30d), so repeated questions skip the model call and shared compounds skip PubChem. - System prompt lives in nl_query_prompt.md, shipped via package-data. - GET /api/nl_query returns the interpretation and resolved structures alongside results for transparency. Eval: nl_query_eval.py + nl_query_eval_cases.json (10 cases); --search executes against the DB to flag zero-result translations. Translation accuracy 10/10. Frontend: a separate /ask page (MainNLSearch) with an "Interpreted as:" panel, useNLQuery hook, types, route, and nav link. Tests: nl_query_test.py and nl_query_eval_test.py (18 tests, fully stubbed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: translation-only cache, fast-fail Redis, query guards Cache only the model's translation (NLQuery), not the search results: repeated identical questions still skip the model call, but the DB query is always re-run so results stay fresh and Redis entries stay small (Greptile P2). Other review fixes and hardening: - Guard the empty-interpretation case: run_query's "no parameters" ValueError now maps to 422 instead of an unhandled 500 (Greptile P1). - Bound the q parameter with Query(min_length=1, max_length=2000) (Greptile P2). - Resolve components concurrently with asyncio.gather (Greptile P2). - Best-effort Redis ops fast-fail via asyncio.timeout (REDIS_OP_TIMEOUT_SECONDS): a missing/slow Redis degrades to a miss in ~1s instead of stalling each request ~10-20s. Measured against the live DB without Redis: per-compound resolve dropped from ~21s to ~2s. - Bump ord-schema to >=0.7.1 for the new NCI/CADD CIR resolver; resolve_name now cascades PubChem -> CIR -> OPSIN, so a PubChem 503 falls through to CIR (which resolves benzene/aspirin/ibuprofen/palladium acetate that OPSIN cannot). Eval: add a per-phase time breakdown (translate/resolve/search) and a per-search asyncio timeout so a pathologically slow query is reported, not hung. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Move /ask frontend to a follow-up branch The natural-language web UI (the /ask page, useNLQuery hook, types, route, and nav link) is split out to the nl-query-frontend branch so this PR is backend-only (the /api/nl_query endpoint and its translation/resolution/caching/eval). The frontend follow-up depends on this endpoint and lands separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Eval: exact identifier match, YAML cases, SMILES coverage - Match component identifiers exactly (case- and whitespace-insensitive) instead of by substring, so the model must reproduce the specific compound -- e.g. "4-aminophenol", not "aminophenol". This surfaced a real prompt bug: the model emitted "pyridine ring" for "a pyridine ring"; the prompt now instructs it to strip descriptive words ("ring"/"group"/"moiety"/"scaffold") and to pass a user-supplied SMILES/SMARTS through verbatim. - Move eval cases from JSON to YAML (nl_query_eval_cases.yaml); add pyyaml as a dependency and ship *.yaml via package-data. - Add SMILES-identifier cases (CCO, an aspirin SMILES, a C(=O)O substructure). - Reformat the system prompt into sections (Components / Match mode / Filters / Output). Translation accuracy: 13/13 with Haiku. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Eval: infer SUBSTRUCTURE from "contain" without the literal word Drop "substructure" from the C(=O)O case so it reads "products contain C(=O)O"; verifies the model maps "contain" to SUBSTRUCTURE mode on its own. Still 13/13. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Restore /ask frontend into this PR Recombine the natural-language web UI (the /ask page, useNLQuery hook, types, route, and nav link) with the backend so the feature can be reviewed and iterated on as a single full-stack change. Reverses the earlier split to the nl-query-frontend branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Eval: complete the YAML/exact-match switch (load yaml, prompt, pyyaml) These changes belong with the earlier eval commit but were dropped when its git add aborted on the already-removed JSON pathspec, leaving the branch in a broken state (JSON deleted while nl_query_eval still loaded it): - nl_query_eval.py: load cases from YAML; exact (case/whitespace-insensitive) identifier matching via the renamed `identifier` field. - nl_query_eval_test.py: assert exact matching ("aminophenol" != "4-aminophenol"). - nl_query_prompt.md: sectioned format; strip descriptive words from identifiers and pass user SMILES/SMARTS through verbatim. - pyproject.toml / uv.lock: add pyyaml; ship *.yaml via package-data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix component-spec parsing for SMARTS containing ";" The component spec is "pattern;target;mode", but SMARTS uses ";" as a low-precedence AND, so a model- or user-supplied SMARTS like "[#6;R]" broke the str.split(";") unpacking and surfaced as an unhandled 500. Split from the right (rsplit(";", 2)) since target and mode are the trailing fields and never contain ";". Adds a regression test. (Flagged by Greptile on #203; their suggested split(";", 2) would mis-parse a leading-";" SMARTS, so rsplit is the correct form.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Emit JSON component specs; add /ask dev mode, in-dev banner, unlist nav - nl_query: build_query_params now emits JSON ComponentSpec strings (matching the new /query encoding from #206) instead of "smiles;target;mode". - Dev mode: GET /api/nl_query?dry_run=true translates and resolves but skips the search, returning the would-be-executed query_components for inspection. The /ask page gets a URL-persisted "Dry run" toggle that shows the structured query instead of results. - Add an "in development" banner to the /ask page. - Remove the "Ask" link from the nav bar; the /ask route still resolves directly. - Apply ruff format to the eval test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Route compound classes to SMARTS/SUBSTRUCTURE; clarify interpretation UI Functional-group and element classes ("brominated products", "an aryl boronic acid") were translated as exact name lookups and failed in the resolver, which only handles specific named compounds. Teach the prompt to express such classes as a model-authored SMARTS or SUBSTRUCTURE pattern instead, and never send them for resolution. SUBSTRUCTURE fragments are now SMILES the model writes directly (e.g. a pyridine ring -> c1ccncc1). Validate model-authored SMARTS with RDKit in _resolve_component so a bad pattern is a clean 422 with the pattern echoed, rather than a 400 deep in query execution that a dry run would skip entirely. Eval matching is now structure-aware: SMARTS and SMILES identifiers are canonicalized with RDKit so equivalent patterns compare equal, while names RDKit cannot parse keep the case- and whitespace-insensitive string compare. The /ask interpretation box now separates provenance: a "From the model" section (the build_query tool call, with the raw JSON available) and a "Resolved to structures" section that lists only identifiers a network resolver actually handled, headed by the resolvers used. The redundant dry-run query dump is dropped since the box already shows it. Bump the translation cache version to v3 for the prompt change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Map malformed model/cached payloads to 502/miss, not 500 NLQuery.model_validate in translate() and model_validate_json in _translation_cache_get can raise pydantic.ValidationError. Catch it explicitly: a tool payload that fails schema validation becomes a 502 (consistent with the no-tool-call case), and a schema-mismatched cache entry degrades to a miss. ValidationError subclasses ValueError in our Pydantic, so the cache path was already covered, but the translate() path was unwrapped entirely and the explicit catch is robust to Pydantic's inheritance quirk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Validate reaction SMARTS up front; resync /ask input on navigation Validate reaction_smarts with RDKit in build_query_params so a bad pattern is a clear 422 in both normal and dry-run mode, instead of a misleading "no constraints" 422 from run_query (normal) or an unvalidated pass-through (dry run, which skips run_query). ReactionFromSmarts both returns None and raises ValueError for different malformed inputs, so handle both. On the /ask page, sync the text input to the ?q= URL parameter via an effect so browser back/forward navigation updates the visible query rather than leaving a stale value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * eval: fix stale cases filename and cover all numeric filters The harness docstring referenced nl_query_eval_cases.json; the cases are YAML. Add similarity_threshold and limit to the over-extraction check (and to CaseExpectation) so the model is flagged for extracting either without the question asking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Tighten verbose comments in nl_query Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 9e4a6b7 commit afeae73

15 files changed

Lines changed: 2240 additions & 7 deletions

app/src/App.tsx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ import About from './views/About';
2424
import MainBrowse from './views/browse/MainBrowse';
2525
import MainSelectedSet from './views/browse/selected-set/MainSelectedSet';
2626
import MainSearch from './views/search/MainSearch';
27+
import MainNLSearch from './views/nl-search/MainNLSearch';
2728
import MainDatasetView from './views/dataset-view/MainDatasetView';
2829
import MainReactionView from './views/reaction-view/MainReactionView';
2930
import './App.scss';
@@ -55,6 +56,10 @@ const AppContent: React.FC = () => {
5556
path="/search"
5657
element={<MainSearch />}
5758
/>
59+
<Route
60+
path="/ask"
61+
element={<MainNLSearch />}
62+
/>
5863
<Route
5964
path="/dataset/:datasetId"
6065
element={<MainDatasetView />}

app/src/components/HeaderNav.tsx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,8 @@ const HeaderNav: React.FC = () => {
5050
>
5151
Search
5252
</Link>
53+
{/* The /ask route exists but is intentionally unlisted while the
54+
natural-language feature is in development. */}
5355
</div>
5456
<div className="nav-item header-nav__nav-item">
5557
<a

app/src/hooks/useNLQuery.ts

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
/**
2+
* Copyright 2026 Open Reaction Database Project Authors
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
import { useQuery } from '@tanstack/react-query';
18+
import reaction_pb from 'ord-schema';
19+
import { base64ToBytes } from '../utils/base64';
20+
import { fetchJson } from '../utils/api';
21+
import type {
22+
NLInterpretation,
23+
NLQueryResponse,
24+
ResolvedComponent,
25+
SearchResult,
26+
} from '../types/search';
27+
28+
export interface NLQueryData {
29+
interpretation: NLInterpretation;
30+
resolvedComponents: ResolvedComponent[];
31+
queryComponents: string[];
32+
results: SearchResult[];
33+
dryRun: boolean;
34+
}
35+
36+
/**
37+
* Runs a natural-language search against `/api/nl_query`.
38+
*
39+
* Unlike structured search, the backend translates and executes in a single
40+
* synchronous request, so this is a plain fetch rather than the submit/poll
41+
* protocol of {@link useSearchTask}. Result protos are deserialized here the
42+
* same way, and the model's interpretation is surfaced so the page can show the
43+
* user how their question was understood.
44+
*/
45+
export function useNLQuery(query: string | null, enabled: boolean, dryRun = false) {
46+
return useQuery<NLQueryData>({
47+
queryKey: ['nl-query', query, dryRun],
48+
enabled: enabled && query !== null && query.trim() !== '',
49+
retry: false,
50+
staleTime: Infinity,
51+
queryFn: async (): Promise<NLQueryData> => {
52+
const url =
53+
`/api/nl_query?q=${encodeURIComponent(query as string)}` +
54+
(dryRun ? '&dry_run=true' : '');
55+
const raw = await fetchJson<NLQueryResponse>(url, undefined, 'nl_query');
56+
const results: SearchResult[] = raw.results.map(r => ({
57+
...r,
58+
data: reaction_pb.Reaction.deserializeBinary(
59+
new Uint8Array(base64ToBytes(r.proto)),
60+
).toObject(),
61+
}));
62+
return {
63+
interpretation: raw.interpretation,
64+
resolvedComponents: raw.resolved_components,
65+
queryComponents: raw.query_components,
66+
results,
67+
dryRun: raw.dry_run,
68+
};
69+
},
70+
});
71+
}

app/src/types/search.ts

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,3 +49,45 @@ export interface Dataset {
4949
description?: string;
5050
num_reactions: number;
5151
}
52+
53+
// Mirrors ord_interface.api.nl_query.NLComponent.
54+
export interface NLComponent {
55+
identifier: string;
56+
target: 'INPUT' | 'OUTPUT';
57+
mode: 'EXACT' | 'SIMILAR' | 'SUBSTRUCTURE' | 'SMARTS';
58+
}
59+
60+
// Mirrors ord_interface.api.nl_query.NLQuery: how the model understood the
61+
// question. Component-level constraints plus optional numeric/flag filters.
62+
export interface NLInterpretation {
63+
components: NLComponent[];
64+
min_yield?: number | null;
65+
max_yield?: number | null;
66+
min_conversion?: number | null;
67+
max_conversion?: number | null;
68+
reaction_smarts?: string | null;
69+
similarity_threshold?: number | null;
70+
use_stereochemistry?: boolean | null;
71+
limit?: number | null;
72+
}
73+
74+
// Mirrors ord_interface.api.nl_query.ResolvedComponent: a component after its
75+
// name has been resolved to a concrete SMILES (shown for transparency).
76+
export interface ResolvedComponent {
77+
identifier: string;
78+
smiles: string;
79+
resolver: string;
80+
target: 'INPUT' | 'OUTPUT';
81+
mode: 'EXACT' | 'SIMILAR' | 'SUBSTRUCTURE' | 'SMARTS';
82+
}
83+
84+
// Raw /api/nl_query payload, before result protos are deserialized.
85+
export interface NLQueryResponse {
86+
query: string;
87+
interpretation: NLInterpretation;
88+
resolved_components: ResolvedComponent[];
89+
// JSON-encoded ComponentSpec strings that would be executed (shown for dry runs).
90+
query_components: string[];
91+
results: Omit<SearchResult, 'data'>[];
92+
dry_run: boolean;
93+
}
Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
/**
2+
* Copyright 2026 Open Reaction Database Project Authors
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
.nl-search {
18+
max-width: 960px;
19+
margin: 0 auto;
20+
padding: 2rem 1rem;
21+
22+
&__title {
23+
margin-bottom: 0.25rem;
24+
}
25+
26+
&__subtitle {
27+
color: #555;
28+
margin-bottom: 1.5rem;
29+
}
30+
31+
&__form {
32+
display: flex;
33+
gap: 0.5rem;
34+
}
35+
36+
&__input {
37+
flex: 1;
38+
padding: 0.6rem 0.75rem;
39+
font-size: 1rem;
40+
border: 1px solid #ccc;
41+
border-radius: 4px;
42+
}
43+
44+
&__button {
45+
padding: 0.6rem 1.25rem;
46+
font-size: 1rem;
47+
border: none;
48+
border-radius: 4px;
49+
background: #1a73e8;
50+
color: #fff;
51+
cursor: pointer;
52+
53+
&:disabled {
54+
opacity: 0.6;
55+
cursor: default;
56+
}
57+
}
58+
59+
&__examples {
60+
margin-top: 0.75rem;
61+
display: flex;
62+
flex-wrap: wrap;
63+
gap: 0.5rem;
64+
}
65+
66+
&__example {
67+
padding: 0.3rem 0.6rem;
68+
font-size: 0.85rem;
69+
border: 1px solid #d0d7de;
70+
border-radius: 999px;
71+
background: #f6f8fa;
72+
color: #1a73e8;
73+
cursor: pointer;
74+
}
75+
76+
&__interpretation {
77+
margin: 1.5rem 0;
78+
padding: 0.75rem 1rem;
79+
background: #f6f8fa;
80+
border: 1px solid #d0d7de;
81+
border-radius: 6px;
82+
}
83+
84+
&__layer + &__layer {
85+
margin-top: 1rem;
86+
padding-top: 1rem;
87+
border-top: 1px solid #e1e4e8;
88+
}
89+
90+
&__interpretation-title {
91+
font-weight: 600;
92+
margin-bottom: 0.4rem;
93+
}
94+
95+
&__provenance {
96+
font-weight: 400;
97+
font-size: 0.8rem;
98+
color: #777;
99+
}
100+
101+
&__interpretation-list {
102+
margin: 0;
103+
padding-left: 1.2rem;
104+
105+
code {
106+
color: #b5179e;
107+
}
108+
}
109+
110+
&__resolution-list {
111+
margin: 0;
112+
padding: 0;
113+
list-style: none;
114+
115+
li {
116+
display: flex;
117+
align-items: baseline;
118+
gap: 0.5rem;
119+
padding: 0.15rem 0;
120+
}
121+
122+
code {
123+
color: #b5179e;
124+
}
125+
}
126+
127+
&__identifier {
128+
font-weight: 600;
129+
}
130+
131+
&__arrow {
132+
color: #999;
133+
}
134+
135+
&__role {
136+
text-transform: uppercase;
137+
font-size: 0.7rem;
138+
letter-spacing: 0.03em;
139+
color: #777;
140+
}
141+
142+
&__mode,
143+
&__resolver {
144+
color: #777;
145+
font-size: 0.85rem;
146+
}
147+
148+
&__muted {
149+
color: #777;
150+
font-size: 0.9rem;
151+
}
152+
153+
&__raw {
154+
margin-top: 0.5rem;
155+
font-size: 0.85rem;
156+
157+
summary {
158+
cursor: pointer;
159+
color: #1a73e8;
160+
}
161+
}
162+
163+
&__raw-json {
164+
margin: 0.5rem 0 0;
165+
padding: 0.6rem 0.75rem;
166+
background: #fff;
167+
border: 1px solid #d0d7de;
168+
border-radius: 6px;
169+
overflow-x: auto;
170+
font-size: 0.8rem;
171+
}
172+
173+
&__error {
174+
margin: 1.5rem 0;
175+
padding: 0.75rem 1rem;
176+
background: #fde8e8;
177+
border: 1px solid #f5b5b5;
178+
border-radius: 6px;
179+
color: #8a1f1f;
180+
}
181+
182+
&__empty {
183+
margin: 1.5rem 0;
184+
color: #555;
185+
}
186+
187+
&__banner {
188+
margin-bottom: 1rem;
189+
padding: 0.6rem 1rem;
190+
background: #fff8e1;
191+
border: 1px solid #f0d58c;
192+
border-radius: 6px;
193+
color: #6b5400;
194+
font-size: 0.9rem;
195+
}
196+
197+
&__dry-run-toggle {
198+
display: flex;
199+
align-items: center;
200+
gap: 0.5rem;
201+
margin: 0.75rem 0 0;
202+
font-size: 0.9rem;
203+
color: #555;
204+
cursor: pointer;
205+
}
206+
207+
&__dry-run {
208+
margin: 1.5rem 0;
209+
}
210+
211+
&__dry-run-title {
212+
font-weight: 600;
213+
color: #555;
214+
}
215+
}

0 commit comments

Comments
 (0)