feat: better describer and human describer

[r-fedorov]

Summary

This PR improves BluNamer’s molecule description and name-token metadata layers. It adds richer, metadata-backed descriptions, introduces a new human-oriented descriptor API, and tightens graph-local token bindings for complex substituents, N-substituent locants, unsaturations, and heteroatom shortcut names.

What changed

• Added DescriptionTokenSummary to summarize final-name token binding metadata by kind, ownership, confidence, and source.

• Expanded describe() output with:

  • token summaries
  • token spans
  • substituent tree rendering
  • ordered trace segment explanations
  • clearer parent, unsaturation, substituent, and rule descriptions

• Added new public describe_human() API and HumanDescription model for compact, chemistry-facing descriptions that avoid token-span/debug terminology.

• Exported the new descriptor types and functions from bluenamer.__init__.

• Added explicit graph-local handling for N-substituent locants so tokens like N bind to the relevant nitrogen atom instead of broad substituent scope.

• Improved token span alignment for:

  • parenthesized unsaturation locants such as 1(6)
  • repeated grouped locants
  • recursive substituent trees
  • heteroatom-centered shortcut substituents
  • oxygen-carbonyl shortcut substituents such as carbonyloxy
  • parent/suffix morphology gaps
  • alkenyl bridge stereo and unsaturation tokens

• Added nested substituent-tree metadata for heteroatom and oxygen-carbonyl shortcut branches.

• Improved substituent renderer token metadata so complex fragments preserve local atom and bond ownership instead of falling back to broad scopes.

Why

The previous describer was useful for high-level naming explanations, but it did not expose enough structured metadata for auditing token-to-graph alignment or for explaining nested substituents clearly. Complex names with recursive substituents, heteroatom shortcut prefixes, and repeated locants could also assign overly broad graph scopes to individual name tokens.

This PR makes the description layer more transparent and makes token bindings more local and auditable.

Testing

Added and updated tests covering:

• token binding summaries in describe()
• nested substituent tree rendering
• ordered trace segment output
• parenthesized unsaturation locants
• heteroatom shortcut ligand trees
• oxygen-carbonyl shortcut substituent trees
• human-oriented descriptions from describe_human()
• N-substituent locant binding
• local parent/suffix morphology scope
• recursive substituent tree token scopes
• repeated grouped locant disambiguation
• local token scopes for hydroxyphenyl, alkenyl, aryl, carbonyl, and oxy fragments
• regression coverage for carbonylamino shortcut trees avoiding incorrect hydroxy children

Summary by Sourcery

Improve name-token to graph binding metadata, add human-readable describer, and refine handling of complex substituents and locants.

New Features:

• Introduce a human-oriented description API and model that explains names from substituent-tree metadata without exposing token-span internals.
• Expose detailed token binding summaries, spans, and substituent trees from the core describer and CLI in an opt-in debug mode.
• Add structured nested substituent trees for heteroatom and oxygen–carbonyl shortcut branches, including ligand and carbonyl core nodes.

Bug Fixes:

• Tighten token binding for N-substituent locants so N tokens bind to their specific nitrogen and local bonds instead of broad substituent scopes.
• Fix binding of parenthesized unsaturation locants and repeated grouped locants so locant tokens map to the correct double-bond endpoints and ligand atoms.
• Prevent shortcut trees such as carbonylamino from inventing spurious hydroxy children in the substituent tree.

Enhancements:

• Refine tokenization and context-aware placement of renderer tokens, including ordered emission, match priorities, and context guards for locants and stereo tokens.
• Improve recursive substituent token metadata so complex fragments reuse tree-local scopes rather than broad substituent-level bindings.
• Extend shortcut parent naming logic to return structured binding roles for homonuclear chains, central parent hydrides, and biphenyl parents to support better auditing.
• Enhance the describer output with substituent-tree rendering, ordered trace segment explanations, and clearer parent, unsaturation, substituent, and rule narratives.

Documentation:

• Update README to document the new human-like description API and demonstrate its usage.

Tests:

• Add extensive tests covering token grammar utilities, ordered renderer token placement, N-substituent locants, unsaturation locants, recursive substituent scopes, and shortcut parent bindings.
• Add integration tests for the new human-oriented descriptor and for debug token summaries in CLI and library describe flows.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: 94ff6dcf-3f8d-4392-b54c-04519cd6944a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Reviewer's Guide

Refines BluNamer’s description and token-binding layers by introducing richer token metadata (including summaries and spans), a new human-oriented descriptor API, and more local, graph-accurate handling of complex substituents, locants, and shortcut parents, while also adding metadata-free fast paths for production naming.

Sequence diagram for the new describe_human API

sequenceDiagram
    actor User
    participant HumanDescriptor as human_descriptor
    participant NamingEngine as DEFAULT_NAMING_ENGINE
    participant Result as NamingResult

    User->>HumanDescriptor: describe_human(smiles)
    HumanDescriptor->>NamingEngine: run(NamingRequest(smiles, include_trace=True))
    NamingEngine-->>HumanDescriptor: NamingResult
    HumanDescriptor->>Result: access result.substituent_tree
    HumanDescriptor->>Result: access result.name
    HumanDescriptor-->>User: HumanDescription

Loading

File-Level Changes

Change	Details	Files
Improve name token span construction, rewriting, and locant/unsaturation binding to ensure local, auditable graph metadata for final-name tokens.	Adjust rewrite pipeline to rebuild spans only when text changes and to split replacement text into lexical tokens via a shared scanner. Introduce per-replacement-token copying, token-context matching, and ordered renderer token placement with render_order and match_priority fields. Add specialized unsaturation span handling for parenthesized locants and enforce standalone/grammar-aware locant detection. Group and merge identical-span tokens, collapsing metadata (kind, ownership, confidence, source) and preferring higher-priority matches. Disambiguate repeated and element locant clusters by governing structural scope, suffix context, and N-substituent preference, pruning broad or redundant helper matches. Treat punctuation-only gaps and parent/suffix morphology gaps as explicit grammar tokens with scoped parent/suffix bindings.	src/bluenamer/name_assembly.py src/bluenamer/token_grammar.py
Enrich name-atom binding metadata and emitted tokens, especially for unsaturations, charges, recursive substituents, and shortcut parents, while sharing lexical tokenization across components.	Add render_order, match_priority, left_context, and right_context to NameTokenBinding and plumb them through postprocessing and binding trace serialization. Emit structured unsaturation locant and unsaturation tokens based on parent locant/bond maps, and avoid charge-suffix fallback bindings. Use shared token_grammar helpers for binding term tokenization and locant predicates, and add multiplier + oxy tokens for multi-locanted hydroxy-like substituents. Derive recursive substituent-tree tokens (including parent locants, substituent suffixes, hydro operations) to keep nested fragment scopes local instead of broad substituent scopes. Ensure special replacement parents (homonuclear chains, central hydrides, biphenyl, sulfoxides) yield role-specific bindings, not a single broad fallback binding.	src/bluenamer/name_bindings.py src/bluenamer/substituent_tokens.py src/bluenamer/special_cases.py src/bluenamer/assembly_parts.py src/bluenamer/trace_helpers.py src/bluenamer/tests/test_analysis.py
Introduce richer description APIs and models, separating debug-oriented descriptions from human-facing explanations and exposing structured token and substituent-tree metadata.	Extend Description with token summaries, token spans, substituent tree, and a debugging_tokens flag; gate token output in to_dict() and CLI by a debug flag. Implement token and trace segment collectors that dedupe tokens from decisions and trees, summarize binding statistics, and render structured prose for components, trees, and trace segments. Add a new HumanDescription model and describe_human() API that explains naming using substituent_tree and graph metadata without relying on token spans, and expose them from the package init. Update the CLI describe command to accept --debug-tokens, wiring through to describe() and JSON output. Add integration tests for describe()/describe_human(), including token-debug behavior, tree rendering, and explanation ordering.	src/bluenamer/describer.py src/bluenamer/human_descriptor.py src/bluenamer/cli.py src/bluenamer/__init__.py tests/integration/test_describer.py tests/integration/test_human_descriptor.py
Refine recursive substituent, heteroatom shortcut, and oxygen–carbonyl shortcut handling so trees and tokens stay graph-local and reuse metadata selectively, with optional metadata-free fast paths.	Allow subgraph and component branch naming to run in a metadata-free mode (emit_metadata flag) that skips traces, trees, tokens, and final assembly auditing when not needed. Adjust name_subgraph and collect_subgraph_substituents/component_branch_substituents to respect emit_metadata, only building DecisionTrace, substituent_tree, trace_segments, and graph_bound_substituent_tokens when requested. Build nested substituent-tree children for heteroatom-centered shortcuts and oxygen-carbonyl (carbonyloxy/carbonothioyloxy) fragments, including ligand fragments, carbonyl cores, and terminal hydroxy/halo/amino substituents with locants. Generate heteroatom and oxygen-carbonyl shortcut NameTokenBindings that separate center, ligands, alkenyl bridges, aryl branches, stereo, and locant scopes, including contextual match priorities and left/right contexts. Add tests to ensure heteroatom shortcuts, carbonylamino trees, and hydroxyphenyl/ethenyl/aryl branches do not invent spurious hydroxy children and keep local atom/bond ownership.	src/bluenamer/namer.py src/bluenamer/component_namer.py src/bluenamer/substituent_tokens.py src/bluenamer/tests/test_analysis.py tests/integration/test_describer.py
Tighten N-substituent locant handling and parent/suffix morphology gaps so that N tokens and parent fragments bind to the correct local atoms and bonds.	Inject explicit N-substituent locant tokens for component N-substituents, binding them to the nitrogen atom and branch bonds, and preserving them through postprocessing. Prefer N-substituent locant bindings in locant disambiguation logic, giving them their own binding_key and source. Introduce parent_suffix_morphology_gap resolution that binds gap tokens (e.g., acet in acetamide) to combined parent + suffix scope while avoiding prefix scope bleed-through. Update tests to assert that N tokens bind to nitrogen atoms and relevant bonds, and that parent/suffix gap tokens have local, not broad, scopes.	src/bluenamer/component_modifiers.py src/bluenamer/name_assembly.py src/bluenamer/tests/test_analysis.py
Miscellaneous infrastructure and test updates to support new behavior and keep round-tripping stable.	Expose shared lexical token grammar utilities (lexical_token_spans, binding_term_tokens, is_locant_token, locant_tokens_in_text, is_locant_binding_token) and update callers to use them instead of local regexes. Relax assertions that every token has binding_indices by exempting grammar tokens, and adjust expectations for oxazolidine spelling changes and biphenyl role names. Route py2opsin roundtrip tests through a temporary-file path to avoid potential state issues in OPSIN bindings. Document the new describe_human() API in the README with a usage example and sample output.	src/bluenamer/token_grammar.py src/bluenamer/tests/test_analysis.py src/bluenamer/tests_roundtrip/roundtrip_helpers.py README.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've left some high level feedback:

• There are now multiple _is_locant_like_token / _is_locant_search_token-style helpers with subtly different regexes across modules (e.g. in name_bindings and substituent_tokens); consider centralizing the locant token definition in a shared helper to avoid divergence and make future changes safer.
• Some of the new decision/heuristic functions (for example _disambiguate_locant_exact_matches, _token_spans_from_native_matches, and _tree_substituent_emitted_tokens) have grown quite long and multi-purpose; consider splitting them into smaller, single-responsibility helpers to make the naming and token-binding rules easier to understand and maintain.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- There are now multiple `_is_locant_like_token` / `_is_locant_search_token`-style helpers with subtly different regexes across modules (e.g. in `name_bindings` and `substituent_tokens`); consider centralizing the locant token definition in a shared helper to avoid divergence and make future changes safer.
- Some of the new decision/heuristic functions (for example `_disambiguate_locant_exact_matches`, `_token_spans_from_native_matches`, and `_tree_substituent_emitted_tokens`) have grown quite long and multi-purpose; consider splitting them into smaller, single-responsibility helpers to make the naming and token-binding rules easier to understand and maintain.

---

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[r-fedorov]

@sourcery-ai review

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• Several of the new helpers in name_assembly (e.g. _disambiguate_locant_exact_matches, _filter_locant_matches_by_governing_scope, _should_split_broad_exact_match) have grown quite long and tightly interdependent; consider extracting a small locant-binding/trace-scoping helper class or submodule to group this logic and make it easier to reason about and evolve.
• Functions like _visible_ligand_substituent_locants and others that build regexes inside loops recompile patterns each call; if these are hit frequently in naming hot paths, it would be worth hoisting the compiled regexes to module level to avoid repeated compilation overhead.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- Several of the new helpers in `name_assembly` (e.g. `_disambiguate_locant_exact_matches`, `_filter_locant_matches_by_governing_scope`, `_should_split_broad_exact_match`) have grown quite long and tightly interdependent; consider extracting a small locant-binding/trace-scoping helper class or submodule to group this logic and make it easier to reason about and evolve.
- Functions like `_visible_ligand_substituent_locants` and others that build regexes inside loops recompile patterns each call; if these are hit frequently in naming hot paths, it would be worth hoisting the compiled regexes to module level to avoid repeated compilation overhead.

## Individual Comments

### Comment 1
<location path="src/bluenamer/special_cases.py" line_range="173-182" />
<code_context>
+            continue
+        tokens.insert(
+            0,
+            NameTokenBinding(
+                text=locant,
+                token_kind="locant",
</code_context>
<issue_to_address>
**issue (bug_risk):** The second biphenyl attachment token’s text does not match its locant, which may confuse downstream matching.

Here the biphenyl parent’s second attachment token ends up as:

```python
NameTokenBinding(
    text="1",
    token_kind="locant",
    ...
    locants=("1'",),
    render_order=1,
)
```

That means the visible text (`"1"`) and the logical locant (`"1'"`) disagree. Because native-token matching elsewhere relies on `token.text`, the `1'` position may be treated as `1`, or not found if the name contains `"1'"`. Please make `text` match the locant (e.g. `text="1'"`) so the rendered token and locant metadata stay consistent.
</issue_to_address>

---

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

issue (bug_risk): The second biphenyl attachment token’s text does not match its locant, which may confuse downstream matching.

Here the biphenyl parent’s second attachment token ends up as:

NameTokenBinding(
    text="1",
    token_kind="locant",
    ...
    locants=("1'",),
    render_order=1,
)

That means the visible text ("1") and the logical locant ("1'") disagree. Because native-token matching elsewhere relies on token.text, the 1' position may be treated as 1, or not found if the name contains "1'". Please make text match the locant (e.g. text="1'") so the rendered token and locant metadata stay consistent.

[r-fedorov]

@sourcery-ai review

[sourcery-ai]

Hey - I've found 1 issue

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="README.md" line_range="170" />
<code_context>
+
+## Human-like description
+
+OpenBlue can generate uncanny human-like description.
+```python
+
</code_context>
<issue_to_address>
**suggestion (typo):** Consider using the plural "descriptions" for grammatical agreement.

Suggested wording: "OpenBlue can generate uncanny human-like descriptions."

```suggestion
OpenBlue can generate uncanny human-like descriptions.
```
</issue_to_address>

---

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}