Skip to content

Steve-Allison/isanlp_rst

 
 

Repository files navigation

Python License Apple Silicon

IsaNLP RST Parser

End-to-end Rhetorical Structure Theory (RST) parser. Predicts discourse trees from raw text or pre-segmented EDUs across 11 languages via the unirst multilingual model, plus three monolingual / bilingual models (rstdt, gumrrg, rstreebank). Pixi-managed, MPS-aware, with real tests and CI.

Table of contents

Performance

The parser achieves strong end-to-end performance across standard RST corpora.

Supported languages (unirst): English (eng), Czech (ces), German (deu), Basque (eus), Persian (fas), French (fra), Dutch (nld), Brazilian Portuguese (por), Russian (rus), Spanish (spa), Chinese (zho).

Click to view detailed end-to-end performance metrics
Tag / Version Languages Train Data Test Data Seg S N R Full
rstdt eng eng.rst.rstdt eng.rst.rstdt 97.8 75.6 65.0 55.6 53.9
gumrrg eng, rus eng.erst.gum, rus.rst.rrg eng.erst.gum 95.5 67.4 56.2 49.6 48.7
rus.rst.rrg 97.0 67.1 54.6 46.5 45.4
rstreebank rus rus.rrt rus.rst.rrt 92.1 66.2 53.1 46.1 46.2
unirst all all ces.rst.crdt 94.5 59.1 41.2 28.6 28.0
deu.rst.pcc 96.5 67.3 47.4 34.1 32.1
eng.erst.gum 95.3 67.3 55.6 48.5 47.4
eng.rst.oll 92.5 55.7 39.0 27.5 26.3
eng.rst.rstdt 98.1 76.7 65.5 55.2 53.6
eng.rst.sts 91.2 43.3 31.3 19.4 18.7
eng.rst.umuc 88.8 52.6 40.6 26.2 25.8
eus.rst.ert 92.5 66.0 50.3 34.9 34.7
fas.rst.prstc 94.7 63.0 50.2 40.8 40.7
fra.sdrt.annodis 91.3 58.6 48.9 30.6 30.3
nld.rst.nldt 98.0 61.8 49.8 36.8 35.8
por.rst.cstn 93.9 68.4 52.8 44.9 44.5
rus.rst.rrg 96.4 67.4 54.0 46.3 45.1
rus.rst.rrt 90.7 63.0 49.0 42.3 42.2
spa.rst.rststb 93.4 63.5 50.3 36.0 36.0
spa.rst.sctb 85.5 55.1 46.8 39.1 39.1
zho.rst.gcdt 93.0 64.5 50.7 45.9 44.6
zho.rst.sctb 95.4 67.5 51.5 39.9 39.9

Full per-corpus UniRST metrics: UniRST_Metrics.md.

Installation & quick start

1. Install

The recommended path is pixi (provisions Python + all dependencies + the iinemo/isanlp runtime into a locked env):

git clone https://github.com/Steve-Allison/isanlp_rst.git
cd isanlp_rst
pixi install

Alternative (raw venv / pip):

pip install git+https://github.com/iinemo/isanlp.git    # required runtime dep
pip install git+https://github.com/Steve-Allison/isanlp_rst.git

# The format-native entry points (parse_docling / parse_doclang /
# parse_markdown) need the optional `formats` extra; the core RST Parser does not:
pip install "isanlp_rst[formats] @ git+https://github.com/Steve-Allison/isanlp_rst.git"

The pydantic extra adds the typed RstNode tree model (isanlp_rst.utils.serialization_pydantic); the dependency-free tree_to_dict / tree_from_dict helpers need nothing:

pip install "isanlp_rst[pydantic] @ git+https://github.com/Steve-Allison/isanlp_rst.git"

pixi install already includes both extras, so the pixi path needs nothing extra.

2. Basic usage

from isanlp_rst.parser import Parser

# Choose a model version
version = 'gumrrg'  # one of: 'gumrrg', 'rstdt', 'rstreebank', 'rrtrrg', 'unirst'

# Initialise the parser (downloads weights from HF on first call)
parser = Parser(hf_model_name='tchewik/isanlp_rst_v3',
                hf_model_version=version,
                device='auto')  # 'auto' (default) | 'cpu' | 'mps' | 'cuda' | 'cuda:N'

text = """
On Saturday, in the ninth edition of the T20 Men's Cricket World Cup, Team India won against South Africa by seven runs.
The final match was played at the Kensington Oval Stadium in Barbados. This marks India's second win in the T20 World Cup,
which was co-hosted by the West Indies and the USA between June 2 and June 29.

After winning the toss, India decided to bat first and scored 176 runs for the loss of seven wickets.
Virat Kohli top-scored with 76 runs, followed by Axar Patel with 47 runs. Hardik Pandya took three wickets,
and Jasprit Bumrah took two wickets.
"""

res = parser(text)  # res['rst'] contains the binary discourse tree
print(vars(res['rst'][0]))

For the multilingual unirst model, specify the relation inventory:

parser = Parser(hf_model_name='tchewik/isanlp_rst_v3',
                hf_model_version='unirst',
                device='auto',
                relinventory='eng.erst.gum')  # see UniRST_Metrics.md for options

Loading from a local checkpoint

For offline / air-gapped use, point Parser at a directory containing the checkpoint:

# Family auto-detected:
#   data_manager_*.pickle or config.json with `data.corpora`  -> UniRST
#   relation_table.txt                                        -> DMRST
parser = Parser(model_dir='/path/to/checkpoint', device='auto')

# Override auto-detection:
parser = Parser(model_dir='/path/to/checkpoint', family='dmrst', device='auto')

Device selection (device=)

device= chooses the compute backend (default "auto"):

  • "auto" (default) → CUDA if present, else MPS on Apple Silicon, else CPU
  • "cpu" → CPU
  • "mps" → Apple Silicon Metal backend (raises if unavailable)
  • "cuda" / "cuda:N" → a specific NVIDIA device (raises if no CUDA)

The integer cuda_device= parameter is deprecated but still accepted (-1 → CPU, >= 0 → best available accelerator); passing it emits a DeprecationWarning. Migrate to device=.

PyTorch has no MPS kernel for torch.linalg.qr (used by torch.nn.init.orthogonal_ during weight init). The parser routes this via CPU automatically — no env-var hacks required.

Mixed precision (dtype=)

Forward passes go through torch.autocast, so the model runs in float32, float16, or bfloat16 without changing the trained weights:

import torch
parser = Parser(hf_model_version='gumrrg', device='auto',
                dtype=torch.bfloat16)   # also accepts 'bf16', 'fp16', 'fp32'

Default is float32 on every device. On Apple Silicon (M-series, PyTorch 2.11) at ~1k-char inputs, float32 beats bfloat16 / float16 for every published model — per-op autocast dispatch overhead dominates the matmul speedup at this scale. On large-batch CUDA workloads with native bf16 (Hopper / Ada Tensor Cores), bfloat16 is likely faster — measure with pixi run bench before pinning a choice.

Tree topology and EDU segmentation are bit-equivalent across all three dtypes for every published model; relation labels are not — on a near-tied node, reduced precision (bf16/fp16) can flip the argmax without any structural change. See tests/test_integration.py for the equivalence suite.

Apple Silicon perf (M-series, PyTorch 2.11, ~1k char input)
Model CPU fp32 MPS fp32 MPS bf16 MPS fp16
gumrrg 143 ms 120 ms 156 ms 157 ms
rstdt 168 ms 123 ms 161 ms 165 ms
rstreebank 113 ms 59 ms 94 ms 94 ms
rrtrrg 118 ms 61 ms 104 ms 95 ms
unirst 127 ms 153 ms 218 ms 221 ms

The 18-corpus unirst model is faster on CPU than on MPS — multi-corpus classifier dispatch costs more than MPS's matmul speedup recovers. Run pixi run bench --version unirst on your hardware to verify before pinning a device choice for that model.

Verifying on NVIDIA CUDA hardware

CI runs on macOS Apple Silicon, so the CUDA dispatch path isn't exercised in CI. To verify on an NVIDIA host:

pixi run cuda-smoke

The script confirms torch.cuda.is_available(), loads DMRST and UniRST on cuda:0, parses a sample text, and round-trips a parse_from_edus call. Exits non-zero on any failure.

3. Understanding the output

The parser returns an RST tree with a recursive DiscourseUnit structure. Each node carries:

{
 'id': 21,
 'left':  (id=14, start=1,   end=323),  # child node refs
 'right': (id=20, start=324, end=570),
 'relation':    'elaboration',           # rhetorical relation
 'nuclearity':  'NS',                    # NS / NN / ""
 'entropy':     0.92,                    # split entropy
 'start':       1,                       # original-text character offset
 'end':         570,
 'text':        "On Saturday, ... took two wickets."
}

Leaves are EDUs; internal nodes are relations. Every node has start / end in original-text character coordinates.


Visualising the RST tree

1. Save to RS3

res['rst'][0].to_rs3('filename.rs3')

Open filename.rs3 in external tools like RSTTool or rstWeb for editing.

2. Inline render (Jupyter / Colab)

import io, contextlib
import isanlp_rst

buf = io.StringIO()
with contextlib.redirect_stdout(buf):
    isanlp_rst.render("filename.rs3")

# For Google Colab, sync the cell height:
# isanlp_rst.render("filename.rs3", colab=True)

Illustration of the parsing visualisation

3. Export to PNG or PDF

Requires Playwright:

pip install playwright
playwright install chromium
import isanlp_rst

isanlp_rst.to_png("filename.rs3", "filename.png")
isanlp_rst.to_pdf("filename.rs3", "filename.pdf")

Illustration of English parsing


Advanced usage

Parsing pre-segmented EDUs

my_edus = [
    "On Saturday, Team India won against South Africa.",
    "The final match was played in Barbados."
]

res = parser.from_edus(my_edus)

Memory management for large datasets

When parsing many documents, the resulting DiscourseUnit trees can consume significant memory — each node stores its corresponding text span.

res['rst'][0].clear_textfields()   # drop .text on every node, keep structure
# ... pickle / store ...
res['rst'][0].fill_textfields(full_text)   # repopulate later

Note: .to_rs3() on a tree with cleared text fields will fail.


Docling-native output

For structured documents — PDFs, PPTX decks, VTT transcripts, Markdown, HTML — converted via Docling, isanlp_rst exposes parse_docling(): a single entry point that walks the canonical Docling structure, harvests the text in source-format-appropriate order, runs RST once over the full harvest, and returns the relations indexed by Docling self_ref with per-relation boundary annotations.

from pathlib import Path
from isanlp_rst.docling import parse_docling

result = parse_docling(Path("deck.docling.json"), device="auto")

# result.relations: tuple of RstRelation — flat list with left_id / right_id for tree shape
# result.edus:      tuple of RstEdu     — leaves indexed by Docling self_ref
# result.boundaries: tuple of Boundary  — slides, sections, speaker turns, tables
# result.source_origin: dict            — mirror of doc.origin (mimetype, binary_hash, filename)

Batch use — inject one Parser

The underlying weights are ~2 GB. Constructing a fresh Parser per call would reload them every time. Build one parser, inject it:

from isanlp_rst.parser import Parser
from isanlp_rst.docling import parse_docling

parser = Parser(hf_model_version="gumrrg", device="auto")

for path in document_paths:
    result = parse_docling(path, parser=parser)
    ...

What enters the harvest

Source format Harvested by default Boundary detection
PPTX slide texts (titles, paragraphs, list items); speaker notes (ContentLayer.NOTES); picture VLM descriptions from meta.description.text where present one slide-N boundary per slide group; one slide-N-notes when the slide carries notes
PDF body texts, section headers, list items, picture-children (OCR / chart labels) section-N opened at each section_header; any pre-header content lives in a leading document boundary
HTML / Markdown same as PDF same as PDF
VTT every transcript line one turn-N per contiguous-same-voice run (coalesce_speaker_turns=True default)
Tables (any format) analysed two-level: cells are excluded from the main harvest; each TableItem gets its own RST mini-parse in result.table_analyses. Cell self_refs are real JSON pointers (#/tables/N/data/table_cells/M) that resolve against the source. Knob off via include_table_cells=False. one table-N boundary per TableItem; self_refs is (#/tables/N, <cell pointers>). The synthetic #/tables/N marker carries no harvest span so it cannot land in relation refs.

Toggle the harvest with keyword arguments: include_picture_descriptions=False, include_slide_notes=False, include_furniture=True, include_table_cells=False. See the walkthrough for the full set of knobs and a tree-reconstruction example.

Caveats — read before relying on it

  • RST was trained on prose. Quality is highest on continuous narrative (VTT transcripts, decks with VLM descriptions, prose-heavy PDFs). On bullet-only slide decks without VLM enrichment, or on highly procedural / list-heavy PDFs, relations are dominated by joint / organization chains — structurally valid but rhetorically thin. Use the result with appropriate scepticism on non-prose input.
  • Long inputs: parse_docling raises InputTooLargeError above max_harvest_chars=200_000 (configurable). The largest fixture exercised end-to-end is ~18 KB; the parser handles 40 KB+ cleanly in smoke tests but degradation at extreme sizes is empirical.
  • Cross-boundary relations are preserved. A relation may touch two slides or sections; boundary_memberships lists all touched boundaries. Filter on len(boundary_memberships) == 1 for within-boundary relations only.

DocLang-native output

For documents authored in the DocLang 0.5 XML format — the AI-native document standard from IBM, ABBYY, RedHat, HumanSignal, NVIDIA, and Forgis — isanlp_rst exposes parse_doclang() alongside parse_docling. Both are first-class entry points with honestly different schemas; neither coerces into the other.

from pathlib import Path
from isanlp_rst.doclang import parse_doclang

result = parse_doclang(Path("document.dclg.xml"), device="auto")

# result.relations:   tuple of RstRelation, indexed by local-name XPath
# result.edus:        tuple of RstEdu      with the same xpath addressing
# result.boundaries:  tuple of Boundary    — headings, pages, groups, tables, field_regions
# result.source_origin: {"format": "doclang", "namespace": ..., "version": "0.5", "head_children": [...]}

How addresses work

Every element is addressed by a local-name canonical XPath of the form /doclang[1]/heading[2]/text[1] — 1-based position predicates per local name, namespaces stripped. The same document with or without xmlns="https://www.doclang.ai/ns/v0" produces identical paths. (lxml.etree.ElementTree.getpath() is not used; it emits /*/*[N] wildcards on default-namespaced documents.) Paths round-trip 100% against the upstream 40-fixture valid corpus.

What enters the DocLang harvest

Element Default Notes
<text>, <heading>, <footnote> harvested one span per element; <heading> also opens a heading-N boundary
<list> items harvested per <ldiv/> marker item text aggregates marker.tail + intervening sibling tails
<picture><caption> harvested include_picture_captions=False to skip
<code> skipped source code is not prose — include_code_blocks=True to opt in
<formula> skipped raw LaTeX is not prose — include_formulas=True to opt in
<table> cells analysed two-level: excluded from the main harvest; each table gets its own RST mini-parse in result.table_analyses, cells addressed by marker xpath with row/col grid positions one table-N boundary per table; xpaths is (table_xpath, <cell marker xpaths>). include_table_cells=False skips the analyses. <index> and <tabular> remain boundary-only.
<field_region> body skipped include_field_regions=True to opt in
<page_header> / <page_footer> skipped include_furniture=True to include
<layer value="background"> items skipped include_background=True to include
<layer value="furniture"> items skipped covered by include_furniture=True

Toggle each via the corresponding keyword. See the walkthrough for the full set, a tree-reconstruction example, and <thread> handling for cross-fragment continuation.

Boundary kinds

DocLang doesn't model slides or speaker turns — the boundary set reflects what the spec does model: heading-N, page-N (between <page_break/> markers), group-N (with group-N-M for one level of nesting), table-N, field_region-N, and a document fallback. If your input is PPTX or VTT, use parse_docling on the Docling JSON form — those formats give you slide-N / slide-N-notes / turn-N instead.

Validation

parse_doclang(..., validate_xml=True) (default) gates the file through the doclang PyPI package's validate(path) before parsing. The doclang package is validator-only (no DOM) — we parse with lxml ourselves. If doclang is not importable in your env, validation is silently skipped.

Caveats

  • RST was trained on prose. Same caveat as parse_docling: bullet-only documents and form-shaped content yield rhetorically thin relations dominated by joint / organization chains.
  • Stable addressing across producers. The local-name XPath is reproducible from the parsed XML alone, but it depends on document order. A producer that reorders elements between runs will produce different addresses for the same logical content. DocLang 0.5 has no stable per-element identifier in the spec.
  • <thread> is continuation, not identity. Two spans sharing thread_id are fragments of one logical paragraph (typically across a <page_break/>). The RST mapper aggregates dedup'd thread ids on each relation as nucleus_thread_ids / satellite_thread_ids.

Markdown-native output

For plain CommonMark / GFM markdown — READMEs, design notes, MkDocs / MyST corpora — isanlp_rst exposes parse_markdown() as a first-class entry point that reads .md directly, with no Docling round-trip. CommonMark parsing is handled in pure Python via markdown-it-py plus the mdit-py-plugins front-matter and GFM table plugins.

from pathlib import Path
from isanlp_rst.markdown import parse_markdown

result = parse_markdown(Path("design-notes.md"), device="auto")

# result.relations:      tuple of RstRelation, indexed by #/blocks/N — the document tree
# result.edus:           tuple of RstEdu      with the same block_ref addressing
# result.table_analyses: tuple of TableAnalysis — one RST mini-parse per table (two-level)
# result.boundaries:     tuple of Boundary    — sections, tables, code blocks, document fallback
# result.source_origin:  {"format": "markdown", "gfm": True, "front_matter": "<raw text>", "front_matter_format": "yaml"}

How block_refs work

Every main-harvest unit gets a sequential #/blocks/N reference in document order — parallel to Docling's #/texts/N. Table cells live in their own namespace, #/tables/T/cells/K (K = grid position, row-major, stable past empty cells). The synthetic #/tables/T marker appears only on the table boundary; it carries no harvest span so it cannot leak into relation refs.

What enters the markdown harvest

The cross-format directive is analyse everything by default — but two-level: prose enters the document tree; each table gets its own RST mini-parse in table_analyses, so table discourse never distorts prose discourse.

Construct Default Notes
Headings (ATX + Setext) harvested level ∈ {1..6}; opens section-N boundary
Paragraphs harvested one span per paragraph
List items harvested one span per item; nested items collapse into outer
Blockquote content harvested include_blockquotes=False gates the whole quoted region (paragraphs, headings, lists, code, HTML, tables). A quoted heading never opens a section.
Fenced + indented code blocks harvested include_code_blocks=False for prose-only; code_block-N boundary still emitted
Raw HTML blocks tags stripped, text harvested include_html=False to skip
GFM tables per-table mini-parse in table_analyses include_table_cells=False to skip; cells carry row/col grid positions
GFM strikethrough wrappers dropped, text kept enabled under gfm=True — no literal ~~ in the harvest
Image alt text flattened into parent paragraph inline content; not its own span
Front-matter (YAML only) stripped from harvest raw text preserved in source_origin.front_matter
Thematic break (---) skipped divider, not prose

Markdown boundary kinds

  • section-N — opened at each heading; level carries 1–6. Quoted headings excluded.
  • table-T — one per table; block_refs is (#/tables/T, <cell refs…>); matches table_analyses[].id.
  • code_block-N — one per code block; one-cell block_refs.
  • document — pre-heading bucket, or fallback when no headings exist.

Markdown caveats

  • RST was trained on prose. Tables of textual content (definitions, comparisons, feature lists) parse reasonably; tables of pure numbers produce noisy, low-value mini-trees. The two-level split keeps that noise out of the document tree either way.
  • No docling runtime dep. The markdown entry path is pure Python — markdown-it-py + mdit-py-plugins only. If you need full Docling fidelity on markdown (heavy layout / OCR pipeline), convert to Docling JSON first and use parse_docling.
  • Front-matter is raw YAML text. No PyYAML / tomllib dependency — the consumer parses source_origin.front_matter if structured access is needed. TOML/JSON front-matter is not supported.

See the walkthrough for tree reconstruction, per-section grouping, table-analysis usage, and the full knob set, and the design plan for the cross-format consistency directive.


Quality diagnostics

pixi run rst-diag <paths> parses any mix of .md / *.docling.json / *.dclg.xml sources (files or directories; one shared model load) and emits per-document proxy metrics — no gold annotations required:

  • joint ratio — share of relations labelled joint / same-unit / organization (high = rhetorically thin chaining)
  • tree skew — max depth ÷ log₂(EDUs) (≫ 1 = degenerate chain)
  • cross-boundary ratio — relations spanning more than one section / slide / turn
  • note ratio — lopsided EDU/span alignment
  • table analyses — per-table mini-parse count

Use it to A/B any harvest-policy change before trusting it. --json for machine-readable output; --model-version, --device, --dtype as in the entry points.

All three entry points also accept cache_dir= — an on-disk result cache keyed by source bytes + model identity + knobs, so batch re-runs skip unchanged documents entirely — and dtype= for mixed-precision overrides.


Project status & licence

This repository is Steve Allison's evolution of the IsaNLP RST Parser. The original RST research code and the trained model weights are by Elena Chistova; the MIT-licensed source code carries her copyright. This repository adds pixi-managed builds, a pytest test suite, GitHub Actions CI, MPS / Apple-Silicon support, mixed-precision dispatch, and ongoing roadmap work (see docs/plans/).

  • Source code: MIT — see LICENSE. Copyright Elena Chistova 2020; Steve Allison contributions also under MIT.
  • Model weights (downloaded from tchewik/isanlp_rst_v3 on HuggingFace): CC BY-NC 4.0 — research and non-commercial use only. See LICENSE_MODELS. Commercial use requires either retraining weights under a permissive licence or replacing the models entirely.

Issues and pull requests: please open them on Steve-Allison/isanlp_rst. For questions about the underlying RST research, see Elena Chistova's papers cited below.


Citation

The published model weights are by Elena Chistova. If you use them in research, please cite:

For rstdt, gumrrg, and rstreebank:

@inproceedings{chistova-2024-bilingual,
 title = "Bilingual Rhetorical Structure Parsing with Large Parallel Annotations",
 author = "Chistova, Elena",
 booktitle = "Findings of the Association for Computational Linguistics ACL 2024",
 month = aug,
 year = "2024",
 address = "Bangkok, Thailand and virtual meeting",
 publisher = "Association for Computational Linguistics",
 url = "https://aclanthology.org/2024.findings-acl.577",
 pages = "9689--9706"
}

For unirst:

@inproceedings{chistova-2025-bridging,
  title = "Bridging Discourse Treebanks with a Unified Rhetorical Structure Parser",
  author = "Chistova, Elena",
  booktitle = "Proceedings of the 6th Workshop on Computational Approaches to Discourse, Context and Document-Level Inferences (CODI 2025)",
  month = nov,
  year = "2025",
  address = "Suzhou, China",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2025.codi-1.17/",
  pages = "197--208"
}

About

RST Discourse Parsers

Resources

License

Stars

0 stars

Watchers

0 watching

Forks

Packages

 
 
 

Contributors

Languages

  • Python 96.0%
  • JavaScript 2.9%
  • Other 1.1%