End-to-end Rhetorical Structure Theory (RST) parser. Predicts discourse trees from raw text or pre-segmented EDUs across 11 languages via the unirst multilingual model, plus three monolingual / bilingual models (rstdt, gumrrg, rstreebank). Pixi-managed, MPS-aware, with real tests and CI.
- Performance
- Installation & quick start
- Visualising the RST tree
- Advanced usage
- Docling-native output
- DocLang-native output
- Markdown-native output
- Quality diagnostics
- Project status & licence
- Citation
The parser achieves strong end-to-end performance across standard RST corpora.
Supported languages (unirst): English (eng), Czech (ces), German (deu), Basque (eus), Persian (fas), French (fra), Dutch (nld), Brazilian Portuguese (por), Russian (rus), Spanish (spa), Chinese (zho).
Click to view detailed end-to-end performance metrics
| Tag / Version | Languages | Train Data | Test Data | Seg | S | N | R | Full |
|---|---|---|---|---|---|---|---|---|
rstdt |
eng | eng.rst.rstdt | eng.rst.rstdt | 97.8 | 75.6 | 65.0 | 55.6 | 53.9 |
gumrrg |
eng, rus | eng.erst.gum, rus.rst.rrg | eng.erst.gum | 95.5 | 67.4 | 56.2 | 49.6 | 48.7 |
| rus.rst.rrg | 97.0 | 67.1 | 54.6 | 46.5 | 45.4 | |||
rstreebank |
rus | rus.rrt | rus.rst.rrt | 92.1 | 66.2 | 53.1 | 46.1 | 46.2 |
unirst |
all | all | ces.rst.crdt | 94.5 | 59.1 | 41.2 | 28.6 | 28.0 |
| deu.rst.pcc | 96.5 | 67.3 | 47.4 | 34.1 | 32.1 | |||
| eng.erst.gum | 95.3 | 67.3 | 55.6 | 48.5 | 47.4 | |||
| eng.rst.oll | 92.5 | 55.7 | 39.0 | 27.5 | 26.3 | |||
| eng.rst.rstdt | 98.1 | 76.7 | 65.5 | 55.2 | 53.6 | |||
| eng.rst.sts | 91.2 | 43.3 | 31.3 | 19.4 | 18.7 | |||
| eng.rst.umuc | 88.8 | 52.6 | 40.6 | 26.2 | 25.8 | |||
| eus.rst.ert | 92.5 | 66.0 | 50.3 | 34.9 | 34.7 | |||
| fas.rst.prstc | 94.7 | 63.0 | 50.2 | 40.8 | 40.7 | |||
| fra.sdrt.annodis | 91.3 | 58.6 | 48.9 | 30.6 | 30.3 | |||
| nld.rst.nldt | 98.0 | 61.8 | 49.8 | 36.8 | 35.8 | |||
| por.rst.cstn | 93.9 | 68.4 | 52.8 | 44.9 | 44.5 | |||
| rus.rst.rrg | 96.4 | 67.4 | 54.0 | 46.3 | 45.1 | |||
| rus.rst.rrt | 90.7 | 63.0 | 49.0 | 42.3 | 42.2 | |||
| spa.rst.rststb | 93.4 | 63.5 | 50.3 | 36.0 | 36.0 | |||
| spa.rst.sctb | 85.5 | 55.1 | 46.8 | 39.1 | 39.1 | |||
| zho.rst.gcdt | 93.0 | 64.5 | 50.7 | 45.9 | 44.6 | |||
| zho.rst.sctb | 95.4 | 67.5 | 51.5 | 39.9 | 39.9 |
Full per-corpus UniRST metrics: UniRST_Metrics.md.
The recommended path is pixi (provisions Python + all dependencies + the iinemo/isanlp runtime into a locked env):
git clone https://github.com/Steve-Allison/isanlp_rst.git
cd isanlp_rst
pixi installAlternative (raw venv / pip):
pip install git+https://github.com/iinemo/isanlp.git # required runtime dep
pip install git+https://github.com/Steve-Allison/isanlp_rst.git
# The format-native entry points (parse_docling / parse_doclang /
# parse_markdown) need the optional `formats` extra; the core RST Parser does not:
pip install "isanlp_rst[formats] @ git+https://github.com/Steve-Allison/isanlp_rst.git"The pydantic extra adds the typed RstNode tree model (isanlp_rst.utils.serialization_pydantic); the dependency-free tree_to_dict / tree_from_dict helpers need nothing:
pip install "isanlp_rst[pydantic] @ git+https://github.com/Steve-Allison/isanlp_rst.git"pixi install already includes both extras, so the pixi path needs nothing extra.
from isanlp_rst.parser import Parser
# Choose a model version
version = 'gumrrg' # one of: 'gumrrg', 'rstdt', 'rstreebank', 'rrtrrg', 'unirst'
# Initialise the parser (downloads weights from HF on first call)
parser = Parser(hf_model_name='tchewik/isanlp_rst_v3',
hf_model_version=version,
device='auto') # 'auto' (default) | 'cpu' | 'mps' | 'cuda' | 'cuda:N'
text = """
On Saturday, in the ninth edition of the T20 Men's Cricket World Cup, Team India won against South Africa by seven runs.
The final match was played at the Kensington Oval Stadium in Barbados. This marks India's second win in the T20 World Cup,
which was co-hosted by the West Indies and the USA between June 2 and June 29.
After winning the toss, India decided to bat first and scored 176 runs for the loss of seven wickets.
Virat Kohli top-scored with 76 runs, followed by Axar Patel with 47 runs. Hardik Pandya took three wickets,
and Jasprit Bumrah took two wickets.
"""
res = parser(text) # res['rst'] contains the binary discourse tree
print(vars(res['rst'][0]))For the multilingual unirst model, specify the relation inventory:
parser = Parser(hf_model_name='tchewik/isanlp_rst_v3',
hf_model_version='unirst',
device='auto',
relinventory='eng.erst.gum') # see UniRST_Metrics.md for optionsFor offline / air-gapped use, point Parser at a directory containing the checkpoint:
# Family auto-detected:
# data_manager_*.pickle or config.json with `data.corpora` -> UniRST
# relation_table.txt -> DMRST
parser = Parser(model_dir='/path/to/checkpoint', device='auto')
# Override auto-detection:
parser = Parser(model_dir='/path/to/checkpoint', family='dmrst', device='auto')device= chooses the compute backend (default "auto"):
"auto"(default) → CUDA if present, else MPS on Apple Silicon, else CPU"cpu"→ CPU"mps"→ Apple Silicon Metal backend (raises if unavailable)"cuda"/"cuda:N"→ a specific NVIDIA device (raises if no CUDA)
The integer cuda_device= parameter is deprecated but still accepted (-1 → CPU, >= 0 → best available accelerator); passing it emits a DeprecationWarning. Migrate to device=.
PyTorch has no MPS kernel for torch.linalg.qr (used by torch.nn.init.orthogonal_ during weight init). The parser routes this via CPU automatically — no env-var hacks required.
Forward passes go through torch.autocast, so the model runs in float32, float16, or bfloat16 without changing the trained weights:
import torch
parser = Parser(hf_model_version='gumrrg', device='auto',
dtype=torch.bfloat16) # also accepts 'bf16', 'fp16', 'fp32'Default is float32 on every device. On Apple Silicon (M-series, PyTorch 2.11) at ~1k-char inputs, float32 beats bfloat16 / float16 for every published model — per-op autocast dispatch overhead dominates the matmul speedup at this scale. On large-batch CUDA workloads with native bf16 (Hopper / Ada Tensor Cores), bfloat16 is likely faster — measure with pixi run bench before pinning a choice.
Tree topology and EDU segmentation are bit-equivalent across all three dtypes for every published model; relation labels are not — on a near-tied node, reduced precision (bf16/fp16) can flip the argmax without any structural change. See tests/test_integration.py for the equivalence suite.
| Model | CPU fp32 | MPS fp32 | MPS bf16 | MPS fp16 |
|---|---|---|---|---|
gumrrg |
143 ms | 120 ms | 156 ms | 157 ms |
rstdt |
168 ms | 123 ms | 161 ms | 165 ms |
rstreebank |
113 ms | 59 ms | 94 ms | 94 ms |
rrtrrg |
118 ms | 61 ms | 104 ms | 95 ms |
unirst |
127 ms | 153 ms | 218 ms | 221 ms |
The 18-corpus unirst model is faster on CPU than on MPS — multi-corpus classifier dispatch costs more than MPS's matmul speedup recovers. Run pixi run bench --version unirst on your hardware to verify before pinning a device choice for that model.
CI runs on macOS Apple Silicon, so the CUDA dispatch path isn't exercised in CI. To verify on an NVIDIA host:
pixi run cuda-smokeThe script confirms torch.cuda.is_available(), loads DMRST and UniRST on cuda:0, parses a sample text, and round-trips a parse_from_edus call. Exits non-zero on any failure.
The parser returns an RST tree with a recursive DiscourseUnit structure. Each node carries:
{
'id': 21,
'left': (id=14, start=1, end=323), # child node refs
'right': (id=20, start=324, end=570),
'relation': 'elaboration', # rhetorical relation
'nuclearity': 'NS', # NS / NN / ""
'entropy': 0.92, # split entropy
'start': 1, # original-text character offset
'end': 570,
'text': "On Saturday, ... took two wickets."
}Leaves are EDUs; internal nodes are relations. Every node has start / end in original-text character coordinates.
res['rst'][0].to_rs3('filename.rs3')Open filename.rs3 in external tools like RSTTool or rstWeb for editing.
import io, contextlib
import isanlp_rst
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
isanlp_rst.render("filename.rs3")
# For Google Colab, sync the cell height:
# isanlp_rst.render("filename.rs3", colab=True)Requires Playwright:
pip install playwright
playwright install chromiumimport isanlp_rst
isanlp_rst.to_png("filename.rs3", "filename.png")
isanlp_rst.to_pdf("filename.rs3", "filename.pdf")my_edus = [
"On Saturday, Team India won against South Africa.",
"The final match was played in Barbados."
]
res = parser.from_edus(my_edus)When parsing many documents, the resulting DiscourseUnit trees can consume significant memory — each node stores its corresponding text span.
res['rst'][0].clear_textfields() # drop .text on every node, keep structure
# ... pickle / store ...
res['rst'][0].fill_textfields(full_text) # repopulate laterNote: .to_rs3() on a tree with cleared text fields will fail.
For structured documents — PDFs, PPTX decks, VTT transcripts, Markdown, HTML — converted via Docling, isanlp_rst exposes parse_docling(): a single entry point that walks the canonical Docling structure, harvests the text in source-format-appropriate order, runs RST once over the full harvest, and returns the relations indexed by Docling self_ref with per-relation boundary annotations.
from pathlib import Path
from isanlp_rst.docling import parse_docling
result = parse_docling(Path("deck.docling.json"), device="auto")
# result.relations: tuple of RstRelation — flat list with left_id / right_id for tree shape
# result.edus: tuple of RstEdu — leaves indexed by Docling self_ref
# result.boundaries: tuple of Boundary — slides, sections, speaker turns, tables
# result.source_origin: dict — mirror of doc.origin (mimetype, binary_hash, filename)The underlying weights are ~2 GB. Constructing a fresh Parser per call would reload them every time. Build one parser, inject it:
from isanlp_rst.parser import Parser
from isanlp_rst.docling import parse_docling
parser = Parser(hf_model_version="gumrrg", device="auto")
for path in document_paths:
result = parse_docling(path, parser=parser)
...| Source format | Harvested by default | Boundary detection |
|---|---|---|
| PPTX | slide texts (titles, paragraphs, list items); speaker notes (ContentLayer.NOTES); picture VLM descriptions from meta.description.text where present |
one slide-N boundary per slide group; one slide-N-notes when the slide carries notes |
| body texts, section headers, list items, picture-children (OCR / chart labels) | section-N opened at each section_header; any pre-header content lives in a leading document boundary |
|
| HTML / Markdown | same as PDF | same as PDF |
| VTT | every transcript line | one turn-N per contiguous-same-voice run (coalesce_speaker_turns=True default) |
| Tables (any format) | analysed two-level: cells are excluded from the main harvest; each TableItem gets its own RST mini-parse in result.table_analyses. Cell self_refs are real JSON pointers (#/tables/N/data/table_cells/M) that resolve against the source. Knob off via include_table_cells=False. |
one table-N boundary per TableItem; self_refs is (#/tables/N, <cell pointers>). The synthetic #/tables/N marker carries no harvest span so it cannot land in relation refs. |
Toggle the harvest with keyword arguments: include_picture_descriptions=False, include_slide_notes=False, include_furniture=True, include_table_cells=False. See the walkthrough for the full set of knobs and a tree-reconstruction example.
- RST was trained on prose. Quality is highest on continuous narrative (VTT transcripts, decks with VLM descriptions, prose-heavy PDFs). On bullet-only slide decks without VLM enrichment, or on highly procedural / list-heavy PDFs, relations are dominated by
joint/organizationchains — structurally valid but rhetorically thin. Use the result with appropriate scepticism on non-prose input. - Long inputs:
parse_doclingraisesInputTooLargeErrorabovemax_harvest_chars=200_000(configurable). The largest fixture exercised end-to-end is ~18 KB; the parser handles 40 KB+ cleanly in smoke tests but degradation at extreme sizes is empirical. - Cross-boundary relations are preserved. A relation may touch two slides or sections;
boundary_membershipslists all touched boundaries. Filter onlen(boundary_memberships) == 1for within-boundary relations only.
For documents authored in the DocLang 0.5 XML format — the AI-native document standard from IBM, ABBYY, RedHat, HumanSignal, NVIDIA, and Forgis — isanlp_rst exposes parse_doclang() alongside parse_docling. Both are first-class entry points with honestly different schemas; neither coerces into the other.
from pathlib import Path
from isanlp_rst.doclang import parse_doclang
result = parse_doclang(Path("document.dclg.xml"), device="auto")
# result.relations: tuple of RstRelation, indexed by local-name XPath
# result.edus: tuple of RstEdu with the same xpath addressing
# result.boundaries: tuple of Boundary — headings, pages, groups, tables, field_regions
# result.source_origin: {"format": "doclang", "namespace": ..., "version": "0.5", "head_children": [...]}Every element is addressed by a local-name canonical XPath of the form /doclang[1]/heading[2]/text[1] — 1-based position predicates per local name, namespaces stripped. The same document with or without xmlns="https://www.doclang.ai/ns/v0" produces identical paths. (lxml.etree.ElementTree.getpath() is not used; it emits /*/*[N] wildcards on default-namespaced documents.) Paths round-trip 100% against the upstream 40-fixture valid corpus.
| Element | Default | Notes |
|---|---|---|
<text>, <heading>, <footnote> |
harvested | one span per element; <heading> also opens a heading-N boundary |
<list> items |
harvested per <ldiv/> marker |
item text aggregates marker.tail + intervening sibling tails |
<picture><caption> |
harvested | include_picture_captions=False to skip |
<code> |
skipped | source code is not prose — include_code_blocks=True to opt in |
<formula> |
skipped | raw LaTeX is not prose — include_formulas=True to opt in |
<table> cells |
analysed two-level: excluded from the main harvest; each table gets its own RST mini-parse in result.table_analyses, cells addressed by marker xpath with row/col grid positions |
one table-N boundary per table; xpaths is (table_xpath, <cell marker xpaths>). include_table_cells=False skips the analyses. <index> and <tabular> remain boundary-only. |
<field_region> body |
skipped | include_field_regions=True to opt in |
<page_header> / <page_footer> |
skipped | include_furniture=True to include |
<layer value="background"> items |
skipped | include_background=True to include |
<layer value="furniture"> items |
skipped | covered by include_furniture=True |
Toggle each via the corresponding keyword. See the walkthrough for the full set, a tree-reconstruction example, and <thread> handling for cross-fragment continuation.
DocLang doesn't model slides or speaker turns — the boundary set reflects what the spec does model: heading-N, page-N (between <page_break/> markers), group-N (with group-N-M for one level of nesting), table-N, field_region-N, and a document fallback. If your input is PPTX or VTT, use parse_docling on the Docling JSON form — those formats give you slide-N / slide-N-notes / turn-N instead.
parse_doclang(..., validate_xml=True) (default) gates the file through the doclang PyPI package's validate(path) before parsing. The doclang package is validator-only (no DOM) — we parse with lxml ourselves. If doclang is not importable in your env, validation is silently skipped.
- RST was trained on prose. Same caveat as
parse_docling: bullet-only documents and form-shaped content yield rhetorically thin relations dominated byjoint/organizationchains. - Stable addressing across producers. The local-name XPath is reproducible from the parsed XML alone, but it depends on document order. A producer that reorders elements between runs will produce different addresses for the same logical content. DocLang 0.5 has no stable per-element identifier in the spec.
<thread>is continuation, not identity. Two spans sharingthread_idare fragments of one logical paragraph (typically across a<page_break/>). The RST mapper aggregates dedup'd thread ids on each relation asnucleus_thread_ids/satellite_thread_ids.
For plain CommonMark / GFM markdown — READMEs, design notes, MkDocs / MyST corpora — isanlp_rst exposes parse_markdown() as a first-class entry point that reads .md directly, with no Docling round-trip. CommonMark parsing is handled in pure Python via markdown-it-py plus the mdit-py-plugins front-matter and GFM table plugins.
from pathlib import Path
from isanlp_rst.markdown import parse_markdown
result = parse_markdown(Path("design-notes.md"), device="auto")
# result.relations: tuple of RstRelation, indexed by #/blocks/N — the document tree
# result.edus: tuple of RstEdu with the same block_ref addressing
# result.table_analyses: tuple of TableAnalysis — one RST mini-parse per table (two-level)
# result.boundaries: tuple of Boundary — sections, tables, code blocks, document fallback
# result.source_origin: {"format": "markdown", "gfm": True, "front_matter": "<raw text>", "front_matter_format": "yaml"}Every main-harvest unit gets a sequential #/blocks/N reference in document order — parallel to Docling's #/texts/N. Table cells live in their own namespace, #/tables/T/cells/K (K = grid position, row-major, stable past empty cells). The synthetic #/tables/T marker appears only on the table boundary; it carries no harvest span so it cannot leak into relation refs.
The cross-format directive is analyse everything by default — but two-level: prose enters the document tree; each table gets its own RST mini-parse in table_analyses, so table discourse never distorts prose discourse.
| Construct | Default | Notes |
|---|---|---|
| Headings (ATX + Setext) | harvested | level ∈ {1..6}; opens section-N boundary |
| Paragraphs | harvested | one span per paragraph |
| List items | harvested | one span per item; nested items collapse into outer |
| Blockquote content | harvested | include_blockquotes=False gates the whole quoted region (paragraphs, headings, lists, code, HTML, tables). A quoted heading never opens a section. |
| Fenced + indented code blocks | harvested | include_code_blocks=False for prose-only; code_block-N boundary still emitted |
| Raw HTML blocks | tags stripped, text harvested | include_html=False to skip |
| GFM tables | per-table mini-parse in table_analyses |
include_table_cells=False to skip; cells carry row/col grid positions |
| GFM strikethrough | wrappers dropped, text kept | enabled under gfm=True — no literal ~~ in the harvest |
| Image alt text | flattened into parent paragraph | inline content; not its own span |
| Front-matter (YAML only) | stripped from harvest | raw text preserved in source_origin.front_matter |
Thematic break (---) |
skipped | divider, not prose |
section-N— opened at each heading;levelcarries 1–6. Quoted headings excluded.table-T— one per table;block_refsis(#/tables/T, <cell refs…>); matchestable_analyses[].id.code_block-N— one per code block; one-cellblock_refs.document— pre-heading bucket, or fallback when no headings exist.
- RST was trained on prose. Tables of textual content (definitions, comparisons, feature lists) parse reasonably; tables of pure numbers produce noisy, low-value mini-trees. The two-level split keeps that noise out of the document tree either way.
- No
doclingruntime dep. The markdown entry path is pure Python —markdown-it-py+mdit-py-pluginsonly. If you need full Docling fidelity on markdown (heavy layout / OCR pipeline), convert to Docling JSON first and useparse_docling. - Front-matter is raw YAML text. No PyYAML /
tomllibdependency — the consumer parsessource_origin.front_matterif structured access is needed. TOML/JSON front-matter is not supported.
See the walkthrough for tree reconstruction, per-section grouping, table-analysis usage, and the full knob set, and the design plan for the cross-format consistency directive.
pixi run rst-diag <paths> parses any mix of .md / *.docling.json / *.dclg.xml sources (files or directories; one shared model load) and emits per-document proxy metrics — no gold annotations required:
- joint ratio — share of relations labelled joint / same-unit / organization (high = rhetorically thin chaining)
- tree skew — max depth ÷ log₂(EDUs) (≫ 1 = degenerate chain)
- cross-boundary ratio — relations spanning more than one section / slide / turn
- note ratio — lopsided EDU/span alignment
- table analyses — per-table mini-parse count
Use it to A/B any harvest-policy change before trusting it. --json for machine-readable output; --model-version, --device, --dtype as in the entry points.
All three entry points also accept cache_dir= — an on-disk result cache keyed by source bytes + model identity + knobs, so batch re-runs skip unchanged documents entirely — and dtype= for mixed-precision overrides.
This repository is Steve Allison's evolution of the IsaNLP RST Parser. The original RST research code and the trained model weights are by Elena Chistova; the MIT-licensed source code carries her copyright. This repository adds pixi-managed builds, a pytest test suite, GitHub Actions CI, MPS / Apple-Silicon support, mixed-precision dispatch, and ongoing roadmap work (see docs/plans/).
- Source code: MIT — see
LICENSE. Copyright Elena Chistova 2020; Steve Allison contributions also under MIT. - Model weights (downloaded from
tchewik/isanlp_rst_v3on HuggingFace): CC BY-NC 4.0 — research and non-commercial use only. SeeLICENSE_MODELS. Commercial use requires either retraining weights under a permissive licence or replacing the models entirely.
Issues and pull requests: please open them on Steve-Allison/isanlp_rst. For questions about the underlying RST research, see Elena Chistova's papers cited below.
The published model weights are by Elena Chistova. If you use them in research, please cite:
For rstdt, gumrrg, and rstreebank:
@inproceedings{chistova-2024-bilingual,
title = "Bilingual Rhetorical Structure Parsing with Large Parallel Annotations",
author = "Chistova, Elena",
booktitle = "Findings of the Association for Computational Linguistics ACL 2024",
month = aug,
year = "2024",
address = "Bangkok, Thailand and virtual meeting",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-acl.577",
pages = "9689--9706"
}For unirst:
@inproceedings{chistova-2025-bridging,
title = "Bridging Discourse Treebanks with a Unified Rhetorical Structure Parser",
author = "Chistova, Elena",
booktitle = "Proceedings of the 6th Workshop on Computational Approaches to Discourse, Context and Document-Level Inferences (CODI 2025)",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.codi-1.17/",
pages = "197--208"
}
