Classical probabilistic NLP models implemented from scratch — N-gram LM · HMM POS Tagger · Viterbi · Word Segmentation
This project implements classical statistical NLP models from scratch, without relying on high-level NLP libraries. The goal is to build a concrete understanding of probabilistic sequence modeling at the algorithm level.
| Model | Description | Key technique |
|---|---|---|
| N-gram LM | Character/word-level language model | Add-k smoothing |
| HMM Tagger | Part-of-speech tagging | Bigram transitions + MLE emission |
| Viterbi | Exact decoding for HMM | Dynamic programming, log-space |
| Segmentation | Chinese/unsegmented word segmentation | DP with bigram cost |
Evaluated on Universal Dependencies English EWT:
| Metric | Score |
|---|---|
| POS Tagging Accuracy | 88.26% |
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -e .Run the HMM POS tagger:
python examples/run_hmm.pystatistical-nlp/
├── statnlp/
│ ├── ngram.py # N-gram language model with add-k smoothing
│ ├── hmm.py # HMM: transition + emission + Viterbi
│ ├── segmentation_fast.py # DP word segmentation (optimised)
│ └── segmentation_iterative.py
├── examples/
│ └── run_hmm.py
├── requirements.txt
└── setup.py
Transition: P(tag_t | tag_{t-1}) — bigram tag LM with add-k smoothing
Emission: P(word | tag) — MLE, log-space to prevent underflow
Decoding: Viterbi algorithm — DP over (time × tag) trellis
The decoder fills a (T × N_tags) trellis with log-probabilities and backtracks to recover the optimal tag sequence. Log-space computation prevents numerical underflow on long sequences.
- Unknown word handling (morphological features / suffix heuristics)
- Trigram HMM for higher-order context
- Confusion matrix analysis
- Neural baseline comparison (BiLSTM-CRF)
本项目从零实现了经典统计 NLP 模型，目标是深入理解概率序列建模原理，而不依赖现成框架。
| 模型 | 说明 | 核心技术 |
|---|---|---|
| N-gram 语言模型 | 字/词级别语言模型 | 加性平滑（add-k） |
| HMM 词性标注器 | 序列标注 | Bigram 转移 + MLE 发射 |
| Viterbi 解码 | HMM 精确解码 | 动态规划 + 对数空间 |
| 词语切分 | 基于 DP 的分词 | Bigram 代价最小化 |
在 Universal Dependencies English EWT 数据集上，词性标注准确率达到 88.26%。
- 转移概率：基于 bigram 的标签语言模型，add-k 平滑
- 发射概率：极大似然估计，对数空间计算防止数值下溢
- Viterbi：
(时间步 × 标签数)网格动态规划，回溯最优路径
- 未登录词处理（词缀特征等）
- Trigram HMM
- 混淆矩阵分析
- 神经网络基线对比（BiLSTM-CRF）