Skip to content

Bundle Punkt models for offline sentence tokenization - #5726

Draft
markbackman wants to merge 2 commits into
mainfrom
feat/bundled-punkt-data
Draft

Bundle Punkt models for offline sentence tokenization#5726
markbackman wants to merge 2 commits into
mainfrom
feat/bundled-punkt-data

Conversation

@markbackman

Copy link
Copy Markdown
Contributor

Summary

  • Bundle all 19 Punkt models so sentence aggregation works without runtime downloads or an external NLTK data directory. Read parameters directly from the compressed archive without changing NLTK data paths.
  • Preserve lazy NLTK imports and background warm-up. Include the pinned archive and provenance in both wheels and source distributions.

Release consideration

NLTK explicitly identifies the Punkt model data license as unclear, separately from its code license: https://github.com/nltk/nltk_data/blob/gh-pages/LICENSE-OVERVIEW.md. Redistribution licensing needs resolution before release; this PR is a draft for review.

Testing

  • 33 tests passed covering offline loading, all 19 models, lazy imports, sentence detection, and streaming aggregation.
  • Built a wheel from the source distribution and verified tokenization directly from the wheel with external NLTK data disabled and downloads blocked.

Related: #5720.

@codecov

codecov Bot commented Sep 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
src/pipecat/utils/string.py 90.17% <100.00%> (+2.79%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant