Skip to content

docs: clarify tokenizer considerations for new-language finetuning - #2332

Open
mukeshr06 wants to merge 1 commit into
Lightning-AI:mainfrom
mukeshr06:patch-1
Open

mukeshr06 wants to merge 1 commit into
Lightning-AI:mainfrom
mukeshr06:patch-1

Conversation

@mukeshr06

Copy link
Copy Markdown

Summary

Addresses #225 by adding a focused "Finetuning on a new language" section to the existing fine-tuning guide.

  • Explain why Falcon-7B's byte-level BPE can encode Chinese without adding vocabulary entries, and why encoding support does not imply language proficiency.
  • Clarify why embedding parameter counts do not grow with new training text and how full finetuning differs from standard LoRA.
  • Describe what intentional vocabulary expansion requires: stable token IDs, compatible embedding/output dimensions, training and saving the added-token rows, and matching tokenizer/checkpoint artifacts.

This is documentation-only. It does not introduce an automatic resizing API or change training behavior.

Validation

  • pytest --check-links tutorials/finetune.md --check-links-ignore "http*": 5 passed.
  • codespell tutorials/finetune.md: passed.
  • git diff --check: passed.
  • Loaded the published tiiuae/falcon-7b/tokenizer.json with Hugging Face tokenizers; verified exact encode/decode round-trips for Simplified Chinese, Traditional Chinese, and mixed English/Chinese samples, in-range token IDs, and an unchanged 65,024-token vocabulary.
  • Cross-checked the embedding dimensions, LoRA freezing, and adapter-only checkpoint saving against the current implementation.

No model weights were downloaded and no training or full test suite was run.

Explain Falcon Chinese tokenization, unchanged embedding parameter counts, and the requirements and LoRA limitations of intentional vocabulary expansion.

Refs Lightning-AI#225.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant