Skip to content

Fix sentencepiece >= 0.2.2 version compatibility #2847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Nikhilyammani007 wants to merge 4 commits into
base: master
Choose a base branch
from
+20 −10
Draft

Fix sentencepiece >= 0.2.2 version compatibility #2847

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b287798
Fix sentencepiece >= 0.2.2 compatibility natively
Nikhilyammani007 Jul 19, 2026
1b78252
Format sentence_piece_tokenizer.py
Nikhilyammani007 Jul 19, 2026
251d1a6
Update sentence_piece_tokenizer to be compliant with Keras API guidel…
Nikhilyammani007 Jul 19, 2026
8066cf8
Implement try-except fallback for older sentencepiece versions
Nikhilyammani007 Jul 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions keras_hub/src/tokenizers/sentence_piece_tokenizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,13 +148,12 @@ def _set_proto_tf(self, proto):
)

def _set_proto_spm(self, proto):
self._sentence_piece_spm = spm.SentencePieceProcessor()
self._sentence_piece_spm.Init(
out_type = str if is_string_dtype(self.compute_dtype) else int
self._sentence_piece_spm = spm.SentencePieceProcessor(
model_proto=proto,
out_type=str if is_string_dtype(self.compute_dtype) else int,
out_type=out_type,
add_bos=self.add_bos,
add_eos=self.add_eos,
alpha=1.0,
)

@gemini-code-assist gemini-code-assist Bot Jul 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The proposed change removes the .Init() call and passes the initialization arguments directly to the spm.SentencePieceProcessor constructor. While this works for sentencepiece >= 0.2.2, it will raise a TypeError on older versions of sentencepiece (e.g., < 0.2.2) because their constructor does not accept these keyword arguments. Since the version constraint has been completely unpinned (rather than being set to >=0.2.2), we must maintain backward compatibility for users with older versions of sentencepiece installed.

We can achieve this robustly by using a try-except block to attempt the new constructor-based initialization first, and fall back to the .Init() method if a TypeError is raised.

Suggested change
out_type = str if is_string_dtype(self.compute_dtype) else int
self._sentence_piece_spm = spm.SentencePieceProcessor(
model_proto=proto,
out_type=str if is_string_dtype(self.compute_dtype) else int,
out_type=out_type,
add_bos=self.add_bos,
add_eos=self.add_eos,
alpha=1.0,
)
out_type = str if is_string_dtype(self.compute_dtype) else int
try:
self._sentence_piece_spm = spm.SentencePieceProcessor(
model_proto=proto,
out_type=out_type,
add_bos=self.add_bos,
add_eos=self.add_eos,
)
except TypeError:
self._sentence_piece_spm = spm.SentencePieceProcessor()
self._sentence_piece_spm.Init(
model_proto=proto,
out_type=out_type,
add_bos=self.add_bos,
add_eos=self.add_eos,
alpha=1.0,
)
References
  1. Demand Robustness: Do not accept fragile code. If the proposed code is not robust enough or lacks proper error handling, explicitly tell the author why the current approach is brittle and what must be done to reinforce it.


def set_proto(self, proto):
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ dependencies = [
"rich",
"kagglehub!=1.0.1",
"kagglesdk==0.1.28",
"sentencepiece<0.2.2",
"sentencepiece",
"tokenizers",
"tensorflow-text;platform_system != 'Windows'",
]
Expand Down
2 changes: 1 addition & 1 deletion requirements-common.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ rich
kagglehub!=1.0.1
kagglesdk==0.1.28
tokenizers
sentencepiece<0.2.2
sentencepiece
# Tooling deps.
pre-commit
astor
Expand Down
Loading