Skip to content

Replace custom HF tokenizer with C++ bindings for HF tokenizer #153

@jackzhxng

Description

@jackzhxng

Refer to these C++ bindings for HF tokenizer by mlc-ai: https://github.com/mlc-ai/tokenizers-cpp/

Metadata

Metadata

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions