do we have plan support java ，we need you

[mullerhai]

Hi

now we also need load llm model and to chat， but not have officeal tokenizers package ，please support us 。thanks

[ArthurZucker]

Let me think a bit, not impossible!

[mullerhai]

Let me think a bit, not impossible!

ahah， so terrible

[ArthurZucker]

the most likely outcome is we'll provide good C abi support for anyone to build on top ! (cpp was asked a lot)

[mullerhai]

the most likely outcome is we'll provide good C abi support for anyone to build on top ! (cpp was asked a lot)

if your team could support cpp, it also perfect ,because it could. recompiled to java

[IgnaciodelaTorreArias]

Hi everyone! I’ve been following this thread and wanted to share some progress I’ve made on this front.
​A few months ago, I needed a C#/.NET implementation, but I ran into some friction with the approach in #1615 (specifically regarding separate packages for different architectures). To solve this, I developed tokenizers_proto.
​It’s essentially a C ABI written in Rust that uses Protobuf to handle communication between languages. By using pointers to pass protobuf-encoded buffers, any language with FFI capabilities can hook into it. I’ve already used it to power:

• ​.NET bindings. (solving the multi-arch packaging issue).

• TypeScript bindings. Using Bun/Deno ffi.

​I’m sharing this in case the architecture or the C ABI logic can serve as a reference (or a starting point) for the official implementation. I’m also planning to explore Kotlin bindings next when time permits!
​Would love to hear your thoughts or feedback if this aligns with what you're looking for.

[zhimin-z]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.