Add MLX (Apple Silicon) inference example to the docs#265
Open
Lee-Gihun wants to merge 1 commit into
Open
Conversation
Shows the token-level pattern for running Mistral models with mlx-lm: encode_chat_completion, generate_step, then decode with the same tokenizer. Appended to docs/examples/inference.md, no nav change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an MLX (Apple Silicon) inference example to
docs/examples/inference.md: encode with mistral-common, generate at token level with mlx-lm, decode with the same tokenizer. No nav change.Context
With transformers v5,
AutoTokenizerloads Mistral models that ship atekken.jsonasMistralCommonBackend. Wrappers that assume fast-tokenizer attributes currently break on this (ml-explore/mlx-lm#1576), and there is no guidance on the mistral-common side for MLX users. The token-level pattern documented here is the path that works today and keeps prompt construction on the official tokenizer.Example validated verbatim with mistral-common 1.11.5, mlx-lm 0.31.3, and
mlx-community/Ministral-3-3B-Instruct-2512-4bit.