Add MLX multimodal inference support#10383
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29a59a24a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Some( | ||
| loaded | ||
| .model | ||
| .prepare_input(&processor_input) | ||
| .map_err(mlx_error)?, |
There was a problem hiding this comment.
Account for media tokens before capping generation
When has_media is true, this branch prepares a processor input that can expand each image into additional model positions, but the earlier context check and the later mlx_max_tokens(..., prompt_tokens.len()) still use the plain text token count from the marker-only prompt. For image requests near the model context limit, Goose can allow a prompt that only fits before image expansion and then either fail during generation or overrun the intended context budget; use the prepared input's effective length or an image-token estimate when validating and computing headroom.
Useful? React with 👍 / 👎.
Summary
Why
Recent safemlx releases add multimodal model processors and change prepared-input generation APIs. Goose needs to use those APIs so MLX vision models receive decoded image data alongside the rendered prompt while text-only models continue using the existing token input path.
Validation
cargo fmtcargo check -p goose-local-inference --features mlxcargo test -p goose-local-inference --features mlx(130 tests passed)cargo check -p goose-local-inference