This add-on makes it possible to obtain detailed descriptions for images, user interface controls, and other visually inaccessible content.
Leveraging the multimodal capabilities of advanced AI models and computer vision algorithms, we aim to deliver best-in-class content descriptions and increase independence overall. For more information about the underlying models, refer to the corresponding section of this document.
- Describe the focus object, navigator object, entire screen, or snap a photo from the onboard camera
- Describe any image that has been copied to the clipboard, be it a picture from an email or a path in windows explorer
- Indicate whether the user's face is positioned at the center of the frame using computer vision algorithms (does not require paid API access)
- Use supported models to control the active application with mouse and keyboard actions for potentially inaccessible tasks
- Free to use by default, optionally add your own API key for more models
- Supports multiple providers (OpenAI's GPT and the free Pollinations tier, Google's Gemini, Mistral's Pixtral Large, Anthropic's Claude, xAI's Grok, Moonshot AI's Kimi, vivo BlueLM Vision via NVDA-CN, Ollama, llama.cpp, LiteLLM Proxy, and Seer)
- Supports a wide variety of formats including PNG (.png), JPEG (.jpeg and .jpg), WEBP (.webp), and non-animated GIF (.gif)
- Optionally caches responses to preserve API quota
- For advanced use, customize the prompt and token count to tailor information to your needs
- Ask follow-up questions and attach additional images
- Markdown rendering to easily access structured information (just enable the "open results in a browsable dialog" setting and embed e.g. "respond in Markdown" at the end of your prompts)
There were a few primary motivations behind this project.
NVDA is capable of performing optical character recognition (OCR) out of the box, which is a game changer. If you are trying to get text out of an image or PDF document, this is what you're looking for.
However, OCR is only able to analyze data that might be text. It falls short at considering the context, objects and relationships conveyed in those images. And the internet is full of them. Logos, portraits, memes, icons, charts, diagrams, bar/line graphs... You name it. They're everywhere, and usually not in a format that screen reader users can interpret. Until recently, there has been an unwavering reliance on content authors providing alternative text descriptions. While this is still a must, it's difficult to change the fact that a high standard of quality happens to be the exception, not the rule.
Now, the possibilities are almost endless. You might:
- Visualize the desktop or a specific window to understand the placement of icons when training others
- Get detailed info about the status of games, virtual machines, etc when sound is insufficient or unavailable
- Figure out what is displayed in a graph
- Demystify screenshots or screen shares in Zoom or Microsoft Teams
- Ask a supported model to help operate an inaccessible app or complete a task in the foreground window
- Ensure your face is looking clearly at the camera and that your background is professional before recording videos or participating in online meetings
- Pollinations: The team behind Pollinations.AI generously sponsors free GPT4 access for this project, so users do not need to supply their own API key. Note that the service has been observed to be buggy, and we highly recommend using your own keys to realize the full benefits of this project.
- OpenAI GPT and reasoning models: Requires an OpenAI API key. Includes GPT-4 turbo, GPT-4o, the GPT-4.1 family (4.1, 4.1 mini, 4.1 nano), the GPT-5 family (5, 5 mini, 5 nano, and 5 chat), the GPT-5.4 family (5.4, 5.4 mini, 5.4 nano), the GPT-5.5 family (5.5 and 5.5 pro), and the reasoning models O3, O3 pro, O3 mini, and O4 mini. Computer use is supported by the GPT-5.4 and GPT-5.5 families.
- Google Gemini, including 2.5 Flash, 2.5 Flash-Lite, 2.5 Pro, 3 Flash Preview, 3.1 Flash-Lite Preview, and 3.1 Pro Preview models.
- Anthropic Claude, including Claude 4 (Sonnet, Opus), 4.1 Opus, 4.5 (Haiku, Sonnet, Opus), 4.6 (Sonnet, Opus), and 4.7 Opus. Computer use is supported by Claude 4.1 Opus, Claude 4.5 models, Claude 4.6 models, and Claude 4.7 Opus.
- Pixtral Large
- Grok 2, Grok 4, and Grok 4 Fast (reasoning and non-reasoning)
- Kimi, Moonshot AI's multimodal models, including Kimi K3, Kimi K2.6, and Kimi K2.5.
- vivo BlueLM Vision: a multimodal model from vivo, accessed via a free NVDA-CN account. See the setup section below.
- Ollama (unstable)
- llama.cpp (extremely unstable and slow depending on your hardware, tested to work with llava-v1.5/1.6, BakLLaVA, Obsidian, and MobileVLM 1.7B/3B models)
- LiteLLM Proxy: Access multiple AI models through a unified proxy server. Requires a LiteLLM proxy server URL, optionally an API key depending on your proxy configuration. Appears as "LiteLLM Proxy" in the model configuration dialog. Supports dynamic model selection and follow-up questions. Compatible with all formats (PNG, JPEG, WEBP, GIF).
- Seer: Runs PaliGemma2 locally via the Seer daemon. No API key or internet connection required. Note: this is a captioning-only model. Prompts and follow-up questions are not supported.
Follow the instructions provided below to get each of these working.
Download the latest release of the add-on from this link. Click on the file on a computer with NVDA installed, and proceed with the installation as prompted.
As of version 2025.06.05, usage of GPT4 is free thanks to the generosity of the community behind PollinationsAI.
If you have the resources and interest in exploring additional models, you can always use your own API key and reduce requests to their servers. If you don't, feel free to skip down to the usage section of this document.
Follow the instructions below to obtain an API key from a supported provider.
We used to provide recommendations on the cheapest and highest quality options, but the landscape is changing so rapidly that it isn't possible to keep this up-to-date.
The short answer is that most of the state-of-the-art models have gotten to a point where they are accurate for most every-day use cases, so choose the provider that you are familiar with.
Most people want to strike a balance between accuracy and cost. The LLM arena leaderboard (specifically the vision category) measures correctness, while LLM pricing calculator outlines pricing.
If you want to use computer control, choose one of the supported OpenAI or Anthropic models listed above. Free Pollinations access is useful for image descriptions, but it does not support computer use.
- Go to open-AI's API key page
- If you don't yet have an account, create one. If you do, log in.
- On the API keys page, click to create a new secret key. Copy it to your clipboard.
- Fund the account with at least $1
- In the NVDA settings dialog, scroll down to the AI Content Describer category, then choose "manage models (alt+m)", select any of the OpenAI models (for example "GPT-4 omni") as the provider, tab into the API key field, and paste the key you just generated here.
At the time of this writing, Open-AI issues credits to new developer accounts that can be used for three months, after which they are lost. Following this period, you will have to purchase credits. Typical usage shouldn't ever exceed $5.00 per month. For a point of reference, the original version of this add-on was developed for slightly under a dollar. It is always possible to login to your OpenAI account and click on "usage" to get your quota.
- You will first need to create a Google workspace project by going to Google cloud console. Make sure you are logged in to your google account.
- Create a name between four and thirty characters, like "Gemini" or "NVDA add-on"
- Navigate to Google AI studio API keys page
- Click "create API key"
- In the NVDA settings dialog, scroll down to the AI Content Describer category, then choose "manage models (alt+m)", select "Google Gemini" as your provider, tab into the API key field, and paste the key you just generated here.
- Login to the Anthropic console.
- Click on your profile -> API keys.
- Click Create Key.
- Enter a name for the key, like "AIContentDescriber", then click on "Create Key" and copy the value that shows up. This is what you will paste into the API key field under the Ai Content Describer category of the NVDA settings dialog -> manage models -> any Claude 4.x model (for example "Claude 4.6 Sonnet").
- If you haven't already, purchase at least $5 in credits under the anthropic plans page.
- Sign into or create a MistralAI account by going to the MistralAI login page.
- If you are creating or logging into an account for the first time, add a workspace as prompted, supplying a name and accepting the terms and conditions.
- Once logged in, select "API keys" from the menu.
- Click "create a new key", and copy it to your clipboard. This value is what you will paste into the API key field under the Ai Content Describer category of the NVDA settings dialog -> manage models -> Pixtral Large.
- Fund the account, if applicable.
- Sign into or create an account on the Kimi developer platform.
- Once logged in, open the API keys section of the console and create a new key.
- Copy the key to your clipboard. This value is what you will paste into the API key field under the AI Content Describer category of the NVDA settings dialog -> manage models -> Kimi K3 (or another Kimi model).
- Fund the account, if applicable.
This service is made available free of charge through a partnership between VIVO (vivo.com.cn) and the NVDA Chinese Community (NVDACN). It provides high-quality multimodal recognition and is the recommended model for all users, especially those getting started.
To use this model, you only need a free NVDA-CN account.
- Create an Account: Go to the NVDA-CN registration page: https://nvdacn.com/admin/register.php.
- Note: The page is currently in Chinese. We recommend using your browser's built-in translation feature to complete the form.
- You will be asked for a username, password, and a valid email address. Be sure to store your password securely, as automated password recovery is not yet implemented.
- Verify Your Email: Check your inbox for a verification email and click the link inside to activate your account.
- Configure the Add-on:
- Open the NVDA settings dialog and navigate to the "AI Content Describer" category.
- Select the "Manage models" button.
- From the list of providers, choose "vivo BlueLM Vision (NVDA-CN)".
- Enter your NVDA-CN username and password into the corresponding fields.
- Click OK to save your credentials.
You are now set up to use the VIVO model. For any account-related issues, you can contact the NVDA-CN team at support@nvdacn.com.
This is currently the preferred option for a local setup.
Though the Ollama integration has been tested more extensively than llama.cpp, it is still less stable than calling out to an API and has been known to behave oddly under some configurations, up to and including crashes on machines that lack the required specifications. At bare minimum, when trying this out for the first time, save all documents and anything of importance before continuing in case this happens to you.
Start by ensuring that you are able to interact with your preferred vision-capable model using the commandline interface. The steps for doing so are as follows:
- Download the Ollama for Windows setup file from the Ollama downloads page.
- Run this setup file. It will handle grabbing all the dependencies that your machine will need.
- Locate the model you would like to use. A list can be found on ollama.com -> models -> vision, or here directly.
- Download and launch this model by opening a command prompt and typing
ollama run [model_name], of course replacing "[model_name]" with the one you chose in step 3. For example,ollama run llama3.2-vision. - Assuming the process completed successfully, you will be placed in an interactive shell wherein it is possible to type queries and get responses from the model, think a localized (and limited) ChatGPT. Test it out by asking something (anything) to see whether it works, then type "/bye" to exit this interface.
- Back in your console window, type
ollama list. The first column will provide a name like "llama3.2-vision:latest". - Navigate to AI Content Describer settings -> manage models -> Ollama. In the model name field, enter this value and click OK -> OK. You are all set! Switch to Ollama in the model submenu and it should work after some time.
This provider is currently somewhat buggy, and your mileage may vary. It should really only be attempted by advanced users with an interest in running local self-hosted models, and the hardware to do so.
- Download llama.cpp. At the time of this writing, this pull request removes multimodal capabilities so you will want to use the last version with support for this. If you are running on an Nvidia graphics adapter with CUDA support, download these prebuilt binaries: llama-b2356-bin-win-cublas-cu12.2.0-x64.zip and cudart-llama-bin-win-cu12.2.0-x64.zip The steps for working with a different graphics adapter are out of scope, but can be found in the llama.cpp readme.
- Extract both of these files into the same folder.
- Locate the quantized formats of the models you'd like to use from Huggingface. For LLaVA 1.6 Vicuna 7B: llava-v1.6-vicuna-7b.Q4_K_M.gguf and mmproj-model-f16.gguf
- Put these files in the folder with the rest of the llama.cpp binaries.
- From a command prompt, run the llava.cpp server binary, passing the .gguf files for the model and multimodal projector (as follows):
server.exe -m llava-v1.6-vicuna-7b.Q4_K_M.gguf --mmproj mmproj-model-f16.gguf
- In the NVDA settings dialog, scroll down to the AI Content Describer category, then choose "manage models (alt+m)", select "llama.cpp" as your provider, tab into the base URL field, and enter the endpoint shown in the console (defaults to "http://localhost:8080").
- Alternatively, you may omit some of these steps and run llama.cpp on a remote server with higher specs than your local machine, then enter that endpoint instead.
LiteLLM Proxy provides a unified interface to access multiple AI models through a single endpoint, simplifying model management and allowing easy switching between providers.
- Set up a LiteLLM proxy server following the LiteLLM proxy documentation. You can run the proxy locally or use a remote server.
- If running locally, the quickest way to get started is:
pip install 'litellm[proxy]'
litellm --model gpt-4o
This will start a proxy server at http://localhost:4000 that forwards requests to OpenAI's GPT-4o.
3. For production use, create a config.yaml file to configure multiple models and authentication. See the LiteLLM configuration guide for details.
4. In the NVDA settings dialog, scroll down to the AI Content Describer category, then choose "manage models (alt+m)", select "LiteLLM Proxy" as your provider.
5. Enter your proxy server URL in the base URL field (e.g., "http://localhost:4000" for local or your remote server URL).
6. If your proxy requires authentication, enter the API key in the API key field. If not, leave it blank.
7. Click the "List models" button to fetch available models from your proxy, then select the model you want to use from the dropdown.
8. Adjust other settings like prompt, max tokens, and timeout as needed, then click OK.
Note: The models available and their capabilities depend on your LiteLLM proxy configuration. Ensure your proxy is configured with vision-capable models for image description.
Seer runs PaliGemma2 on your own machine with no API key or cloud connection required. Note that this is a captioning-only model; prompts and follow-up questions are not supported.
- Install the Seer daemon. A one-command installer is provided for Windows (.bat) and Linux/macOS (.sh).
- In the NVDA settings dialog, navigate to the AI Content Describer category, choose "manage models (alt+m)", and select "Seer (requires installation)".
- The base URL defaults to
http://127.0.0.1:11435. Leave it as-is unless you changed the daemon's port. - Click OK. The daemon must be running before you attempt a description.
Six hotkeys are bound by default:
- NVDA+shift+i: Pops up a menu asking whether to describe the current focus, navigator object, physical camera, entire screen, or control the computer with AI.
- NVDA+shift+u: Describe the contents of the current navigator object using AI.
- NVDA+shift+y: Describe the image (or file path to an image) on the clipboard using AI.
- NVDA+shift+j: Describe the position of your face in the frame of the selected camera. If you have multiple cameras connected, navigate to the AI content describer menu (NVDA+shift+i) and choose the one you would like to use with the "select camera" item in the face detection submenu.
- NVDA+alt+c: Open the AI conversation dialog to ask follow-up questions.
- NVDA+control+shift+p: Pause or resume an active computer control session.
Four gestures are unbound:
- Describe the contents of the currently focused item using AI.
- Take a screenshot, then describe it using AI.
- Snap a picture using the selected camera, then describe it using AI.
- Cancel the active computer control session.
Don't hesitate to customize these at any time from the input gestures dialog.
Sometimes, the response you get back from an AI is insufficient. Perhaps the image is low quality, incomplete, or includes unwanted details. Maybe you want to hone in on just a certain section or take a clearer photo without losing context. After receiving a description, you can press NVDA+alt+c, or select "Follow-up on previous description" from the AI Content Describer context menu (NVDA+shift+i). By default, the focus is set at the message field. To add an additional image, just keep the conversation window open and use the add-on as you normally would. When a picture is taken (be it from the camera, system control, screenshot, etc.) you will be asked if you want to attach it to the current session or start a new one.
Computer use lets AI Content Describer control the active application with the mouse and keyboard after you describe a task. This can help with tasks that are difficult or inaccessible to complete manually.
Because this gives an AI model control of your computer, use it carefully. The add-on asks for consent before starting, reports actions as they happen, and asks before potentially risky actions. You are responsible for what it does, so monitor the session and avoid using it for sensitive or hard-to-reverse work.
You can pause or resume computer control at any time with NVDA+Control+Shift+P. While paused, the add-on will not continue with the next action until you resume it.
The add-on uses tones to indicate the control state. A high beep means the model is taking control or resuming control. A low beep means control has paused, either because you paused it or because the model needs input. A short low error beep means the session stopped because of an API error.
While a control session is active, the model will announce what it is about to do before sending input. Nothing happens until that speech has finished, unless you press Control to interrupt the announcement and let the action continue sooner.
If the model is unsure how to complete an action, needs more information, or otherwise yields the turn back to you, the session pauses and the Computer Use dialog comes to the foreground. Type a follow-up message there to continue the conversation, or close the dialog to stop the session.
This feature only works with models that provide a computer-use tool: GPT-5.4, GPT-5.4 mini, GPT-5.4 nano, GPT-5.5, GPT-5.5 pro, Claude 4.1 Opus, Claude 4.5 Sonnet, Claude 4.5 Opus, Claude 4.5 Haiku, Claude 4.6 Sonnet, Claude 4.6 Opus, and Claude 4.7 Opus. Other models can describe images, but they do not provide the structured mouse and keyboard actions needed for computer control.
To use computer control:
- Configure and select one of the supported models.
- Open the application or window you want the model to work with.
- Press
NVDA+shift+i, then choose "Control computer...". - Type the task you want performed and press Send.
- Review the consent prompt. If you allow the session, monitor the actions and approve or cancel any risky action prompts.
Note: This should be considered a beta feature, and we are actively taking steps to harden the implementation.
To create the add-on package from source, you will need:
- a Python distribution (3.7 or later is recommended). Check the Python Website for Windows Installers. Please note that at present, preparing the NVDA source code and included third party modules requires the 32-bit version of Python 3.7.
- Scons - Website - version 4.3.0 or later. You can install it via PIP.
pip install scons - Markdown 3.3.0 or later.
pip install markdown
Then open your terminal of choice:
git clone https://github.com/cartertemm/AI-content-describer.git
cd AI-content-describer
scons
After the scons command is finished executing, a *.nvda-addon file will be placed in the root of this repository ready for testing and release.
If you add additional strings that need to be translated, it is important to rebuild the .pot file like so:
scons pot
On a windows machine:
- download poedit. This is the software you will use to translate each message from English.
- download the .pot file with all the strings here
- Open the file you just downloaded in the poedit program. Click "Create new translation" in the window that appears, then select the target language.
- Go through and convert the contents of the source text into the target language, then paste it into the translation field. For extra help, feel free to right click the list item -> code occurrences, then go up a line to read the comment starting with "# Translators: ". These comments are additionally made available in one place in the .pot file.
- When done, click file -> save or press ctrl+s then choose a location for the new .mo and .po file to be stored. These are the files that should be emailed to me or attached in a pull request.
- Translate the contents of readme.md (this file). Attach it too!
All are highly appreciated and will be credited. The following people have worked on the addon.
- Cary Rowen: made the add-on function for Chinese users via dependency mirroring, added support for the Vivo Blue provider
- Mazen: markdown implementation, other code contributions
- Kostenkov-2021: Russian translation
- Nidza07: Serbian translation
- Heorhii Halas and George-br: Ukrainian translation
- Umut Korkmaz: Turkish translation
- Platinum_Hikari: French translation
- Lukas: Czech translation
- Michaela: Slovak translation
Facing a problem? Submit it to the issue tracker
Have a suggestion for a new feature? Create a ticket for that as well, and we can talk about implementing it. Pull requests without associated issues will be reviewed, but are likely to take up more time for everyone, especially if I decide the new fix or functionality needs to work differently than what was proposed.
Translations are welcomed with open arms. The more people that can access this powerful technology, the better!
If you don't have Github, or prefer not to use it, you can shoot me an email - cartertemm (at) gmail (dot) com.
Thanks for the support!