v0.22.4

What's Changed

• Feat/Add annotation function by @mlikasam-askui in #188
• docs: add docstring to AndroidVisionAgent class by @mlikasam-askui in #192
• feat(chat): select model of run with request params by @adi-wan-askui in #193

🚀 Features

• Element Annotation:

  • annotate() method: Generate interactive HTML files that visualize detected UI elements on screenshots. The generated HTML allows users to:
    • View bounding boxes around all detected elements
    • Hover over elements to see their names and text values
    • Click on elements to copy their text values to the clipboard

  from askui import VisionAgent
  
  with VisionAgent() as agent:
      # Annotate current screen and save to default 'annotations' directory
      agent.annotate()
  
      # Or specify custom screenshot and output directory
      agent.annotate(screenshot="screenshot.png", annotation_dir="htmls")

  Also works with AndroidVisionAgent:

  from askui import AndroidVisionAgent
  
  with AndroidVisionAgent() as agent:
      agent.annotate()

  • locate_all_elements() method: Retrieve all detected elements programmatically as a list of DetectedElement objects:

  from askui import VisionAgent
  
  with VisionAgent() as agent:
      detected_elements = agent.locate_all_elements()
      print(f"Found {len(detected_elements)} elements: {detected_elements}")
  
      # Access element properties
      for element in detected_elements:
          print(f"Name: {element.name}, Text: {element.text}")
          print(f"Position: {element.center}, Size: {element.width}x{element.height}")

  • New Data Models:
    • DetectedElement: Represents a detected UI element with name, text, and bounding_box properties, plus convenience properties for center, width, and height
    • BoundingBox: Represents element coordinates with xmin, ymin, xmax, ymax, plus convenience properties for width, height, and center

• Chat API Model Selection: Chat runs can now specify which model to use via the model parameter in the run creation request, allowing dynamic model selection per run instead of using only the configured default model.

📜 Documentation

• AndroidVisionAgent class: Added comprehensive docstring with detailed parameter descriptions and usage examples for the AndroidVisionAgent class (src/askui/android_agent.py:41-62).

Full Changelog: v0.22.3...v0.22.4