SwiftAutoGUI

A Swift library for macOS automation — mouse, keyboard, screenshots, image recognition, and AI-powered agents.

This repository is inspired by pyautogui.

Demo

AI Agent that autonomously observes the screen and executes actions to achieve a goal.

sagui agent "Open Safari and search for Swift"

Requirements

• macOS 26.0+
• Swift 6.0+

Installation

Swift Package Manager

SwiftAutoGUI is available through Swift Package Manager.

in Package.swift add the following:

dependencies: [
    // Dependencies declare other packages that this package depends on.
    .package(url: "https://github.com/NakaokaRei/SwiftAutoGUI", branch: "master")
],
targets: [
    .target(
        name: "MyProject",
        dependencies: [..., "SwiftAutoGUI"]
    )
    ...
]

Homebrew (sagui CLI)

The sagui command-line tool is available via the Homebrew tap:

brew install NakaokaRei/tap/sagui

After install, grant Accessibility permission to your terminal in System Settings → Privacy & Security → Accessibility.

sagui --version
sagui key list
sagui key shortcut return
sagui mouse click --x 100 --y 200
sagui mouse click --right --x 100 --y 200

Example Usage

If you would like to know more details, please refer to the DocC Style Document.

AI Agent

SwiftAutoGUI includes an Agent that can autonomously observe the screen, reason about what it sees, and execute actions in a loop until a goal is achieved. This follows the ReAct (Observe → Think → Act) pattern using a vision-capable LLM.

import SwiftAutoGUI

let backend = OpenAIVisionBackend(apiKey: "sk-...", model: "gpt-5.6-sol")
let agent = Agent(
    backend: backend,
    maxIterations: 15,
    visionMode: .automatic
)

let result = try await agent.run(goal: "Open Safari and search for Swift")
print("Completed: \(result.completed), Steps: \(result.iterationsUsed)")

The CLI prints the effective reasoning effort when it starts and the model-provided reasoning summary for every agent step:

sagui agent "Open Safari and search for Swift" --reasoning-effort low
sagui agent "Press the Save button" --vision-mode automatic

--reasoning-effort accepts none, low, medium, high, xhigh, or max. It defaults to low for GPT-5.6 models. The per-step Reasoning: line is the agent's concise explanation of its chosen actions, not the model's hidden chain of thought.

When screen context is enabled, actionable Accessibility elements receive step-local identifiers such as [#12]. The agent can target these identifiers directly, resolves them again immediately before execution, and rejects stale elements safely. Each action returns a structured result describing the execution method, failure, UI change, and focus change. If an action changes the UI, the remaining batch is stopped and the agent observes the new state before continuing.

--vision-mode accepts always, automatic, or never. automatic omits the screenshot when actionable Accessibility elements are available; always preserves the original behavior and remains the default.

Basic Usage

import SwiftAutoGUI

// Execute single actions
await Action.leftClick.execute()
await Action.write("Hello, World!").execute()
await Action.keyShortcut([.command, .a]).execute()  // Select all

// Build and execute action sequences
let actions: [Action] = [
    .move(to: CGPoint(x: 100, y: 100)),
    .wait(0.5),
    .leftClick,
    .write("Hello, SwiftAutoGUI!"),
    .keyShortcut([.returnKey])
]
await actions.execute()

Claude Code Plugin

SwiftAutoGUI ships as a Claude Code plugin so Claude can control macOS GUI applications via the sagui CLI — taking screenshots, clicking buttons, typing text, scrolling, and more.

Install from the marketplace

Inside Claude Code:

/plugin marketplace add NakaokaRei/SwiftAutoGUI
/plugin install swift-auto-gui@swift-auto-gui

This installs the macos-control skill, which is invoked as /swift-auto-gui:macos-control. The skill walks Claude through installing the sagui binary the first time it's needed (Swift 6.2+ toolchain required).

Permissions

Grant the application running Claude Code (Terminal.app, iTerm, etc.) both:

• Accessibility — System Settings → Privacy & Security → Accessibility
• Screen Recording — System Settings → Privacy & Security → Screen Recording

For full skill details, see plugins/swift-auto-gui/skills/macos-control/SKILL.md.

Contributors

• NakaokaRei

License

MIT license. See the LICENSE file for details.