sonichi/sutando

macos-use

GUI control for macOS apps via mediar-ai's mcp-server-macos-use. Click, type, scroll, key-press, open apps — driven by accessibility tree, works in non-interactive Claude Code mode. Use this for any Sutando task that needs to drive another macOS application (Safari, Zoom, Mail, Finder, etc.).

87CollectingClaude CodeRuns scriptsNetwork access

See how to use it View GitHub source

npx skills add https://github.com/sonichi/sutando --skill "skills/macos-use"

Quick start

Start using it in three steps

Install it or open the source, trigger it with a clear task, then follow the source workflow.

Install the Skill

npx skills add https://github.com/sonichi/sutando --skill "skills/macos-use"

Describe the task

Use macos-use to help me with: [describe your task]. Before you begin, tell me what input you need, the steps you will follow, and the expected output.

Follow the workflow

No structured workflow was detected; follow the original SKILL.md below.

Continue to the workflow

Direct answers

Answers to review before you install

What is macos-use?

GUI control for macOS apps via mediar-ai's mcp-server-macos-use. Click, type, scroll, key-press, open apps — driven by accessibility tree, works in non-interactive Claude Code mode.

Who should use macos-use?

It is relevant to workflows involving Testing, Engineering, Design.

How do you install macos-use?

SkillSignal detected this source-specific command: npx skills add https://github.com/sonichi/sutando --skill "skills/macos-use". Inspect the repository and command before running it.

Which Agent platforms does it support?

claude code

What permissions or risks should you review?

Static analysis detected exec-script, network signals. Review the cited source lines before installing; these signals are not a security audit.

What are the current evidence limits?

This page combines upstream documentation with deterministic repository, quality, and static-risk signals. It is not described as a manual test or security review.

SkillSignal brief

Decide whether it fits your work first

GUI control for macOS apps via mediar-ai's mcp-server-macos-use. Click, type, scroll, key-press, open apps — driven by accessibility tree, works in non-interactive Claude Code mode.

Useful in these contexts

Not yet included in a workflow collection

Core capabilities

TestingEngineeringDesign

Distilled from the source

Understand this Skill in one minute

About 3 min · 7 sections

When it is worth using

•
"Open Safari and navigate to github.com" — anything requiring real GUI interaction with an app
•
"Click the Join button on the Zoom invite dialog"
•
"Type this into the Discord message box"
•
"Scroll the frontmost window to the bottom"

Repository stars

359

Repository forks

Quality

87/100

Source repository last pushed

2026-07-28

Quality breakdown

Based on traceable docs and repository signals; stars are not treated as quality.

87/100

Documentation26/30

Specificity19/25

Maintenance20/20

Trust signals22/25

Compare before choosing

Related Agent Skills and source variants

These links are selected from shared tasks, functions, stacks, platforms, and same-name variants. Compare the source owner, documentation, permissions, and maintenance signals.

ab-testing by coreyhaines31

When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation program

design-review by event4u-app

Use when the user says "review the design", "check the UI", or wants a comprehensive UI/UX review. Uses a 7-phase methodology covering interaction, responsiveness, accessibility, and more.

hexagonal-architecture by affaan-m

Design, implement, and refactor Ports & Adapters systems with clear domain boundaries, dependency inversion, and testable use-case orchestration across TypeScript, Java, Kotlin, and Go services.

playwright-architect by event4u-app

Use when shaping a Playwright suite — locator strategy, Page Object boundaries, fixture composition, flake-prevention architecture, CI-vs-local split — even on 'design our E2E tests'.

playwright-testing by event4u-app

Use when writing Playwright E2E tests — browser automation, visual regression testing, Page Objects, fixtures, and reliable test patterns.

View original Skill.mdThis page is parsed directly from the repository SKILL.md without editorial rewriting. Collected: Jul 28, 2026 · about 3 min

macos-use

Drive macOS applications from Claude Code via mediar-ai's mcp-server-macos-use. A Swift MCP server that wraps the macOS Accessibility API. Unlike Claude's built-in computer-use, this works in non-interactive mode (which is how Sutando's proactive loop and task bridge run), does not hold a machine-wide lock, and does not require a Pro/Max subscription.

When to use

"Open Safari and navigate to github.com" — anything requiring real GUI interaction with an app
"Click the Join button on the Zoom invite dialog"
"Type this into the Discord message box"
"Scroll the frontmost window to the bottom"
Any task that currently falls back to AppleScript + Quartz mouse events in src/inline-tools.ts

Prefer this skill over:

bash src/screen-capture.sh — that captures screenshots; macos-use actually interacts
AppleScript tell application blocks — more reliable, better error handling
cliclick — lower-level, no accessibility context
Claude's built-in computer-use — that mode requires interactive sessions and holds a lock that contends with Sutando's own loop

Tools exposed

After install, these appear as mcp__macos-use__* in Claude Code:

Tool	Parameters	Purpose
`open_application_and_traverse`	`identifier` (name/bundle ID/path)	Launch or activate an app, return its a11y tree
`click_and_traverse`	`pid`, `x`, `y`	Click at coordinates in a target app, return updated tree
`type_and_traverse`	`pid`, `text`	Type into the frontmost element
`press_key_and_traverse`	`pid`, `key`	Press a named key (Return, Tab, Escape, arrows, ...)
`scroll_and_traverse`	`pid`, `direction`, `amount`	Scroll in a direction
`refresh_traversal`	`pid`	Re-read the a11y tree without acting

Every tool returns an accessibility-tree snapshot of the target app — structured UI elements with roles, titles, positions, and identifiers. No pixels. Model reasons over the tree, not over screenshots.

Install

Two steps, one-time:

# 1. Build the Swift binary (~35s)
bash skills/macos-use/scripts/build.sh

# 2. Register with Claude Code's MCP config (writes ~/.claude.json)
bash skills/macos-use/scripts/install-mcp.sh

# 3. Grant Accessibility permission
#    System Settings → Privacy & Security → Accessibility
#    Click +, navigate to ~/.macos-use-mcp/.build/release/mcp-server-macos-use, enable.

Restart Claude Code after install for the MCP tools to appear.

Gotchas

Swift 6 build fragility: the swift-sdk transitive dep has data-race errors that Swift 6.3+ strict-concurrency trips on. build.sh uses -Xswiftc -swift-version -Xswiftc 5 as a workaround. When upstream fixes this, remove the flag.
Accessibility permission: the binary must be added to System Settings → Privacy & Security → Accessibility, or every tool call will return "not authorized". First-run error is obvious; owner must click through once.
Apps without good a11y trees: Canvas / Electron / games degrade badly. For those, fall back to screen-capture.sh + Claude vision.
Build dep pulled from GitHub: air-gapped Macs won't work. No prebuilt releases yet.
Multi-node: each node builds its own binary. Not synced via sutando-memory.git (binaries are machine-specific). Run build.sh + install-mcp.sh on Mac Mini and MacBook separately.

Quick self-test

After install + restart:

Sutando, open Safari and navigate to https://github.com/sonichi/sutando

You should see Claude invoke mcp__macos-use__open_application_and_traverse with identifier: "Safari", then type_and_traverse into the URL bar, then press_key_and_traverse with Return.

Research + decision memo: notes/issue-65-computer-use-research.md
Issue: #65 Add Claude Computer Use support
Upstream: https://github.com/mediar-ai/mcp-server-macos-use

Source repo: sonichi/sutando
Skill path: skills/macos-use/SKILL.md
Commit SHA: 6a8f0fccd32e
Repository license: MIT
Data collected: Jul 28, 2026