claudegoodies
Skill

computer-use

From NousResearch

|

Facts

Repository
NousResearch/hermes-agent
Status
Actively maintained
Last commit
Source file
skills/computer-use/SKILL.md

Source preview

The instructions Claude Code reads when this skill runs.

# Computer Use (universal, any-model, cross-platform)

You have a `computer_use` tool that drives the user's desktop in the
**background** — your actions do NOT move the user's cursor, steal
keyboard focus, or switch virtual desktops / Spaces. The user can keep
typing in their editor while you click around in a browser in another
window. This is the opposite of pyautogui-style automation.

Everything here works with any tool-capable model — Claude, GPT, Gemini,
or an open model on a local OpenAI-compatible endpoint. There is no
Anthropic-native schema to learn.

Hermes drives [cua-driver](https://github.com/trycua/cua) under the hood
for the platform plumbing. The Hermes-side `computer_use` tool exposed
in this skill is a higher-level Hermes vocabulary; the raw cua-driver
MCP tools (which a different agent harness would see) are NOT what you
call — call the `computer_use` actions documented below.

## The canonical workflow

**Step 1 — Capture first.** Almost every task starts with:

```
computer_use(action="capture", mode="som", app="<the app you're driving>")
```

Returns a screenshot with numbered overlays on every interactable
element AND an AX-tree index like:

```
#1  AXButton 'Back' @ (12, 80, 28, 28) [Chrome]
#2  AXTextField 'Address bar' @ (80, 80, 900, 32) [Chrome]
#7  Link 'Sign In' @ (900, 420, 80, 24) [Chrome]
...
```

The role names match the host platform's acces
View full source on GitHub →

Other skills