High-risk tools in ScreenHand

Run an AppleScript command. For controlling Finder, Safari, Mail, Notes, etc. (macOS only). WARNING: Executes arbitrary AppleScript — can perform destructive actions (delete fil...

Launch a macOS/Windows application by bundle ID (e.g.,

Execute JavaScript in a Chrome/Electron tab. Returns the result. WARNING: This runs arbitrary JS in the browser context — avoid on sensitive pages (banking, email). All executio...

Navigate the active Chrome/Electron tab to a URL

Wait for a condition on a Chrome/Electron page

Launch an application by bundle ID

Navigate a browser to a URL, or open an app via

Start the observer daemon to continuously watch an app window. Captures frames via CGWindowListCreateImage, runs OCR only when pixels change, detects popups. Zero overhead on en...

Stop the observer daemon.

Start the multi-agent orchestrator daemon. Manages parallel worker slots: web tasks (CDP) run in parallel, native tasks (AX/keyboard) are serialized per-app. Survives restarts.

Stop the orchestrator daemon. Running tasks finish before exit.

Macro recorder: start/stop/trim/clean recorded playbooks. Use

Execute a saved playbook by ID or auto-match by task description. Playbooks run deterministically without AI calls. If a step fails, AI automatically recovers and patches the pl...

Start recording user actions to auto-generate a playbook. Do the task manually while recording, then call recording_stop to save.

Stop recording and save the captured actions as a new playbook.

Start a new automation session. Returns a sessionId needed by all other tools. Automatically attaches to the frontmost app.

Run a complete task autonomously. Starts an observe→decide→act loop that uses the accessibility tree (not screenshots) to see the UI and Claude to decide each action. The loop c...

Wait for a condition: element appears/disappears, text appears, URL changes, window title matches, etc.

Wait until a condition is met on screen: text appears, text disappears, or element becomes available. Polls at intervals using the fallback chain.

Bring a running application to the foreground.

Inject anti-detection patches into Chrome/Electron page. Call once after navigating to a protected site. Hides webdriver flag, patches plugins/languages/permissions.

Fast swipe/flick gesture (for iOS home gesture etc)

Focus/activate an application by bundle ID

Press a key combination

Autonomously explore an app or website. Maps all interactive elements, tries each one, records working selectors and broken paths. Outputs a reference JSON.

Register a dialog watch rule: when a dialog matching the pattern appears, auto-execute an action.

Register a watch rule: when element with matching title appears, execute an action. Use for automated responses to known UI states.

Start the state watcher polling loop. Evaluates registered watch rules every 2s against the world model.

Stop the state watcher polling loop.

Find a UI element by title and press/click it via accessibility

Click an element in Chrome/Electron by CSS selector. Uses CDP Input.dispatchMouseEvent for realistic mouse events.

Fill a form field with human-like typing (anti-detection). Uses real keyboard events via CDP Input domain.

Alias for browser_click — both use realistic mouseMoved → mousePressed → mouseReleased events. Prefer browser_click directly.

Open a URL in Chrome/Electron (creates new tab)

Type into an input field in Chrome/Electron. Uses CDP Input.dispatchKeyEvent for real keyboard events (works with React/Angular).

Click at screen coordinates

Find text on a window via OCR and click it. Handles Retina + shadow coordinate mapping.

Click a target by text using the canonical fallback chain: AX → CDP → OCR. Automatically retries and falls through methods.

Drag from one point to another (slow, smooth)

Send a keyboard shortcut. Keys:

Click a menu item in an app

Submit a task to the orchestrator. Web tasks (CDP) run in parallel, native tasks queue per-app. Returns immediately — task is processed asynchronously.

Click/press a UI element. Finds the element by text, role, selector, or coordinates, then clicks it.

Scroll at a position

Scroll within an element or the active window using the canonical fallback chain: AX → CDP → coordinates. Scrolls until target text is visible, or by a fixed amount.

Select an option from a dropdown/menu using the canonical fallback chain: AX → CDP. Finds the control, opens it, and picks the specified option.

Type text into a UI element (text field, search box, etc). Locates the field, optionally clears it, then types.

Type text using keyboard

Type text into a target field using the canonical fallback chain: AX → CDP → coordinates. Finds the field by label/placeholder, focuses it, then types.

PREFERRED: Find and press/click a UI element by its title via Accessibility. Faster and more reliable than click_text — no screenshot needed.