Browser Use — act inside the user's real Chrome
Wisp does not launch an automation browser. It talks to a small
extension inside the user's own Chrome/Chromium, so every action runs in
their real profile — existing cookies, logins, extensions, and normal
fingerprint all apply. That is the whole point: you can operate pages the
user is already signed into.
Every web_scan and web_execute_js call needs the user's approval by
design. Do not treat that as a bug to route around.
Before anything: confirm the bridge is live
Call browser_setup. If status is not connected, relay its steps
(load the unpacked extension from extension_path, verbatim) and stop
until the popup shows Connected to Wisp. Never invent the path.
The loop
web_open_tab {url} — open the page (works even with no tab
open yet). Returns the new tab id.
web_scan — read the page. Returns page.text, page.title, and
page.elements[], where each element carries a unique selector,
its visible text/aria_label, and a rect [x,y,w,h]. Use these
selectors directly — do not guess. Use tabs_only:true first when you
are unsure which tab to target; pass switch_tab_id:<id> to pin one.
web_execute_js — act, then re-scan to confirm the effect.
Recipes (web_execute_js script)
| Goal | script |
|---|
| Click | document.querySelector('<selector>').click() |
| Type into a field | const e=document.querySelector('<sel>'); e.value='text'; e.dispatchEvent(new Event('input',{bubbles:true})); e.dispatchEvent(new Event('change',{bubbles:true})) |
| Submit a form | click the submit control by its selector, then re-scan |
| Navigate current tab | location.href='https://example.com' |
| Read a value | document.querySelector('<sel>').textContent |
script may instead be a JSON command:
| Goal | JSON command |
|---|
| Switch to & focus a tab (so the user sees it) | {"cmd":"tabs","method":"switch","tabId":<id>} |
| List tabs | {"cmd":"tabs"} (or just web_scan tabs_only) |
| Close tabs you opened | {"cmd":"tabs","method":"close","tabIds":[<id>,...]} — returns closed + remaining |
Trusted click when .click() is ignored | {"cmd":"cdp","method":"Input.dispatchMouseEvent","params":{"type":"mousePressed","x":<x>,"y":<y>,"button":"left","clickCount":1}} then the same with "type":"mouseReleased" — use the element's rect centre from web_scan |
Prefer plain JS. Reach for cmd:cdp only when a page blocks synthetic
events or you truly need trusted input.
Seeing the page — web_screenshot
web_scan gives text and elements; web_screenshot gives sight. Use it
when structure isn't enough: rendered layout, a chart or diagram, a
canvas/WebGL page, a QR code, a PDF or image viewer, or a page that looks
broken. It captures the visible viewport of the tab — to see below the
fold, scroll first (web_execute_js scrollTo(0, 1200)) and capture again.
Pass question to say what to read out of it, e.g.
{"question":"is the login QR code visible and not expired?"}.
It goes through the configured vision model, so web_scan stays the cheaper
default — screenshot when you need eyes, not for every step.
Tab hygiene — track what you open, offer to close it
Browsing tasks (searching papers, opening a dozen results) leave the user
with a pile of tabs to close by hand. So:
- Every
web_open_tab returns tab.id. Keep a running list of the ids
you opened in this task, in your own message text — e.g. after a batch
write opened tabs: 1234, 1235, 1236. {"cmd":"tabs"} cannot tell you
which tabs are yours, only what exists.
- When the task is done, before your final answer, ask the user:
name the count and offer to close them, e.g. "我为这次检索开了 6 个标签
页,需要我关掉吗?" Do not close anything without a yes.
- On a yes, close them in one call:
{"cmd":"tabs","method":"close","tabIds":[1234,1235,1236]}. Report
closed; ids already gone are skipped silently.
Close only ids you opened yourself. Tabs the user had open, or ones
they opened during the task, are theirs — never include them, and never
close a tab mid-task that later steps still need.
Stop conditions (do not automate through these)
- Human verification / CAPTCHA: if
web_scan returns
human_intervention.required=true, stop, ask the user to complete the
challenge in the visible tab, and wait for their confirmation before
scanning again.
- Credentials: never type passwords, card numbers, or one-time codes
yourself. If a step needs a password, have the user sign in directly in
the browser and continue once they confirm.
- Irreversible / outward actions (send, pay, post, delete): confirm
with the user before clicking the control.
- Downloads: for multiple-file downloads, first surface the browser
settings from
browser_setup (download_automation) and wait for the
user to confirm; until then trigger at most one download.