ego-browser

ego-browser

ego-browser gives AI agents a CLI-accessible Node.js runtime, with built-in helpers — snapshotText, click, js, cdp, and more — that agents call directly inside JS scripts to observe pages, interact with UI, evaluate browser-side JavaScript, and drive a real browser for any web automation task.

For setup, install, or connection problems, read references/install.md.

Use the Bash tool to run all browser operations via ego-browser nodejs <<'EOF' ... EOF heredoc. Do not write code to a .js file first.

Quick start

ego-browser nodejs <<'EOF'
// Name the task space for the whole user task, then reuse that space across heredoc rounds.
const task = await useOrCreateTaskSpace('inspect example page')
cliLog('task space id: ' + task.id)

await openOrReuseTab('https://example.com', { wait: true, timeout: 20 })

cliLog(await snapshotText())
EOF

The heredoc body runs as a Node.js script that controls the selected ego-browser task space. All ego-browser helpers are preloaded into that script.

Common helpers

• Task spaces: listTaskSpaces, useOrCreateTaskSpace, claimTaskSpace, handOffTaskSpace, takeOverTaskSpace, waitForAgentControl, completeTaskSpace
• Navigation / state: listTabs, openOrReuseTab, closeTab, gotoAndWait, currentTab, switchTab, gotoUrl, pageInfo, ensureRealTab
• Observation: snapshotText, captureScreenshot, drainEvents
• Scroll / mouse: scrollBy, scrollToBottomUntil, scroll, click, doubleClick, hover, dragMouse
• Keyboard & input: typeText, fillInput, pressKey, dispatchKey
• File: uploadFile
• Wait: wait, waitForLoad, waitForElement, waitForNetworkIdle
• Fetch: serverFetch, browserFetch
• CDP / evaluate: js, cdp
• Output: cliLog, help

Notes:

• cliLog(value) — prints to the terminal; it is the only output mechanism inside a heredoc, and all final results must go through it.
• await pageInfo() — normally resolves to { url, title, w, h, sx, sy, pw, ph }; if a native browser dialog is open, resolves to { dialog: ... } instead because page JavaScript is blocked.
• If await pageInfo() resolves to { dialog: ... }, handle the dialog with await cdp('Page.handleJavaScriptDialog', { accept: true }) or accept: false before running page JavaScript.
• await ensureRealTab() — switches to an existing non-internal page tab if needed and resolves to it; resolves to null when none exists. It does not create a tab — use await openOrReuseTab(...) for that.
• await closeTab(target?) — closes the given target id / tab object, or the current tab when omitted.
• await drainEvents() — consumes and returns the async event queue produced by the page (navigation events, network events, etc.).
• await serverFetch(url, options) — issues a request from Node and returns the response body.
• await browserFetch(url, options) — issues a request from the current browser page context and returns the response body.
• help(name) — prints usage for a given helper, e.g. cliLog(help('click')).

Task spaces

A task space is an isolated browsing context that ego-browser provides for AI Agents. Each task space has its own set of tabs but inherits the current user's login state by default, so Agents can operate on authenticated sites without competing with or disturbing the user's normal browser windows.

Closing all tabs in a task space is equivalent to closing that task space.

A task often takes multiple heredoc rounds to complete. Because the Node.js runtime exits after each heredoc and retains no state, normal working heredocs should start with an explicit call to useOrCreateTaskSpace(nameOrId) to reuse the same space — this lets you operate continuously and reuse tabs across rounds. The exception is resuming after a handoff: once the user confirms "continue" (through an Ask or in chat), start the next heredoc with takeOverTaskSpace(nameOrId) instead.

nameOrId can be a task space name, numeric id, or digit-only numeric id string. String values match name/taskId first, then digit-only strings fall back to numeric id. Number values match existing numeric ids only; if no matching id exists, useOrCreateTaskSpace fails instead of creating a new space.

Use a short name for the active user goal when creating a new task space. Keep reusing that task space for follow-up questions, corrections, refinements, re-checks, and result validation, even if you previously thought the task was complete. Choose a new task space only when the user clearly starts a separate, unrelated goal. Prefer using the numeric id returned by useOrCreateTaskSpace (for example, task.id) to resume a known task in later rounds and avoid name collisions.

For any follow-up on the same user goal — including continue, corrections, retries, validation, user-reported problems, or work after completeTaskSpace(..., { keep: true }) — resume the original task space first if it still exists. Do not create a new task space for the same goal unless the user asks for a fresh space, starts an unrelated goal, or the original space is unavailable after checking. If a new space is necessary, state why.

After explicit user confirmation, to continue work from an existing user-owned, inactive, or unassigned task space, use await listTaskSpaces() to find the space, call await claimTaskSpace(id) to take ownership and select it, then use await listTabs() and await switchTab(targetId) to select the exact tab before acting.

Ownership policy — every task space has ownership: 'agent' | 'agentDelegatedToUser' | 'user'; the helpers treat user-owned spaces differently:

handOffTaskSpace and completeTaskSpace resolve { done: true } when the operation actually happened. Check done before telling the user the handoff/cleanup is finished — a skipped result usually means you targeted a space that was never yours.

completeTaskSpace(nameOrId, { keep }) must occupy its own dedicated final heredoc, and run only after a prior heredoc's output has confirmed the task is genuinely done. keep is required and defaults by policy to false: close the task space after completion unless there is a concrete reason to leave the live page visible.

Use { keep: true } only when the user explicitly asks to keep the page open, the task needs manual user action in that exact page, or the result cannot be delivered well as a URL, file, artifact, or summary. Do not keep a task space open merely because a page was visited, a document was created, or a screenshot was used for verification.

When passing a string that may create a new task space, the string should reflect the task's intent (e.g. 'search github issues'); don't use literal placeholders.

If the task space needs to be preserved after the task ends, keep only the tabs that need to be shown to the user. Keep loose awareness of how many tabs are open — a quick (await listTabs()).length is enough; there's no need to spend a dedicated round just to check. When scratch tabs (search-result pages, cross-check pages, and other one-off pages) pile up, close them as you go rather than letting them all accumulate for the end. When finishing with { keep: true } to leave pages for the user, clear out the remaining scratch tabs so only the pages worth showing stay open. Close a single tab with await closeTab(targetId) (targetId comes from listTabs() or an openOrReuseTab return value).

Control handoff

Only one side — agent or user — holds control of a task space at any time. While the user holds control, any browser operation by the agent fails with a "user is controlling" message — do not retry it; follow the steps below to resume.

A "user is controlling" error is a hard stop on the whole task — not an obstacle to route around. It means the user has deliberately taken the browser back, often because your current approach is going wrong. Honoring it is the correct outcome here; pushing the goal forward anyway is the failure. The only thing you may do is ask the user and wait.

An "inactive", "not assigned to an agent", or similar task-space error is also a hard stop with the same confirmation requirement. Resume only after explicit user confirmation, then start with await claimTaskSpace(id).

Handing off: When the task requires user intervention (e.g. login, captcha, manual confirmation), call await handOffTaskSpace([nameOrId]) to give control to the user, and tell them exactly what to do. Omitting nameOrId uses the currently selected task space; pass task.id across heredoc rounds to avoid ambiguity.

Regaining control: Take control back only after the user explicitly confirms — through an Ask (your harness's button/option prompt, e.g. "Continue" vs "Finish task") or a "continue" message in chat. Then start a new heredoc with await takeOverTaskSpace([nameOrId]) and resume; if the user chooses to finish, close out with await completeTaskSpace(nameOrId, { keep }). Never call takeOverTaskSpace on your own to grab control back — it has no ownership check and will seize the browser away from the user.

Unexpected takeover: The user can take over at any time via the browser GUI — the same effect as the agent calling handOffTaskSpace. Do not retry the failed operation and do not auto-takeover; surface the Ask above (Continue / Finish) and resume only when the user picks Continue.

await waitForAgentControl(nameOrId) is a read-only blocking poll (it never takes control); use it only to wait inside the current heredoc for a handoff you initiated.

Scroll / mouse

// DOM scroll
await scrollBy(900)
await scrollToBottomUntil(
  async () => await js(String.raw`document.querySelectorAll('article').length`) >= 20,
  { step: 900, wait: 1, maxSteps: 20 }
)

// Real wheel event
await scroll({ dy: 900 })

Element-target helpers such as click, doubleClick, hover, dragMouse, fillInput, uploadFile, and waitForElement accept the same selector/ref surface: raw CSS, xpath=..., @N / ref=N, and loc=... values from snapshotText() (loc=css:..., loc=role:..., loc=href:...). @N refs are for ego-browser helpers only; they are not valid selectors inside document.querySelector(...).

click, doubleClick, hover, and dragMouse share these target formats. Coordinates are in CSS pixels:

• string — CSS selector, xpath=..., @N / ref=N, or loc=...; clicks the element's center.
• [x, y] or {x, y} — viewport coordinates.
• {selector} — CSS selector, xpath=..., @N / ref=N, or loc=...; clicks the element's center.
• {selector, x, y} — offset from the element's top-left corner by x/y.
• options.label (optional) — a 3-6 word action description; triggers a visual highlight animation.

await click('@21', { label: 'check login status' })
await click('button.primary', { label: 'click submit button' })
await click([420, 260])
await click({ x: 420, y: 260 })
await click({ selector: 'canvas#stage', x: 12, y: 8 })
await hover('@5', { label: 'hover to reveal menu' })
await dragMouse([from, to], { label: 'drag card' })

uploadFile

await uploadFile('input[type="file"]', "/absolute/path/to/file.pdf")

js

js() is essentially Runtime.evaluate and takes a string. You can pass a function, but doing so triggers a one-time warning and wraps it via .toString() — closures are not captured and there is no argument channel. Do not use js() the way you would Puppeteer / Playwright's page.evaluate(fn, ...args).

When you need to run multi-step logic inside the browser, wrap it in a single self-invoking closure and return once — don't split it across multiple await js() calls:

const data = await js(String.raw`(() => {
  const items = [...document.querySelectorAll('article')]
  return items.map(el => ({
    text: el.innerText,
    links: [...el.querySelectorAll('a')].map(a => a.href),
  }))
})()`)

Recommended workflow

ego-browser has three main workflows. Pick the workflow that fits the page and task before acting.

Use the semantic workflow first for ordinary websites with real DOM controls. For canvas-like productivity apps and rich editors — including Google Docs, Google Sheets, Lark/Feishu Docs, Notion, Figma, whiteboards, maps, and other virtualized editors — use the visual workflow first for the main editing surface. These apps often expose toolbars, title inputs, hidden textareas, offscreen iframes, or canvas layers in the DOM that do not represent the actual user-editable document or grid. Do not rely on await fillInput(...), DOM selectors, or snapshotText() refs for the main editing surface unless a small write probe proves the text lands in the intended place.

Before writing substantial content into a rich editor, perform a tiny write probe, then verify it with await captureScreenshot(), an export/readback path, or another reliable visual/state check. If the probe appears in the title bar, toolbar search, hidden input, or any wrong field, stop using DOM/input helpers for that surface and switch to screenshot-guided mouse actions plus real keyboard operations.

1. Semantic workflow: snapshotText() + refs / locators — default for most pages with normal text, links, buttons, forms, tables, and lists.

   • Reuse or create a task space: const task = await useOrCreateTaskSpace(name).
   • Open or switch pages with await openOrReuseTab(url, { wait: true }); use await gotoAndWait(url, { timeout, settle }) only when navigating inside the current tab.
   • Observe with await snapshotText() to get a full-page semantic tree annotated with [ref=N, loc=..., url=...].
   • Act with await click('@N'), await fillInput('@N', ...), or stable loc=... values. Use direct DOM logic only when it is simpler than helper calls.
   • After meaningful clicks, input, or navigation, observe again with await snapshotText(), await pageInfo(), or await captureScreenshot() before assuming success.

2. Visual workflow: await captureScreenshot() + coordinate/keyboard actions — use when the page is primarily visual, canvas-like, heavily virtualized, or when accessibility / semantic structure is incomplete.

   • Inspect the screenshot, act with viewport coordinates such as await click([x, y]), await doubleClick([x, y]), await pressKey(...), and await typeText(...), then verify with another screenshot or a reliable export/readback path.
   • Prefer this path for rich editors, spreadsheets, visual menus, map/canvas UIs, drag interactions, and targets that are obvious visually but poor in the DOM/AX tree.

3. Direct DOM / CDP workflow: await js(...) / await cdp(...) — use when you need browser state, compact data extraction, custom DOM traversal, or raw browser capabilities.

   • Keep browser-side logic in one explicit IIFE and return once.
   • Use await cdp(...) for browser protocol operations that helpers do not cover.

These workflows can be combined. A task may take multiple heredoc rounds when the next step depends on fresh page state or user handoff. In each round, write a coherent script that advances the task: observe, act or extract, verify, and report with cliLog(...). Avoid tiny probe scripts, but don't force the whole task into one oversized script.

Caveats

• wait(...) and timeout values are in seconds; only parameters whose names end in Ms are milliseconds.
• snapshotText() defaults to scope: 'full_page', covering the whole page. Use the default in almost every case; only pass scope: 'only_within_viewport' when the task needs only visible content.
• @N refs are only valid for the most recent snapshotText call — every call rebuilds the refMap. Ref numbers come from the CDP backendNodeId, so the same element keeps the same number across calls; but to use @N, N must appear in the latest snapshotText output. An element scrolled out of the viewport, a DOM re-render, or a previous call with scope:'only_within_viewport' that didn't cover the element will all cause Unknown ref. For elements you need to reference long-term, use the loc=... value from snapshotText output as a stable selector, or write a CSS selector directly.
• js() returns the evaluated result, not a JSON string — don't wrap it with JSON.parse(...).
• Inside a js(...) template string, regex backslashes must be doubled (e.g. \\d, \\s), or use String.raw.
• If the source passed to js() contains a top-level return, it will be auto-wrapped in an IIFE; return inside nested callbacks can also trigger this accidentally. For complex expressions, prefer the explicit (() => { ... })() form.
• If await pageInfo() reports w: 0 or h: 0, do not continue coordinate actions or screenshots until the viewport is fixed. Try switching to the real tab, reloading, or using CDP viewport metrics, then verify with await pageInfo() and await captureScreenshot().
• Code in the heredoc body runs in Node.js; code inside js(...) runs in the browser page. Navigation, waits, and cliLog(...) belong in the heredoc body; document, window, and page selectors belong inside js(...).
• Always call completeTaskSpace(name, { keep }) when the task is done — do not leave the space hanging. Default to { keep: false }; use { keep: true } only for the concrete live-page cases described in Task spaces.
• When the user explicitly asks to use ego-browser, assume both ego-browser and the repo runtime are ready. Do not pre-check which ego-browser, node -v, package metadata, or help output. Only investigate environment issues if the first run produces an error.
• If the first run reports command not found / a missing environment (most likely ego lite isn't installed yet), or the user explicitly asks to install ego lite, first read references/install.md and follow its flow to complete the install, then return to the original task — do not give up, and do not keep retrying the same heredoc.