browser-skill

browser-skill

Drive the user's real Chromium browser (with their logins and cookies) through the bsk CLI. The extension opens an isolated Agent Window for automation; the user's normal windows stay protected unless you explicitly borrow a tab.

When to use

• Open pages, read titles/text, scrape structured data from sites the user can already access
• Fill forms, click through multi-step flows, smoke-test a UI change
• Understand pages with bsk snapshot first; use bsk get-html or bsk screenshot only when the snapshot is insufficient
• Operate on a specific user tab they point you at (after bsk tab borrow)

When NOT to use

• Tasks with no browser involved (files, APIs, databases only)
• Installing or configuring the extension (point the user to setup docs instead)
• Credential harvesting — never run bsk evaluate on banking, SSO, or password-manager pages to extract tokens, cookies, or secrets
• Long-lived control of a user's personal login window — borrow only for the immediate step, then bsk tab return or end the session
• Replacing the user's manual browsing when they only wanted an explanation

Prerequisites

1. bsk on PATH (Rust CLI from browser-skill)
2. browser-skill extension loaded in Chromium and connected (popup shows green)
3. Any bsk command auto-starts background services as needed; use bsk doctor if anything fails

Mandatory workflow

Every automation task must follow this lifecycle. Do not rely on idle timeouts (default session idle is 5 minutes).

1. bsk session start              → capture the 4-letter session id printed on stdout
2. … every tool command …        → always pass --session <id>
3. bsk session stop <id>          → REQUIRED when done (even on error paths)

Optional: bsk session start --browser <instance-id-or-label> when multiple browsers are connected (bsk browsers / error output lists them). Add --no-focus to open the Agent Window in the background without stealing focus from the user's current window.

Emergency cleanup: bsk session stop --all or the Agent Window overlay Stop all.

Stop when the goal is met

Every task is a bounded goal, not open-ended browsing. The goal may come from the user's request, a recorded trace.json, or both.

1. Define success first — one concrete, observable condition derived from the user's words, purpose, or the last meaningful step in a trace (e.g. "form submitted", "item added to cart", "playback started").
2. Take the shortest path — snapshot → act → at most one check. Do not wander, re-try unrelated actions, or stack exploratory steps.
3. Stop as soon as success is reached — run bsk session stop <id> immediately unless the user explicitly asked to keep the session open (e.g. "don't close yet", "keep browsing").
4. No post-success work — once the goal is met, do not click, refresh, navigate, re-search, switch tabs, or "double-check" that it worked. Further verification is a new task.
5. When blocked, pause — do not brute-force — if the page requires human input (login, captcha, OTP, payment confirmation) or an action fails twice with no progress, call bsk request-help instead of retrying blindly. See Ask the human for help below.
6. When unsure — at most one extra bsk snapshot. If success looks met, stop. If not, ask the user; do not keep clicking.

With a trace: replay steps in order using target role/name/tag and raw value/selection fields. After the last step (or when its effect.navigated_to / success hint is satisfied), apply rules 3–4 immediately. The trace guides execution; it does not extend control beyond the goal.

Without a trace: the user's request is the success condition. Satisfying it ends the task — same stop rules apply.

Core interaction loop

Write operations only affect tabs in the Agent Window (or tabs you borrowed into it).

bsk navigate <url> --session <id>
bsk observe --session <id>           → primary semantic VOM view; reveals hover/focus surfaces
bsk snapshot --session <id>          → static aria tree fallback when VOM is insufficient
bsk hover @e3 --session <id>          → reveal hover-triggered menus before re-observing/clicking
bsk click @e4 --session <id>          → or bsk fill, bsk select, bsk press
bsk observe --session <id>             → again after navigation / DOM change

Refs invalidate after navigation — always re-snapshot before clicking, filling, or selecting on a new page.

Prefer @eN refs from the latest snapshot over raw CSS selectors. Use --ref / --selector when ambiguous (bsk click --help).

When VOM renders [hover first: …] on an element, the listed items are not currently clickable refs. Run bsk hover <that-ref> --session <id>, then immediately run bsk snapshot or bsk observe again and click the newly visible menu item ref. Do not click the trigger itself unless the user explicitly wants the trigger action.

Observation priority

Start with bsk observe to understand page structure, text, controls, element refs, and conditional hover/focus surfaces. Use bsk snapshot only when you need the stricter static accessibility tree or VOM is insufficient. Only escalate to raw HTML or screenshots when the latest observation cannot answer the question:

1. bsk observe — primary semantic VOM observation; may run bounded perception probes such as hover-surface discovery
2. bsk snapshot — strict static accessibility tree fallback
3. bsk get-html — when hidden DOM, metadata, or markup details are required
4. bsk screenshot — when visual layout, canvas/image content, or styling cannot be inferred from the observation. Use --ref @eN (from the latest snapshot/observe) to crop to one element; omit --ref for the full visible tab.

Do not call bsk get-html or bsk screenshot first just to inspect a page.

Sandbox rules

Global flags

Auto-update is on by default — the background daemon upgrades bsk itself when a new release is available (postponed while a session is active). Set BSK_AUTO_UPDATE=off to disable it and upgrade manually with bsk update.

Command-specific flags (timeouts, --tab-id, --wait-until, …): bsk <cmd> --help

CLI command reference (one line each)

Details and flags: bsk <cmd> --help

Diagnostics

Session

Window (require --session <id>)

Device emulation — bsk emulate (requires --session <id>)

Emulate a mobile device environment on the agent tab via CDP — viewport, User-Agent, and touch — to debug a page's mobile layout and behaviour:

bsk emulate --session <id> --device iphone-14
bsk emulate --session <id> --width 390 --height 844 --dpr 3 --mobile --ua "Mozilla/5.0 (iPhone…" --touch
bsk emulate --session <id> --off

• Presets (--device): iphone-14, iphone-14-pro-max, iphone-se, pixel-7, galaxy-s23, ipad-mini, galaxy-tab-s8. Manual flags (--width/--height/--dpr/--mobile/--ua/--accept-language/--touch/--max-touch-points) also work without a preset, or override individual preset fields; --no-mobile/--no-touch turn a preset's mobile viewport / touch emulation back off.
• Repeated runs merge field by field onto the tab's current emulation state — only the flags you pass change. E.g. after --device iphone-14, bsk emulate --session <id> --width 390 --height 844 keeps the preset's dpr (3) and mobile viewport. (The extension remembers the state per tab; after an extension reload the next run applies only the fields it carries.)
• --off clears every override (viewport, UA, touch) and the remembered state, restoring the tab's real environment.
• Scope: overrides apply to one tab only (CDP per-target) and are not inherited by new tabs — re-run bsk emulate after opening or switching to another tab (default target is the session's active tab; --tab-id overrides).
• Emulation covers viewport/UA/touch only; it does not throttle the network, fake geolocation, or synthesise real touch-event streams.

Tabs (require --session <id>)

Observation (require --session unless noted)

Console & network debugging (read-only; require --session)

Both capture from the moment the tab is attached and read a bounded per-tab buffer: --since <seq> pages from a cursor (next_since in the result), --limit (default 50, max 200), --max-text-chars (default 1000, max 4096), --tab-id to target a non-active tab. Both are strictly read-only — they never intercept or modify traffic, and request/response headers, bodies, and timings are not captured.

Navigation

(bsk navigate back / bsk navigate forward are equivalent subcommands.)

Interaction

Scripting & timing

Ask the human for help — bsk request-help

When a step needs a human (captcha, login, OTP) or you want the user to confirm an important action, pause and ask:

bsk request-help --session <id> --prompt "Solve the captcha, then click Done only after the site accepts it" \
  --title "Captcha required" --target @e7 --target "#submit" --timeout 5m

• --prompt (required): what the user should do.
• --title (optional): custom title for the overlay panel. When omitted, the extension shows its default localized title.
• --target (repeatable): a snapshot ref (@e7) or CSS selector (#submit) to scroll to and flash-highlight. Strongly recommended — whenever the prompt refers to a concrete element (a button to click, a field to fill, a checkbox to toggle), pass its @eN ref / selector so the user is guided straight to the right spot instead of hunting for it. For interaction scenarios, always include the relevant target(s); reserve a prompt with no --target for cases where there is genuinely no specific element to point at (e.g. "wait for the page to finish loading").
• --timeout (default 5m): how long to wait.
• --completion-criteria (optional): JSON success detector. Use it only when there is a concrete post-help success signal, e.g. {"any":[{"url_contains":"/dashboard"},{"selector_exists":"[data-testid='account-menu']"}],"stable_for_ms":1000}.

The target tab is brought to the foreground; the page stays interactive while the agent control mask is hidden. The call blocks until the user explicitly acts, the timeout expires, cancellation arrives, or explicit completion criteria match. Page reloads, SPA route changes, and captcha refreshes do not return control by themselves. The result outcome is one of:

• continued — the user finished and clicked Done / return control (treat as confirm).
• cancelled — the user clicked Cancel (treat as reject/abort).
• timed_out — nobody acted within the timeout.
• completed — the explicit --completion-criteria matched while the user had control.
• navigated — deprecated legacy outcome. Do not rely on navigation as a completion signal.

note carries any text the user typed back. resolved_targets reports which refs/selectors matched a live element.

request-help does not refresh the page model after the user returns control. After a continued or completed result, issue a separate observation tool call (usually bsk snapshot --session <id>) before using new refs or reasoning about the post-help page state.

Disabling request-help (unattended mode)

Set BSK_REQUEST_HELP=off on unattended servers: bsk request-help then returns immediately with outcome="disabled" (no overlay, no waiting, exit 0). Any other value keeps it enabled. If you get disabled, do not retry — complete the task autonomously or stop gracefully.

Recording — bsk record

Capture the user's own actions in the Agent Window to a trace bundle, for later LLM-driven automation. New CLI builds request trace v3 (page observations + action chain); older extensions may still return trace v2 (actions only), which the CLI exports as a single trace.json without states/.

bsk record start --browser <instance-id-or-label> \
  [--url https://…] [--purpose "publish a wiki doc"] \
  [--max-page-tokens 3000] [--redact-values] \
  [--output trace]
# `--url` is optional; default https://example.com/ when omitted (must be http(s)).
# Blocks until the user clicks Finish in the recording panel, then writes:
#   trace/trace.json    — action chain (+ state index when v3)
#   trace/states/       — v3 only: one `sN.txt` observe snapshot per settled page state

bsk record stop [--output trace]   # terminal fallback if the browser panel is unavailable

• v3 bundle (preferred): --output is a directory (default ./trace) containing trace.json and states/. trace.json lists states[] (page observation ids) and steps[] (each step binds state = observe snapshot before the action and result.state = snapshot after settle). Page bodies live in states/sN.txt using the same VOM format as bsk observe.
• v2 fallback: when the connected extension is older, export may contain only legacy trace.json with pages[] action context and no states/ observation files. Update the extension for full v3 bundles.
• Each states[] entry is one settled page observation (captured after recording start, navigation landing, or action settle — not on a timer).
• target.ref values like @e12 exist only inside the bundle for disambiguation; do not copy @eN refs into SKILL.md or agent runbooks — use visible names from the observation text instead.
• --purpose is optional context metadata; it does not change what gets captured.
• --redact-values masks all form values in page files as [filled] / [empty].
• Do not record on banking/SSO/password-manager pages; passwords are redacted but traces may still contain sensitive text.

Error handling

Exit codes (echo $? after bsk …)

Human errors print error: + hint: on stderr; --json includes code, message, hint, exit_code.

When to run diagnostics

Always bsk session stop <id> in a finally-style path so the Agent Window closes and borrowed tabs return.

Red lines

1. No token theft — do not bsk evaluate on sensitive sites to read localStorage, cookies, or auth headers for exfiltration.
2. No long borrow — do not leave a user's personal tab in the Agent Window across unrelated tasks.
3. No skip stop — always bsk session stop <id>; never assume idle timeout will clean up.
4. No post-success control — once the user’s goal (or last trace step) is met, do not keep operating the page; stop the session unless they asked to keep it open.
5. No raw observe escalation before snapshot/observe — use bsk snapshot first; use bsk observe when VOM semantics or conditional surfaces help. Only use bsk get-html or bsk screenshot when snapshot/observe is insufficient. Element screenshots (--ref @eN) still require a fresh snapshot/observe ref — never skip observation just to grab a visual.
6. evaluate is powerful and risky — use only when snapshot + click/fill/select cannot suffice; never on credential surfaces.

More detail for any command: bsk <cmd> --help