feat: snapshot related prompts

This commit is contained in:
Richard Tang
2026-04-18 19:39:00 -07:00
parent 0d13c805b1
commit f2e51157dc
3 changed files with 18 additions and 19 deletions
@@ -33,9 +33,9 @@ All tools are prefixed with `browser_`:
**`browser_snapshot`** — compact accessibility tree of interactive elements. Fast, cheap, good for static or form-heavy pages where the DOM matches what's visually rendered (documentation, simple dashboards, search results, settings pages).
**`browser_screenshot`** — visual capture + metadata (`cssWidth`, `devicePixelRatio`, scale fields). **Use this on any complex SPA** — LinkedIn, Twitter/X, Reddit, Gmail, Notion, Slack, Discord, any site using shadow DOM, virtual scrolling, React reconciliation, or dynamic layout. On these pages, snapshot refs go stale in seconds, shadow contents aren't in the AX tree, and virtual-scrolled elements disappear from the tree entirely. Screenshot is the **only** reliable way to orient yourself.
**`browser_screenshot`** — visual capture + metadata (`cssWidth`, `devicePixelRatio`, scale fields). Use this when `browser_snapshot` does not show the thing you need, when refs look stale, or when visual position/layout matters. This often happens on complex SPAs — LinkedIn, Twitter/X, Reddit, Gmail, Notion, Slack, Discord and on sites using shadow DOM, virtual scrolling, React reconciliation, or dynamic layout.
Neither tool is "preferred" universally — they're for different jobs. Default to snapshot on text-heavy static pages, screenshot on SPAs and anything shadow-DOM-heavy. Activate the `browser-automation` skill for the full decision tree.
Neither tool is "preferred" universally — they're for different jobs. Start with snapshot for page structure and ordinary controls; use screenshot as the fallback when snapshot can't find or verify the visible target. Activate the `browser-automation` skill for the full decision tree.
## Coordinate rule
@@ -44,9 +44,9 @@ Every browser tool that takes or returns coordinates operates in **fractions of
## System prompt tips for browser nodes
```
1. On LinkedIn / X / Reddit / Gmail / any SPA — use browser_screenshot to orient,
not browser_snapshot. Shadow DOM and virtual scrolling make snapshots unreliable.
2. For static pages (docs, forms, search results), browser_snapshot is fine.
1. Start with browser_snapshot or the snapshot returned by the latest interaction.
2. If the target is missing, ambiguous, stale, or visibly present but absent from the tree,
use browser_screenshot to orient and then click by fractional coordinates.
3. Before typing into a rich-text editor (X compose, LinkedIn DM, Gmail, Reddit),
click the input area first with browser_click_coordinate so React / Draft.js /
Lexical register a native focus event, then use browser_type_focused(text=...)
@@ -66,7 +66,7 @@ Every browser tool that takes or returns coordinates operates in **fractions of
"tools": {"policy": "all"},
"input_keys": ["search_url"],
"output_keys": ["profiles"],
"system_prompt": "Navigate to the search URL via browser_navigate(wait_until='load', timeout_ms=20000). Wait 3s for SPA hydration. On LinkedIn, use browser_screenshot to see the page — browser_snapshot misses shadow-DOM and virtual-scrolled content. Paginate through results by scrolling and screenshotting; extract each profile card by reading its visible layout..."
"system_prompt": "Navigate to the search URL via browser_navigate(wait_until='load', timeout_ms=20000). Wait 3s for SPA hydration. Use the returned snapshot to look for result cards first. If the cards are missing, stale, or visually present but absent from the tree, use browser_screenshot to orient; paginate through results by scrolling and use screenshots only when the snapshot cannot find or verify the visible cards..."
}
```
+10 -11
View File
@@ -26,18 +26,17 @@ Follow these rules for reliable, efficient browser interaction.
- **`browser_snapshot`** compact accessibility tree. Fast, cheap, good
for static / text-heavy pages where the DOM matches what's visually
rendered (docs, forms, search results, settings pages).
- **`browser_screenshot`** visual capture + scale metadata. Use on any
complex SPA (LinkedIn, X / Twitter, Reddit, Gmail, Notion, Slack,
Discord) and on any site using shadow DOM or virtual scrolling. On
those pages, snapshot refs go stale in seconds, shadow contents
aren't in the AX tree, and virtual-scrolled elements disappear from
the tree entirely screenshots are the only reliable way to orient.
- **`browser_screenshot`** visual capture + scale metadata. Use when
the snapshot does not show the thing you need, when refs look stale,
or when you need visual position/layout to act. This is common on
complex SPAs (LinkedIn, X / Twitter, Reddit, Gmail, Notion, Slack,
Discord), shadow DOM, and virtual scrolling.
Neither tool is "preferred" universally they're for different jobs.
Default to snapshot on static pages, screenshot on SPAs and
shadow-heavy sites. Interaction tools (click/type/fill/scroll) return
a snapshot automatically, so don't call `browser_snapshot` separately
after an interaction unless you need a fresh view.
Use snapshot first for structure and ordinary controls; switch to
screenshot when snapshot can't find or verify the target. Interaction
tools (click/type/fill/scroll) return a snapshot automatically, so
don't call `browser_snapshot` separately after an interaction unless
you need a fresh view.
Only fall back to `browser_get_text` for extracting small elements by
CSS selector.
@@ -31,7 +31,7 @@ browser_shadow_query(...) → rect → same
## Screenshot + coordinates is shadow-agnostic — prefer it on shadow-heavy sites
On sites that use Shadow DOM heavily (Reddit's faceplate Web Components, LinkedIn's `#interop-outlet` messaging overlay, some X custom elements), **coordinate-based operations reach elements that selector-based tools can't see.**
Start with `browser_snapshot` when you need to inspect the page structure or find ordinary controls. If the snapshot does not show the thing you need, shows stale or misleading refs, or cannot prove where a visible target is, take `browser_screenshot` and use the screenshot + coordinate path. This is especially useful on sites that use Shadow DOM heavily
Why:
@@ -114,7 +114,7 @@ Even after `wait_until="load"`, React/Vue SPAs often render their real chrome in
- **Prefer `browser_snapshot` over `browser_get_text("body")`** — returns a compact ~15 KB accessibility tree vs 100+ KB of raw HTML.
- Interaction tools (`browser_click`, `browser_type`, `browser_type_focused`, `browser_fill`, `browser_scroll`, etc.) return a page snapshot automatically in their result. Use it to decide your next action — do NOT call `browser_snapshot` separately after every action. Only call `browser_snapshot` when you need a fresh view without performing an action, or after setting `auto_snapshot=false`.
- Complex pages (LinkedIn, Twitter/X, SPAs with virtual scrolling) have DOMs that don't match what's visually rendered — snapshot refs may be stale, missing, or misaligned with visible layout. On these pages, `browser_screenshot` is the only reliable way to orient yourself.
- Complex pages (LinkedIn, Twitter/X, SPAs with virtual scrolling) can have DOMs that don't match what's visually rendered — snapshot refs may be stale, missing, or misaligned with visible layout. Try the available snapshot first; when the target is not present in that snapshot or visual position matters, switch to `browser_screenshot` to orient yourself.
- Only fall back to `browser_get_text` for extracting specific small elements by CSS selector.
## Typing and keyboard input