website-scraping

Website scraping

A workflow for turning websites into structured data. Bias toward the lightest tool that works: static HTML → JSON-in-page → reverse-engineered API → browser automation, in that order. Skipping recon and reaching straight for Playwright is the single most common way to waste hours.

When to use

This skill applies to any task shaped like "given a URL or a list of URLs, produce structured records". Examples:

• Competitor price monitoring (hotels, flights, retail, rentals, student housing, SaaS)
• E-commerce product catalogue extraction
• Real-estate or job-listing harvesting
• Reviews, ratings, availability calendars
• One-off "pull this table into a spreadsheet" jobs

Out of scope: database persistence, scheduling, dashboards, downstream normalisation against a fixed schema. This skill produces JSONL — what the caller does with it is their problem.

Workflow

Follow these steps in order. Most failures come from skipping step 1.

1. Frame the goal before touching code

Pin down the answers in writing before opening an editor:

• What records do you want? One row per what? Per product? Per (product × variant × date)? Per (hotel × room-type × check-in)? Be explicit — many sites force you to choose between "one row per listing card" and "one row per price-variant", and the choice cascades through the whole scraper.
• What fields per record? Required vs nice-to-have. Mark anything the source might not publish on every record (sometimes-missing fields are normal, not a bug).
• Input shape: a single URL? A list of URLs? A search query you need to execute first? A seed listing page you crawl from?
• Output shape: this skill recommends JSONL (one JSON object per line, caller-defined keys) plus a sidecar manifest file. See "Output convention" below.
• Scale & cadence: how many records, how often? This one answer drives every downstream cost decision. A one-off harvest of 100 pages and a daily refresh of 10,000 SKUs are different projects: the first just needs a polite time.sleep; the second has to justify proxy/managed-service economics (see step 4, Tier 5+). Pin down volume and repeat-frequency now, not after you've built a single-shot scraper that can't keep up.
• What's "ground truth"? Pick one specific item you'll eyeball in the browser end-to-end to verify the scraper matches.

If the user hasn't been explicit about any of these, ask. A 30-second clarification saves an hour of wrong-shape extraction.

2. Recon the site — pick the lightest tool

Before writing the decision tree below, run two cheap sanity checks. Either can make the whole scraper unnecessary.

2.0a — Is the data already one tool-call away? You may be running inside an agent runtime that already exposes a fetch/scrape/SERP capability — Hermes's web_search + web_extract tools, a built-in search, or a scraping MCP server (Bright Data, Firecrawl, etc.). For a one-off, low-volume job, calling a tool that already exists is the lightest path of all — lighter than writing any code. Check what's available before you open an editor. (For repeatable, version-controlled, or high-volume work, still write a script — you want something you can re-run, diff, and hand off.)

2.0b — Is this a hostile mega-platform? A handful of sites — Amazon, LinkedIn, Instagram, TikTok, Facebook, YouTube, Zillow, Google Maps, Crunchbase, and similar — invest heavily in defeating DIY scraping and change their internals constantly. For these, hand-rolling a scraper is often poor ROI: it works for a week, then breaks. The lightest tool here may not be urllib at all — it's a commercial structured-data product (Bright Data's dataset/Web-Scraper APIs, Apify actors, etc.) that already maintains the extraction for that platform. Surface this trade-off to the user ("I can hand-roll this, but for $SITE a maintained data API will be more reliable and probably cheaper than the upkeep — your call") rather than silently grinding on a fragile custom scraper. If they want DIY anyway, proceed — but go in expecting the anti-bot ladder in step 4.

If neither shortcut applies, recon normally. Open the target URL in a real browser with DevTools open. Walk this decision tree top to bottom; stop at the first match. The earlier you stop, the simpler and more robust the scraper.

Is the data you need visible in "View Source" (right-click → View Page Source)?
├─ YES → Static HTML. Use urllib / requests + an HTML parser. Done.
│
└─ NO → Is there a <script type="application/ld+json"> block with what you need?
    │
    ├─ YES → JSON-LD. Universal schema (schema.org), present on most modern
    │        sites for SEO. Often has product, price, address, rating, etc.
    │        Parse the JSON block directly. No JS execution needed.
    │
    └─ NO → Is there a JSON blob in a <script id="__NEXT_DATA__">, __NUXT__,
            drupal-settings-json, window.__INITIAL_STATE__, or similar?
        │
        ├─ YES → Framework JSON-in-page. Pull the blob, json.loads it, navigate
        │        to the fields you want. Still no browser needed.
        │
        └─ NO → Open DevTools Network tab. Reload. Filter to XHR/Fetch.
                Does a clear JSON endpoint return the data?
            │
            ├─ YES → Reverse the API. Copy as cURL, port to Python.
            │        Often the cleanest path — same data the site uses, no scraping
            │        of HTML at all. Watch for auth tokens / CSRF / rate limits.
            │
            └─ NO → It's client-rendered. Browser automation required.
                    Use Playwright. See references/playwright.md.

Hard rule: only use Playwright when one of the earlier paths genuinely doesn't work. Playwright is 10-50× slower, 10× more memory-hungry, and 10× more fragile than urllib. Sites change their HTML structure every few months but tend not to change their JSON-LD or their backing API as often.

For an automated recon helper that hits the URL and reports which paths look viable, see scripts/recon.py. Run it as the first thing you do on any new site.

3. Build the smallest possible extractor

Start with one item, not the whole catalogue. Get the field extraction correct against a single known target, then generalise to the list. Two reasons:

1. Field-extraction bugs are easier to spot on one item than buried in 200.
2. You'll iterate the parser 5-10 times before it's right — doing that against the whole site burns rate-limit budget and trains the site's WAF on your fingerprint.

Code shape for the inner loop:

def extract_one(html_or_json) -> dict | None:
    # one item -> one dict; None if this item is unparseable.
    ...

def extract_many(items_iter) -> list[dict]:
    out, seen = [], set()
    for item in items_iter:
        try:
            rec = extract_one(item)
        except Exception as e:
            log.warning("extract failed for %s: %s", _label(item), e)
            continue
        if rec is None:
            continue
        key = rec.get("id") or rec.get("url") or _hash(rec)
        if key in seen:
            continue
        seen.add(key)
        out.append(rec)
    return out

Three things this shape enforces and explanations of why each matters:

• try/except per item, not over the whole batch. One malformed listing should not lose you the other 199. Log the failure with enough context to find the item later (the URL or a stable id), then move on. Silently swallowing exceptions is worse — the failure rate becomes invisible.
• Dedup by stable canonical id, not by record-equality. If the site exposes a numeric id, a SKU, a URL slug — use that. Don't dedup by hash(record_dict), because real-world records have minor variations (a whitespace change, a price tick) that you'd miss as duplicates.
• None means "this item is intentionally skipped", exception means "I tried and failed". Different signals, log them differently.

Concurrency. The loop above is sequential, which is the right default — it's the politest to the site and the easiest to debug. Reach for parallelism only when sequential is genuinely too slow, and bound it:

• Up to ~20 URLs: stay sequential with a small time.sleep between requests. Not worth the complexity.
• ~20–200 URLs: bounded concurrency — an asyncio.Semaphore(N) (or a thread pool of N) with N small (4–8). Never fire all requests at once; an unbounded asyncio.gather over 200 URLs is a self-inflicted DoS that gets your IP blocked.
• Hundreds+: you're now firmly in anti-bot territory (step 4). The semaphore size is capped by whatever the site tolerates, not by your CPU — start low and raise it only while the success rate holds.

The concurrency ceiling is set by the site's rate-limiting (step 4, Tier 4), not by your hardware. When in doubt, slower and complete beats faster and blocked.

4. Handle anti-bot defences with proportional response

Modern sites push back on scraping in escalating ways. Match your defence to what you actually see, not what you fear.

Don't reach for residential proxies on day 1. They cost real money and signal that you're operating at a scale or aggression level the site is actively trying to prevent. Make sure the use case justifies it.

The managed-unblocker off-ramp. Between "configure your own residential proxies" and "give up" sits a whole managed tier worth knowing about: services like Bright Data's Web Unlocker, Zyte API, ScraperAPI, and similar take a URL and hand back clean HTML — proxy rotation, TLS fingerprinting, and CAPTCHA solving all handled server-side. When DIY stalls on a site that's worth the spend, the right move is usually to swap only the fetcher and leave everything else alone: your extract_one, dedup, JSONL writer, and manifest stay byte-for-byte identical, you just change how the HTML arrives. Record the swap in the manifest ("tool": "managed-unblocker" or the specific vendor) so consumers know the provenance. This keeps the skill vendor-neutral — the managed service is an implementation detail behind the same fetch boundary, not a rewrite. See references/anti-bot.md for where it sits on the escalation ladder — including two disciplines specific to this tier: retrying an incomplete render (a 200 stub that passed the block-page gate but lacks your target field) via a validate-and-refetch hook, and asserting the currency/locale, since the unblocker's exit country changes the returned data and not just access.

Reference: references/anti-bot.md has the full escalation ladder and the specific patterns for surviving Cloudflare/Turnstile.

5. Capture the raw payload, not just the parsed output

For every item you scrape, save the raw HTML / JSON response alongside the parsed record. This is the single highest-value piece of robustness you can build in. Reasons:

• Replay: when your parser has a bug (and it will), you can re-parse from the saved raw payload without re-fetching. The site might be down, rate-limiting you, or have changed by the time you notice.
• Forensics: when the site changes and the parser breaks, the diff between the working raw payload and the broken one tells you what changed.
• Cheap: HTML compresses well. A 1MB page is ~100KB gzipped.

Convention: write raw payloads to raw/<id_or_hash>.html or .json next to your output JSONL. Reference the filename from the parsed record so you can find it later.

6. Verify against ground truth

Before declaring done:

0. Block-page gate (do this at fetch time, not just at the end). A 200 OK does not mean you got real content. Before parsing, assert the payload is non-empty and free of block-page signatures — "Just a moment", "Performing security verification", "Access denied", "unusual traffic", a Cloudflare/Turnstile iframe, or a body that's suspiciously short for the page. A scraper that silently writes 200 challenge-page records looks successful (status: "ok", 200 rows) but contains zero usable data. Fail loud — count these as errors in the manifest, don't let them masquerade as records. This is the single most common silent-corruption failure.

0a. Sanitize for LLM consumption (if scraped content will feed into an LLM). Hidden elements (<script>, <style>, <template>, hidden divs) can carry prompt injection payloads that execute when the scraped text enters an LLM prompt. Strip these before saving or passing to any model. If Scrapling is installed, use scrapling extract get <url> output.md --ai-targeted — it handles this automatically. Otherwise, apply the manual sanitization function in references/scrapling.md (Concept 1). This is mandatory whenever scraped content will enter an LLM context window.

0b. Assert the money/locale convention at fetch time (if the data is localised). For anything priced — travel, retail, marketplaces — the egress country (your proxy/unblocker exit, or just where the code runs) can change the page's currency, language, and tax treatment, so a £1,200 can silently arrive as $1,200 or ¥1,200. Read the currency the page actually rendered and compare it to what you asked for; refuse to record a mismatch rather than logging a converted number. A silently converted price is a wrong number that looks right — worse than a gap. Unlike a stub (0-ish, retry it), this is determinate: skip the item and flag it, don't re-fetch. Detail in references/anti-bot.md.

1. Smoke test: run the scraper against the single known-good item from step 1. Eyeball every field of the resulting record against what the browser shows.
2. Sample of N: pull 5-10 records at random from a full run and verify each in the browser. Look for: column-shift bugs (every field off by one), unit confusions (£ vs $, weekly vs monthly), date-format errors (MM/DD vs DD/MM), and silently-empty fields.
3. Count check: if the site reports a total ("234 properties in this city"), check your output has roughly that many records (allowing for things the site lists but you filtered out).
4. Re-run determinism: scrape the same URL twice within a few minutes. If the records or field values differ randomly, you've got a race condition in browser-driven extraction — see the double-render gotcha in references/extraction.md.

7. Write output in JSONL + manifest

The skill recommends this output convention because it's the simplest format that survives schema evolution:

output/
├── records.jsonl          # one JSON object per line, caller-defined keys
├── manifest.json          # run metadata: timestamp, source URLs, status, counts
└── raw/                   # raw payloads, named by stable id
    ├── abc123.html
    └── def456.json

JSONL: each line is json.dumps(record) + "\n". No header. Add fields freely between runs without breaking older consumers. Stream-readable.

Manifest structure (write this even on partial-failure runs):

{
  "scrape_started_at": "2026-05-24T10:34:00Z",
  "scrape_finished_at": "2026-05-24T10:36:12Z",
  "source_urls": ["https://example.com/listings"],
  "status": "ok",                  // "ok" | "partial" | "failed"
  "records_written": 187,
  "items_attempted": 200,
  "items_skipped": 13,
  "errors": [
    {"url": "...", "reason": "..."}
  ],
  "user_agent": "Mozilla/5.0 ...",
  "tool": "urllib" | "playwright" | "playwright+stealth"
}

The manifest lets the caller (and you, three months later) know whether to trust the data. A run with status: "partial" and items_skipped: 13 is a different consumption story to status: "ok".

References

Deeper material is in references/ — load as needed, don't read upfront:

• references/recon.md — the recon decision tree in full, with concrete examples of how to spot each pattern in the wild (JSON-LD shapes, Next.js / Nuxt / Drupal blob locations, common framework signatures, hidden REST APIs). Also covers the two pre-recon shortcuts: reusing an existing runtime fetch/scrape tool, and recognising hostile mega-platforms where a maintained commercial data product beats DIY.
• references/playwright.md — browser automation patterns: when to use it, domcontentloaded vs networkidle (and why the latter is usually wrong), waiting on selectors, stealth, headless vs headed, injecting page-side JS scrapers for SPAs.
• references/anti-bot.md — escalation ladder for Cloudflare, Turnstile, rate limits, and IP-based blocking. Specific patterns (fresh context per request, stealth library setup, the managed-unblocker off-ramp and how to swap only the fetcher, when to give up).
• references/extraction.md — pulling JSON-LD, finding JSON-in-page blobs in Next.js/Nuxt/Drupal/etc., handling double-rendered DOMs, polling for value-change on click, robust dedup, parsing common field shapes (size ranges, price ranges, dates, postcodes/zip codes, addresses).
• references/agentic-browsing.md — the narrow case where the blocker is multi-step navigation rather than parsing (login flows, filter wizards, calendar pickers). Covers Microsoft's Webwright agentic browser framework: when it earns its keep, when it's overkill, install/CLI, and the "use it once to harden the script, then run that script deterministically" pattern. Not for bulk extraction — its own docs say so.
• references/named-div-extraction.md — extracting server-rendered text from <div id="..."> containers (common on ASP.NET/legacy sites). Regex vs BeautifulSoup vs browser fallback, nested-div pitfalls, multi-language variant handling, and the bulk-urllib + browser-fallback hybrid pattern.
• references/scrapling.md — Load when the site changed structure and CSS selectors broke (adaptive element relocation), when scraped content will feed into an LLM (prompt injection sanitization), or when you need a CLI quick-test before writing a full scraper.

Helper scripts

• scripts/recon.py — python recon.py <url> fetches the page with plain HTTP, then with Playwright if needed, and reports which extraction strategies look viable (static HTML hits, JSON-LD blocks, framework blobs, anti-bot signatures). Run this on every new site before writing anything.

Anti-patterns to avoid

Things that look like good ideas but cost more than they save:

• Starting with Playwright "just in case". Recon first. If urllib works, every minute of Playwright debugging is wasted.
• Aggressive global try/except. Catches your own bugs and ships silent corruption. Per-item handling with logged failures is the right granularity.
• Hard-coding values that vary by tenant. Internal IDs in URLs (e.g. academicYear=82, region_id=14), CSRF tokens, signed query parameters — read them from the page rather than hard-coding. Sites rotate them and you'll wake up to "scraper returns 0 rows" with no obvious cause.
• Relying on CSS class names that look generated (e.g. .css-1a2b3c4). They're build-hash artifacts and change on every deploy. Prefer semantic selectors ([data-product-id], [itemprop=price], <h1>, structured data, ARIA roles).
• time.sleep(N) as a substitute for waiting on a selector. Fixed sleeps are the single biggest source of flake. Always wait on a specific DOM condition.
• Discarding the raw payload. Re-fetching to debug a parse bug is slow, rate-limit-burning, and often impossible (the site may have changed). Keep raw payloads cheap.
• Caring about every field on every item. Some sites legitimately have missing data on some items. Mark fields as optional in step 1 and don't crash on absence.

A note on ethics and ToS

This skill helps with the mechanics of scraping, not the question of whether you should. Before running a scraper at any meaningful scale, the caller is responsible for:

• Checking the site's robots.txt and Terms of Service.
• Considering whether a public API is available that does the same job.
• Throttling traffic to a respectful level (a single user's worth of browsing, not a DDoS).
• Not scraping personal data without a lawful basis.
• Not scraping content behind a login they don't have explicit permission to access programmatically.

Flag any of these as a concern if the user's scope crosses a line. "We can do this, but check that ToS first" is a legitimate response.

Evals

Two eval sets live under evals/:

• evals.json — runnable end-to-end scraping tasks against stable practice sites (static HTML and JS-rendered), with the expected extraction strategy.
• routing-fixtures.json — lightweight contract fixtures: sample request → expected routing (including when a request belongs to entity-research or people-enrichment instead), required output fields, and forbidden patterns.

The routing fixtures are specs, not run against a live model; tests/test_routing_fixtures.py validates they stay well-formed.