The scraping landscape we worked in three years ago is gone. CSS selectors that worked in 2023 break weekly. Cloudflare and DataDome catch headless browsers in seconds. Sites stream their data through React Server Components and the HTML you see in DevTools is not the HTML the page loads with.
The good news: the tools got better too. Playwright is now the standard, residential proxy networks are commodity, and LLM extractors solve the brittle-selector problem. Here is how we actually build scrapers in 2026.
Why Playwright Won
Three years ago, Puppeteer was the default and Selenium was the old guard. In 2026, Playwright is what every serious scraping team uses. Reasons:
- Cross-browser support out of the box (Chromium, Firefox, WebKit) helps when one is fingerprinted harder than another.
- Auto-waiting (no manual sleeps) cuts flakiness by 80% in our experience.
- Network interception is first-class, so capturing API responses is trivial.
- The ecosystem of stealth plugins (playwright-extra, playwright-stealth) is now more mature than Puppeteer's.
Static vs. Dynamic: Pick the Cheapest Path
The first question on any scraping project: do you actually need a browser? Many sites that look JS-heavy still ship server-rendered HTML. If curl returns the data you want, use curl (or the equivalent in your language). Playwright is 100x slower and 10x more expensive than HTTP requests.
Heuristic we use: try a plain HTTP fetch first, parse with cheerio or BeautifulSoup. If the data is there, ship it. Reach for Playwright only when the data only loads after JavaScript runs, or when interaction (click, scroll, login) is required.
Anti-Bot Defenses You Will Hit in 2026
Modern protection systems (Cloudflare Turnstile, DataDome, Akamai Bot Manager, PerimeterX) layer detection signals:
- TLS fingerprinting. Your client's TLS handshake is unique. Default Node fetch and default Python requests have signatures that scream "bot".
- HTTP/2 fingerprinting. Chromium sends headers in a specific order and frame priority. Anything else gets flagged.
- Browser fingerprinting. Canvas, WebGL, audio context, fonts, and screen dimensions are combined into a fingerprint hash.
- Behavioral signals. Mouse movement, scroll velocity, timing between actions. Real humans are messy; bots are precise.
Counter-Measures That Actually Work
- Use real browsers. Headless Chrome with the
--headless=newmode plus playwright-stealth covers 80% of cases. - Residential proxies. $5/GB residential IPs route around datacenter IP bans. Smartproxy, Bright Data, IPRoyal are the players.
- Rotate carefully. Switching IP every request looks botlike. Rotate per session (5 to 50 requests per IP) instead.
- Match a real user agent and viewport. Default Playwright user agents are flagged. Use a current Chrome UA and a 1920x1080 viewport.
- Add human-like delays. Random sleeps between 800ms and 3500ms between actions. Mouse jitter helps on the hardest sites.
One pattern we use: the browserforge library generates realistic fingerprints (headers, navigator properties, font lists) per session. It defeats most off-the-shelf bot detection.
The Big 2026 Shift: LLM Extractors
For years, scrapers were tied to CSS selectors and XPath expressions. The site updated its CSS class names, your scraper broke, you fixed it, repeat forever.
The 2026 pattern: scrape the page HTML, hand it to a small LLM, ask for structured data. The LLM does not care if the class name changed from .product-price to .css-x4f7a9. It reads the visible text the same way a human would.
const html = await page.content()
const cleaned = stripScriptsAndStyles(html) // small token saving
const result = await claude.messages.create({
model: 'claude-haiku-4-5',
max_tokens: 2000,
messages: [{
role: 'user',
content: [
{ type: 'text', text: 'Extract the product as JSON: name, price, sku, in_stock.' },
{ type: 'text', text: cleaned }
]
}]
})
const product = JSON.parse(result.content[0].text)Tradeoffs:
- Cost: $0.001 to $0.01 per page with Haiku-tier models. Negligible for low-volume, real for millions of pages.
- Speed: 1 to 3 seconds per extraction. Run in parallel.
- Reliability: Massively higher when sites change layout. We have scrapers running 18+ months without maintenance using this pattern.
For high-volume jobs, we use a hybrid: try CSS selectors first, fall back to LLM extraction when selectors return nothing. Cost stays low, reliability stays high.
The Legal and Ethical Floor
The hiQ vs LinkedIn ruling and subsequent case law have been clarifying:
- Scraping public data from a site you can access without a login is generally legal in the US.
- Bypassing technical access controls (like login walls) crosses into CFAA territory.
- Respect robots.txt as a courtesy and a legal hedge.
- Do not republish copyrighted content as your own. Fair use is narrower than people think.
We refuse projects that involve breaking ToS in spirit, scraping personal data without consent, or building tools designed for stalking or harassment. We accept projects for competitive intelligence, price monitoring, lead enrichment from public sources, and academic research.
Where We Go From Here
If you have a scraping project, the right tool is not always Playwright, and the right architecture is not always a single script on a cron. We have built scrapers ranging from one-off Excel-to-database imports to 24/7 distributed pipelines pulling from 50+ sites with ML-based deduplication. Our web scraping practice covers the full range. Tell us what data you need and we will respond within 24 hours with a feasibility assessment and quote.