A fullstack dev takes a serious look at web scraping

I'd never scraped a website before last week. I knew it was a thing people did - I vaguely filed it next to “parsing HTML with regex” in the drawer of things you're not supposed to admit to. Then a role came up that lives at the intersection of web data and AI, and instead of just reading about the field, I gave myself two days to build my way through it. This is what surprised me.

Scraping is reverse-engineering someone else's CSS

My first scraper was almost boring: fetch a page, load it into cheerio, walk the selectors, collect 1,000 books into JSON. The surprise wasn't that it worked - it's what the work actually is. You're reverse-engineering the structure of a page that nobody promised you. The CSS classes you hook into are an internal implementation detail of someone else's frontend, and they can change without notice. Every selector is a small bet that a stranger's codebase stays still.

The server is judging you

Here's where it got personal. My capstone at Codam was building an HTTP/1.1 server from scratch in C++ - I wrote the request parser myself, hundreds of lines of header handling, chunked encoding, multipart uploads. I thought I knew HTTP. But I'd only ever stood on the server side, parsing whatever arrived. Scraping put me on the client side, and the first thing you learn there is that the server isn't just parsing your request - it's judging it. Bare Node fetch announces itself with User-Agent: node. Every header you don't think about is a tell.

And the judging happens twice. Once at the edge, before your request ever reaches the site - IP reputation, TLS fingerprint - and again inside the browser, where the page's own JavaScript examines you. I pointed a headless browser at a fingerprinting test page and got back a wall of red: navigator.webdriver: true, “HeadlessChrome” in the user agent, zero plugins. The browser confesses.

The insight that reframed the whole field for me: anti-bot systems can't read intent, only fingerprints. A completely legitimate scraper has to clear exactly the same walls as a malicious one. There is no “good bot” lane. That's why proxies aren't a hack - they're infrastructure. I had never once thought about IP addresses having reputations, that one IP makes you credible and another gets you blocked before anyone reads a byte of your request. An entire industry exists in that gap.

Where it meets AI

The part that hooked me most: I fed the same product page to my cheerio selectors and to an LLM with a structured output schema, then changed the page layout. The selectors returned null. The LLM read the page like a human would and extracted every field anyway. Selectors are the cheap, fast happy path; LLM extraction is the fallback that survives structural change - at a real cost per page. The honest architecture uses both, and realizing that was the moment “AI needs web data” stopped being a slogan for me.

Then I went two steps further: a small RAG pipeline over pages I'd scraped - like talking to a PDF, except the corpus is live web data you gathered yourself - and finally an MCP server wrapping my scraper as a tool, so an AI assistant can decide on its own when to go fetch a page. That last one is the clearest picture I've seen of where this is heading: not people running scrapers to feed models, but models reaching for the web themselves.

What I'd tell past-me

Web scraping isn't a trick, it's a discipline - equal parts HTTP archaeology, adversarial thinking, and infrastructure. Two days wasn't enough to master any of it, but it was enough to replace a caricature with a map. The code is public: github.com/PauliusMinialga/scraping-lab - four small projects, one lesson each, scraped politely from practice sites built for exactly this.