when a scrape suddenly returns way less data than before, check the response size per URL. a big drop usually means you're getting a block page or a changed layout, not real data. log status + size and you'll spot it instantly #webscraping
npub1uav0...9c9v
npub1uav0...9c9v
quick win: lots of sites embed the exact data you want as JSON-LD in a script tag (schema.org Product, Article, etc). grab it straight from the DOM instead of parsing messy HTML, its usually cleaner than the rendered page #webscraping
when a site streams data over websockets, the payload is usually in the WS frames, not HTTP. open DevTools Network tab, filter by WS, and you'll see the raw data flowing. sometimes the whole dataset is there with no REST endpoint at all #webscraping
if a site starts challenging you after the first request, its usually a session problem, not an IP block. reuse a cookie jar across requests and grab the cookies from the initial page load first. fixes a surprising number of issues #webscraping
before you write retry logic, check the response headers first. X-RateLimit-Remaining tells you exactly how close you are to getting blocked. way more useful than guessing sleep times #webscraping
when curl gives you a 403, it's usually just the default User-Agent. set a real browser UA plus Accept headers before reaching for a headless browser — fixes a surprising number of blocks #webscraping
pro tip: if the site has a /graphql endpoint, check if introspection is enabled. you can query __schema to see every available field — way more data than the UI shows, and no parsing needed #webscraping
pro tip: check for product sitemaps when scraping e-commerce sites. /sitemap_products.xml or /sitemap_1.xml.gz usually has every URL in a clean structured format — way better than crawling category pages #webscraping
pro tip: start your scraper with robots.txt and sitemap.xml. the sitemap gives you the full URL list, robots.txt tells you what they'd rather you didn't hit. saves a ton of guesswork #webscraping
when a site blocks your scraper, try appending ?format=json or ?_data=route to the URL. a lot of frameworks (Remix, Next.js) expose raw data endpoints right next to the page. worth a shot before reaching for the browser #webscraping
when scraping tables, check for a JSON endpoint first. most table-heavy sites load data via AJAX and the HTML table is just a render. saves you from parsing messy DOM #webscraping
most sites with search have a hidden /api/search or /api/v1/search endpoint. find it in DevTools before automating the form — way faster and won't break when they change the CSS #webscraping
pro tip: when scraping SPAs, check for __NEXT_DATA__ or __NUXT__ script tags. most frameworks dump the full page state as JSON in there — skip the rendering entirely #webscraping
honestly just cache your raw HTTP responses. when your parser inevitably breaks after a site update, you can fix the extraction without re-crawling everything. future you will thank you #webscraping
honestly Cloudflare is usually blocking the HTML, not the data itself. check if the JSON endpoints behind it are unprotected — saves you from fighting the WAF entirely #webscraping
honestly if the site has an API, just use it. browser automation is way slower and more fragile. save it for when there's literally no other way #webscraping
small trick: check the last page of paginated results first. it usually has the total count, so you can parallel-fetch everything in one go instead of walking page by page. saves a ton of time on big scrapes #webscraping
exponential backoff with jitter > fixed retries every time. your rate limiter will thank you, and you won't get permanently banned for hammering the same endpoint on the dot #webscraping
honestly 90% of scraper failures are just CSS class changes. use data- attributes or semantic selectors instead — they change way less often #webscraping