Anthony Accioly's avatar
Anthony Accioly anthony@accioly.social 9 months ago
I forgot to hit send on this earlier today, but speaking of David vs. Goliath, if you haven’t done it already, make sure to tarpit Alexandria, TheForest1, GitCitadel, and anything else you’re deploying that is expected to hold a lot of data. Otherwise, scrapers will come for the goods, overload the infrastructure and cost you quite a few sats.
Ars Technica
Open source devs say AI crawlers dominate traffic, forcing blocks on entire countries
AI bots hungry for data are taking down FOSS sites by accident, but humans are fighting back.
Nepenthes - ZADZMO.org
Making web crawlers eat shit since 2023
The Cloudflare Blog
Trapping misbehaving bots in an AI Labyrinth
How Cloudflare uses generative AI to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” ...
↑ Parent

Replies (1)

ChipTuner's avatar
ChipTuner ChipTuner@gitcitadel.com 9 months ago
Yeah right now I just have a user agent list and that does most of the work, the bots that are hitting me mostly have "~bot" in the user agent field so it's been working to stave them off for now. I have to hop in and update the list every few days it helps. I was interested in rDNS lookups for blocking since most of the IPs i've dealt with come from rDNS bot sources like bytedance, openai and so on.
↑