The unwritten rules of respecting robots.txt and not scooping up everything have already crumpled due to AI scraping.
Seems to be the trend that if you can access it it will be used. Encrypt and deny access might become the new reality, unfortunately.
Business Insider
Meta has 2 new sneaky bots scooping up free AI-training data from the web
Meta's new AI bots, Meta-ExternalAgent and Meta-ExternalFetcher, scrape web data and may bypass robots.txt rules.