Robots.txt
1 post
AI
Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)
Hundreds of sites have put old Anthropic scrapers on their blocklist, while leaving a new one unblocked.
