Robots.txt
2 posts
AI
Developer Creates Infinite Maze That Traps AI Training Bots
"Nepenthes generates random links that always point back to itself - the crawler downloads those new links. Nepenthes happily just returns more and more lists of links pointing back to itself."
AI
Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)
Hundreds of sites have put old Anthropic scrapers on their blocklist, while leaving a new one unblocked.