Hacker News · AI·· 10 h agoAI score42
AI Crawlers Outrequested All Search Engines Combined, and a Third of "Googlebots" Were Fake
AI crawlers read my site more than Google does. A third of Googlebots were fake
AI summary
In just under two weeks of nginx logs, AI crawlers made almost three times as many requests as all search engines combined, and more than a third of "Googlebots" were fake. Five rented Google Cloud servers used 30 bot names for 2,496 requests, probing for password and key files after first reading robots.txt and sitemaps. No AI assistant sent a single visitor, while Google sent 108.
Source:Hacker News · AI · opensource-analytics.com