Server Log Analysis Finds AI Crawlers Outpace Search Engines
Start here
An author at opensource-analytics.com analyzed just under two weeks of their site's nginx logs and found AI crawlers made almost three times as many requests as all search engines combined. The author counted a request as real only if its address belonged to the company behind the bot, verified through published lists, DNS or the company's registered network. More than a third of requests claiming to be "Googlebots" were fake. Five rented Google Cloud servers used 30 bot names for 2,496 requests, reading robots.txt and sitemaps before probing for password and key files. No AI assistant sent a single visitor, while Google sent 108.
Generated from reports · updated 10 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIAI Crawlers Outrequested All Search Engines Combined, and a Third of "Googlebots" Were Fake
In just under two weeks of nginx logs, AI crawlers made almost three times as many requests as all search engines combined, and more than a third of "Googlebots" were fake. Five rented Google Cloud servers used 30 bot names for 2,496 requests, probing for password and key files after first reading robots.txt and sitemaps. No AI assistant sent a single visitor, while Google sent 108.
Heat for this event
- Current, comparable range
- 8
- Peak, comparable range
- 10November 17 23:00
- Change in 24 hours
- –
The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.