Chris Shiflett: AI Crawlers Still Rely on Basic HTML
Start here
Chris Shiflett argues that AI crawlers such as GPTBot and ClaudeBot behave much like the text-only Googlebot of 2004: they fetch HTML and largely skip JavaScript, though OpenAI's crawlers recently appeared to run scripts on some sites. He built two tools to test whether AI can find a site and whether it actually recommends it. Shiflett's conclusion is that robots.txt, sitemaps, real HTML content and structured data matter more than any llms.txt tricks. The finding is his own assessment from testing with those tools, not a measured ranking effect.
Generated from reports · updated 8 h ago
Report timeline
Follow the reports to see the event from different sides.
- Hacker News · AIChris Shiflett: Ready or Not, Here Comes AI
Chris Shiflett argues that AI crawlers such as GPTBot and ClaudeBot resemble the text-only Googlebot of 2004, fetching HTML and largely skipping JavaScript, though OpenAI's crawlers recently appeared to run scripts on some sites. He built two tools to check whether AI can find a site and whether it actually recommends it, finding that basics like robots.txt, sitemaps, real HTML content and structured data matter more than any llms.txt tricks.
Heat for this event
- Current, comparable range
- 8
- Peak, comparable range
- 10November 18 00:00
- Change in 24 hours
- –
The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.