Skip to content
Hot eventDeveloping

Chris Shiflett: AI Crawlers Still Rely on Basic HTML

1 reports1 reporting sourcesUpdated 9 h ago

Start here

AI digest

Chris Shiflett argues that AI crawlers such as GPTBot and ClaudeBot behave much like the text-only Googlebot of 2004: they fetch HTML and largely skip JavaScript, though OpenAI's crawlers recently appeared to run scripts on some sites. He built two tools to test whether AI can find a site and whether it actually recommends it. Shiflett's conclusion is that robots.txt, sitemaps, real HTML content and structured data matter more than any llms.txt tricks. The finding is his own assessment from testing with those tools, not a measured ranking effect.

Generated from reports · updated 8 h ago

Report timeline

Follow the reports to see the event from different sides.

November 17
  1. Hacker News · AI
    Chris Shiflett: Ready or Not, Here Comes AI

    Chris Shiflett argues that AI crawlers such as GPTBot and ClaudeBot resemble the text-only Googlebot of 2004, fetching HTML and largely skipping JavaScript, though OpenAI's crawlers recently appeared to run scripts on some sites. He built two tools to check whether AI can find a site and whether it actually recommends it, finding that basics like robots.txt, sitemaps, real HTML content and structured data matter more than any llms.txt tricks.

Heat for this event

Current, comparable range
8
Peak, comparable range
10November 18 00:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.