Skip to content
Hacker News · AI·· 9 h agoAI score33

Chris Shiflett: Ready or Not, Here Comes AI

Ready or Not, Here Comes AI, by Chris Shiflett

AI summary

Chris Shiflett argues that AI crawlers such as GPTBot and ClaudeBot resemble the text-only Googlebot of 2004, fetching HTML and largely skipping JavaScript, though OpenAI's crawlers recently appeared to run scripts on some sites. He built two tools to check whether AI can find a site and whether it actually recommends it, finding that basics like robots.txt, sitemaps, real HTML content and structured data matter more than any llms.txt tricks.

Source:Hacker News · AI · shiflett.org