Hacker News · AI·· 9 h agoAI score33
Chris Shiflett: Ready or Not, Here Comes AI
Ready or Not, Here Comes AI, by Chris Shiflett
AI summary
Chris Shiflett argues that AI crawlers such as GPTBot and ClaudeBot resemble the text-only Googlebot of 2004, fetching HTML and largely skipping JavaScript, though OpenAI's crawlers recently appeared to run scripts on some sites. He built two tools to check whether AI can find a site and whether it actually recommends it, finding that basics like robots.txt, sitemaps, real HTML content and structured data matter more than any llms.txt tricks.
Source:Hacker News · AI · shiflett.org