Hacker News · AI·· 21 h agoAI score55
AI Access Index: How 684 of the Top 1,000 Hostnames Treat AI Crawlers
AI Access Index: how 684 of the top thousand hostnames treat AI crawlers
AI summary
Crawl Readiness published the first edition of its AI Access Index, measuring on November 1, 2026 how the top 1,000 Tranco hostnames treat AI crawlers, with 684 sites answering. 79.7% of those 684 are open to both AI search and AI training, 8.8% allow AI search but block training, 3.8% are selective and 7.7% are closed to AI. The index also compares robots.txt with server behavior: 11.3% of 547 tested sites refuse a crawler their robots.txt allows, and 63.7% of 124 tested sites serve a crawler their robots.txt blocks.
Source:Hacker News · AI · crawlreadiness.com