Skip to content
Hot eventDeveloping

AI Access Index: 684 Top Sites Measured for AI Crawler Access

1 reports1 reporting sourcesUpdated 21 h ago

Start here

AI digest

Crawl Readiness published the first edition of its AI Access Index, measuring on November 1, 2026 how the top 1,000 Tranco hostnames treat AI crawlers; 684 sites answered. Of those 684, 79.7% are open to both AI search and AI training, 8.8% allow AI search but block training, 3.8% are selective and 7.7% are closed to AI. The index also compares robots.txt declarations with actual server behavior: among 547 tested sites, 11.3% refuse a crawler that their robots.txt allows, and among 124 tested sites, 63.7% serve a crawler that their robots.txt blocks. The published files are licensed CC BY-NC 4.0, free to share and adapt for non-commercial use with credit; commercial use, such as in a product or a dataset for sale, requires Crawl Readiness's permission. Sites outside the sample can be checked the same way for free, and a site in the sample can ask to have its row removed from the published files.

Generated from reports · updated 20 h ago

Report timeline

Follow the reports to see the event from different sides.

November 19
  1. Hacker News · AI
    AI Access Index: How 684 of the Top 1,000 Hostnames Treat AI Crawlers

    Crawl Readiness published the first edition of its AI Access Index, measuring on November 1, 2026 how the top 1,000 Tranco hostnames treat AI crawlers, with 684 sites answering. 79.7% of those 684 are open to both AI search and AI training, 8.8% allow AI search but block training, 3.8% are selective and 7.7% are closed to AI. The index also compares robots.txt with server behavior: 11.3% of 547 tested sites refuse a crawler their robots.txt allows, and 63.7% of 124 tested sites serve a crawler their robots.txt blocks.

Heat for this event

Current, comparable range
6
Peak, comparable range
10November 19 07:00
Change in 24 hours
–

The trend compares the same participants over hours observed continuously, so its range can be narrower than the current heat count. Hover or tap the chart for hourly heat; the arrow keys move through it.