Figures / Holders
6,094
Sites whose robots.txt the census has read in its current pass
The sites that answered with a robots.txt in the newest pass of the census (3 daily sweeps from 2026-10-15 to 2026-10-17, 12,000 requested), out of the most visited 300,000 sites on the Tranco list in the regions this business works in. A pass reads the whole list in 46 daily slices.
Updated 17 Oct 2026, from its newest record.
The 3 records it is computed from, newest first
- As stated219 of 1,898 sites that answered shut at least one AI crawler outDated17 Oct 2026SourceTranco list of the most visited sites…
- As stated244 of 2,089 sites that answered shut at least one AI crawler outDated16 Oct 2026SourceTranco list of the most visited sites…
- As stated233 of 2,107 sites that answered shut at least one AI crawler outDated15 Oct 2026SourceTranco list of the most visited sites…