Research project

Crawl Policy Index

A longitudinal record of which AI crawlers a website allows, blocks, or restricts — as declared in robots.txt.

This site is the project’s public identity, not the tracker yet. Numbers will appear here only when they come from a named, frozen panel and a documented parse.

• Bot policy — who we are, what we fetch, how to be excluded

v1 measures robots.txt only. HTTP 402 / pay-per-crawl is out of scope.