ai.robots.txt is a blocklist for AI agents and robots that you can use in your robots.txt file.
The project addresses the problem of unwanted AI crawlers accessing your website by providing a curated list of user-agent strings and patterns that correspond to known AI agents and bots. Rather than manually researching and maintaining your own list of AI crawler identifiers, developers can reference this project's compiled list to block these agents at the robots.txt level, preventing them from crawling and potentially training on your site's content.
The tool suits anyone concerned about AI systems scraping their website without permission, whether for privacy reasons, to prevent unauthorized training data collection, or to control how their content is used. It works best as a straightforward addition to an existing robots.txt file and is particularly valuable for sites that want a maintained, community-driven list rather than attempting to keep pace with new AI agents independently. The README does not compare this approach to alternatives.
The project accepts contributions from the community to expand and maintain the blocklist as new AI agents emerge. Development activity shows ongoing updates to the agent list in response to newly identified crawlers.
