robotstxt is a C++ library that parses and matches robots.txt files according to the Robots Exclusion Protocol standard.
The library addresses the problem that different implementations of the robots.txt standard have historically parsed and matched rules inconsistently, creating confusion for developers building crawlers and tools. It solves this by exposing Google's production robots.txt parser, the same code used by Googlebot, as an open-source library. The parser handles the full syntax of robots.txt files including percent-encoded characters and unicode patterns. The library requires that URIs passed to its matching functions already conform to RFC3986 format, as it performs only pattern matching rather than full URI normalization.
Developers should choose this library when building tools that need to respect robots.txt rules in the same way Google's crawler does, ensuring compatibility with how webmasters expect their directives to be interpreted. It suits any project that crawls the web or needs to validate robots.txt compliance. The library includes a command-line binary for testing individual URLs and user-agents against a robots.txt file, which is useful for webmasters and developers validating their configurations. The project supports multiple build systems, with Bazel as the official build system and CMake as a community-supported alternative, and runs on major platforms including Linux, Windows, and macOS with any C++14-compatible compiler.
The project maintains a focused scope on parsing and matching logic without attempting to handle URI normalization, placing responsibility on callers to provide properly formatted input. Development activity shows consistent attention to the core parsing specification with careful documentation of edge cases and expected behavior.
