gau is a command-line tool that fetches known URLs for a given domain from multiple public sources including AlienVault's Open Threat Exchange, the Wayback Machine, Common Crawl, and URLScan.
The tool solves the problem of discovering historical and indexed URLs associated with a domain, which is useful for security research, reconnaissance, and understanding a site's exposure across the web. It aggregates results from four different URL sources, allowing users to query them individually or in combination, and provides filtering capabilities to narrow results by status codes, MIME types, file extensions, and date ranges.
Security researchers and penetration testers should choose this tool when they need to enumerate URLs for a target domain across multiple archival and threat intelligence sources. It suits projects that require broad URL discovery without manually querying each source separately. The tool was inspired by waybackurls, offering similar functionality with the addition of multiple data sources and more granular filtering options. Users can control which providers to query, filter by HTTP status codes and content types, remove parameter variations from endpoints, and output results in JSON format. Configuration can be managed through a TOML file or command-line flags, with support for proxy settings, custom timeouts, and parallel processing via configurable thread counts.
The project shows consistent maintenance with regular updates addressing user-reported issues and feature requests. Development activity includes responsiveness to bug reports and incorporation of community feedback, as evidenced by documented workarounds for known conflicts with other tools. The codebase remains actively refined with attention to usability details such as configuration file handling and cross-platform compatibility.
