x-ray is a web scraper that extracts structured data from HTML by filtering out noise through a flexible selector-based approach.
The tool solves the problem of converting unstructured HTML into clean, organized data by letting you define the shape of output you want independently from the page structure. It uses a jQuery-like selector syntax enhanced with attribute selection, allowing you to pull data into strings, arrays, objects, or nested structures of your choosing. The API is composable, meaning you can build complex scraping logic by combining simpler operations.
x-ray suits projects that need to extract data from multiple pages or crawl through paginated content responsibly. It includes built-in support for pagination, concurrency control, request delays, throttling, and timeouts to avoid overwhelming target servers. The tool supports pluggable drivers, with HTTP and PhantomJS drivers currently available, letting you choose the right scraper for your needs. If you need to handle JavaScript-heavy sites, the PhantomJS driver option provides that capability. The breadth-first crawler follows a predictable flow when moving from one page to the next, and scraped results can be streamed to files to protect against data loss if errors occur mid-crawl.
Development activity in the project is dominated by enhancement-focused work in the issue tracker.
