Pholcus is a distributed web crawler framework written in pure Go that handles high-concurrency data collection tasks.
The tool addresses the need to scrape web content at scale by providing three execution modes: standalone for immediate use, server mode for task distribution, and client mode for task execution. It supports multiple download engines including a high-concurrency HTTP client, PhantomJS, and headless Chrome for JavaScript-heavy sites. Data extraction rules can be written as compiled Go code for performance or as dynamic JavaScript/XML rules that load without recompilation. The framework handles common crawling challenges through intelligent cookie management, proxy IP rotation, random delays to simulate human behavior, automatic deduplication, retry logic, and persistent progress tracking for resumable crawls.
Developers should choose this tool if they need distributed crawling with fine-grained control over concurrency and output formats. It suits projects requiring multiple data destinations—MySQL, MongoDB, Kafka, Beanstalkd, CSV, or Excel—and those needing flexible rule definition. The tool provides three interfaces: a web UI for cross-platform browser control, a native Windows GUI, and command-line access for batch scheduling. The README includes over thirty built-in example rules to accelerate development.
The project shows active maintenance with regular updates addressing Go version compatibility and framework improvements. Development focuses on expanding engine capabilities and refining the distributed communication layer, with particular attention to supporting modern browser automation through Chrome integration. The codebase demonstrates ongoing refinement of core features like cookie handling, proxy management, and the full-duplex socket framework for distributed communication.
