node-crawler is a web crawler and spider for Node.js that provides server-side DOM manipulation with automatic jQuery insertion via Cheerio.
The tool addresses the need to extract data from websites by queuing HTTP requests and parsing HTML responses with jQuery-like syntax on the server side. It handles the mechanics of web crawling: managing connection pools, controlling request rates, retrying failed requests, detecting and converting character encodings, and prioritizing requests in a queue. Developers write callback functions that receive parsed DOM objects, allowing them to extract data using familiar jQuery selectors without running a browser.
The project suits teams building data extraction pipelines, monitoring systems, or aggregation tools that need to crawl many pages efficiently. It works well for projects where server-side parsing is sufficient and browser automation overhead is unnecessary. The tool is designed as a modern ESM-only package requiring Node.js 22 or above, reflecting a commitment to current JavaScript standards. It includes an agent skill for integration with AI agents, enabling them to perform large-scale crawling tasks. Developers migrating from the earlier version should expect breaking changes, particularly around form data handling and the removal of CommonJS support, though a beta version with dual ESM and CommonJS support exists for gradual transitions.
Development activity shows consistent maintenance with regular updates addressing dependency changes and TypeScript improvements. The project maintains comprehensive documentation including migration guides for users upgrading from prior versions. The codebase demonstrates attention to modern tooling practices through its ESM-first approach and TypeScript implementation. Active engagement with the AI agent ecosystem through the ClawHub integration indicates responsiveness to emerging use cases beyond traditional web scraping.
