Crawlee is a web scraping and browser automation library for Node.js that helps developers build reliable crawlers in JavaScript and TypeScript.
The tool addresses the challenge of extracting data from websites while evading bot detection and handling the complexity of modern web infrastructure. It works by providing multiple crawler implementations that can operate in headful or headless modes, supporting various backends including Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP requests. The library handles proxy rotation, file downloads in multiple formats, and manages request queuing and result storage automatically, allowing developers to focus on extraction logic rather than infrastructure concerns.
Crawlee suits projects that need to scrape data for AI, LLMs, RAG systems, or similar applications where reliable data extraction is critical. It works well for teams building production crawlers that must handle anti-bot measures and scale across multiple requests. The tool is particularly valuable when you need flexibility in choosing your underlying browser automation or HTTP library, since it abstracts over multiple backends rather than locking you into a single approach. The library provides both a CLI for quick experimentation and manual installation for integration into existing projects, with configurable storage backends for results and requests.
The project maintains active development with continuous integration testing and a community presence on Discord. Documentation is comprehensive with tutorials, guides, and examples available on the project website. The codebase is written in TypeScript, providing type safety for users building crawlers. The tool is distributed as an NPM package with modular architecture, allowing developers to install only the components they need.
