Crawlee is a web scraping and browser automation library for Python that helps developers build reliable crawlers.
The tool addresses the challenge of extracting data from websites while evading bot detection and handling the complexity of modern web scraping. It works by providing a unified interface that supports multiple extraction methods—raw HTTP requests, Playwright for browser automation, BeautifulSoup for parsing, and Parsel for CSS and XPath selection. The library handles proxy rotation, both headless and headful browser modes, and can download various file types including HTML, PDF, and images. It abstracts away technical details like session management, request handling, and data persistence so developers can focus on defining what to scrape rather than how to scrape it.
Crawlee suits projects that need to extract data at scale for AI, machine learning, or retrieval-augmented generation pipelines. It is particularly valuable when crawlers must avoid detection by modern anti-bot systems, as the default configuration is designed to make requests appear human-like. The tool's rich configuration options allow fine-tuning of nearly every aspect, making it adaptable to both simple scraping tasks and complex crawling scenarios. A TypeScript implementation exists for developers working in that ecosystem.
The project maintains active development with passing build status and code coverage tracking. The tool is distributed as a PyPI package with optional extras to keep core dependencies minimal, and comprehensive documentation and tutorials are available on the project website.
