Trafilatura is a Python package and command-line tool for gathering text and metadata from the web through crawling, scraping, and extraction into structured formats.
The tool addresses the challenge of converting raw HTML into meaningful, noise-free content. It combines web discovery capabilities including sitemap and feed support with a robust extraction engine that isolates main text while filtering out boilerplate elements like headers and footers. The extraction process balances precision against recall, and the tool supports parallel processing of both live URLs and previously downloaded HTML files. Output can be converted to multiple formats including plain text, Markdown, CSV, JSON, HTML, and XML-TEI.
Trafilatura suits projects that need to build text corpora from web sources at scale without requiring a database backend. It works well for news aggregation, content mining, and preparing training data for language models. The tool is particularly valuable for teams prioritizing extraction accuracy, as it consistently outperforms other open-source libraries in published benchmarks across multiple evaluation frameworks.
The project maintains steady activity with regular updates addressing both bug fixes and feature enhancements. Development includes ongoing refinement of the extraction algorithms and expansion of output format support. The maintainers actively respond to issues and incorporate community feedback into the tool's evolution.
