WikiExtractor is a Python tool for extracting and cleaning plain text from Wikipedia database backup dumps.
The tool solves the problem of converting raw Wikipedia XML dumps into usable plain text by preprocessing the entire dump to extract and expand template definitions, then processing articles in parallel using multiprocessing. A cache of parsed templates is maintained to speed up repeated extractions. The tool offers a `--no-templates` option that significantly accelerates processing by skipping template expansion entirely, and a separate Cirrus Extractor variant that works with pre-expanded Cirrus dumps from the cirrussearch repository.
WikiExtractor requires only Python 3 with no additional library dependencies, making it straightforward to deploy. It suits projects that need to bulk-process Wikipedia content for natural language processing, machine learning training data, or text analysis. The tool can be invoked directly as a script or installed via package managers. Output is distributed across multiple files of similar size in a specified directory, each containing documents in a structured format. Users working with large dumps should consider whether template expansion is necessary for their use case, as disabling it provides substantial performance gains.
Issues and pull requests often wait weeks or longer for a first response.
