The nlp-progress repository serves as a comprehensive tracking system for state-of-the-art results and benchmark datasets across Natural Language Processing tasks. Maintained by Sebastian Ruder, the repository provides researchers and practitioners with a centralized reference point for understanding current performance levels on standard NLP benchmarks. The project is accessible both through its GitHub repository and via a dedicated website at nlpprogress.com, making it a widely-referenced resource in the NLP research community.
The repository's scope is extensive, covering both foundational NLP tasks and contemporary challenges. For English, it tracks over forty distinct tasks including automatic speech recognition, constituency parsing, coreference resolution, dialogue systems, machine translation, named entity recognition, question answering, sentiment analysis, and summarization, among many others. Beyond English, the repository extends coverage to multiple languages including Vietnamese, Hindi, Chinese, French, Russian, Spanish, Portuguese, Korean, Nepali, Bengali, Persian, Turkish, German, and Arabic, with language-specific task documentation for each.
The organizational structure centers on detailed markdown files for each task, where researchers can find descriptions of benchmark datasets, evaluation metrics, annotated examples, and tables of state-of-the-art results sorted by performance. The repository emphasizes published results from peer-reviewed papers, with exceptions made for influential preprints. Contributors are encouraged to add implementation links, distinguishing between official and unofficial code implementations when available.
Community contribution is facilitated through straightforward processes documented in the repository. Users can edit task files directly through GitHub's interface to add new results, or submit pull requests for entirely new tasks and datasets. The contribution guidelines specify that results should come from published papers, datasets should have been evaluated in at least one published work beyond their introduction, and code links should be provided when available. The repository maintains a wish list of missing tasks including bilingual dictionary induction, discourse parsing, and knowledge base population, indicating areas where community contributions would be particularly valuable.
The repository provides structured data export capabilities, allowing users to extract all tracked information into machine-readable JSON format with parsed tasks, descriptions, and state-of-the-art tables. This functionality supports programmatic access to the data for researchers building tools or conducting meta-analyses of NLP progress.
According to GitGenius activity tracking, the repository shows median issue and pull request response latency of 22 hours across tracked items, with mean latency of 182.3 hours, indicating active but variable engagement patterns. The most active contributors tracked include Siddharth-Latthe-07 and sebastianruder. The repository shares contributors with other significant projects including kaldi-asr/kaldi, streamlit/streamlit, and tensorflow/models, reflecting its position within a broader ecosystem of machine learning and NLP infrastructure projects. The repository is classified across multiple domains including natural language processing, benchmarking, state-of-the-art tracking, research progress documentation, datasets, evaluation metrics, and performance comparison.
