ARC-task-gen is a task generator that creates original ARC-AGI-1-style problems matched to the distribution of the public evaluation set.
The tool addresses the limitation that public benchmarks cannot fully isolate few-shot rule induction from potential prior familiarity with existing tasks. It generates a private evaluation set with similar statistical properties to the public ARC-AGI-1 dataset, enabling researchers to measure model performance on genuinely novel problems that follow the same underlying distribution. The generated tasks are output in standard ARC format, making them compatible with existing evaluation harnesses.
This tool is suited for researchers evaluating reasoning models on problems they are unlikely to have encountered during training. It provides a complementary measure to public-benchmark performance, particularly valuable when assessing models that might have been exposed to public ARC tasks. The generated task set enables controlled experiments studying what models learn from demonstrations without confounding factors from task familiarity.
The project maintains focused development around task generation and distribution matching, with documentation provided in an instructions file for users to generate their own task sets. The work is grounded in peer-reviewed research and has undergone independent reproduction by multiple researchers with established credentials in machine learning and benchmark design.
