karpathy/autoresearch

autoresearch is a framework for autonomous AI agents to conduct machine learning research experiments overnight on a single GPU.

The project addresses the challenge of exploring the vast hyperparameter and architectural space in neural network training by delegating experimentation to AI agents. The approach works by giving an agent a simplified single-GPU LLM training setup based on nanochat, where the agent can modify the training code, run experiments within a fixed five-minute time budget, evaluate results using validation bits per byte as the metric, and automatically keep or discard changes. Rather than directly editing Python files, researchers program Markdown files that provide context and instructions to the agents, establishing what amounts to an autonomous research organization.

Adoption makes sense for researchers interested in exploring how AI agents can accelerate the experimental cycle in model development, or those curious about delegating hyperparameter and architectural search to autonomous systems. The fixed five-minute training window and vocabulary-size-independent evaluation metric ensure fair comparison across different architectural choices. The project is deliberately minimal, with only three core files: a data preparation script that remains fixed, a training file that agents modify, and a Markdown program file that humans edit to guide agent behavior. This is an exploratory tool rather than a production system, suited to experimentation with autonomous research workflows rather than immediate practical deployment.

Development activity shows active iteration on the core concept with refinement of the agent-editable training loop and the instruction framework. The project maintains a deliberately stripped-down codebase to keep the scope manageable and the agent's task well-defined. Documentation includes contextual references to the underlying ideas and practical guidance for setup and execution.