Engram is a neural architecture module that introduces conditional memory via scalable lookup as a sparsity mechanism for large language language models.
The project addresses the limitation that Transformers lack a native primitive for knowledge lookup, unlike Mixture-of-Experts which enables conditional computation. Engram modernizes classic n-gram embeddings to achieve constant-time lookup, allowing models to retrieve static n-gram memory and fuse it with dynamic hidden states. The module uses deterministic addressing to enable offloading of massive embedding tables to host memory with minimal inference overhead during inference.
Engram suits researchers and practitioners building large language models who want to explore sparsity beyond mixture-of-experts approaches. The tool is particularly relevant for those investigating how to allocate capacity between neural computation and static memory, as the project identifies a U-shaped scaling law that guides optimal allocation between these two axes. The implementation demonstrates that under equivalent parameter and compute budgets, models augmented with Engram show consistent improvements across knowledge, reasoning, code, and math domains compared to mixture-of-experts baselines.
The project provides the official implementation accompanying a research paper, with architecture diagrams and evaluation results on large-scale pretraining experiments. The codebase is written in Python and made available under an open-source license, enabling direct reproduction of the reported scaling laws and empirical comparisons.
