Attention Residuals is a drop-in replacement for standard residual connections in Transformers that enables selective aggregation of earlier layer representations through learned attention over depth.
Standard residual connections accumulate all layer outputs with uniform weights, which causes hidden-state magnitudes to grow unboundedly as depth increases and dilutes each layer's contribution. Attention Residuals replaces this fixed accumulation with softmax attention computed over preceding layer outputs, where each layer uses a learned pseudo-query to selectively attend to earlier representations. The approach includes a Block variant that partitions layers into groups and applies attention only over block-level representations, reducing memory requirements from O(Ld) to O(Nd) while retaining most of the performance gains.
The tool suits projects training large Transformers where scaling efficiency matters. Block Attention Residuals serves as a practical drop-in replacement with marginal computational overhead and recovers most of the full variant's benefits. The README demonstrates improvements across scaling laws and downstream benchmarks including reasoning, code generation, and general knowledge tasks, with particularly large gains on multi-step reasoning problems.
Development activity shows consistent focus on the core technique with empirical validation across multiple model scales and downstream tasks. The repository includes detailed pseudocode and training dynamics analysis that document how the approach mitigates known issues with PreNorm architectures. The project maintains references to the underlying research paper and arXiv preprint alongside implementation details.
