Attention Residuals (AttnRes) enables content-dependent retrieval over network depth, which can be viewed as single-stream softmax attention along the depth axis. Inspired by the success of multi-head attention along the sequence axis, we introduce Multi-Stream Attention Residuals (MSAR), an extension of AttnRes that generalizes this mechanism to multi-head depth attention, yielding substantial performance gains with minimal additional overhead. MSAR draws stream diversity from sequence context through multi-scale causal convolutions, allowing different heads to retrieve historical representations through distinct contextual views while preserving the integrity of the hidden representation. The retrieved streams are combined through lightweight adaptive mixing to generate the single-stream layer input. By retaining single-stream sources and expanding them on demand, MSAR supports multi-stream computation without requiring persistent multi-stream storage or communication. Across all 12 downstream benchmarks at the 18B and 28B MoE scales, MSAR consistently outperforms prior residual variants, including standard residual connections, manifold-constrained Hyper-Connections (mHC), and AttnRes. Scaling-law experiments yield an iso-loss compute speedup of $1.58\times$ over the standard residual baseline, compared with $1.35\times$ for AttnRes and $1.30\times$ for mHC. These results make MSAR a promising residual architecture for large-scale pre-training.
Transformers represent belief state geometry in their residual stream
Simplex（辛普莱克斯）；PIBBSS；Pitzer and Scripps College（皮策与斯克里普斯学院）；University College London（伦敦大学学院）；BITS Principles of Intelligent Behaviour in Biological and Social Systems（BITS生物与社会系统智能行为原理中心）
Uncovering Uncontrolled Repetition through Residual Stream Dynamics
Beijing University of Posts and Telecommunications（北京邮电大学）；JIUTIAN Research（中移九天）；Nanyang Technological University（南洋理工大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
School of AI, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Dots Studio, Xiaohongshu Inc.（小红书公司点点工作室）；University of Science and Technology of China（中国科学技术大学）；School of CS, Peking University（北京大学计算机科学学院）；The Chinese University of Hong Kong（香港中文大学）
Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；Adelaide University（阿德莱德大学）；Naval Group Pacific（太平洋海军集团）；Responsible AI Research Centre（负责任人工智能研究中心）
Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture
SciAI Center, Cornell University, USA（SciAI中心，康奈尔大学，美国）；Neural Coding and Brain Computing Unit, OIST, Japan（神经编码与脑计算单元，海洋研究所，日本）；SciAI Center, Center for Applied Mathematics, School of Civil and Environmental Engineering, Cornell University, USA（SciAI中心，应用数学中心，土木与环境工程学院，康奈尔大学，美国）
