MIDUS：记忆融合深度上采样
MIDUS: Memory-Infused Depth Up-Scaling
AI总结：
本文提出MIDUS，通过将复制的FFN分支替换为记忆层，利用轻量级检索残差能力提升模型性能与效率，引入Head-wise Memory Layer和HIVE实现头条件残差扩展。
摘要：
扩展预训练语言模型是一种增加容量而不从头训练大模型的实用方法。深度上采样（DUS）通过复制Transformer块并插入到预训练骨干中实现。此过程也复制了FFN密集块，增加了参数和计算成本，同时通过块级密集残差分支增加容量。然而，先前工作表明，增加的容量不必局限于密集FFN分支，而注意力头往往扮演异质角色，这促使更高效的头级残差修正。我们提出记忆融合深度上采样（MIDUS），将复制的FFN分支替换为记忆层，将增加的深度转化为轻量级检索基于残差容量。我们引入头级记忆层（HML），结合多头产品键记忆与头级隐式值扩展（HIVE）。HML为每个头分配不同的键空间，而HIVE通过紧凑投影从共享潜在银行中实现头特定的值。除了在性能和效率上的实证改进，我们头重要性和固定检索结构分析表征HML与HIVE作为结构上不同的头条件替代方案，替代基于FFN的残差扩展。
英文摘要：
Expanding pre-trained language models offers a practical way to increase capacity without training larger models from scratch. Depth Up-Scaling (DUS) does so by duplicating Transformer blocks and inserting them into a pre-trained backbone. This process also duplicates FFN-heavy blocks, increasing parameter and compute cost while adding capacity through a block-level dense residual branch. Yet prior work suggests that added capacity need not remain tied to dense FFN branches, while attention heads often play heterogeneous roles, motivating more efficient head-level residual corrections. We propose Memory-Infused Depth Up-Scaling (MIDUS), which replaces the duplicated FFN branches with memory layers and turns added depth into lightweight retrieval-based residual capacity. We introduce a Head-wise Memory Layer (HML), which combines multi-head product-key memory with Head-wise Implicit Value Expansion (HIVE). HML assigns each head a distinct key space, while HIVE realizes head-specific values from a shared latent bank through compact projections. Alongside empirical improvements in performance and efficiency, our head-importance and fixed-retrieval structural analyses characterize HML with HIVE as a structurally distinct, head-conditioned alternative to FFN-based residual expansion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
