arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.06324cs.LGcs.CLcs.NE
收藏

B'MOJO:具备遗觉记忆与衰减记忆的基础模型混合状态空间实现

B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory

AWS AI Labs(亚马逊云科技人工智能实验室)

Luca Zancato, Arjun Seshadri, Yonatan Dukler, Aditya Golatkar, Yantao Shen, Benjamin Bowman, Matthew Trager, Alessandro Achille, Stefano Soatto

AI总结:

本研究提出B'MOJO混合架构,无缝融合遗觉与衰减记忆,涵盖现有主流模型为特例,在直推推理任务表现更优,语言建模效率与效果兼具,长序列推理能力更强。

摘要:

我们提出了一类支持直推式推理的架构,该架构允许记忆增长至一个有限但先验未知的边界,同时能高效利用有限资源完成推理。当前的架构要么在有限跨度内以遗觉方式表征数据(即Transformer中的“上下文”),要么在无限跨度内以衰减方式表征数据(即状态空间模型,简称SSM)。近期的混合架构虽结合了遗觉记忆与衰减记忆,但存在局限性:设计者或学习过程无法无缝调节二者,也无法扩展遗觉记忆的跨度。\n我们借助随机实现理论的思想,开发了一类名为B'MOJO的模型,可在一个基础的可组合模块内无缝融合遗觉记忆与衰减记忆。该整体架构可用于实现多种模型,这些模型能通过原生整合异步更新记忆的检索功能,访问“上下文内”的短期遗觉记忆、“权重内”的永久结构记忆、“状态内”的衰减记忆,以及“存储内”的长期遗觉记忆。我们证明,Transformer、Mamba等现有SSM,以及Jamba等混合架构都是B'MOJO的特例;我们还描述了一个将开源的基础实现,该实现可高效堆叠并在硬件上扩展。\n我们在联想回忆等直推式推理任务上测试了B'MOJO,其表现优于现有SSM和混合模型;作为基线,我们在普通语言建模任务上进行了测试,在最高14亿参数的规模下,B'MOJO的困惑度与同等规模的Transformer和SSM相当,训练速度最高快10%。最后我们发现,B'MOJO调节遗觉记忆与衰减记忆的能力,使其在最长32K token的测试长序列上拥有更优的推理效果,该长度是训练期间见过的最长序列的四倍。

英文摘要:

We describe a family of architectures to support transductive inference by allowing memory to grow to a finite but a-priori unknown bound while making efficient use of finite resources for inference. Current architectures use such resources to represent data either eidetically over a finite span ("context" in Transformers), or fading over an infinite span (in State Space Models, or SSMs). Recent hybrid architectures have combined eidetic and fading memory, but with limitations that do not allow the designer or the learning process to seamlessly modulate the two, nor to extend the eidetic memory span. We leverage ideas from Stochastic Realization Theory to develop a class of models called B'MOJO to seamlessly combine eidetic and fading memory within an elementary composable module. The overall architecture can be used to implement models that can access short-term eidetic memory "in-context," permanent structural memory "in-weights," fading memory "in-state," and long-term eidetic memory "in-storage" by natively incorporating retrieval from an asynchronously updated memory. We show that Transformers, existing SSMs such as Mamba, and hybrid architectures such as Jamba are special cases of B'MOJO and describe a basic implementation, to be open sourced, that can be stacked and scaled efficiently in hardware. We test B'MOJO on transductive inference tasks, such as associative recall, where it outperforms existing SSMs and Hybrid models; as a baseline, we test ordinary language modeling where B'MOJO achieves perplexity comparable to similarly-sized Transformers and SSMs up to 1.4B parameters, while being up to 10% faster to train. Finally, we show that B'MOJO's ability to modulate eidetic and fading memory results in better inference on longer sequences tested up to 32K tokens, four-fold the length of the longest sequences seen during training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑