arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00398cs.LG
收藏

MemoryLLM: 可插拔的可解释性前馈记忆用于Transformer

MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers

Apple(苹果公司)

Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Arnav Kundu, Mehrdad Farajtabar, Minsik Cho

AI总结:

MemoryLLM通过解耦前馈模块与自注意力机制,实现无上下文的token级神经检索记忆,提升大语言模型的可解释性和推理效率。

摘要:

理解Transformer组件在大语言模型(LLM)中的运作机制非常重要,因为这是近年来人工智能技术进步的核心。在本工作中,我们重新审视与前馈模块(FFN)可解释性相关的挑战,并提出了MemoryLLM,旨在将FFN与自注意力机制解耦,使我们能够将解耦的FFN作为无上下文的token级神经检索记忆进行研究。具体而言,我们探讨了输入token如何访问FFN参数内的内存位置以及FFN内存在不同下游任务中的重要性。MemoryLLM通过直接使用token嵌入训练FFN,使其在不依赖自注意力的情况下实现无上下文的FFN。这种方法使FFN可以预计算为token级查找(ToLs),从而实现按需在VRAM和存储之间转移,进一步提高推理效率。我们还引入了Flex-MemoryLLM,将其定位在传统Transformer设计和MemoryLLM之间。该架构弥合了由于使用无上下文的token级嵌入训练FFN而导致的性能差距。

英文摘要:

Understanding how transformer components operate in LLMs is important, as it is at the core of recent technological advances in artificial intelligence. In this work, we revisit the challenges associated with interpretability of feed-forward modules (FFNs) and propose MemoryLLM, which aims to decouple FFNs from self-attention and enables us to study the decoupled FFNs as context-free token-wise neural retrieval memory. In detail, we investigate how input tokens access memory locations within FFN parameters and the importance of FFN memory across different downstream tasks. MemoryLLM achieves context-free FFNs by training them in isolation from self-attention directly using the token embeddings. This approach allows FFNs to be pre-computed as token-wise lookups (ToLs), enabling on-demand transfer between VRAM and storage, additionally enhancing inference efficiency. We also introduce Flex-MemoryLLM, positioning it between a conventional transformer design and MemoryLLM. This architecture bridges the performance gap caused by training FFNs with context-free token-wise embeddings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑