arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.07905cs.AIcs.LG
收藏

对数记忆网络(LMNs):面向资源受限环境的高效长程序列建模

Logarithmic Memory Networks (LMNs): Efficient Long-Range Sequence Modeling for Resource-Constrained Environments

Mohamed A. Taha

AI总结:

针对传统长程序列模型效率低的问题,提出采用分层对数树结构的对数记忆网络LMNs,将注意力复杂度降至O(log(n)),适配资源受限环境,代码已开源。

摘要:

长程序列建模是自然语言处理与时间序列分析的关键环节,但循环神经网络(RNNs)、Transformer等传统模型存在计算与内存效率低下,处理长序列时问题尤为突出。本文提出对数记忆网络(LMNs),这一新型架构借助分层对数树结构高效存储与检索过往信息。LMNs可动态总结历史上下文,将注意力机制的内存占用与计算复杂度从O(n²)大幅降至O(log(n))。该模型采用单向量定向注意力机制访问存储信息,内存块构建工作器(摘要器)层具备两种运行模式:训练阶段采用并行执行模式以高效处理分层树结构,推理阶段采用顺序执行模式,充当内存管理系统。它还能隐式编码位置信息,无需显式位置编码。这些特性使LMNs成为资源受限环境下处理长程序列的鲁棒可扩展方案,在效率与可扩展性方面带来实际提升。代码以MIT许可证在GitHub公开:https://github.com/AhmedBoin/LogarithmicMemory。

英文摘要:

Long-range sequence modeling is a crucial aspect of natural language processing and time series analysis. However, traditional models like Recurrent Neural Networks (RNNs) and Transformers suffer from computational and memory inefficiencies, especially when dealing with long sequences. This paper introduces Logarithmic Memory Networks (LMNs), a novel architecture that leverages a hierarchical logarithmic tree structure to efficiently store and retrieve past information. LMNs dynamically summarize historical context, significantly reducing the memory footprint and computational complexity of attention mechanisms from O(n2) to O(log(n)). The model employs a single-vector, targeted attention mechanism to access stored information, and the memory block construction worker (summarizer) layer operates in two modes: a parallel execution mode during training for efficient processing of hierarchical tree structures and a sequential execution mode during inference, which acts as a memory management system. It also implicitly encodes positional information, eliminating the need for explicit positional encodings. These features make LMNs a robust and scalable solution for processing long-range sequences in resource-constrained environments, offering practical improvements in efficiency and scalability. The code is publicly available under the MIT License on GitHub: https://github.com/AhmedBoin/LogarithmicMemory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑