arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.03474cs.LG
收藏

基于格的可微随机存取存储器

Differentiable Random Access Memory using Lattices

University of Cambridge(剑桥大学);InBalance

Adam P. Goucher, Rajan Troll

摘要:

我们介绍了一种可微分随机存取存储器模块与$O(1)$的性能,无论大小,扩展到数十亿的条目。该设计将条目存储在所选网格的点上,以通过利用对称性有效地计算任意点的最近邻。在此基础上,用单个存储器层来增强标准神经网络架构,我们可以将参数计数扩展到存储器限制,而计算开销可以忽略不计,从而以类似的成本提供更好的准确性。在大型语言建模任务中,这些具有更大容量的增强模型显著优于未修改的Transformer基线。我们发现,随着内存大小的不断扩大,测试的限制。

英文摘要:

We introduce a differentiable random access memory module with $O(1)$ performance regardless of size, scaling to billions of entries. The design stores entries on points of a chosen lattice to calculate nearest neighbours of arbitrary points efficiently by exploiting symmetries. Augmenting a standard neural network architecture with a single memory layer based on this, we can scale the parameter count up to memory limits with negligible computational overhead, giving better accuracy at similar cost. On large language modelling tasks, these enhanced models with larger capacity significantly outperform the unmodified transformer baseline. We found continued scaling with memory size up to the limits tested.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑