arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.13073cs.LGq-bio.GNq-bio.QM
收藏

Double-Take:在双向记忆中提升长上下文下的容量、绑定与选择性

Double-Take: Improving Capacity, Binding, and Selectivity In Long-Context Within Bidirectional Memory

Rutgers University(罗格斯大学);University of South Florida(南佛罗里达大学)

Vakul Raghavan, Utkarsh Grover, Mingyang Mao, Chengwen Wang, Xiaomin Lin

AI总结:

该研究针对长序列处理中远距离信息可用性问题,提出Double-Take模型,通过多步非固定更新等技术提升记忆的容量、绑定与选择性,在基因组数据上取得良好性能并实现零样本迁移。

摘要:

处理长序列并不保证远距离信息仍可使用。随着上下文增长,固定大小的记忆必须保留大量关联、连接不同位置的信息,并在不被所有观测内容淹没的情况下保留相关结构,这些是不同的要求:记忆可存储更多信息却不具备检索所需的关系,也可传递这些关系却不保留重要的关系。因此,挑战并非简单延长上下文长度,而是构建随上下文增长仍保持可用的记忆。我们在大分块测试时训练(TTT)中研究该问题,发现两个基本缺陷:单步写入会累积干扰,限制容量;自关联写入无法形成跨位置绑定。我们引入Double-Take(DT),通过多步非固定更新和跨位置混合修复这些缺陷,并添加双向变化敏感写入以实现选择性保留。仅在32kb基因组上下文上训练时,DT在131kb时达到0.927的AUROC,在近1Mb时达到0.856,且实现从人类到小鼠的零样本迁移。这些结果共同表明,容量、绑定和选择性是长程记忆可用的三个必要条件。

英文摘要:

Processing a long sequence does not guarantee that distant information remains usable. As context grows, a fixed-size memory must retain many associations, connect information across positions, and preserve relevant structure without being overwhelmed by everything it observes. These are distinct requirements: a memory may store more information without representing the relationships needed for retrieval, and it may transport those relationships without preserving the ones that matter. Thus, the challenge is not simply extending context length, but constructing a memory that remains usable as context grows. We study this problem in large-chunk test-time training (TTT) and identify two basic failures: single-step writes accumulate interference, limiting capacity, while self-associative writes cannot form cross-position bindings. We introduce Double-Take (DT), which repairs these failures with multi-step unpinned updates and cross-position mixing, and adds bidirectional change-sensitive writes for selective retention. Capacity alone leaves end-to-end recall at the $0.10$ chance floor; adding binding raises it to $0.92$. Trained only on $32\,\mathrm{kb}$ genomic contexts, DT reaches $0.927$ AUROC at $131\,\mathrm{kb}$ and $0.856$ near $1\,\mathrm{Mb}$, while transferring zero-shot from human to mouse. Together, these results identify capacity, binding, and selectivity as three requirements for usable long-range memory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑