本文提出 Extended Mind Transformers，通过改进 Memorizing Transformers 的位置编码更新机制免除了微调需求，利用模型自身的键/查询系统在多数解码器层检索外部记忆，并在新基准上平均超越 SOTA 6%。
摘要：
预训练语言模型展现出通用智能和常识，但在推理时，长输入很快成为记忆信息的瓶颈。我们重新提出一种简单的方法 Memorizing Transformers（Wu et al., 2022），该方法使模型能够访问一个预计算的记忆库。我们表明，通过批判性地评估检索到的键和值的位置编码应如何更新，可以解决原始方法的许多缺陷，例如需要微调。这种直观的方法利用模型自身的键/查询系统在每个生成步骤中选择并关注最相关的记忆，而不是使用外部嵌入。我们证明了外部信息在大多数解码器层中被检索的重要性，这与之前的工作相反。我们开源了一个新的反事实长程检索基准，并表明 Extended Mind Transformers 平均比当今的最先进水平高出 6%。
英文摘要：
Pre-trained language models demonstrate general intelligence and common sense, but long inputs quickly become a bottleneck for memorizing information at inference time. We resurface a simple method, Memorizing Transformers (Wu et al., 2022), that gives the model access to a bank of pre-computed memories. We show that it is possible to fix many of the shortcomings of the original method, such as the need for fine-tuning, by critically assessing how positional encodings should be updated for the keys and values retrieved. This intuitive method uses the model's own key/query system to select and attend to the most relevant memories at each generation step, rather than using external embeddings. We demonstrate the importance of external information being retrieved in a majority of decoder layers, contrary to previous work. We open source a new counterfactual long-range retrieval benchmark, and show that Extended Mind Transformers outperform today's state of the art by 6% on average.
Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis
EvolvTrip: Enhancing Literary Character Understanding with Temporal Theory-of-Mind Graphs
The University of Manchester（曼彻斯特大学）；King’s College London（伦敦国王学院）；Huawei London Research Centre（华为伦敦研究中心）；Huawei Technologies Co., Ltd.（华为技术有限公司）；The Alan Turing Institute（艾伦·图灵研究所）
Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM
University of Southern California（南加州大学）；The University of Queensland（昆士兰大学）；University of California, San Diego（加州大学圣地亚哥分校）；University of California, Merced（加州大学默塞德分校）
A Transformer-Based Framework for Greek Sign Language Production using Extended Skeletal Motion Representations
School of ECE, National Technical University of Athens（雅典国立技术大学电气与计算机工程学院）；Robotics Institute, Athena Research Center（雅典娜研究中心机器人研究所）；Institute of Computer Science, Foundation for Research & Technology - Hellas (FORTH)（希腊研究与技术基金会计算机科学研究所）
