让大语言模型（LLM）能够处理长上下文是当前的研究热点。大多数LLM基于旋转位置嵌入（rotary position embedding, RoPE）——一种流行的位置编码方法——构建，因此一个重要的研究方向是将在相对较短文本上训练的RoPE外推到长得多的文本上。大量研究通过扩展RoPE的公式来提升外推效果，但很少有研究全面揭示其内在工作机制。本文从注意力视角、结合两项基准任务，对RoPE扩展提供直观且深入的理解。大量实验得出了若干有价值的发现：1）保持预训练长度下的注意力模式可提升外推效果；2）较大的注意力不确定性会导致检索错误；3）为RoPE扩展使用更长的持续预训练长度可降低注意力不确定性，并显著增强外推能力。
英文摘要：
Enabling LLMs to handle lengthy context is currently a research hotspot. Most LLMs are built upon rotary position embedding (RoPE), a popular position encoding method. Therefore, a prominent path is to extrapolate the RoPE trained on comparably short texts to far longer texts. A heavy bunch of efforts have been dedicated to boosting the extrapolation via extending the formulations of the RoPE, however, few of them have attempted to showcase their inner workings comprehensively. In this paper, we are driven to offer a straightforward yet in-depth understanding of RoPE extensions from an attention perspective and on two benchmarking tasks. A broad array of experiments reveals several valuable findings: 1) Maintaining attention patterns to those at the pretrained length improves extrapolation; 2) Large attention uncertainty leads to retrieval errors; 3) Using longer continual pretraining lengths for RoPE extensions could reduce attention uncertainty and significantly enhance extrapolation.
Mixture of In-Context Experts Enhance LLMs' Long Context Awareness
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；XiaoMi AI Lab（小米AI实验室）；Career Science Lab, BOSS Zhipin（BOSS直聘职业科学实验室）；NLP Center, BOSS Zhipin（BOSS直聘自然语言处理中心）
DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search
TJUNLP Lab, College of Intelligence and Computing, Tianjin University（天津大学智能计算学院）；School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）
The University of Hong Kong（香港大学）；University of Michigan, Ann Arbor（密歇根大学安阿伯分校）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）
Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information
School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Research Center for SCIR, Harbin Institute of Technology（哈尔滨工业大学SCIR研究中心）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Yunnan University（云南大学）
MaLei at MultiClinSUM: Summarisation of Clinical Documents using Perspective-Aware Iterative Self-Prompting with LLMs
University of Manchester, UK（曼彻斯特大学）；Modul University Vienna, Austria（维也纳Modul大学）；Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands（莱顿先进计算机科学研究所（LIACS）、莱顿大学）；Leiden University Medical Center, The Netherlands（莱顿大学医学中心）
Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
School of Computer Science, Fudan University（复旦大学计算机科学学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Institute of Modern Languages and Linguistics, Fudan University（复旦大学现代语言与语言学研究所）；Institute of Trustworthy Embodied Artificial Intelligence, Fudan University（复旦大学可信具身人工智能研究所）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Pengcheng Laboratory（鹏城实验室）；Hikvision Inc（海康威视公司）
Perspective Transition of Large Language Models for Solving Subjective Tasks
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China（计算机科学与技术系，人工智能研究院，清华大学，北京，中国）；Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China（人工智能产业研究院（AIR），清华大学，北京，中国）；Jiuquan Satellite Launch Center (JSLC), Gansu, China（酒泉卫星发射中心（JSLC），甘肃，中国）；Harbin Institute of Technology, Harbin, China（哈尔滨工业大学，哈尔滨，中国）；College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China（计算机科学与软件工程学院，深圳大学，深圳，中国）；Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China（江苏省语言能力协同创新中心，江苏，中国）；Tencent Inc, China（腾讯公司，中国）
Test-time Scaling of LLMs: A Survey from A Subproblem Structure Perspective
Department of Computer Science and Engineering, The Pennsylvania State University（宾夕法尼亚州立大学计算机科学与工程系）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
