Long-context language models (LCLMs), characterized by their extensive context window, are becoming popular. However, despite the fact that they are nearly perfect at standard long-context retrieval tasks, our evaluations demonstrate they fail in some basic cases. Later, we find they can be well addressed with a sufficient number of reasoning steps, guided by specific CoT prompts. This result emphasizes the potential necessity of solving specific long-context tasks using long-CoT methods, while previous long-context benchmarks always ignore the necessity of long reasoning for long-context tasks and treat them as direct QA tasks.
Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；National University of Singapore（新加坡国立大学）
Efficient Uncertainty Estimation with Gaussian Process for Reliable Dialog Response Retrieval
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；University of Science and Technology of China（中国科学技术大学）
深度神经网络在基于检索的对话系统中取得了卓越的性能，但它们被证明是校准不良的。虽然像Monte Carlo Dropout和Entrance这样的基本校准方法可以很好地校准，但这些方法在训练或推理阶段非常耗时。为了应对这些挑战，我们提出了一种用于基于BERT的对话式搜索的高效不确定性校准框架GPF-BERT，该框架在BERT架构之上采用高斯过程层和焦点损失来实现高质量的神经排序器。大量的实验验证了该方法的有效性。与基本标定方法相比，GPF-BERT在三个域内数据集和分布移位任务中实现了最低的经验标定误差（ECE），而在大多数情况下产生了最高的$R_{10}@1$和MAP性能。在时间消耗方面，我们的GPF-BERT具有8 $\times$的加速比。
LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；London School of Economics（伦敦经济学院）；BUPT（北京邮电大学）；Harbin Institute of Technology（哈尔滨工业大学）；SUTD（新加坡科技设计大学）；University of Oxford（牛津大学）
The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes
Singapore Institute of Technology（新加坡理工大学）；Nvidia AI Center (SNAIC)（英伟达人工智能中心（SNAIC））；MIDAS Lab, IIIT Delhi（IIIT德里MIDAS实验室）；MIDAS Lab, IIT Mandi（IIT曼迪MIDAS实验室）；Owl Autonomous Imaging, Inc.（Owl自主成像公司）；College of Computing & Data Science, NTU Singapore（新加坡南洋理工大学计算与数据科学学院）；NVIDIA AI Technology Centre, Singapore（英伟达新加坡人工智能技术中心）；Department of Computer Science and Engineering, IIT Kanpur（IIT坎普尔计算机科学与工程系）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Amazon.com Inc.（亚马逊公司）；USC Information Sciences Institute（南加州大学信息科学研究所）；Argonne National Laboratory（阿贡国家实验室）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；University of Chicago（芝加哥大学）
