本文探讨了扩展输入长度对大语言模型（Large Language Models, LLMs）能力的影响。尽管近年来LLMs取得了诸多进展，但人们对其在不同输入长度下性能的一致性仍缺乏充分了解。我们通过提出一种专为评估输入长度影响而设计的新型QA推理框架来研究这一问题。我们使用同一样本的多个版本来隔离输入长度的影响，每个版本都通过不同长度、类型和位置的填充进行扩展。研究结果表明，在远短于其技术最大输入长度的情况下，LLMs的推理性能就出现了显著下降。我们发现这种下降趋势在我们数据集的每个版本中都存在，只是强度有所不同。此外，本研究还揭示，传统的下一词预测指标与LLMs在我们推理数据集上的性能呈负相关。我们对结果进行了分析，并识别出了失效模式，这些模式可作为未来研究的有用参考，有望为解决LLMs中观察到的局限性提供策略参考。
英文摘要：
This paper explores the impact of extending input lengths on the capabilities of Large Language Models (LLMs). Despite LLMs advancements in recent times, their performance consistency across different input lengths is not well understood. We investigate this aspect by introducing a novel QA reasoning framework, specifically designed to assess the impact of input length. We isolate the effect of input length using multiple versions of the same sample, each being extended with padding of different lengths, types and locations. Our findings show a notable degradation in LLMs' reasoning performance at much shorter input lengths than their technical maximum. We show that the degradation trend appears in every version of our dataset, although at different intensities. Additionally, our study reveals that the traditional metric of next word prediction correlates negatively with performance of LLMs' on our reasoning dataset. We analyse our results and identify failure modes that can serve as useful guides for future research, potentially informing strategies to address the limitations observed in LLMs.
On the Token Value Inequality in Efficient Reasoning
Purdue University（普渡大学）；Rochester Institute of Technology（罗切斯特理工学院）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；University of Missouri-Kansas City（密苏里大学堪萨斯城分校）；Rutgers University（罗格斯大学）；Meta AI
Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University College London, University of London（伦敦大学大学学院）；Dalian University of Technology（大连理工大学）
SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoning
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；BandAI, Bytedance（字节跳动BandAI）
Reasoning-Token Spikes Under Prompted Untruthful Responses in Large Language Models
Chapman University（查普曼大学）；INSERM U992, Cognitive Neuroimaging Unit, NeuroSpin Center（法国国家健康与医学研究院U992认知神经影像单元，神经自旋中心）；University of California, Los Angeles（加州大学洛杉矶分校）；California Institute of Technology（加州理工学院）
Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
University of California, Santa Barbara（加州大学圣芭芭拉分校）；University of California, Santa Cruz（加州大学圣克ruz分校）；University of California, Los Angeles（加州大学洛杉矶分校）；Purdue University（普渡大学）；LMSYS Org（LMSYS组织）；Microsoft（微软）
KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Amplify Adjacent Token Differences: Enhancing Long Chain-of-Thought Reasoning with Shift-FFN
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（认知与决策智能复杂系统实验室，自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Baichuan Inc（百川公司）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学数学科学学院）；Institute of Natural Sciences, Shanghai Jiao Tong University（上海交通大学自然科学研究院）
SeLaR: Selective Latent Reasoning in Large Language Models
Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology（广东省超高清沉浸式媒体技术重点实验室）；Shenzhen Graduate School, Peking University（北京大学深圳研究生院）
