超越思维链：大语言模型潜在空间推理能力的基准测试
Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
AI总结：
本研究针对大语言模型内部推理能力量化不足的问题，引入含4000项的基准，通过让模型选择非英语初始响应令牌指示推理答案，评估18个模型后发现GPT-4.5准确率最高，揭示了模型内部推理策略及安全相关研究方向。
摘要：
大语言模型（LLMs）既能在其潜在空间内部执行推理计算，也能通过生成思维链等显式令牌序列在外部执行推理计算。通过扩展测试时计算，在增强推理能力方面已取得显著进展。然而，理解和量化模型内部推理能力——即模型在单个令牌预测之间做出的推理“跳跃”——仍然至关重要。本研究引入了一个基准（n = 4000项），旨在量化不同领域的模型内部推理能力。我们通过让LLMs不通过描述性文本，而是通过选择其初始响应令牌的特定语言来指示推理问题的正确解决方案，从而实现这一目标，该语言不同于基准语言英语。这不仅要求模型在其上下文窗口之外进行推理，还要求它们克服默认以提示相同语言响应的倾向，从而带来额外的认知负担。我们评估了18个LLMs，结果显示性能存在显著差异，GPT-4.5的准确率最高（74.7%），优于Grok-2（67.2%）和Llama 3.1 405B（65.6%）等模型。控制实验和难度缩放分析表明，尽管LLMs参与内部推理，但我们不能排除在某些条件下的启发式利用，这标志着未来研究的一个方向。我们的实验表明，LLMs可以通过潜在空间计算“思考”，揭示了需要进一步理解的模型内部推理策略，特别是在安全相关问题方面，如无显式令牌痕迹的隐蔽规划、目标寻求或欺骗。
英文摘要：
Large language models (LLMs) can perform reasoning computations both internally within their latent space and externally by generating explicit token sequences like chains of thought. Significant progress in enhancing reasoning abilities has been made by scaling test-time compute. However, understanding and quantifying model-internal reasoning abilities - the inferential "leaps" models make between individual token predictions - remains crucial. This study introduces a benchmark (n = 4,000 items) designed to quantify model-internal reasoning in different domains. We achieve this by having LLMs indicate the correct solution to reasoning problems not through descriptive text, but by selecting a specific language of their initial response token that is different from English, the benchmark language. This not only requires models to reason beyond their context window, but also to overrise their default tendency to respond in the same language as the prompt, thereby posing an additional cognitive strain. We evaluate a set of 18 LLMs, showing significant performance variations, with GPT-4.5 achieving the highest accuracy (74.7%), outperforming models like Grok-2 (67.2%), and Llama 3.1 405B (65.6%). Control experiments and difficulty scaling analyses suggest that while LLMs engage in internal reasoning, we cannot rule out heuristic exploitations under certain conditions, marking an area for future investigation. Our experiments demonstrate that LLMs can "think" via latent-space computations, revealing model-internal inference strategies that need further understanding, especially regarding safety-related concerns such as covert planning, goal-seeking, or deception emerging without explicit token traces.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
