大型语言模型的推理停滞：对前沿模型能力的调查
Large Language Models' Reasoning Stalls: An Investigation into the Capabilities of Frontier Models
AI总结：
研究评估了2023年12月至2024年8月间前沿LLM在PRONTOQA上的ATP推理能力，发现其推理进展停滞，提升主要归因于系统提示或思维链策略，且模型最擅长前向链策略，正确推理与正确结论仅呈低正相关。
摘要：
研究了检验大型语言模型（LLMs）使用自动定理证明器（ATP）推理策略能力的经验方法。我们评估了2023年12月和2024年8月的最新模型在PRONTOQA蒸汽压路机推理问题上的表现。为此，我们开发了评估LLM响应准确性和正确答案相关性的方法。结果显示，在九个月期间，提升LLM推理能力的进展已停滞。通过追踪补全标记，我们表明自GPT-4发布以来，推理能力的几乎所有提升都可归因于隐藏的系统提示，或训练模型自动使用通用的思维链提示策略。在尝试的ATP推理策略中，我们发现当前的前沿LLM最能遵循自下而上（也称为前向链）策略。在包含正确推理的LLM响应与得出正确结论之间，发现了低正相关。
英文摘要：
Empirical methods to examine the capability of Large Language Models (LLMs) to use Automated Theorem Prover (ATP) reasoning strategies are studied. We evaluate the performance of State of the Art models from December 2023 and August 2024 on PRONTOQA steamroller reasoning problems. For that, we develop methods for assessing LLM response accuracy and correct answer correlation. Our results show that progress in improving LLM reasoning abilities has stalled over the nine month period. By tracking completion tokens, we show that almost all improvement in reasoning ability since GPT-4 was released can be attributed to either hidden system prompts or the training of models to automatically use generic Chain of Thought prompting strategies. Among the ATP reasoning strategies tried, we found that current frontier LLMs are best able to follow the bottom-up (also known as forward-chaining) strategy. A low positive correlation was found between an LLM response containing correct reasoning and arriving at the correct conclusion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
