创造力还是蛮力？以脑筋急转弯为窗口探究大语言模型的问题解决能力
Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models
AI总结：
本研究引入长篇叙事脑筋急转弯基准，从多层推理维度探究大语言模型的解题策略，发现其具备一定创造性解题能力，但仍存在依赖蛮力的情况，为推理能力改进指明方向。
摘要：
准确率仍是评估AI系统的标准指标，但它对模型如何得出解决方案的揭示作用有限。本研究中，我们引入了一个基于长篇叙事形式脑筋急转弯的基准，以更深入地探究模型所使用的推理策略类型。脑筋急转弯十分契合这一目标，因为它们可通过多种方法求解，比如运用创造性洞见的少步骤解法，或是采用更多蛮力的更长解法。我们从多层推理维度研究大语言模型（LLMs），不仅关注正确性，还关注其解决方案的质量与创造性。我们探究了推理过程的多个方面：（1）将脑筋急转弯语义解析为精确的数学竞赛风格格式；（2）基于这些数学形式生成解决方案；（3）依据标准答案（gold solutions）自我修正解决方案；（4）生成解决方案的逐步梗概；（5）利用提示（hints）。我们发现，LLMs在很多情况下能够找到具有创造性、富有洞见的脑筋急转弯解法，这表明它们具备了以创造性方式解决新问题所需的部分能力。尽管如此，仍存在一些情况：尽管有更高效、更具创造性的解法可用，它们却依赖蛮力，这为LLMs推理能力的提升指明了一个潜在方向。
英文摘要：
Accuracy remains a standard metric for evaluating AI systems, but it offers limited insight into how models arrive at their solutions. In this work, we introduce a benchmark based on brainteasers written in long narrative form to probe more deeply into the types of reasoning strategies that models use. Brainteasers are well-suited for this goal because they can be solved with multiple approaches, such as a few-step solution that uses a creative insight or a longer solution that uses more brute force. We investigate large language models (LLMs) across multiple layers of reasoning, focusing not only on correctness but also on the quality and creativity of their solutions. We investigate many aspects of the reasoning process: (1) semantic parsing of the brainteasers into precise mathematical competition style formats; (2) generating solutions from these mathematical forms; (3) self-correcting solutions based on gold solutions; (4) producing step-by-step sketches of solutions; and (5) making use of hints. We find that LLMs are in many cases able to find creative, insightful solutions to brainteasers, suggesting that they capture some of the capacities needed to solve novel problems in creative ways. Nonetheless, there also remain situations where they rely on brute force despite the availability of more efficient, creative solutions, highlighting a potential direction for improvement in the reasoning abilities of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
