MacGyver：大型语言模型是创造性问题解决者吗？
MacGyver: Are Large Language Models Creative Problem Solvers?
AI总结：
本研究构建了含1600余道现实问题的MACGYVER数据集，对比人类与LLM的创造性问题解决能力，分析了二者的互补性挑战，并验证了新型提示技术提升LLM该能力的潜力，补充了机器智能评估维度。
摘要：
我们在一种新型受限场景下探究了现代大型语言模型（LLM）的创造性问题解决能力。为此，我们构建了MACGYVER数据集——一个自动生成的数据集，包含1600余道现实世界问题，专门设计用于激发对物品的创新性使用，并要求跳出固有思维模式。随后我们将该数据集同时提供给LLM和人类，对比分析二者的问题解决能力。MACGYVER对两类群体都具有挑战性，但挑战的形式独特且互补。例如，人类在熟悉的任务中表现出色，但在领域特定知识相关任务中存在困难，导致表现方差更高。相比之下，接触过各类专业知识的LLM会尝试解决更广泛的问题，但会因提出物理上不可行的行动而失败。最后，我们对LLM进行了详细的错误分析，并证明了通过迭代式逐步反思、发散-聚合思维等新型提示技术提升其问题解决能力的潜力。本研究（1）引入了一个聚焦物理推理、规划与非常规思维复杂维度的智能体新竞技场，补充了现有的机器智能评估体系；（2）为人类与AI的受限问题解决能力提供了洞见。
英文摘要：
We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. To this end, we create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. MACGYVER is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar with but struggle with domain-specific knowledge, leading to a higher variance. In contrast, LLMs, exposed to a variety of specialized knowledge, attempt broader problems but fail by proposing physically-infeasible actions. Finally, we provide a detailed error analysis of LLMs, and demonstrate the potential of enhancing their problem-solving ability with novel prompting techniques such as iterative step-wise reflection and divergent-convergent thinking. This work (1) introduces a fresh arena for intelligent agents focusing on intricate aspects of physical reasoning, planning, and unconventional thinking, which supplements the existing spectrum of machine intelligence; and (2) provides insight into the constrained problem-solving capabilities of both humans and AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
