OMEGA：大语言模型能否在数学中跳出框架思考？评估探索性、组合性与变革性泛化能力
OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization
AI总结：
OMEGA基准从探索、组合、变革三个轴心评估大模型数学分布外泛化，发现复杂问题下性能骤降，微调仅提升探索性泛化。
摘要：
近期，具有长思维链推理能力的大规模语言模型（LLMs），如DeepSeek-R1，在奥林匹克级别的数学基准测试中取得了令人瞩目的成绩。然而，它们往往依赖一套狭窄的策略，在面对需要新颖思维方式的问题时表现挣扎。为了系统性地研究这些局限性，我们引入了OMEGA——基于3个泛化轴心的分布外数学问题评估基准。这是一个受控且多样化的基准，旨在评估三个维度的分布外泛化能力，其灵感来源于Boden的创造力类型学：（1）探索性泛化——将已知的问题解决技能应用于同一问题领域内更复杂的实例；（2）组合性泛化——将先前单独学习的、不同的推理技能结合起来，以解决需要以新颖且连贯的方式整合这些技能的新问题；（3）变革性泛化——采用新颖且往往非常规的策略，超越熟悉的方法，从而更有效地解决问题。OMEGA包含由模板化问题生成器程序化生成的训练-测试对，覆盖几何、数论、代数、组合数学、逻辑和谜题等领域，并通过符号、数值或图形方法验证解答。我们评估了前沿（或顶级）LLMs，并观察到随着问题复杂度的增加，性能急剧下降。此外，我们在所有泛化设置下对Qwen系列模型进行了微调，并观察到探索性泛化有显著提升，而组合性泛化仍然有限，变革性推理几乎没有改善。通过隔离并量化这些细粒度的失败，OMEGA为推进LLMs迈向超越机械熟练度的真正数学创造力奠定了基础。
英文摘要：
Recent large-scale language models (LLMs) with long Chain-of-Thought reasoning-such as DeepSeek-R1-have achieved impressive results on Olympiad-level mathematics benchmarks. However, they often rely on a narrow set of strategies and struggle with problems that require a novel way of thinking. To systematically investigate these limitations, we introduce OMEGA-Out-of-distribution Math Problems Evaluation with 3 Generalization Axes-a controlled yet diverse benchmark designed to evaluate three axes of out-of-distribution generalization, inspired by Boden's typology of creativity: (1) Exploratory-applying known problem solving skills to more complex instances within the same problem domain; (2) Compositional-combining distinct reasoning skills, previously learned in isolation, to solve novel problems that require integrating these skills in new and coherent ways; and (3) Transformative-adopting novel, often unconventional strategies by moving beyond familiar approaches to solve problems more effectively. OMEGA consists of programmatically generated training-test pairs derived from templated problem generators across geometry, number theory, algebra, combinatorics, logic, and puzzles, with solutions verified using symbolic, numerical, or graphical methods. We evaluate frontier (or top-tier) LLMs and observe sharp performance degradation as problem complexity increases. Moreover, we fine-tune the Qwen-series models across all generalization settings and observe notable improvements in exploratory generalization, while compositional generalization remains limited and transformative reasoning shows little to no improvement. By isolating and quantifying these fine-grained failures, OMEGA lays the groundwork for advancing LLMs toward genuine mathematical creativity beyond mechanical proficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
