arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.03017cs.CL
收藏

MMFormalizer: 多模态自动形式化

MMFormalizer: Multimodal Autoformalization in the Wild

The University of Hong Kong(香港大学);University of Michigan, Ann Arbor(密歇根大学安娜堡分校);University of Edinburgh(爱丁堡大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);Ohio State University(俄亥俄州立大学);University of California, Los Angeles(加州大学洛杉矶分校)

Jing Xiong, Qi Han, Yunta Hsieh, Hui Shen, Huajian Xin, Chaofan Tao, Chenyang Zhao, Hengyuan Zhang, Taiqiang Wu, Zhen Zhang, Haochen Wang, Zhongwei Wan, Lingpeng Kong, Ngai Wong

AI总结:

MMFormalizer通过整合适应性定位与现实世界数学物理领域实体,实现多模态自动形式化,首次处理经典力学、相对论、量子力学和热力学等复杂领域。

摘要:

自动形式化将自然语言数学转化为形式语句以实现机器推理,但在真实世界中由于物理世界的多模态特性,面临根本性挑战。物理需要从视觉元素推断隐藏的约束(例如质量或能量)。为此,我们提出了MMFormalizer,它通过整合适应性定位与现实世界数学和物理领域的实体,将自动形式化扩展到文本之外。MMFormalizer通过递归定位和公理组合,从视觉定位的原始元素递归构建形式命题,通过适应性递归终止确保每个抽象都有视觉证据支持,并以维度或公理定位为锚点。我们在新的基准测试PhyX-AF上评估了MMFormalizer,该基准测试包含来自MathVerse、PhyX、合成几何和解析几何的115个精心挑选的样本,涵盖多样化的多模态自动形式化任务。结果表明,前沿模型如GPT-5和Gemini-3-Pro在编译和语义准确性上达到最高水平,其中GPT-5在物理推理方面表现突出,而几何学仍然是最具有挑战性的领域。总体而言,MMFormalizer提供了一个可扩展的统一多模态自动形式化框架,连接感知与形式推理。据我们所知,这是首个能够处理经典力学(源自哈密顿量)、相对论、量子力学和热力学的多模态自动形式化方法。更多细节请访问我们的项目页面:MMFormalizer.github.io

英文摘要:

Autoformalization, which translates natural language mathematics into formal statements to enable machine reasoning, faces fundamental challenges in the wild due to the multimodal nature of the physical world, where physics requires inferring hidden constraints (e.g., mass or energy) from visual elements. To address this, we propose MMFormalizer, which extends autoformalization beyond text by integrating adaptive grounding with entities from real-world mathematical and physical domains. MMFormalizer recursively constructs formal propositions from perceptually grounded primitives through recursive grounding and axiom composition, with adaptive recursive termination ensuring that every abstraction is supported by visual evidence and anchored in dimensional or axiomatic grounding. We evaluate MMFormalizer on a new benchmark, PhyX-AF, comprising 115 curated samples from MathVerse, PhyX, Synthetic Geometry, and Analytic Geometry, covering diverse multimodal autoformalization tasks. Results show that frontier models such as GPT-5 and Gemini-3-Pro achieve the highest compile and semantic accuracy, with GPT-5 excelling in physical reasoning, while geometry remains the most challenging domain. Overall, MMFormalizer provides a scalable framework for unified multimodal autoformalization, bridging perception and formal reasoning. To the best of our knowledge, this is the first multimodal autoformalization method capable of handling classical mechanics (derived from the Hamiltonian), as well as relativity, quantum mechanics, and thermodynamics. More details are available on our project page: MMFormalizer.github.io

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑