Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models
The Hong Kong University of Science and Technology(香港科技大学);University of Washington(华盛顿大学);Xi’an Jiaotong University(西安交通大学);University of Notre Dame(圣母大学);Carnegie Mellon University(卡内基梅隆大学)
We propose Knowledge Crosswords, a geometric knowledge reasoning benchmark consisting of incomplete knowledge networks bounded by structured factual constraints, where LLMs are tasked with inferring the missing facts to meet all constraints. The novel setting of geometric knowledge reasoning necessitates new LM abilities beyond existing atomic/linear multi-hop QA, such as backtracking, verifying facts and constraints, reasoning with uncertainty, and more. Knowledge Crosswords contains 2,101 individual problems, covering diverse knowledge domains, and is further divided into three difficulty levels. We conduct extensive experiments to evaluate existing LLMs and approaches on Knowledge Crosswords. Results demonstrate that baseline approaches struggle with larger knowledge networks and semantically-equivalent entity distractors. In light of their limitations, we propose two new approaches, Staged Prompting and Verify-All, to augment LLMs' abilities for error-aware backtracking and constraint verification. Our Verify-All significantly outperforms prior methods and is more robust towards problems in the hard subset. Further analysis shows that geometric knowledge reasoning poses new challenges to LLMs' knowledge abilities, particularly in robustness towards varying option orders, complex structural constraints in knowledge networks, "none of the above" scenarios, and more.
NeSyGeo: A Neuro-Symbolic Framework for Multimodal Geometric Reasoning Data Generation
School of Intelligence Science and Technology, Nanjing University(智能科学与技术学院,南京大学);National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家实验室,南京大学);School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学)
Symbolic Intermediaries as a Linguistic-Numerical Interface for LLM-Driven Geometric Reasoning
Idiap Research Institute(Idiap研究 institute);École Polytechnique Fédérale de Lausanne(瑞士联邦理工学院);Honda Research Institute Europe(本田欧洲研究院);Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);National Biomarker Centre, CRUK-MI, University of Manchester(曼彻斯特大学国家生物标记中心)
TrustGeoGen: Formal-Verified Data Engine for Trustworthy Multi-modal Geometric Problem Solving
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院)
Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructions
Shenyang institute of computing technology, Chinese academy of sciences(沈阳计算技术研究所,中国科学院);Shanghai AI Laboratory(上海人工智能实验室);University of Chinese Academy of Sciences(中国科学院大学)
GeoSketch: A Neural-Symbolic Approach to Geometric Multimodal Reasoning with Auxiliary Line Construction and Affine Transformation
Fudan University(复旦大学);IFLYTEK CO.LTD(若lytek有限公司);Zhejiang University(浙江大学);The Hong Kong University of Science and Technology(香港科学与技术大学);National University of Defense Technology(国防科技大学);Hainan University(海南大学)
How RL Unlocks the Aha Moment in Geometric Interleaved Reasoning
ByteDance(字节跳动);Shenyang Institute of Computing Technology, Chinese Academy of Sciences(沈阳计算技术研究所,中国科学院);Westlake University(西交大学);University of Chinese Academy of Science(中国科学院大学);Key Laboratory of Computing Power Network and Information Security, Ministry of Education(教育部计算能力网络与信息安全重点实验室);Shandong Computer Science Center (National Supercomputer Center in Jinan)(山东省计算机科学中心(济南国家超算中心));Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院))
GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning
Sun Yat-sen University(中山大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Dark Matter AI Inc.(暗物智能科技有限公司);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Geometrically-Constrained Agent for Spatial Reasoning
School of Software, Beihang University(北京航空航天大学软件学院);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Shanghai Innovation Institute(上海创新研究院);ZIP Lab, Zhejiang University(浙江大学ZIP实验室)
Self-Evolving Spatial Reasoning in Vision Language Models via Geometric Logic Consistency
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The City University of New York(纽约城市大学);The Chinese University of Hong Kong(香港中文大学);Data61 CSIRO(Data61澳大利亚国家科学研究院);University of New South Wales(新南威尔士大学);Australian National University(澳大利亚国立大学)
TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics
Beihang University(北洋大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(北京大学计算机科学学院多媒体信息处理国家重点实验室)
Thinking with Constructions: A Benchmark and Policy Optimization for Visual-Text Interleaved Geometric Reasoning
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);School of ECE, Peking University(北京大学电子工程学院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);Tencent Youtu Lab(腾讯优图实验室)
GeoAlign: Geometric Feature Realignment for MLLM Spatial Reasoning
National Engineering Research Center of Visual Technology, National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(视觉技术国家工程研究中心、多媒体信息处理国家重点实验室、计算机学院、北京大学);Meituan Inc.(美团公司);National Biomedical Imaging Center, Peking University(生物医学成像国家中心、北京大学)
JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles
University of Southern California(南加州大学);National University of Singapore(新加坡国立大学);Adobe Research(奥多比研究院);Texas A&M University(德克萨斯农工大学);Rice University(莱斯大学);The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)
Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);Meituan(美团);National University of Singapore(新加坡国立大学);Beihang University(北航);LMMs-Lab(LMMs实验室)
PointCoT: A Multi-modal Benchmark for Explicit 3D Geometric Reasoning
Xi'an Jiaotong University(西安交通大学);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Institute of Automation, CASIA(中国科学院自动化研究所);Taiyuan University of Technology(太原理工大学);Shanghai AI Laboratory(上海人工智能实验室)
REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Model
Tsinghua University(清华大学);City University of Hong Kong(香港城市大学);Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学);Beihang University(北航);Baidu Inc(百度公司)
本文提出 REMA 框架,将正确推理表征建模为低维推理流形,通过 k 近邻距离和逐层偏差定位 LLM 推理失效的起点。