The advancement of large language models (LLMs) for real-world applications hinges critically on enhancing their reasoning capabilities. In this work, we explore the reasoning abilities of large language models (LLMs) through their geometrical understanding. We establish a connection between the expressive power of LLMs and the density of their self-attention graphs. Our analysis demonstrates that the density of these graphs defines the intrinsic dimension of the inputs to the MLP blocks. We demonstrate through theoretical analysis and toy examples that a higher intrinsic dimension implies a greater expressive capacity of the LLM. We further provide empirical evidence linking this geometric framework to recent advancements in methods aimed at enhancing the reasoning capabilities of LLMs.
NeSyGeo: A Neuro-Symbolic Framework for Multimodal Geometric Reasoning Data Generation
School of Intelligence Science and Technology, Nanjing University(智能科学与技术学院,南京大学);National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家实验室,南京大学);School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学)
Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models
The Hong Kong University of Science and Technology(香港科技大学);University of Washington(华盛顿大学);Xi’an Jiaotong University(西安交通大学);University of Notre Dame(圣母大学);Carnegie Mellon University(卡内基梅隆大学)
Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism
Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University(上海交通大学自然科学院);School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学数学学院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Huawei Noah’s Ark Lab(华为诺亚实验室);Key Laboratory of Marine Intelligent Equipment and System, Ministry of Education(教育部海洋智能装备与系统重点实验室)
Hán Dān Xué Bù (Mimicry) or Qīng Chū Yú Lán (Mastery)? A Cognitive Perspective on Reasoning Distillation in Large Language Models
School of Philosophy, Anhui University(安徽大学哲学学院);Institute of Neuroscience, Chinese Academy of Sciences(中国科学院神经科学研究所);Faculty of Education, University of Cambridge(剑桥大学教育学院);School of Foreign Studies, South China Normal University(华南师范大学外语学院);Division of Psychology and Language Sciences, University College London(伦敦大学学院心理学与语言科学系)
A Decomposition Perspective to Long-context Reasoning for LLMs
Large Language Model Department, Tencent(腾讯大语言模型部门);Xi'an Jiaotong University(西安交通大学);Tencent(腾讯);Fudan University(复旦大学);City University of Hong Kong(香港城市大学)
Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructions
Shenyang institute of computing technology, Chinese academy of sciences(沈阳计算技术研究所,中国科学院);Shanghai AI Laboratory(上海人工智能实验室);University of Chinese Academy of Sciences(中国科学院大学)
Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information
School of Computer Science and Engineering, Central South University(中南大学计算机科学与工程学院);Research Center for SCIR, Harbin Institute of Technology(哈尔滨工业大学SCIR研究中心);Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));Yunnan University(云南大学)
Mind the Perspective: Let's Reason Recursively for Theory of Mind
School of Computing and Information Systems, The University of Melbourne, Australia(墨尔本大学计算与信息系统学院);SensiLab, Monash University, Australia(蒙纳士大学SensiLab)
MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人机混合增强智能国家重点实验室、视觉信息与应用国家工程研究中心、人工智能与机器人研究所);Nankai University(南开大学);The Hong Kong University of Science and Technology(Guangzhou)(香港科技大学(广州));School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院)
Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective
MaiNLP;Center for Information and Language Processing, LMU Munich(信息与语言处理中心,慕尼黑大学);LMU Munich, Germany(慕尼黑大学,德国);Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心(MCML),慕尼黑,德国);Language Technology Lab, University of Cambridge, United Kingdom(语言技术实验室,剑桥大学,英国)
GeoSketch: A Neural-Symbolic Approach to Geometric Multimodal Reasoning with Auxiliary Line Construction and Affine Transformation
Fudan University(复旦大学);IFLYTEK CO.LTD(若lytek有限公司);Zhejiang University(浙江大学);The Hong Kong University of Science and Technology(香港科学与技术大学);National University of Defense Technology(国防科技大学);Hainan University(海南大学)
Tree of Agents: Improving Long-Context Capabilities of Large Language Models through Multi-Perspective Reasoning
School of Computer and Information Science, Southwest University(西南大学计算机与信息科学学院);School of Information Technology, Murdoch University(默多克大学信息科技学院);School of Computing Technologies, RMIT University(皇家墨尔本理工大学计算技术学院);University of Technology Sydney(悉尼技术大学)
针对大语言模型长上下文处理中的中间信息丢失等问题,提出多智能体推理框架Tree of Agents,通过分块处理、树状协作推理及缓存剪枝策略,在低开销下提升长上下文任务性能,小模型驱动版本即可媲美大型商业模型。
Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);College of Intelligence Science and Technology, National University of Defense Technology(智能科学与技术学院,国防科技大学)
Geometrically-Constrained Agent for Spatial Reasoning
School of Software, Beihang University(北京航空航天大学软件学院);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Shanghai Innovation Institute(上海创新研究院);ZIP Lab, Zhejiang University(浙江大学ZIP实验室)