Reasoning ability is one of the most crucial capabilities of a foundation model, signifying its capacity to address complex reasoning tasks. Chain-of-Thought (CoT) technique is widely regarded as one of the effective methods for enhancing the reasoning ability of foundation models and has garnered significant attention. However, the reasoning process of CoT is linear, step-by-step, similar to personal logical reasoning, suitable for solving general and slightly complicated problems. On the contrary, the thinking pattern of an expert owns two prominent characteristics that cannot be handled appropriately in CoT, i.e., high-order multi-hop reasoning and multimodal comparative judgement. Therefore, the core motivation of this paper is transcending CoT to construct a reasoning paradigm that can think like an expert. The hyperedge of a hypergraph could connect various vertices, making it naturally suitable for modelling high-order relationships. Inspired by this, this paper innovatively proposes a multimodal Hypergraph-of-Thought (HoT) reasoning paradigm, which enables the foundation models to possess the expert-level ability of high-order multi-hop reasoning and multimodal comparative judgement. Specifically, a textual hypergraph-of-thought is constructed utilizing triple as the primary thought to model higher-order relationships, and a hyperedge-of-thought is generated through multi-hop walking paths to achieve multi-hop inference. Furthermore, we devise a visual hypergraph-of-thought to interact with the textual hypergraph-of-thought via Cross-modal Co-Attention Graph Learning for multimodal comparative verification. Experimentations on the ScienceQA benchmark demonstrate the proposed HoT-based T5 outperforms CoT-based GPT3.5 and chatGPT, which is on par with CoT-based GPT4 with a lower model size.
Iterative Forward Tuning Boosts In-Context Learning in Language Models
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Alibaba Group（阿里巴巴集团）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Shanghai AI Laboratory（上海人工智能实验室）
Multi-choice Machine Reading Comprehension (MRC) requires model to decide the correct answer from a set of answer options when given a passage and a question. Thus in addition to a powerful Pre-trained Language Model (PrLM) as encoder, multi-choice MRC especially relies on a matching network design which is supposed to effectively capture the relationships among the triplet of passage, question and answers. While the newer and more powerful PrLMs have shown their mightiness even without the support from a matching network, we propose a new DUal Multi-head Co-Attention (DUMA) model, which is in…
Step Back to Leap Forward: Self-Backtracking for Boosting Reasoning of Language Models
Nanjing University（南京大学）；National Key Laboratory for Novel Software Technology（现代软件工程国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
Ubiquitous Knowledge Processing Lab, Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt（德意志联邦共和国达姆施塔特技术大学）；School of Computing Technologies, Royal Melbourne Institute of Technology, Australia（澳大利亚皇家墨尔本理工学院）；Data Science Group, University of Hamburg（汉堡大学）；Department of Data Science & AI, Monash University, Australia（澳大利亚墨尔本大学）
Unlocking Recursive Thinking of LLMs: Alignment via Refinement
Soochow University（苏州大学）；Zhipu AI（智谱AI）；Tsinghua University（清华大学）；Key Laboratory of Data Intelligence and Advanced Computing, Soochow University（苏州大学数据智能与先进计算重点实验室）
DeepRAG: Thinking to Retrieve Step by Step for Large Language Models
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Pattern Recognition Center, WeChat AI, Tencent Inc, China（腾讯人工智能部模式识别中心）
SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoning
Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly（联合NTU-UBC老龄化积极生活卓越研究中心）；Alibaba-NTU Global e-Sustainability CorpLab (ANGEL)（阿里巴巴-NTU全球可持续性企业实验室（ANGEL））；College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院）；KTH Royal Institute of Technology, Sweden（瑞典皇家理工学院）
The Evolution of Thought: Tracking LLM Overthinking via Reasoning Dynamics Analysis
State Key Laboratory of AI Safety,Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost
NLP（自然语言处理）；CT Lab, Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models
Zhejiang Univeristy（浙江大学）；MAPLE Lab, Westlake University（西湖大学MAPLE实验室）；Matterwave Intelligence；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖研究所以高级技术研究所）
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Allen Institute for AI（艾伦人工智能研究所）
Thinking aloud is an effective meta-cognitive strategy human reasoners apply to solve difficult problems. We suggest to improve the reasoning ability of pre-trained neural language models in a similar way, namely by expanding a task's context with problem elaborations that are dynamically generated by the language model itself. Our main result is that dynamic problem elaboration significantly improves the zero-shot performance of GPT-2 in a deductive reasoning and natural language inference task: While the model uses a syntactic heuristic for predicting an answer, it is capable (to some degree…
School of Informatics, Xiamen University, China（厦门大学信息学院）；Department of Digital Media Technology, Xiamen University, China（厦门大学数字媒体技术系）；Baidu Inc., Beijing, China（百度公司）
Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting. It first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. Self-consistency leverages the intuition that a complex reasoning problem typically admits multiple different ways of thinking leading to i…
Sun Yat-sen University（中山大学）；Harbin Institute of Technology（哈尔滨工业大学）；Microsoft Research Asia（微软亚洲研究院）
This paper presents ReasonFormer, a unified reasoning framework for mirroring the modular and compositional reasoning process of humans in complex decision-making. Inspired by dual-process theory in cognitive science, the representation module (automatic thinking) and reasoning modules (controlled thinking) are decoupled to capture different levels of cognition. Upon the top of the representation module, the pre-trained reasoning modules are modular and professional in specific and fundamental reasoning skills (e.g., logic, simple QA, etc). To mimic the controlled compositional thinking proces…
BAPPA: Benchmarking Agents, Plans, and Pipelines for Automated Text-to-SQL Generation
Center for Computational & Data Sciences, Independent University, Bangladesh（计算与数据科学中心，独立大学，孟加拉国）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs
UC Santa Cruz（加州大学圣克鲁兹分校）；University of British Columbia（不列颠哥伦比亚大学）；Nanyang Technological University（南洋理工大学）；Stanford University（斯坦福大学）；Weill Cornell Medicine（威尔康奈尔医学院）；NYU Langone Health（纽约大学兰贡医学中心）；Chungnam National University Sejong Hospital（忠南大学世宗医院）；Vector Institute（矢量研究院）；Pusan National University Hospital（釜山国立大学医院）；Massachusetts General Hospital（麻省总医院）
Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
ByteDance Seed（字节跳动种子）；Fudan University（复旦大学）；Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究院（AIR），清华大学）；Nanjing University（南京大学）；Shanghai Jiao Tong University（上海交通大学）；SIA-Lab of Tsinghua AIR and ByteDance Seed（清华大学AIR实验室和字节跳动种子）
