The advancement of Large Language Models (LLMs) has brought substantial attention to the Chain of Thought (CoT) approach, primarily due to its ability to enhance the capability of LLMs on complex reasoning tasks. Moreover, the significance of CoT approaches extends to the application of LLMs for multi-modal tasks. However, the selection of optimal CoT demonstration examples in multi-modal reasoning remains less explored for LLMs due to the inherent complexity of multi-modal examples. In this paper, we introduce a novel approach that addresses this challenge by using retrieval mechanisms to dynamically and automatically select demonstration examples based on cross-modal and intra-modal similarities. Furthermore, we employ a Stratified Sampling method of categorising demonstration examples into groups based on their types and then retrieving examples from different groups respectively to promote the diversity of demonstration examples. Through a series of experiments on two popular benchmark datasets: ScienceQA and MathVista, we demonstrate that our approach significantly improves the performance of GPT-4 by 6% on ScienceQA and 12.9% on MathVista, and enhances the performance of GPT-4V on two datasets by 2.7%, substantially improving the performance of the most advanced LLMs and LMMs for complex multi-modal reasoning tasks.
Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Study
School of Informatics, Xiamen University, China（厦门大学信息学院）；Baidu Inc., Beijing, China（百度公司）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China（福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Beijing Key Laboratory of Big Data Management and Analysis Methods（北京大数据管理与分析方法重点实验室）
RATT: A Thought Structure for Coherent and Correct LLM Reasoning
Portland State University（波特兰州立大学）；Renmin University of China（中国人民大学）；Pennsylvania State University（宾夕法尼亚州立大学）；Dalian Maritime University（大连海事大学）；University of Kansas（堪萨斯大学）
Soft-Prompting with Graph-of-Thought for Multi-modal Representation Learning
School of Computer Science, Wuhan University（武汉大学计算机学院）；National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；School of Electronic Information Engineering, Henan Polytechnic Institute（河南工业职业技术学院电子信息工程学院）；JD Explore Academy（京东探索研究院）
HGOT: Hierarchical Graph of Thoughts for Retrieval-Augmented In-Context Learning in Factuality Evaluation
Smith School of Business, Queen’s University（女王大学史密斯商学院）；Ingenuity Labs Research Institute, Queen’s University（女王大学 Ingenuity 实验室研究所）；Department of Electrical and Computer Engineering, Queen’s University（女王大学电气与计算机工程系）
M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
Research Center for Social Computing and Information Retrieval（社会计算与信息检索研究中心）；Harbin Institute of Technology（哈尔滨工业大学）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Shanghai AI Laboratory（上海人工智能实验室）
Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought
Speechmatics, Apollo Research（Speechmatics与Apollo研究机构）；University of Oxford（牛津大学）；NYU, Anthropic（纽约大学与Anthropic）；NYU（纽约大学）；Anthropic, NYU（Anthropic与纽约大学）
CoT-RAG: Integrating Chain of Thought and Retrieval-Augmented Generation to Enhance Reasoning in Large Language Models
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China（华中科技大学计算机科学与技术学院）；Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China（华中科技大学光电实验室）；Department of Computer Science, Bowling Green State University, Ohio, USA（Bowling Green State University 计算机科学系）；Department of Computer Science, Aalborg University, Aalborg, Denmark（Aalborg University 计算机科学系）；Huawei Technologies Co., Ltd, Shenzhen, China（华为技术有限公司）
Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes
Raymond G. Perelman Center for Cellular and Molecular Therapeutics, Children's Hospital of Philadelphia（费城儿童医院雷蒙德·G·佩雷尔曼细胞与分子治疗中心）；University of Pennsylvania（宾夕法尼亚大学）；Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）
RAR$^2$: Retrieval-Augmented Medical Reasoning via Thought-Driven Retrieval
Department of Computing, The Hong Kong Polytechnic University（计算机系，香港理工大学）；Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology（可信自主系统研究院和计算机科学与工程系，南方科技大学）
CheckRLM: Effective Knowledge-Thought Coherence Checking in Retrieval-Augmented Reasoning
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Beijing Key Laboratory of Artificial Intelligence for Education（北京人工智能教育重点实验室）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Department of Computer Science and Technology, Institute for AI, Tsinghua University（清华大学人工智能研究院计算机科学与技术系）；Northeastern University（东北大学）
School of Computer Science and Technology, Beijing Institute of Technology, China（计算机科学与技术学院，北京理工大学，中国）；Nanyang Technological University, Singapore（南洋理工大学，新加坡）；WeChat AI, Tencent Inc., China（微信AI，腾讯公司，中国）
Recent research has shown that rationales, or step-by-step chains of thought, can be used to improve performance in multi-step reasoning tasks. We reconsider rationale-augmented prompting for few-shot in-context learning, where (input -> output) prompts are expanded to (input, rationale -> output) prompts. For rationale-augmented prompting we demonstrate how existing approaches, which rely on manual prompt engineering, are subject to sub-optimal rationales that may harm performance. To mitigate this brittleness, we propose a unified framework of rationale-augmented ensembles, where we identify…
Vision-and-language multi-modal pretraining and fine-tuning have shown great success in visual question answering (VQA). Compared to general domain VQA, the performance of biomedical VQA suffers from limited data. In this paper, we propose a retrieval-augmented pretrain-and-finetune paradigm named RAMM for biomedical VQA to overcome the data limitation issue. Specifically, we collect a new biomedical dataset named PMCPM which offers patient-based image-text pairs containing diverse patient situations from PubMed. Then, we pretrain the biomedical multi-modal model to learn visual and textual re…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
Human brains integrate linguistic and perceptual information simultaneously to understand natural language, and hold the critical ability to render imaginations. Such abilities enable us to construct new abstract concepts or concrete objects, and are essential in involving practical knowledge to solve problems in low-resource scenarios. However, most existing methods for Natural Language Understanding (NLU) are mainly focused on textual signals. They do not simulate human visual imagination ability, which hinders models from inferring and learning efficiently from limited data samples. Therefo…
