大语言模型(LLMs)的最新进展已在一系列自然语言处理任务中展现出令人瞩目的能力,尤其是在作为实现通用人工智能(AGI)基石的推理任务上。然而,常用基准可能无法全面体现这些模型在真实场景中的推理能力。为填补这一空白,本文提出了一种新型问答(QA)任务,名为“冗余信息推理(Reasoning with Redundant Information Provided, RRIP)”。本研究设计了小学数学8K(GSM-8K)数据集的改进版本,该版本包含多个聚焦冗余信息不同属性的变体。本次研究评估了两种主流大语言模型——LlaMA2-13B-chat和生成式预训练Transformer 3.5(GPT-3.5),对比了它们在传统问答任务与RRIP任务上的性能。结果表明,尽管这些模型在标准问答基准上取得了中等水平的成绩,但在RRIP任务上的评估性能显著下降。本研究不仅揭示了当前大语言模型在处理冗余信息方面的局限性,还指出未来对这些模型的训练应注重将冗余信息纳入训练数据,以提升其在RRIP任务上的表现。
英文摘要:
Recent advancements in Large Language Models (LLMs) have demonstrated impressive capabilities across a range of natural language processing tasks, especially in reasoning, a cornerstone for achieving Artificial General Intelligence (AGI). However, commonly used benchmarks may not fully encapsulate the inferential abilities of these models in real-world scenarios. To address this gap, a new form of Question-Answering (QA) task, termed Reasoning with Redundant Information Provided (RRIP), is introduced. The study designed a modified version of the grade school math 8K (GSM-8K) dataset which has several variants focusing on different attributes of redundant information. This investigation evaluates two popular LLMs, LlaMA2-13B-chat and generative pre-trained transformer 3.5 (GPT-3.5), contrasting their performance on traditional QA tasks against the RRIP tasks. Findings indicate that while these models achieved moderate success on standard QA benchmarks, their performance notably declines when assessed on RRIP tasks. The study not only highlights the limitations of current LLMs in handling redundant information but also suggests that future training of these models should focus on incorporating redundant information into the training data to increase the performance on RRIP tasks.
Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteria
POSTECH;Seoul National University(首尔国立大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Kyungpook National University(庆北国立大学);LG AI Research(LG AI研究院)
SEEK: Steering LLM Reasoning for RAG via Internal Reasoning Sketches
School of Computer Science and Engineering, Northeastern University, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学计算机科学与技术系,人工智能研究院);School of Intelligent Science and Technology, Nanjing University, China(南京大学智能科学与技术学院)
Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Department of Computer Science, National University of Singapore(新加坡国立大学计算机科学系);Ant Group(蚂蚁集团)
Distill Visual Chart Reasoning Ability from LLMs to MLLMs
School of Computer Science, Fudan University(复旦大学计算机学院);Weixin Group, Tencent(腾讯Weixin部门);Shanghai Innovation Institute(上海创新研究院);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室)
On the Token Value Inequality in Efficient Reasoning
Purdue University(普渡大学);Rochester Institute of Technology(罗切斯特理工学院);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);University of Missouri-Kansas City(密苏里大学堪萨斯城分校);Rutgers University(罗格斯大学);Meta AI
Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning
Zhejiang University(浙江大学);Westlake University(西湖大学);Westlake Institute for Advanced Study(西湖高等研究院);Johns Hopkins University(约翰斯·霍普金斯大学);Imperial College London(帝国理工学院)
Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning
Renmin University of China(中国人民大学);Xiaohongshu Inc.(小红书公司);Institute of Automation,Chinese Academy of Sciences(中国科学院自动化研究所);University of Southern California(南加州大学);Shandong University(山东大学)
Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);School of Artificial Intelligence and Data Science, USTC(中国科学技术大学人工智能与数据科学学院);Kuaishou Technology(快手科技);College of Future Information Technology, Fudan University(复旦大学未来信息技术学院)
Semantic Exploration with Adaptive Gating for Efficient Problem Solving with Language Models
Department of Computer Science and Engineering, POSTECH, South Korea(计算机科学与工程系,POSTECH,韩国);Graduate School of Artificial Intelligence, POSTECH, South Korea(人工智能研究生院,POSTECH,韩国)
CoTJudger: A Graph-Driven Framework for Automatic Evaluation of Chain-of-Thought Efficiency and Redundancy in LRMs
University of Science and Technology of China(科学技术大学);Shenzhen University of Advanced Technology(深圳先进技术大学);Shenzhen Institutes of Advanced Technology, CAS(深圳先进技术研究所,中国科学院);Southeast University(东南大学);Nanjing University(南京大学);Beihang University(北航);University of Manchester(曼彻斯特大学)
An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning
School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);Shanghai Innovation Center for Processor Technologies(上海处理器技术创新中心);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学与技术学院)