arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.04039cs.CL
收藏

大语言模型的冗余信息推理能力分析

Analysis of the Reasoning with Redundant Information Provided Ability of Large Language Models

Beijing University of Technology(北京工业大学);Beijing-Dublin International College(北京-都柏林国际学院)

Wenbei Xie

AI总结:

本研究针对现有基准无法全面反映大语言模型真实场景推理能力的问题,提出冗余信息推理(RRIP)任务并构建GSM-8K改进数据集,评估发现LlaMA2-13B-chat与GPT-3.5在该任务上性能显著下降,指出需在训练中加入冗余信息以提升相关能力。

摘要:

大语言模型(LLMs)的最新进展已在一系列自然语言处理任务中展现出令人瞩目的能力,尤其是在作为实现通用人工智能(AGI)基石的推理任务上。然而,常用基准可能无法全面体现这些模型在真实场景中的推理能力。为填补这一空白,本文提出了一种新型问答(QA)任务,名为“冗余信息推理(Reasoning with Redundant Information Provided, RRIP)”。本研究设计了小学数学8K(GSM-8K)数据集的改进版本,该版本包含多个聚焦冗余信息不同属性的变体。本次研究评估了两种主流大语言模型——LlaMA2-13B-chat和生成式预训练Transformer 3.5(GPT-3.5),对比了它们在传统问答任务与RRIP任务上的性能。结果表明,尽管这些模型在标准问答基准上取得了中等水平的成绩,但在RRIP任务上的评估性能显著下降。本研究不仅揭示了当前大语言模型在处理冗余信息方面的局限性,还指出未来对这些模型的训练应注重将冗余信息纳入训练数据,以提升其在RRIP任务上的表现。

英文摘要:

Recent advancements in Large Language Models (LLMs) have demonstrated impressive capabilities across a range of natural language processing tasks, especially in reasoning, a cornerstone for achieving Artificial General Intelligence (AGI). However, commonly used benchmarks may not fully encapsulate the inferential abilities of these models in real-world scenarios. To address this gap, a new form of Question-Answering (QA) task, termed Reasoning with Redundant Information Provided (RRIP), is introduced. The study designed a modified version of the grade school math 8K (GSM-8K) dataset which has several variants focusing on different attributes of redundant information. This investigation evaluates two popular LLMs, LlaMA2-13B-chat and generative pre-trained transformer 3.5 (GPT-3.5), contrasting their performance on traditional QA tasks against the RRIP tasks. Findings indicate that while these models achieved moderate success on standard QA benchmarks, their performance notably declines when assessed on RRIP tasks. The study not only highlights the limitations of current LLMs in handling redundant information but also suggests that future training of these models should focus on incorporating redundant information into the training data to increase the performance on RRIP tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑