如果大语言模型能解决更难的问题,它们就总能解决简单问题吗?
Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
AI总结:
该研究针对大语言模型能解难题却可能在简单题上失败的“难到易”不一致性,构建ConsisEval基准并提出一致性分数量化评估,经多模型实验揭示相关规律与提升方向。
摘要:
大语言模型(LLMs)已展现出令人瞩目的能力,但仍存在不一致性问题(例如,LLMs会对改写、无关紧要的顺序变更等干扰产生不同反应)。除了这些不一致性,我们还观察到一个矛盾现象:LLMs虽能解决难题,却可能在更简单的问题上失败。为评估这种“难到易”的不一致性,我们构建了ConsisEval基准(一致性评估基准),其中每个条目包含一对难度有严格排序的问题。此外,我们提出一致性分数的概念以定量衡量这种不一致性,并通过相对一致性分数分析一致性的提升潜力。基于对多种现有模型的全面实验,我们发现:(1)GPT-4取得了92.2%的最高一致性分数,但仍会因冗余信息干扰、问题误解等在特定问题上表现出不一致;(2)能力更强的模型通常一致性更高,但也存在例外;(3)难例数据对微调与上下文学习的一致性均有提升作用。我们的数据和代码将在GitHub上公开。
英文摘要:
Large language models (LLMs) have demonstrated impressive capabilities, but still suffer from inconsistency issues (e.g. LLMs can react differently to disturbances like rephrasing or inconsequential order change). In addition to these inconsistencies, we also observe that LLMs, while capable of solving hard problems, can paradoxically fail at easier ones. To evaluate this hard-to-easy inconsistency, we develop the ConsisEval benchmark, where each entry comprises a pair of questions with a strict order of difficulty. Furthermore, we introduce the concept of consistency score to quantitatively measure this inconsistency and analyze the potential for improvement in consistency by relative consistency score. Based on comprehensive experiments across a variety of existing models, we find: (1) GPT-4 achieves the highest consistency score of 92.2\% but is still inconsistent to specific questions due to distraction by redundant information, misinterpretation of questions, etc.; (2) models with stronger capabilities typically exhibit higher consistency, but exceptions also exist; (3) hard data enhances consistency for both fine-tuning and in-context learning. Our data and code will be publicly available on GitHub.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。