arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.00624cs.LG
收藏

聊天机器人LLM是否过于冗长?YapBench基准测试

Do Chatbot LLMs Talk Too Much? The YapBench Benchmark

Vadim Borisov, Michael Gröger, Mina Mikhael, Richard H. Schreiber

AI总结:

YapBench基准测试旨在评估LLM在简洁提示下的过度生成问题,通过量化用户可见的冗余长度,揭示模型在不同任务中的表现差异。

摘要:

大型语言模型(如ChatGPT、Claude和Gemini)越来越多地充当通用型助手,但它们在简单的请求上常常产生不必要的长度,添加冗余的解释、让步或套话,增加了认知负荷并提高了基于令牌的推理成本。先前的工作表明,基于偏好的后训练和LLM评估可以导致系统性的长度偏差,即使在质量相当的情况下,更长的回答也会被奖励。我们引入了YapBench,这是一个轻量级的基准测试,用于量化在简洁理想提示下的用户可见的过度生成。每个项目由一个单轮提示、一个精心挑选的最小充分基线回答和一个类别标签组成。我们的主要指标YapScore测量超出基线的响应长度(以字符为单位),使模型之间的比较不依赖于特定的分词器。我们通过YapIndex,即类别级中位YapScore的均匀加权平均,总结模型的表现。YapBench包含超过三百个英文提示,涵盖三种常见的简洁理想设置:(A)最小或模糊的输入,理想行为是简短的澄清;(B)闭式事实性问题,有简短稳定的答案;(C)一行编码任务,单个命令或片段即可。评估76个助手LLM后,我们观察到中位超额长度有数量级的差异,并且存在不同的类别特定失败模式,包括在模糊输入上的真空填充和在一行技术请求上的解释或格式超支。我们发布了该基准测试并维护了一个实时排行榜,以跟踪随时间推移的冗余行为。

英文摘要:

Large Language Models (LLMs) such as ChatGPT, Claude, and Gemini increasingly act as general-purpose copilots, yet they often respond with unnecessary length on simple requests, adding redundant explanations, hedging, or boilerplate that increases cognitive load and inflates token-based inference cost. Prior work suggests that preference-based post-training and LLM-judged evaluations can induce systematic length bias, where longer answers are rewarded even at comparable quality. We introduce YapBench, a lightweight benchmark for quantifying user-visible over-generation on brevity-ideal prompts. Each item consists of a single-turn prompt, a curated minimal-sufficient baseline answer, and a category label. Our primary metric, YapScore, measures excess response length beyond the baseline in characters, enabling comparisons across models without relying on any specific tokenizer. We summarize model performance via the YapIndex, a uniformly weighted average of category-level median YapScores. YapBench contains over three hundred English prompts spanning three common brevity-ideal settings: (A) minimal or ambiguous inputs where the ideal behavior is a short clarification, (B) closed-form factual questions with short stable answers, and (C) one-line coding tasks where a single command or snippet suffices. Evaluating 76 assistant LLMs, we observe an order-of-magnitude spread in median excess length and distinct category-specific failure modes, including vacuum-filling on ambiguous inputs and explanation or formatting overhead on one-line technical requests. We release the benchmark and maintain a live leaderboard for tracking verbosity behavior over time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑