arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.01436cs.CL
收藏

从饱和数据中学习:LLM训练中超越正确性的信号

Learning from Saturated Data: Signals Beyond Correctness for LLM Training

ETH Zurich(苏黎世联邦理工学院)

Hanno Hiss, Jasper Dekoninck, Martin Vechev

AI总结:

本文研究在基准测试和训练数据集饱和的情况下,如何利用超越二元正确性的细粒度质量信号(如成对LLM自判断和token级熵)来提升下游性能,实验表明在简单算术任务上质量信号显著优于SFT,但在复杂任务上效果有限且需谨慎校准。

摘要:

大型语言模型(LLM)能力的不断增强导致许多用于改进它们的基准测试和训练数据集趋于饱和。受此启发,我们研究了那些经验上达到完美准确率的问题是否仍可用于提升下游性能。为此,我们将二元正确性替换为两种更细粒度的质量信号来源:(1)成对LLM自判断,即模型评估自身解决方案的相对质量;(2)token级熵,其中token级不确定性作为解决方案质量的代理。我们将这些信号整合到多种训练算法中,并在Qwen3-1.7B-Base上进行了评估。当仅在一个简单算术任务上训练时,基于质量的信号相比基础模型提升了高达18.6%的性能,显著优于SFT。然而,在GSM8K上,提升较为有限,且强烈依赖于质量信号。例如,自判断与更强的外部判断者一致性较差,甚至可能使性能低于基础模型。总体而言,我们的结果表明,基于质量的训练可以从饱和问题中为基础模型提取有用信号,但将此类信号应用于更复杂的任务需要仔细校准和进一步研究。

英文摘要:

The growing capabilities of large language models (LLMs) have led to the saturation of many benchmarks and training datasets used to improve them. Motivated by this, we investigate whether questions solved with perfect empirical accuracy can nevertheless be used to improve downstream performance. To do so, we replace binary correctness with two sources of more fine-grained quality signals: (1) pairwise LLM self-judgments, in which the model evaluates the relative quality of its own solutions, and (2) token-level entropy, where token-level uncertainty is used as a proxy for solution quality. We incorporate these signals into several training algorithms and evaluate them on Qwen3-1.7B-Base. When training exclusively on a simple arithmetic task, quality-based signals improve performance by up to $18.6\%$ over the base model, substantially outperforming SFT. On GSM8K, however, gains are more modest and depend strongly on the quality signal. For instance, self-judgments show poor agreement with a stronger external judge and can even degrade performance below the base model. Overall, our results suggest that quality-based training can extract useful signal from saturated questions for base models, but that applying such signals to more complex tasks requires careful calibration and further study.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑