arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.02668cs.CL
收藏

通过验证器与生成器对齐改进大语言模型

Improving LLMs via Validator-to-Generator Alignment

Department of Computer Science, The University of Texas at Austin(德克萨斯大学奥斯汀分校计算机科学系);Departments of Linguistics and Computer Science, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校语言学与计算机科学系);Department of Computer Science & Center for Data Science, New York University(纽约大学计算机科学系及数据科学中心)

Juan Diego Rodriguez, Jocelyn Zhang, Katrin Erk, Greg Durrett

AI总结:

研究大语言模型不一致问题,提出基于话语频率的生成器-验证器一致性新公式,介绍\FCPA方法,实验表明该方法能提升生成器性能及两者一致性,且保持验证器质量。

摘要:

大语言模型存在不一致性,如生成器-验证器差距。本文引入基于话语频率的G-V一致性新公式,在理性主体多答案问答模型下,验证器与频率校正生成器分数的一致性自然出现。\FCPA方法实现频率校正G-V一致性,实验显示其显著提升G-V一致性和生成器性能。

英文摘要:

Large language models are inconsistent: varying prompts or including unrelated information can lead to unexpected changes in model outputs. The generator-validator (G-V) gap is one manifestation of this phenomenon, where LLMs generate responses that they then deem as invalid if re-queried to validate them. In this work, we introduce a new formulation of G-V consistency that involves a principled correction for utterance frequency. Specifically, generators often assign low likelihood to valid strings simply because those strings are a priori unlikely, which makes naive notions of G-V consistency unworkable. We show that under a natural model of rational agents answering questions with multiple answers, consistency of the validator with a frequency-corrected generator score emerges naturally. Our method, \emph{\FCPAname} (\FCPA), is a training objective implementing frequency-corrected G-V consistency for real-world LLMs. Our experimental results show that training with \FCPA{} substantially improves both G-V consistency and generator performance over prior methods, with gains of up to $+27$pp in Pearson correlation on IFEval and HumanEval, while preserving validator quality across all evaluated tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑