arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.24693cs.CL
收藏

MUSIC: 多步指令对比用于多轮奖励模型

MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Models

Princeton University(普林斯顿大学);Google DeepMind(谷歌DeepMind)

Wenzhe Li, Shujian Zhang, Wenxuan Zhou, John Lambert, Chi Jin, Andrew Hard, Rajiv Mathews, Lun Wang

AI总结:

MUSIC通过多步指令对比提升多轮奖励模型的性能,有效增强多轮对话的评估能力。

摘要:

评估多轮对话的质量对于开发具有能力的大型语言模型(LLMs)至关重要,但仍然是一个重大挑战,通常需要昂贵的人工评估。多轮奖励模型(RMs)提供了一种可扩展的替代方案,可以为LLM训练提供有价值的信号。尽管最近的工作已经推进了多轮训练技术,但有效自动评估多轮交互仍滞后。我们发现,标准偏好数据集通常仅对比最终对话回合的响应,提供的信号不足以捕捉多轮交互的细微差别。相反,我们发现,纳入跨越多个回合的对比是构建稳健多轮RMs的关键。受此发现的启发,我们提出了MUSIC(多步指令对比),一种无监督的数据增强策略,通过合成表现出多个回合差异的对比对话对。利用MUSIC在Skywork偏好数据集上训练基于Gemma-2-9B-Instruct模型的多轮RM。实验证明,我们的MUSIC增强的RM在多轮对话中优于基线方法,实现了与高级专有LLM评判者判断更高的对齐,关键的是,不损害标准单轮RM基准测试的性能。

英文摘要:

Evaluating the quality of multi-turn conversations is crucial for developing capable Large Language Models (LLMs), yet remains a significant challenge, often requiring costly human evaluation. Multi-turn reward models (RMs) offer a scalable alternative and can provide valuable signals for guiding LLM training. While recent work has advanced multi-turn \textit{training} techniques, effective automated \textit{evaluation} specifically for multi-turn interactions lags behind. We observe that standard preference datasets, typically contrasting responses based only on the final conversational turn, provide insufficient signal to capture the nuances of multi-turn interactions. Instead, we find that incorporating contrasts spanning \textit{multiple} turns is critical for building robust multi-turn RMs. Motivated by this finding, we propose \textbf{MU}lti-\textbf{S}tep \textbf{I}nstruction \textbf{C}ontrast (MUSIC), an unsupervised data augmentation strategy that synthesizes contrastive conversation pairs exhibiting differences across multiple turns. Leveraging MUSIC on the Skywork preference dataset, we train a multi-turn RM based on the Gemma-2-9B-Instruct model. Empirical results demonstrate that our MUSIC-augmented RM outperforms baseline methods, achieving higher alignment with judgments from advanced proprietary LLM judges on multi-turn conversations, crucially, without compromising performance on standard single-turn RM benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑