Evaluating the quality of multi-turn conversations is crucial for developing capable Large Language Models (LLMs), yet remains a significant challenge, often requiring costly human evaluation. Multi-turn reward models (RMs) offer a scalable alternative and can provide valuable signals for guiding LLM training. While recent work has advanced multi-turn \textit{training} techniques, effective automated \textit{evaluation} specifically for multi-turn interactions lags behind. We observe that standard preference datasets, typically contrasting responses based only on the final conversational turn, provide insufficient signal to capture the nuances of multi-turn interactions. Instead, we find that incorporating contrasts spanning \textit{multiple} turns is critical for building robust multi-turn RMs. Motivated by this finding, we propose \textbf{MU}lti-\textbf{S}tep \textbf{I}nstruction \textbf{C}ontrast (MUSIC), an unsupervised data augmentation strategy that synthesizes contrastive conversation pairs exhibiting differences across multiple turns. Leveraging MUSIC on the Skywork preference dataset, we train a multi-turn RM based on the Gemma-2-9B-Instruct model. Empirical results demonstrate that our MUSIC-augmented RM outperforms baseline methods, achieving higher alignment with judgments from advanced proprietary LLM judges on multi-turn conversations, crucially, without compromising performance on standard single-turn RM benchmarks.
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);Zhipu AI(智谱AI);University of Electronic Science and Technology of China(电子科技大学)
Audio-FLAN: An Instruction-Following Dataset for Unified Audio Understanding and Generation of Speech, Music, and Sound
The Hong Kong University of Science(香港科学与技术大学);Inner Mongolia University(内蒙古大学);Beihang University(北京航空航天大学);Queen Mary University of London(伦敦玛丽女王大学);The Chinese University of Hong Kong(香港中文大学);National University of Singapore(新加坡国立大学);University of Surrey(萨里大学);University of Rochester(罗切斯特大学);Independent Researcher(独立研究者)
Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University(上海数据科学 key laboratory,复旦大学计算机科学与人工智能学院);School of Data Science, Fudan University(复旦大学数据科学学院);Ant Group(蚂蚁集团)
Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education
Chosun University(昌原大学);Korea University(韩国大学);Seoul National University(首尔国立大学);Korea Institute for Curriculum and Evaluation(韩国课程与评价研究院);Upstage;Indiana University Bloomington(印第安纳大学布卢明顿分校);Nanyang Technological University(南洋理工大学)
IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);Zhipu AI(智谱AI);University of Electronic Science and Technology of China(电子科技大学);The Knowledge Engineering Group (KEG), Tsinghua University(清华大学知识工程小组(KEG))
MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response
University of Pennsylvania(宾夕法尼亚大学);Queen Mary University of London(伦敦大学玛丽皇后学院);Carnegie Mellon University(卡内基梅隆大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Waterloo(滑铁卢大学)
MusiLingo 通过单一投影层将冻结音乐音频模型 MERT 与冻结 LLM 对齐,并构建 MusicInstruct 数据集,实现音乐描述生成与音乐查询响应的性能提升。
SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data
Zhejiang University(浙江大学);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机与数据科学学院);Hangzhou City University(杭州城市大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院)
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);Nanjing University(南京大学);Fudan University(复旦大学);Nanyang Technological University(南洋理工大学)
MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following
Department of Electrical and Computer Engineering, College of Engineering, University of Tehran(德黑兰大学电气与计算机工程系,工程学院);Department of Statistics, Mathematics and Computer Science, Allameh Tabataba’i University(塔巴蒂大学统计、数学与计算机科学系)
ComposerX: Multi-Agent Symbolic Music Composition with LLMs
University of Rochester(罗切斯特大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Hong Kong University of Science and Technology(香港科技大学);University of Surrey(萨里大学);The University of Manchester(曼彻斯特大学);Queen Mary University of London(伦敦玛丽女王大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(中国科学院计算技术研究所智能信息处理重点实验室);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学)
Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Dalian University of Technology(大连理工大学);University of Science and Technology of China(中国科学技术大学);Alibaba Cloud Computing(阿里云计算);Chinese Academy of Sciences(中国科学院)
HiR 提出了一种样本高效强化学习框架,通过将失败尝试视为成功来提升指令跟随任务的性能,利用双偏好学习实现高效优化。
Centre for Digital Music, Queen Mary University of London(伦敦大学玛丽皇后学院数字音乐中心);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机学院);University of Sheffield(谢菲尔德大学);The University of Manchester(曼彻斯特大学);University of Auckland(奥克兰大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);New York University Shanghai(纽约大学上海分校);University of Tokyo(东京大学)