通过大语言模型多模态奖励分解对齐对话代理
Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition
AI总结：
本文提出了一种基于大语言模型的多模态奖励分解方法，通过分解会话级反馈来提升对话生成质量，无需人工反馈。
摘要：
我们提出了一种基于大语言模型的奖励分解框架，用于仅使用单个会话级反馈信号对齐对话代理。我们利用冻结的、预训练的大语言模型（LLM）的推理能力，通过分解全局会话级反馈来推断细粒度的局部隐式奖励。我们的第一个纯文本变体促使LLM仅使用对话记录进行奖励分解。第二个多模态变体纳入了额外的行为线索，如音调、目光和面部情绪，这些线索以自然语言描述形式呈现。这些推断出的回合级奖励被浓缩成一个轻量级的奖励模型，我们将其用于基于强化学习的对话生成微调。我们评估了纯文本和多模态变体与现有最佳奖励分解方法的对比，并在人类对对话质量的评估中展示了显著的改进，表明LLM是强大的奖励分解器，可以省去手动奖励塑造和细粒度人工反馈的需要。
英文摘要：
We propose a large language model based reward decomposition framework for aligning dialogue agents using only a single session-level feedback signal. We leverage the reasoning capabilities of a frozen, pretrained large language model (LLM) to infer fine-grained local implicit rewards by decomposing global, session-level feedback. Our first \emph{text-only} variant prompts the LLM to perform reward decomposition using only the dialogue transcript. The second \emph{multimodal} variant incorporates additional behavioral cues, such as pitch, gaze, and facial affect, expressed as natural language descriptions. These inferred turn-level rewards are distilled into a lightweight reward model, which we utilize for RL-based fine-tuning for dialogue generation. We evaluate both text-only and multimodal variants against state-of-the-art reward decomposition methods and demonstrate notable improvements in human evaluations of conversation quality, suggesting that LLMs are strong reward decomposers that obviate the need for manual reward shaping and granular human feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
