arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.04068cs.LG
收藏

学习可调节的澄清策略与协作自我扮演

Learning Steerable Clarification Policies with Collaborative Self-play

Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind);Meta

Jonathan Berant, Maximillian Chen, Adam Fisch, Reza Aghajani, Fantine Huot, Mirella Lapata, Jacob Eisenstein

AI总结:

本文提出通过自我扮演训练可调节的澄清策略,以处理模糊查询,通过考虑澄清问题的成本和生成词的成本,提高回答的准确性和奖励

摘要:

为了解决不明确或模糊的查询,AI助手需要一种管理不确定性的策略,以确定(a)何时猜测用户意图并直接回答,(b)何时列举并回答多种可能的意图,以及(c)何时提问澄清。然而,此类策略依赖于用户偏好或模态等上下文因素。例如,在小屏幕上或语音设置中,列举多种用户意图会很麻烦。在本工作中,我们提出使用自我扮演来训练可调节的策略来管理这种不确定性。给定两个代理,一个模拟用户,另一个是AI助手,我们生成对话,其中用户发出可能模糊的查询,助手需要决定如何回应。重要的是,模型的输入是每个澄清问题的数值成本以及每个生成的词,它被要求采取最大化最终奖励的行动,该奖励是成本惩罚的准确性。我们使用强化自我训练(ReST)来训练模型以获得高奖励,并展示这导致一种可调节的策略,其行为可根据提供的成本预测性地变化,从而导致更高的奖励和准确性。此外,我们的程序还能够推广到训练时未观察到的数值成本值。

英文摘要:

To handle underspecified or ambiguous queries, AI assistants need a policy for managing their uncertainty to determine (a) when to guess the user intent and answer directly, (b) when to enumerate and answer multiple possible intents, and (c) when to ask a clarifying question. However, such policies are contextually dependent on factors such as user preferences or modality. For example, enumerating multiple possible user intentions is cumbersome on small screens or in a voice setting. In this work, we propose to train steerable policies for managing this uncertainty using self-play. Given two agents, one simulating a user and the other an AI assistant, we generate conversations where the user issues a potentially ambiguous query, and the assistant needs to determine how to respond. Importantly, the model takes as input the numerical cost of each clarification question, and each generated word, and is asked to take the action that will maximize its final reward, which is the cost-penalized accuracy. We use Reinforced Self-Training (ReST) to train our model to achieve high reward and show this leads to a steerable policy that changes its behavior predictably conditioned on the provided costs, leading to higher reward and accuracy. Moreover, our procedure also generalizes to numerical cost values that were unobserved at training time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑