arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.13709cs.AIcs.LG
收藏

训练大型语言模型智能体以增强人类能力

Training LLM Agents to Empower Humans

UC Berkeley(加州大学伯克利分校);Princeton University(普林斯顿大学)

Evan Ellis, Vivek Myers, Jens Tuyls, Sergey Levine, Anca Dragan, Benjamin Eysenbach

AI总结:

提出Empower方法,通过最大化人类赋权,利用离线数据自监督微调语言模型智能体,使其在辅助人类时更有效,实验显示用户偏好率78%,模拟编程成功率提升192%。

摘要:

辅助智能体不仅应代表人类采取行动,还应在面临重要决策时主动让位并放弃控制权。然而,当前构建辅助智能体的方法,无论是通过模仿人类专家还是基于推断奖励进行强化学习微调,往往鼓励智能体独立完成任务,而非真正辅助人类实现其目标。此外,这些方法通常需要昂贵的显式人类反馈来提供训练信号。我们提出了一种新方法,用于调整辅助语言模型,其核心是最大化人类的赋权(empowerment),即人类在环境中实现期望变化的能力。我们的赋权最大化方法Empower仅需离线文本数据,提供了一种自监督的方法来微调语言模型,使其更好地辅助人类。为研究我们方法的有效性,我们进行了一项18人用户研究,将我们的赋权助手与强基线进行比较。参与者78%的时间偏好我们的助手(p=0.015),接受率高出31%,建议数量减少38%。此外,我们引入了一个新环境,用于使用模拟人类评估多轮代码辅助。利用该环境,我们表明使用Empower训练的智能体在具有挑战性的编码问题上,将模拟人类程序员的成功率平均提高了192%,相对于SFT基线。通过这一赋权目标,我们提供了一个框架,仅使用离线数据即可规模化构建有用且对齐的AI智能体,无需任何额外的人类反馈或可验证奖励。

英文摘要:

Assistive agents should not only take actions on behalf of a human, but also step out of the way and cede control when there are important decisions to be made. However, current methods for building assistive agents, whether via mimicking expert humans or via RL finetuning on an inferred reward, often encourage agents to complete tasks on their own rather than truly assisting the human attain their objectives. Additionally, these methods often require costly explicit human feedback to provide a training signal. We propose a new approach to tuning assistive language models based on maximizing the human's empowerment, their ability to effect desired changes in the environment. Our empowerment-maximizing method, Empower, only requires offline text data, providing a self-supervised method for fine-tuning language models to better assist humans. To study the efficacy of our approach, we conducted an 18-person user study comparing our empowerment assistant with a strong baseline. Participants preferred our assistant 78% of the time (p=0.015), with a 31% higher acceptance rate and 38% fewer suggestions. Additionally, we introduce a new environment for evaluating multi-turn code assistance using simulated humans. Using this environment, we show that agents trained with Empower increase the success rate of a simulated human programmer on challenging coding questions by an average of 192% over an SFT baseline. With this empowerment objective, we provide a framework for useful aligned AI agents at scale using only offline data without the need for any additional human feedback or verifiable rewards.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑