通过从用户编辑中学习潜在偏好来对齐LLM智能体
Aligning LLM Agents by Learning Latent Preference from User Edits
AI总结：
本文提出PRELUDE框架和CIPHER算法，通过从用户历史编辑中推断潜在偏好描述来生成个性化回复，无需微调LLM，在摘要和邮件写作任务上以最低编辑距离成本优于基线。
摘要：
我们研究基于用户对智能体输出所做编辑的、基于LLM的语言智能体交互式学习。在诸如写作助手等典型场景中，用户与语言智能体交互以根据上下文生成回复，并且除了提高正确性之外，还可以选择性地编辑智能体回复，以根据其潜在偏好进行个性化。编辑反馈是自然产生的，这使其成为改进智能体与用户偏好对齐以及随时间降低用户编辑成本的合适候选。我们提出了一个学习框架PRELUDE，它基于历史编辑数据推断用户潜在偏好的描述。推断出的用户偏好描述被用来定义未来生成回复的提示。这避免了微调智能体，因为微调成本高昂、难以随用户数量扩展，甚至可能降低其在其他任务上的性能。此外，学习描述性偏好提高了可解释性，使用户能够查看和修改学习到的偏好。然而，用户偏好可能复杂、微妙，并随上下文而变化，这使得学习具有挑战性。为了解决这个问题，我们提出了一个简单而有效的算法CIPHER，它利用LLM根据用户编辑来推断给定上下文的用户偏好。在未来，CIPHER从历史中最接近的k个上下文中检索推断出的偏好，并形成一个聚合偏好用于回复生成。我们引入了两个交互式环境——摘要和电子邮件写作，并使用GPT-4模拟用户进行评估。在这两项任务上，CIPHER通过实现最低的编辑距离成本优于多个基线，同时在LLM查询成本上只有很小的开销。我们的分析报告表明，CIPHER学习到的用户偏好与真实的潜在偏好显示出显著的相似性。
英文摘要：
We study interactive learning of LLM-based language agents based on user edits made to the agent's output. In a typical setting such as writing assistants, the user interacts with a language agent to generate a response given a context, and may optionally edit the agent response to personalize it based on their latent preference, in addition to improving the correctness. The edit feedback is naturally generated, making it a suitable candidate for improving the agent's alignment with the user's preference, and for reducing the cost of user edits over time. We propose a learning framework, PRELUDE that infers a description of the user's latent preference based on historic edit data. The inferred user preference descriptions are used to define prompts for generating responses in the future. This avoids fine-tuning the agent, which is costly, challenging to scale with the number of users, and may even degrade its performance on other tasks. Furthermore, learning descriptive preference improves interpretability, allowing the user to view and modify the learned preference. However, user preference can be complex, subtle, and vary based on context, making it challenging to learn. To address this, we propose a simple yet effective algorithm named CIPHER that leverages the LLM to infer the user preference for a given context based on user edits. In the future, CIPHER retrieves inferred preferences from the k-closest contexts in the history, and forms an aggregate preference for response generation. We introduce two interactive environments -- summarization and email writing, and use a GPT-4 simulated user for evaluation. On both tasks, CIPHER outperforms several baselines by achieving the lowest edit distance cost while only having a small overhead in LLM query cost. Our analysis reports that user preferences learned by CIPHER show significant similarity to the ground truth latent preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
