arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.23815cs.CLcs.LG

通过对用户写作样本进行偏好预测来对齐 LLM

Aligning LLMs by Predicting Preferences from User Writing Samples

University of Colorado（科罗拉多大学）；Apple（苹果公司）

Stéphane Aroca-Ouellette, Natalie Mackraz, Barry-John Theobald, Katherine Metcalf

备注：
Accepted to ICML 2025. 32 pages total: 9 main, 2 references, 21 appendix. arXiv admin note: substantial text overlap with arXiv:2410.06273
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 PROSE 方法，通过迭代细化偏好描述和跨样本验证，从用户写作样本中更精准地推断个性化偏好，显著提升了写作智能体的生成质量并证明了与 ICL 的互补性。

摘要：

为了创建能够提供个性化和有效交互的对齐 LLM 智能体，满足人类偏好至关重要。最近的研究表明，作为写作智能体的 LLM 有潜力推断出用户偏好的描述。随后，智能体的对齐便来自于基于推断出的偏好描述进行条件化。然而，现有方法通常生成通用的偏好描述，无法捕捉人类偏好独特且个性化的本质。本文介绍了 PROSE，这是一种旨在提高从用户写作样本中推断出的偏好描述精确度的方法。PROSE 包含两个关键要素：(1) 对推断出的偏好进行迭代细化，以及 (2) 在多个用户写作样本中验证推断出的偏好。我们使用多个 LLM（即 Qwen2.5 7B 和 72B Instruct、GPT-mini 和 GPT-4o）在摘要和电子邮件写作任务上对 PROSE 进行了评估。我们发现 PROSE 能更准确地推断出细微的人类偏好，使写作智能体生成内容的质量比 CIPHER（一种最先进的偏好推断方法）提高了 33%。最后，我们证明了 ICL 和 PROSE 是互补的方法，将它们结合起来比单独使用 ICL 提供了高达 9% 的提升。

英文摘要：

Accommodating human preferences is essential for creating aligned LLM agents that deliver personalized and effective interactions. Recent work has shown the potential for LLMs acting as writing agents to infer a description of user preferences. Agent alignment then comes from conditioning on the inferred preference description. However, existing methods often produce generic preference descriptions that fail to capture the unique and individualized nature of human preferences. This paper introduces PROSE, a method designed to enhance the precision of preference descriptions inferred from user writing samples. PROSE incorporates two key elements: (1) iterative refinement of inferred preferences, and (2) verification of inferred preferences across multiple user writing samples. We evaluate PROSE with several LLMs (i.e., Qwen2.5 7B and 72B Instruct, GPT-mini, and GPT-4o) on a summarization and an email writing task. We find that PROSE more accurately infers nuanced human preferences, improving the quality of the writing agent's generations over CIPHER (a state-of-the-art method for inferring preferences) by 33\%. Lastly, we demonstrate that ICL and PROSE are complementary methods, and combining them provides up to a 9\% improvement over ICL alone.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑