谁的船能浮起来？通过推断用户画像提升偏好微调中的个性化
Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas
AI总结：
针对偏好数据缺乏用户偏好原因导致LLMs难以个性化定制响应的问题，提出基于溯因推理的画像推断（PI）与画像定制（PT）方法，通过增强偏好数据有效提升模型个性化能力并更好满足罕见偏好。
摘要：
LLMs通过学习用户对于提示更偏好两个响应中的哪一个，来遵循输入指令以实现对齐。然而，这类偏好数据并未传达用户为何偏好被选中或被拒绝的响应，因此在这些数据集上训练的LLMs无法针对多样的用户需求定制响应。为了揭示这些个性化参数，我们将溯因推理应用于偏好数据，推断可能偏好任一响应的用户需求与兴趣，即画像。我们分两步测试该想法：画像推断（PI），溯因推断偏好被选中或被拒绝输出的用户画像；以及画像定制（PT），训练模型针对PI得到的画像定制输出。我们表明：1）LLMs推断的画像能准确解释为何不同用户可能同时偏好被选中或被拒绝的输出；2）通过PT在经PI画像增强的偏好数据上训练，能提升个性化并泛化至支持用户编写的画像；3）被拒绝响应的画像构成了更难的个性化评估，表明相比典型对齐方法，PT能更好地帮助具有罕见偏好的用户。我们主张以溯因视角看待偏好个性化，不仅追问哪个响应更好，还要追问何时、为何以及为谁更好。
英文摘要：
LLMs are aligned to follow input instructions by learning which of two responses users prefer for a prompt. However, such preference data do not convey why users prefer responses that are chosen or rejected, so LLMs trained on these datasets cannot tailor responses to varied user needs. To surface these parameters of personalization, we apply abductive reasoning to preference data, inferring needs and interests of users, i.e., personas, that may prefer either response. We test this idea in two steps: Persona Inference (PI), abductively inferring personas of users who prefer chosen or rejected outputs, and Persona Tailoring (PT), training models to tailor outputs to personas from PI. We show: 1) LLMs infer personas accurately explaining why different users may prefer both chosen or rejected outputs; 2) Training on preference data augmented with PI personas via PT boosts personalization and generalizes to supporting user-written personas; and 3) Rejected response personas form harder personalization evaluations, showing PT better aids users with uncommon preferences versus typical alignment methods. We argue for an abductive view of preferences for personalization, asking not only which response is better but when, why, and for whom.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
