重新将人类置于LLM个性化中心
Re-Centering Humans in LLM Personalization
AI总结：
研究LLM个性化在合成数据与人类数据上的性能差距，通过收集人类对话和判断揭示系统在属性提取、相关属性配对和个性化响应生成阶段的局限性，并引入轻量级训练干预以缩小差距。
摘要：
尽管兴趣日益增长，但大多数对大型语言模型（LLM）个性化能力的评估都依赖于合成数据。目前尚不清楚当前的个性化系统对真实用户的效果如何。在本文中，我们研究了LLM个性化在使用合成数据与人类数据时的性能差距。我们收集了人类对话（550个对话）和个性化三个阶段的判断：从对话中提取用户属性（5,949个判断），将相关属性与新提示配对（11,919个），以及将相关属性融入个性化响应（1,101个）。纳入人类数据揭示了每个阶段的系统局限性。模型难以从人类对话中提取属性，与人类在相关属性上的判断不一致，并且生成的个性化响应被人类评价为并不优于通用响应（尽管LLM广泛评价为更好）。我们在前两个阶段引入了两种轻量级基于训练的干预措施，使自动化个性化评估更接近人类数据。然而，在第三阶段，我们发现学习到的奖励模型与人类评分的相关性仅达到中等水平，这表明与人类一致的个性化质量判断难以直接建模。我们收集的数据为研究模型如何以人类认为有用的方式提取、选择和整合用户信息提供了基础。
英文摘要：
Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce two lightweight training-based interventions that shift automated personalization evaluation closer to human data in our first two stages. However, in our third stage we find that learned reward models achieve only modest correlation with human ratings, suggesting that human-aligned personalization quality judgments are difficult to model directly. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
