PersonalLLM：为个体偏好定制大语言模型
PersonalLLM: Tailoring LLMs to Individual Preferences
AI总结：
针对大语言模型个性化交互中用户偏好异质且数据稀疏的问题，提出PersonalLLM基准，利用奖励模型模拟多样化用户，为个性化算法提供测试平台。
摘要：
随着大语言模型（LLM）能够处理复杂任务，针对用户细微且独特的偏好进行个性化交互的潜力日益增长。我们提出了一个公开基准数据集 PersonalLLM，专注于调整大语言模型以为特定用户提供最大益处。与现有隐含假设偏好一致的 alignment 基准不同，我们策划了开放式提示，并配以多个高质量答案，用户在这些答案上预计会表现出异质的潜在偏好。我们并非基于高层属性（如用户的种族或回答长度）对 LLM 进行人物角色提示（这相对于人类会产生同质化的偏好），而是开发了一种方法，能够从一组预训练奖励模型中模拟具有多样化偏好的大规模用户群体。我们的数据集和生成的人物角色为开发个性化算法提供了一个创新测试平台，这些算法需要应对持续的数据稀疏性——即来自特定用户的反馈很少——通过利用来自其他（相似）用户的历史数据。我们探索了基本的上下文学习和元学习基线，以展示 PersonalLLM 的实用性，并强调未来方法论发展的必要性。我们的数据集可在 https://huggingface.co/datasets/namkoong-lab/PersonalLLM 获取。
英文摘要：
As LLMs become capable of complex tasks, there is growing potential for personalized interactions tailored to the subtle and idiosyncratic preferences of the user. We present a public benchmark, PersonalLLM, focusing on adapting LLMs to provide maximal benefits for a particular user. Departing from existing alignment benchmarks that implicitly assume uniform preferences, we curate open-ended prompts paired with many high-quality answers over which users would be expected to display heterogeneous latent preferences. Instead of persona-prompting LLMs based on high-level attributes (e.g., user's race or response length), which yields homogeneous preferences relative to humans, we develop a method that can simulate a large user base with diverse preferences from a set of pre-trained reward models. Our dataset and generated personalities offer an innovative testbed for developing personalization algorithms that grapple with continual data sparsity--few relevant feedback from the particular user--by leveraging historical data from other (similar) users. We explore basic in-context learning and meta-learning baselines to illustrate the utility of PersonalLLM and highlight the need for future methodological development. Our dataset is available at https://huggingface.co/datasets/namkoong-lab/PersonalLLM
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
