发表机构
Penn State University; Boston University; Meta Inc.（宾夕法尼亚州立大学; 波士顿大学; Meta公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出从用户历史偏好对中学习个性化奖励模型，通过自编码器压缩属性维度并优化低维权重，在推理时指导扩散模型生成，实现约77%的偏好预测准确率。
AI 中文摘要
扩散模型能够生成高质量的图像，但将其输出与个体用户偏好对齐仍然具有挑战性。一个关键的瓶颈是如何从有限的反馈中准确建模多样化的用户偏好。现有方法通常依赖劳动密集的人工偏好标注或视觉语言模型（VLM）从用户交互历史中提取偏好信息，这引入了大量的标注或计算成本，限制了可扩展性。我们提出了一种方法，直接从用户的历史图像偏好对中学习个性化奖励模型。首先，我们使用自编码器将数百个视觉属性压缩为50个属性锚定的偏好维度，并训练一个评估器沿这些维度对图像进行评分。然后，我们将每个用户的偏好表示为共享维度分数的线性组合，通过最大化其观测到的成对偏好在Bradley-Terry模型下的似然来估计用户特定的权重。这种表述将每个用户的适应简化为优化一个低维权重向量，简化了优化过程，并实现了从稀疏反馈中进行数据高效的个性化。学习到的个性化奖励在推理时指导图像生成，同时保持扩散模型冻结。在真实用户偏好数据上的实验表明，我们的方法达到了约77%的留出成对偏好预测准确率，并改善了生成图像与个体用户偏好的对齐。
英文摘要
Diffusion models can generate high-quality images, yet aligning their outputs with individual user preferences remains challenging. A key bottleneck is accurately modeling diverse user preferences from limited feedback. Existing approaches often rely on labor-intensive manual preference annotations or vision-language models (VLM) to extract preference information from user interaction histories, introducing substantial annotation or computational costs that limit scalability. We propose an approach that learns personalized reward models directly from users' historical image preference pairs. First, we use an autoencoder to compress hundreds of visual attributes into 50 attribute-anchored preference dimensions and train an evaluator to score images along these dimensions. We then represent each user's preferences as a linear combination of the shared dimension scores, estimating the user-specific weights by maximizing the likelihood of their observed pairwise preferences under the Bradley-Terry model. This formulation reduces per-user adaptation to optimizing a low-dimensional weight vector, simplifying optimization and enabling data-efficient personalization from sparse feedback. The learned personalized rewards guide image generation at inference time while keeping the diffusion model frozen. Experiments on real-user preference data show that our approach achieves approximately 77% held-out pairwise preference prediction accuracy and improves the alignment of generated images with individual user preferences.
