个性化多模态大语言模型中的群体偏好崩溃

中文摘要 AI 辅助

个性化多模态大语言模型旨在生成特定于用户的响应，但现有方法主要依赖于配置文件级信息，忽略了多样的用户偏好。我们识别出群体偏好崩溃，即多用户个性化大语言模型由于生成过程中偏好信号被抑制和偏好使用不可靠，对个体偏好变得不敏感并趋向于占主导地位的群体级选择。我们提出了PrefMoE，一个以偏好为中心的框架，将稳定的配置文件信息与偏好相关表示分开。PrefMoE将偏好分解为共享原型和个性化残差，通过不平衡感知学习、反事实伪用户增强和残差去相关来保留个性化残差，并通过单独的LoRA适应路径路由配置文件和偏好因素。跨多个大语言模型主干的实验表明，PrefMoE改善了偏好敏感的个性化，同时大幅减少了偏好崩溃。

英文摘要

Personalized multimodal large language models (MLLMs) aim to generate user-specific responses, but existing methods mainly rely on profile-level information and overlook diverse user preferences. We identify group preference collapse, where multi-user personalized MLLMs become insensitive to individual preferences and drift toward dominant population-level choices due to suppressed preference signals and unreliable preference use during generation. We propose PrefMoE, a preference-centric framework that separates stable profile information from preference-related representations. PrefMoE decomposes preferences into shared prototypes and personalized residuals, preserves individualized residuals with imbalance-aware learning, counterfactual pseudo-user augmentation, and residual decorrelation, and routes profile and preference factors through separate LoRA adaptation paths. Experiments across multiple MLLM backbones show that PrefMoE improves preference-sensitive personalization while substantially reducing preference collapse. Project page: https://prefmoe.github.io/.

发表机构

• Computer Vision Center, Universitat Autònoma de Barcelona（计算机视觉中心，巴塞罗那自治大学）

机构由 AI 辅助整理，请以论文原文为准。