VLM2Rec: 解决视觉语言模型嵌入器在多模态序列推荐中的模态崩溃问题
VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation
AI总结：
本文提出VLM2Rec框架，通过弱模态惩罚对比学习和跨模态关系拓扑正则化，解决多模态序列推荐中模态崩溃问题，提升推荐准确性和鲁棒性。
摘要：
多模态序列推荐（SR）通常依赖于小而固定的预训练编码器，这限制了语义能力并阻碍了协同过滤（CF）信号的完全整合。受大型语言模型（LLMs）在高容量嵌入器方面的成功启发，我们研究了视觉语言模型（VLMs）作为CF-aware多模态编码器在SR中的应用。然而，我们发现标准对比监督微调（SFT）在适应VLMs生成嵌入并注入CF信号时，会加剧其固有的模态崩溃。在这种状态下，优化被单一模态主导，而其他模态退化，最终损害推荐准确性。为了解决这一问题，我们提出了VLM2Rec，一种基于VLM嵌入器的多模态序列推荐框架，旨在确保模态利用的平衡。具体而言，我们引入了弱模态惩罚对比学习以在优化过程中纠正梯度不平衡，并引入了跨模态关系拓扑正则化以保持模态间的几何一致性。广泛实验表明，VLM2Rec在多种场景中均在准确性和鲁棒性上优于最先进的基线。
英文摘要：
Sequential Recommendation (SR) in multimodal settings typically relies on small frozen pretrained encoders, which limits semantic capacity and prevents Collaborative Filtering (CF) signals from being fully integrated into item representations. Inspired by the recent success of Large Language Models (LLMs) as high-capacity embedders, we investigate the use of Vision-Language Models (VLMs) as CF-aware multimodal embedders for SR. However, we find that standard contrastive Supervised Fine-Tuning (SFT), used to adapt VLMs for embedding generation and inject CF signals, can amplify inherent modality imbalance: optimization becomes dominated by one modality while the other degrades, ultimately undermining recommendation accuracy. To address this, we propose VLM2Rec, a VLM embedder-based framework for multimodal sequential recommendation designed to promote balanced modality utilization. Specifically, we introduce Weak-modality Penalized Contrastive Learning to mitigate gradient imbalance during optimization and cross-modal Relational Topology Regularization to preserve geometric consistency between modalities. Experiments show that VLM2Rec consistently improves over strong baselines in both accuracy and robustness across diverse scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
