UMAIR-FPS:融合绘画风格的用户感知多模态动漫插画推荐
UMAIR-FPS: User-aware Multi-modal Animation Illustration Recommendation Fusion with Painting Style
AI总结:
针对现有动漫插画推荐系统图像特征整合不足、多模态推荐受数据集限制的问题,提出UMAIR-FPS模型,通过双输出图像编码器、领域知识微调文本表征及用户感知动态融合机制,在真实数据集上实现性能提升。
摘要:
基于AI的高质量图像生成模型快速发展,催生了海量动漫插画。在海量数据中为用户推荐插画已成为一项具有挑战性的热门任务。然而,现有动漫推荐系统多聚焦文本特征,仍需整合图像特征。此外,多数多模态推荐研究受限于强耦合数据集,限制了其在动漫插画场景的适用性。\n为弥补这些不足,我们提出了融合绘画风格的用户感知多模态动漫插画推荐模型UMAIR-FPS。在特征提取阶段,针对图像特征,我们首次将图像绘画风格特征与语义特征相结合,构建双输出图像编码器以增强表征能力;针对文本特征,我们通过融入领域知识微调Sentence-Transformers(句向量模型)来获取文本嵌入,该领域知识分别从多语言映射、实体关系和术语解释视角构建了多种领域文本对。在多模态融合阶段,我们创新性地提出用户感知多模态贡献度度量机制,根据交互层面的用户特征动态加权多模态特征,并采用DCN-V2模块有效建模有界度多模态交叉。\nUMAIR-FPS在大型真实世界数据集上超越了现有最优基线,展现出显著的性能提升。
英文摘要:
The rapid advancement of high-quality image generation models based on AI has generated a deluge of anime illustrations. Recommending illustrations to users within massive data has become a challenging and popular task. However, existing anime recommendation systems have focused on text features but still need to integrate image features. In addition, most multi-modal recommendation research is constrained by tightly coupled datasets, limiting its applicability to anime illustrations. We propose the User-aware Multi-modal Animation Illustration Recommendation Fusion with Painting Style (UMAIR-FPS) to tackle these gaps. In the feature extract phase, for image features, we are the first to combine image painting style features with semantic features to construct a dual-output image encoder for enhancing representation. For text features, we obtain text embeddings based on fine-tuning Sentence-Transformers by incorporating domain knowledge that composes a variety of domain text pairs from multilingual mappings, entity relationships, and term explanation perspectives, respectively. In the multi-modal fusion phase, we novelly propose a user-aware multi-modal contribution measurement mechanism to weight multi-modal features dynamically according to user features at the interaction level and employ the DCN-V2 module to model bounded-degree multi-modal crosses effectively. UMAIR-FPS surpasses the stat-of-the-art baselines on large real-world datasets, demonstrating substantial performance enhancements.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。