arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.10381cs.IRcs.AI
收藏

UMAIR-FPS:融合绘画风格的用户感知多模态动漫插画推荐

UMAIR-FPS: User-aware Multi-modal Animation Illustration Recommendation Fusion with Painting Style

National Pilot School of Software, Yunnan University(云南大学国家示范性软件学院);Yunnan Key Laboratory of Software Engineering(云南省软件工程重点实验室);Institute of Management of Technology, National Yang Ming Chiao Tung University(国立阳明交通大学科技管理研究所)

Yan Kang, Hao Lin, Mingjian Yang, Shin-Jye Lee

AI总结:

针对现有动漫插画推荐系统图像特征整合不足、多模态推荐受数据集限制的问题,提出UMAIR-FPS模型,通过双输出图像编码器、领域知识微调文本表征及用户感知动态融合机制,在真实数据集上实现性能提升。

摘要:

基于AI的高质量图像生成模型快速发展,催生了海量动漫插画。在海量数据中为用户推荐插画已成为一项具有挑战性的热门任务。然而,现有动漫推荐系统多聚焦文本特征,仍需整合图像特征。此外,多数多模态推荐研究受限于强耦合数据集,限制了其在动漫插画场景的适用性。\n为弥补这些不足,我们提出了融合绘画风格的用户感知多模态动漫插画推荐模型UMAIR-FPS。在特征提取阶段,针对图像特征,我们首次将图像绘画风格特征与语义特征相结合,构建双输出图像编码器以增强表征能力;针对文本特征,我们通过融入领域知识微调Sentence-Transformers(句向量模型)来获取文本嵌入,该领域知识分别从多语言映射、实体关系和术语解释视角构建了多种领域文本对。在多模态融合阶段,我们创新性地提出用户感知多模态贡献度度量机制,根据交互层面的用户特征动态加权多模态特征,并采用DCN-V2模块有效建模有界度多模态交叉。\nUMAIR-FPS在大型真实世界数据集上超越了现有最优基线,展现出显著的性能提升。

英文摘要:

The rapid advancement of high-quality image generation models based on AI has generated a deluge of anime illustrations. Recommending illustrations to users within massive data has become a challenging and popular task. However, existing anime recommendation systems have focused on text features but still need to integrate image features. In addition, most multi-modal recommendation research is constrained by tightly coupled datasets, limiting its applicability to anime illustrations. We propose the User-aware Multi-modal Animation Illustration Recommendation Fusion with Painting Style (UMAIR-FPS) to tackle these gaps. In the feature extract phase, for image features, we are the first to combine image painting style features with semantic features to construct a dual-output image encoder for enhancing representation. For text features, we obtain text embeddings based on fine-tuning Sentence-Transformers by incorporating domain knowledge that composes a variety of domain text pairs from multilingual mappings, entity relationships, and term explanation perspectives, respectively. In the multi-modal fusion phase, we novelly propose a user-aware multi-modal contribution measurement mechanism to weight multi-modal features dynamically according to user features at the interaction level and employ the DCN-V2 module to model bounded-degree multi-modal crosses effectively. UMAIR-FPS surpasses the stat-of-the-art baselines on large real-world datasets, demonstrating substantial performance enhancements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑