arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.08670cs.AI
收藏

Rec-GPT4V:基于大型视觉语言模型的多模态推荐

Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models

University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Lehigh University(里海大学)

Yuqing Liu, Yu Wang, Lichao Sun, Philip S. Yu

AI总结:

本文提出Rec-GPT4V框架及视觉-摘要思维(VST)推理方案,通过利用用户历史作为上下文偏好并生成图像摘要,有效解决了大型视觉语言模型在多模态推荐中缺乏用户偏好知识及处理图像动态困难的问题。

摘要:

大型视觉语言模型(LVLMs)的发展,得益于其对静态图像和文本动态的熟练理解,为解决传统多模态推荐面临的挑战提供了潜力。然而,由于以下复杂性,LVLMs在该领域的应用仍受到限制:首先,LVLMs缺乏用户偏好知识,因为它们是从海量通用数据集中训练而来的。其次,在涉及离散、嘈杂和冗余图像序列的场景中,LVLMs在处理多个图像动态方面遭遇挫折。为了克服这些问题,我们提出了一种名为Rec-GPT4V:视觉-摘要思维(VST)的新型推理方案,利用大型视觉语言模型进行多模态推荐。我们利用用户历史作为上下文用户偏好来解决第一个挑战。接着,我们提示LVLMs生成物品图像摘要,并在自然语言空间中利用图像理解结合物品标题来查询候选物品的用户偏好。我们在四个数据集上使用三种LVLMs进行了全面实验:GPT4-V、LLaVa-7b和LLaVa-13b。数值结果证明了VST的有效性。

英文摘要:

The development of large vision-language models (LVLMs) offers the potential to address challenges faced by traditional multimodal recommendations thanks to their proficient understanding of static images and textual dynamics. However, the application of LVLMs in this field is still limited due to the following complexities: First, LVLMs lack user preference knowledge as they are trained from vast general datasets. Second, LVLMs suffer setbacks in addressing multiple image dynamics in scenarios involving discrete, noisy, and redundant image sequences. To overcome these issues, we propose the novel reasoning scheme named Rec-GPT4V: Visual-Summary Thought (VST) of leveraging large vision-language models for multimodal recommendation. We utilize user history as in-context user preferences to address the first challenge. Next, we prompt LVLMs to generate item image summaries and utilize image comprehension in natural language space combined with item titles to query the user preferences over candidate items. We conduct comprehensive experiments across four datasets with three LVLMs: GPT4-V, LLaVa-7b, and LLaVa-13b. The numerical results indicate the efficacy of VST.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑