arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.13022cs.CLcs.MM
收藏

SoMeLVLM:一种面向社交媒体处理的大视觉语言模型

SoMeLVLM: A Large Vision Language Model for Social Media Processing

Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University(复旦大学类脑智能科学与技术研究院);School of Data Science, Fudan University(复旦大学数据科学学院);Department of Computer Science, University of Rochester(罗切斯特大学计算机科学系);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Research Institute of Intelligent Complex Systems, Fudan University(复旦大学智能复杂体系研究院)

Xinnong Zhang, Haoyu Kuang, Xinyi Mou, Hanjia Lyu, Kun Wu, Siming Chen, Jiebo Luo, Xuanjing Huang, Zhongyu Wei

AI总结:

针对通用大视觉语言模型难以适配社交媒体任务的问题,本文提出具备五大认知能力的SoMeLVLM模型,构建65.4万条多模态指令数据集微调,在多项社交媒体任务上达SOTA,认知能力优于基线。

摘要:

社交媒体以多模态为特征的发展催生了各类现象与挑战,亟需一种有效方法统一解决自动化任务。功能强大的大视觉语言模型(Large Vision Language Models)为同时处理多种任务提供了可能,但即便采用精心设计的提示方法,通用领域模型往往也难以适配社交媒体任务独特的表达风格与语境。本文提出了一种面向社交媒体处理的大视觉语言模型SoMeLVLM,这是一个具备知识与理解、应用、分析、评估、创造五大核心能力的认知框架,旨在理解并生成符合真实情况的社交媒体行为。我们构建了一个包含65.4万条样本的多模态社交媒体指令微调数据集,以支撑该认知框架并微调模型。实验表明,SoMeLVLM在多项社交媒体任务上取得了当前最优(state-of-the-art)性能,进一步分析显示其在认知能力方面相较于基线模型具有显著优势。

英文摘要:

The growth of social media, characterized by its multimodal nature, has led to the emergence of diverse phenomena and challenges, which calls for an effective approach to uniformly solve automated tasks. The powerful Large Vision Language Models make it possible to handle a variety of tasks simultaneously, but even with carefully designed prompting methods, the general domain models often fall short in aligning with the unique speaking style and context of social media tasks. In this paper, we introduce a Large Vision Language Model for Social Media Processing (SoMeLVLM), which is a cognitive framework equipped with five key capabilities including knowledge & comprehension, application, analysis, evaluation, and creation. SoMeLVLM is designed to understand and generate realistic social media behavior. We have developed a 654k multimodal social media instruction-tuning dataset to support our cognitive framework and fine-tune our model. Our experiments demonstrate that SoMeLVLM achieves state-of-the-art performance in multiple social media tasks. Further analysis shows its significant advantages over baselines in terms of cognitive abilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑