arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.09758cs.CVcs.CL
收藏

一段视频价值4096个Token:将视频言语化以实现零样本理解

A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot

Adobe Media and Data Science Research (MDSR)(Adobe媒体与数据科学研究(MDSR));IIIT-Delhi(德里印度理工学院);State University of New York at Buffalo(纽约州立大学布法罗分校)

Aanisha Bhattacharya, Yaman K Singla, Balaji Krishnamurthy, Rajiv Ratn Shah, Changyou Chen

AI总结:

本文提出将长视频言语化为自然语言描述,再利用大语言模型进行零样本视频理解,在十五项任务上超越监督基线,并发布了首个说服策略识别数据集。

摘要:

多媒体内容,如广告和故事视频,展现出创造力与多种模态的丰富融合。它们融合了文本、视觉、音频和叙事技巧等元素,运用情感、象征和口号等手法来传达意义。多媒体领域缺乏大规模标注训练数据集,阻碍了具有满意性能的监督学习模型在现实应用中的发展。另一方面,大语言模型(LLMs)的兴起在各种自然语言处理(NLP)任务中展现出卓越的零样本性能,例如情感分类、问答和主题分类。为了利用这些先进技术弥合多媒体理解中的性能差距,我们提出将长视频言语化,以自然语言生成其描述,然后对生成的故事而非原始视频执行视频理解任务。通过在十五个视频理解任务上的广泛实验,我们证明我们的方法尽管是零样本的,但在视频理解方面取得了显著优于监督基线的结果。此外,为了缓解故事理解基准的缺乏,我们公开发布了计算社会科学中一项关键任务——说服策略识别——的首个数据集。

英文摘要:

Multimedia content, such as advertisements and story videos, exhibit a rich blend of creativity and multiple modalities. They incorporate elements like text, visuals, audio, and storytelling techniques, employing devices like emotions, symbolism, and slogans to convey meaning. There is a dearth of large annotated training datasets in the multimedia domain hindering the development of supervised learning models with satisfactory performance for real-world applications. On the other hand, the rise of large language models (LLMs) has witnessed remarkable zero-shot performance in various natural language processing (NLP) tasks, such as emotion classification, question-answering, and topic classification. To leverage such advanced techniques to bridge this performance gap in multimedia understanding, we propose verbalizing long videos to generate their descriptions in natural language, followed by performing video-understanding tasks on the generated story as opposed to the original video. Through extensive experiments on fifteen video-understanding tasks, we demonstrate that our method, despite being zero-shot, achieves significantly better results than supervised baselines for video understanding. Furthermore, to alleviate a lack of story understanding benchmarks, we publicly release the first dataset on a crucial task in computational social science on persuasion strategy identification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑