arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.12419cs.CV
收藏

利用专业标注视频进行多模态新闻理解(ReutersViLNews)

Multi-modal News Understanding with Professionally Labelled Videos (ReutersViLNews)

Vector Institute(向量研究所);University of British Columbia(不列颠哥伦比亚大学);Western University(西安大略大学);York University(约克大学);Reuters News Agency(路透通讯社);CIFAR(加拿大高等研究院);Toyota Research Institute(丰田研究所)

Shih-Han Chou, Matthew Kowal, Yasmin Niknam, Diana Moyano, Shayaan Mehdi, Richard Pito, Cheng Zhang, Ian Knopke, Sedef Akinli Kocak, Leonid Sigal, Yalda Mohsenzadeh

AI总结:

本文提出由专业人员标注的长篇新闻视频-语言数据集 ReutersViLNews,统计分析其类别与多层标注,并在四类视频-语言任务上评测现有方法,揭示新闻视频高层语义理解仍是重大挑战。

摘要:

尽管视频-语言理解领域已经取得进展,但当前最先进算法在理解高度抽象的视频(如新闻类视频)方面能力仍然有限。相比之下,人类能够轻松融合来自视频和语言的信息,推断出超出像素中视觉可见内容的信息。其中一个例子是观看新闻报道:事件背景在理解报道时所起的作用可能与事件本身同样重要。为了在算法中设计这种能力,我们对路透社(Reuters News Agency)收集的内部数据集进行了大规模分析,该数据集名为 Reuters Video-Language News(ReutersViLNews),聚焦于高层级视频-语言理解,并重点关注长篇新闻。ReutersViLNews 数据集包含由新闻行业专业人员多年来收集并标注的长篇新闻视频,涵盖来自世界各地的重要新闻报道。每个视频涉及一个单一报道,包含真实事件的行动镜头、对与事件相关人员的采访、附近地区的素材等。ReutersViLNews 数据集包含七个主题类别的视频:灾难、财经、娱乐、健康、政治、体育和其他(miscellaneous),并带有从高层级到低层级的标注,包括标题字幕、视觉视频描述、高层级报道描述、关键词和位置。我们首先分析 ReutersViLNews 的数据集统计信息,并与以往数据集进行比较。随后,我们针对四种不同的视频-语言任务对最先进方法进行基准测试。结果表明,新闻类视频对当前视频-语言理解算法构成了重大挑战;最后,我们为设计能够解决 ReutersViLNews 数据集问题的方法提供了未来方向。

英文摘要:

While progress has been made in the domain of video-language understanding, current state-of-the-art algorithms are still limited in their ability to understand videos at high levels of abstraction, such as news-oriented videos. Alternatively, humans easily amalgamate information from video and language to infer information beyond what is visually observable in the pixels. An example of this is watching a news story, where the context of the event can play as big of a role in understanding the story as the event itself. Towards a solution for designing this ability in algorithms, we present a large-scale analysis on an in-house dataset collected by the Reuters News Agency, called Reuters Video-Language News (ReutersViLNews) dataset which focuses on high-level video-language understanding with an emphasis on long-form news. The ReutersViLNews Dataset consists of long-form news videos collected and labeled by news industry professionals over several years and contains prominent news reporting from around the world. Each video involves a single story and contains action shots of the actual event, interviews with people associated with the event, footage from nearby areas, and more. ReutersViLNews dataset contains videos from seven subject categories: disaster, finance, entertainment, health, politics, sports, and miscellaneous with annotations from high-level to low-level, title caption, visual video description, high-level story description, keywords, and location. We first present an analysis of the dataset statistics of ReutersViLNews compared to previous datasets. Then we benchmark state-of-the-art approaches for four different video-language tasks. The results suggest that news-oriented videos are a substantial challenge for current video-language understanding algorithms and we conclude by providing future directions in designing approaches to solve the ReutersViLNews dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑