In this paper we propose a new evaluation challenge and direction in the area of High-level Video Understanding. The challenge we are proposing is designed to test automatic video analysis and understanding, and how accurately systems can comprehend a movie in terms of actors, entities, events and their relationship to each other. A pilot High-Level Video Understanding (HLVU) dataset of open source movies were collected for human assessors to build a knowledge graph representing each of them. A set of queries will be derived from the knowledge graph to test systems on retrieving relationships among actors, as well as reasoning and retrieving non-visual concepts. The objective is to benchmark if a computer system can "understand" non-explicit but obvious relationships the same way humans do when they watch the same movies. This is long-standing problem that is being addressed in the text domain and this project moves similar research to the video domain. Work of this nature is foundational to future video analytics and video understanding technologies. This work can be of interest to streaming services and broadcasters hoping to provide more intuitive ways for their customers to interact with and consume video content.
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
University of Washington（华盛顿大学）；Microsoft Research Asia（微软亚洲研究院）；Hong Kong University of Science and Technology (GZ)（香港科技大学（广州））；Donghua University（东华大学）；Zhejiang University（浙江大学）
Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understanding
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；ICSI（国际计算机科学研究所）；UC Berkeley（加州大学伯克利分校）；Disney Research（迪士尼研究中心）；École Polytechnique de Montréal（蒙特利尔理工学院）；Université de Sherbrooke（舍布鲁克大学）；Twitter（推特）；Université de Montréal（蒙特利尔大学）
Audio Description (AD) provides linguistic descriptions of movies and allows visually impaired people to follow a movie along with their peers. Such descriptions are by design mainly visual and thus naturally form an interesting data source for computer vision and computational linguistics. In this work we propose a novel dataset which contains transcribed ADs, which are temporally aligned to full length movies. In addition we also collected and aligned movie scripts used in prior work and compare the two sources of descriptions. In total the Large Scale Movie Description Challenge (LSMDC) con…
MovieChat+: Question-aware Sparse Memory for Long Video Question Answering
Zhejiang University-University of Illinois at Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校学院）；Zhejiang University（浙江大学）；the University of Washington（华盛顿大学）；Hong Kong University of Science and Technology (GuangZhou)（香港科技大学（广州））；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
University of Southern California（南加州大学）；University of Pennsylvania（宾夕法尼亚大学）；Allen Institute for AI（艾伦人工智能研究所）
We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling. We represent videos as a set of related events, wherein each event consists of a verb and multiple entities that fulfill various roles relevant to that event. To study the challenging task of semantic role labeling in videos or VidSRL, we introduce the VidSitu benchmark, a large-scale video understanding data source with $29K$ $10$-second movie clips richly annotated with a verb and semantic-roles every $2$ seconds. Entities are co-referenced across events within…
The Chinese University of Hong Kong（香港中文大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；SenseTime（商汤科技）
Scene, as the crucial unit of storytelling in movies, contains complex activities of actors and their interactions in a physical environment. Identifying the composition of scenes serves as a critical step towards semantic understanding of movies. This is very challenging -- compared to the videos studied in conventional vision problems, e.g. action recognition, as scenes in movies usually contain much richer temporal structures and more complex semantic information. Towards this goal, we scale up the scene segmentation task by building a large-scale video dataset MovieScenes, which contains 2…
National Taiwan University（国立台湾大学）；University of Toronto（多伦多大学）；Microsoft（微软）
The human ability of deep cognitive skills are crucial for the development of various real-world applications that process diverse and abundant user generated input. While recent progress of deep learning and natural language processing have enabled learning system to reach human performance on some benchmarks requiring shallow semantics, such human ability still remains challenging for even modern contextual embedding models, as pointed out by many recent studies. Existing machine comprehension datasets assume sentence-level input, lack of casual or motivational inferences, or could be answer…
Most modern video recognition models are designed to operate on short video clips (e.g., 5-10s in length). Thus, it is challenging to apply such models to long movie understanding tasks, which typically require sophisticated long-range temporal reasoning. The recently introduced video transformers partially address this issue by using long-range temporal self-attention. However, due to the quadratic cost of self-attention, such models are often costly and impractical to use. Instead, we propose ViS4mer, an efficient long-range video model that combines the strengths of self-attention and the r…
We propose DuoRC, a novel dataset for Reading Comprehension (RC) that motivates several new challenges for neural approaches in language understanding beyond those offered by existing RC datasets. DuoRC contains 186,089 unique question-answer pairs created from a collection of 7680 pairs of movie plots where each pair in the collection reflects two versions of the same movie - one from Wikipedia and the other from IMDb - written by two different authors. We asked crowdsourced workers to create questions from one version of the plot and a different set of workers to extract or synthesize answer…
StoryVideoQA: Scaling Deep Video Understanding with a Large-Scale, Multi-Genre and Auto-Generated Dataset
School of Computer Science, Wuhan University（武汉大学计算机学院）；National Engineering Research Center for Multimedia Software（多媒体软件国家工程研究中心）；Hubei Key Laboratory of Multimedia and Network Communication Engineering（湖北省多媒体与网络通信工程重点实验室）
To understand movies, humans constantly reason over the dialogues and actions shown in specific scenes and relate them to the overall storyline already seen. Inspired by this behaviour, we design ROLL, a model for knowledge-based video story question answering that leverages three crucial aspects of movie understanding: dialog comprehension, scene reasoning, and storyline recalling. In ROLL, each of these tasks is in charge of extracting rich and diverse information by 1) processing scene dialogues, 2) generating unsupervised video scene descriptions, and 3) obtaining external knowledge in a w…
MLVU: Benchmarking Multi-task Long Video Understanding
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Jiao Tong University（上海交通大学）；Zhejiang University（浙江大学）；Peking University（北京大学）
National Taiwan University（国立台湾大学）；Microsoft（微软）；Mobile Drive Technology（摩比驱动科技）
Understanding and comprehending video content is crucial for many real-world applications such as search and recommendation systems. While recent progress of deep learning has boosted performance on various tasks using visual cues, deep cognition to reason intentions, motivation, or causality remains challenging. Existing datasets that aim to examine video reasoning capability focus on visual signals such as actions, objects, relations, or could be answered utilizing text bias. Observing this, we propose a novel task, along with a new dataset: Trope Understanding in Movies and Animations (TrUM…
Understanding web instructional videos is an essential branch of video understanding in two aspects. First, most existing video methods focus on short-term actions for a-few-second-long video clips; these methods are not directly applicable to long videos. Second, unlike unconstrained long videos, e.g., movies, instructional videos are more structured in that they have step-by-step procedure constraining the understanding task. In this paper, we study reasoning on instructional videos via question-answering (QA). Surprisingly, it has not been an emphasis in the video community despite its rich…
Interpersonal language style shifting in dialogues is an interesting and almost instinctive ability of human. Understanding interpersonal relationship from language content is also a crucial step toward further understanding dialogues. Previous work mainly focuses on relation extraction between named entities in texts. In this paper, we propose the task of relation classification of interlocutors based on their dialogues. We crawled movie scripts from IMSDb, and annotated the relation labels for each session according to 13 pre-defined relationships. The annotated dataset DDRel consists of 630…
DrVideo: Document Retrieval Based Long Video Understanding
College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）；Data Science & AI Department, Faculty of IT, Monash University（莫纳什大学信息技术学院数据科学与人工智能系）
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. Next, we introduce a new datase…
LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering
OpenGVLab, Shanghai AI Laboratory, China（OpenGVLab，上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Honor Device Co.,Ltd（荣耀设备有限公司）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）；Nanjing University（南京大学）
Generic Boundary Detection (GBD) aims at locating the general boundaries that divide videos into semantically coherent and taxonomy-free units, and could serve as an important pre-processing step for long-form video understanding. Previous works often separately handle these different types of generic boundaries with specific designs of deep networks from simple CNN to LSTM. Instead, in this paper, we present Temporal Perceiver, a general architecture with Transformer, offering a unified solution to the detection of arbitrary generic boundaries, ranging from shot-level, event-level, to scene-l…
The Allen Institute for AI（艾伦人工智能研究所）；Carnegie Mellon University（卡内基梅隆大学）；Inria（法国国家信息与自动化研究所）；University of Washington（华盛顿大学）
Computer vision has a great potential to help our daily lives by searching for lost keys, watering flowers or reminding us to take a pill. To succeed with such tasks, computer vision methods need to be trained from real and diverse examples of our daily dynamic scenes. While most of such scenes are not particularly exciting, they typically do not appear on YouTube, in movies or TV broadcasts. So how do we collect sufficiently many diverse but boring samples representing our lives? We propose a novel Hollywood in Homes approach to collect such data. Instead of shooting videos in the lab, we ens…
Goldfish: Vision-Language Understanding of Arbitrarily Long Videos
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Harvard University（哈佛大学）；The Swiss AI Lab IDSIA, USI, SUPSI（瑞士IDSIA人工智能实验室（瑞士应用科学大学USI、瑞士南方应用科学大学SUPSI））
Perceive, Reflect and Understand Long Video: Progressive Multi-Granular Clue Exploration with Interactive Agents
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Department of Computer Science and Engineering, Hong Kong University of Science and Technology（香港理工大学计算机科学与工程系）；College of Computer and Information, Hohai University（河海大学计算机与信息学院）
High-level understanding of stories in video such as movies and TV shows from raw data is extremely challenging. Modern video question answering (VideoQA) systems often use additional human-made sources like plot synopses, scripts, video descriptions or knowledge bases. In this work, we present a new approach to understand the whole story without such external sources. The secret lies in the dialog: unlike any prior work, we treat dialog as a noisy source to be converted into text description via dialog summarization, much like recent methods treat video. The input of each modality is encoded …
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science, Peking University（北京大学计算机科学学院）；Center for Data Science, Peking University（北京大学数据科学中心）
