We present a domain- and user-preference-agnostic approach to detect highlightable excerpts from human-centric videos. Our method works on the graph-based representation of multiple observable human-centric modalities in the videos, such as poses and faces. We use an autoencoder network equipped with spatial-temporal graph convolutions to detect human activities and interactions based on these modalities. We train our network to map the activity- and interaction-based latent structural representations of the different modalities to per-frame highlight scores based on the representativeness of the frames. We use these scores to compute which frames to highlight and stitch contiguous frames to produce the excerpts. We train our network on the large-scale AVA-Kinetics action dataset and evaluate it on four benchmark video highlight datasets: DSH, TVSum, PHD2, and SumMe. We observe a 4-12% improvement in the mean average precision of matching the human-annotated highlights over state-of-the-art methods in these datasets, without requiring any user-provided preferences or dataset-specific fine-tuning.
Important ethical concerns arising from computer vision datasets of people have been receiving significant attention, and a number of datasets have been withdrawn as a result. To meet the academic need for people-centric datasets, we propose an analytical framework to guide ethical evaluation of existing datasets and to serve future dataset creators in avoiding missteps. Our work is informed by a review and analysis of prior works and highlights where such ethical challenges arise.
Bioinformatics Institute, A*STAR（A*STAR生物信息研究所）；School of Computing, National University of Singapore（新加坡国立大学计算学院）
Detecting hand actions from ego-centric depth sequences is a practically challenging problem, owing mostly to the complex and dexterous nature of hand articulations as well as non-stationary camera motion. We address this problem via a Hough transform based approach coupled with a discriminatively learned error-correcting component to tackle the well known issue of incorrect votes from the Hough transform. In this framework, local parts vote collectively for the start $\&$ end positions of each action over time. We also construct an in-house annotated dataset of 300 long videos, containing 3,1…
UC Berkeley（加州大学伯克利分校）；ICSI（国际计算机科学研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarland University（萨尔大学）；Technische Universität München（慕尼黑工业大学）；Stanford University（斯坦福大学）；SPIEGEL-Verlag（明镜出版社）
Activity recognition has shown impressive progress in recent years. However, the challenges of detecting fine-grained activities and understanding how they are combined into composite activities have been largely overlooked. In this work we approach both tasks and present a dataset which provides detailed annotations to address them. The first challenge is to detect fine-grained activities, which are defined by low inter-class variability and are typically characterized by fine-grained body motions. We explore how human pose and hands can help to approach this challenge by comparing two pose-b…
Facebook AI Research (FAIR)（Facebook人工智能研究院（FAIR））
To understand the visual world, a machine must not only recognize individual object instances but also how they interact. Humans are often at the center of such interactions and detecting human-object interactions is an important practical and scientific problem. In this paper, we address the task of detecting <human, verb, object> triplets in challenging everyday photos. We propose a novel model that is driven by a human-centric approach. Our hypothesis is that the appearance of a person -- their pose, clothing, action -- is a powerful cue for localizing the objects they are interacting with.…
Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning
National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHIGHLIGHTS) dataset. It consists of over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos. Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant momen…
Recent years have witnessed rapid progress in detecting and recognizing individual object instances. To understand the situation in a scene, however, computers need to recognize how humans interact with surrounding objects. In this paper, we tackle the challenging task of detecting human-object interactions (HOI). Our core idea is that the appearance of a person or an object instance contains informative cues on which relevant parts of an image to attend to for facilitating interaction prediction. To exploit these cues, we propose an instance-centric attention module that learns to dynamically…
Detecting Children with Autism Spectrum Disorder based on Script-Centric Behavior Understanding with Emotional Enhancement
School of Computer Science, Wuhan University（武汉大学计算机学院）；Digital Innovation Research Center, Duke Kunshan University（杜克昆山大学数字创新研究中心）；School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息学院）；Child Development and Behavior Center, Third Affiliated Hospital of Sun Yat-sen University（中山大学第三附属医院儿童发展与行为中心）
Polytechnique Montréal（蒙特利尔高等理工学院）；Institute of Biomedical Engineering, Polytechnique Montréal（蒙特利尔高等理工学院生物医学工程研究所）
We propose a novel unsupervised approach based on a two-stage object-centric adversarial framework that only needs object regions for detecting frame-level local anomalies in videos. The first stage consists in learning the correspondence between the current appearance and past gradient images of objects in scenes deemed normal, allowing us to either generate the past gradient from current appearance or the reverse. The second stage extracts the partial reconstruction errors between real and generated images (appearance and past gradient) with normal object behaviour, and trains a discriminato…
Esports has gained global popularity in recent years and several companies have started offering live streaming videos of esports games and events. This creates opportunities to develop large scale video understanding systems for new product features and services. We present a technique for detecting highlights from live streaming videos of esports game matches. Most video games use pronounced visual effects to emphasize highlight moments; we use CNNs to learn convolution filters of those visual effects for detecting highlights. We propose a cascaded prediction approach that allows us to deal …
The research and data science community has been fascinated with the development of automatic systems for the detection of key events in a video. Special attention in this field is given to sports video analytics which could help in identifying key events during a match and help in preparing a strategy for the games going forward. For this paper, we have chosen Football (soccer) as a sport where we would want to create highlights for a given match video, through a computer vision model that aims to identify important events in a Soccer match to create highlights of the match. We built the mode…
UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learning
FPT Software AI Center（FPT软件AI中心）；Aalborg University（奥尔堡大学）；Pioneer Centre for AI（先锋人工智能中心）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）；AICV Lab, University of Arkansas（阿肯色大学AICV实验室）
