We introduce QuerYD, a new large-scale dataset for retrieval and event localisation in video. A unique feature of our dataset is the availability of two audio tracks for each video: the original audio, and a high-quality spoken description of the visual content. The dataset is based on YouDescribe, a volunteer project that assists visually-impaired people by attaching voiced narrations to existing YouTube videos. This ever-growing collection of videos contains highly detailed, temporally aligned audio and text annotations. The content descriptions are more relevant than dialogue, and more detailed than previous description attempts, which can be observed to contain many superficial or uninformative descriptions. To demonstrate the utility of the QuerYD dataset, we show that it can be used to train and benchmark strong models for retrieval and event localisation. Data, code and models are made publicly available, and we hope that QuerYD inspires further research on video understanding with written and spoken natural language.
Videos are more well-organized curated data sources for visual concept learning than images. Unlike the 2-dimensional images which only involve the spatial information, the additional temporal dimension bridges and synchronizes multiple modalities. However, in most video detection benchmarks, these additional modalities are not fully utilized. For example, EPIC Kitchens is the largest dataset in first-person (egocentric) vision, yet it still relies on crowdsourced information to refine the action boundaries to provide instance-level action annotations. We explored how to eliminate the expensiv…
HowToCaption: Prompting LLMs to Transform Video Annotations at Scale
Goethe University Frankfurt（法兰克福大学）；SIC；University of Bonn（波恩大学）；University of Oxford（牛津大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Saarland University（萨尔大学）
Université de Montréal（蒙特利尔大学）；École Polytechnique de Montéral（蒙特利尔综合理工学院）；Université de Sherbrooke（舍布鲁克大学）
In this work, we introduce a dataset of video annotated with high quality natural language phrases describing the visual content in a given segment of time. Our dataset is based on the Descriptive Video Service (DVS) that is now encoded on many digital media products such as DVDs. DVS is an audio narration describing the visual elements and actions in a movie for the visually impaired. It is temporally aligned with the movie and mixed with the original movie soundtrack. We describe an automatic DVS segmentation and alignment method for movies, that enables us to scale up the collection of a DV…
We present LumièreNet, a simple, modular, and completely deep-learning based architecture that synthesizes, high quality, full-pose headshot lecture videos from instructor's new audio narration of any length. Unlike prior works, LumièreNet is entirely composed of trainable neural network modules to learn mapping functions from the audio to video through (intermediate) estimated pose-based compact and abstract latent codes. Our video demos are available at [22] and [23].
Contextual AD Narration with Interleaved Multimodal Sequence
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；ESAT, KU Leuven（鲁汶大学ESAT学院）；Ant Group（蚂蚁集团）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Over the last decade, online lecture videos have become increasingly popular and have experienced a meteoric rise during the pandemic. However, video-language research has primarily focused on instructional videos or movies, and tools to help students navigate the growing online lectures are lacking. Our first contribution is to facilitate research in the educational domain, by introducing AVLectures, a large-scale dataset consisting of 86 courses with over 2,350 lectures covering various STEM subjects. Each course contains video lectures, transcripts, OCR outputs for lecture frames, and optio…
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China（中国科学院人工智能安全重点实验室，计算技术研究所，中国科学院，北京，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）；Tsinghua University, Beijing, China（清华大学，北京，中国）
