Video description involves the generation of the natural language description of actions, events, and objects in the video. There are various applications of video description by filling the gap between languages and vision for visually impaired people, generating automatic title suggestion based on content, browsing of the video based on the content and video-guided machine translation [86] etc.In the past decade, several works had been done in this field in terms of approaches/methods for video description, evaluation metrics,and datasets. For analyzing the progress in the video description task, a comprehensive survey is needed that covers all the phases of video description approaches with a special focus on recent deep learning approaches. In this work, we report a comprehensive survey on the phases of video description approaches, the dataset for video description, evaluation metrics, open competitions for motivating the research on the video description, open challenges in this field, and future research directions. In this survey, we cover the state-of-the-art approaches proposed for each and every dataset with their pros and cons. For the growth of this research domain,the availability of numerous benchmark dataset is a basic need. Further, we categorize all the dataset into two classes: open domain dataset and domain-specific dataset. From our survey, we observe that the work in this field is in fast-paced development since the task of video description falls in the intersection of computer vision and natural language processing. But still, the work in the video description is far from saturation stage due to various challenges like the redundancy due to similar frames which affect the quality of visual features, the availability of dataset containing more diverse content and availability of an effective evaluation metric.
In this paper, we introduce VCSL (Video Copy Segment Localization), a new comprehensive segment-level annotated video copy dataset. Compared with existing copy detection datasets restricted by either video-level annotation or small-scale, VCSL not only has two orders of magnitude more segment-level labelled data, with 160k realistic video copy pairs containing more than 280k localized copied segment pairs, but also covers a variety of video categories and a wide range of video duration. All the copied segments inside each collected video pair are manually extracted and accompanied by precisely…
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
State Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室）；School of Intelligence Science and Technology（智能科学与技术学院）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；XMU（厦门大学）；HKU（香港大学）；PKU（北京大学）；CUHK（香港中文大学）；ECNU（华东师范大学）；CASIA（中国科学院自动化研究所）
There are substantial instructional videos on the Internet, which enables us to acquire knowledge for completing various tasks. However, most existing datasets for instructional video analysis have the limitations in diversity and scale,which makes them far from many real-world applications where more diverse activities occur. Moreover, it still remains a great challenge to organize and harness such data. To address these problems, we introduce a large-scale dataset called "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains …
Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thought
Institute of Automation, CAS（中国科学院自动化研究所）；School of Artifcial Intelligence, UCAS（中国科学技术大学人工智能学院）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）；Shenzhen International GraduateSchool,Tsinghua University（深圳国际研究生院，清华大学）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，北京大学计算机学院）
Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学清华深圳国际研究生院）
Thanks to the substantial and explosively inscreased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, the most existing datasets in this area have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, we present a large-scale dataset named as "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchic…
SceneScribe-1M: A Large-Scale Video Dataset with Comprehensive Geometric and Semantic Annotations
Shanghai Jiao Tong University（上海交通大学）；Ant Group（蚂蚁集团）；Visual Geometry Group, University of Oxford（牛津大学视觉几何组）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo（宁波数字孪生研究院，东部技术研究院，宁波）；Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin（浙江工业智能与数字孪生重点实验室）
Imperial College London（帝国理工学院）；Hacettepe University（哈杰特佩大学）；Koç University（科奇大学）
Automatic generation of video descriptions in natural language, also called video captioning, aims to understand the visual content of the video and produce a natural language sentence depicting the objects and actions in the scene. This challenging integrated vision and language problem, however, has been predominantly addressed for English. The lack of data and the linguistic properties of other languages limit the success of existing approaches for such languages. In this paper we target Turkish, a morphologically rich and agglutinative language that has very different properties compared t…
ActivityForensics: A Comprehensive Benchmark for Localizing Manipulated Activity in Videos
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Computing and Artificial Intelligence, Jiangxi University of Finance and Economics（江西财经大学计算机与人工智能学院）；Jiangxi Provincial Key Laboratory of Multimedia Intelligent Processing（江西省多媒体智能处理重点实验室）；Faculty of Engineering, Shenzhen MSU-BIT University（深圳北理莫斯科大学工程系）；VinUniversity
SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Models
Department of Computer Science, Beijing University of Technology（北京理工大学计算机科学系）；Inspur Electronic Information Industry Co., Ltd（Inspur电子信息产业有限公司）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；JD Explore Academy（京东探索研究院）
DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning
Georgia Institute of Technology（佐治亚理工学院）；Rice University（Rice大学）；Johns Hopkins University（约翰霍普金斯大学）；University of California, Irvine（加州大学 Irvine分校）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
University of Maryland, College Park（马里兰大学帕克分校）；Microsoft Research Asia（微软亚洲研究院）；Fudan University（复旦大学）
Accelerated by the tremendous increase in Internet bandwidth and storage space, video data has been generated, published and spread explosively, becoming an indispensable part of today's big data. In this paper, we focus on reviewing two lines of research aiming to stimulate the comprehension of videos with deep learning: video classification and video captioning. While video classification concentrates on automatically labeling video clips based on their semantic contents like human actions or complex events, video captioning attempts to generate a complete and natural sentence, enriching the…
SENAI CIMATEC University Center（SENAI CIMATEC大学中心）
Estimating heart rate is important for monitoring users in various situations. Estimates based on facial videos are increasingly being researched because it makes it possible to monitor cardiac information in a non-invasive way and because the devices are simpler, requiring only cameras that capture the user's face. From these videos of the user's face, machine learning is able to estimate heart rate. This study investigates the benefits and challenges of using machine learning models to estimate heart rate from facial videos, through patents, datasets, and articles review. We searched Derwent…
The University of Tokyo（东京大学）；Australian National University（澳大利亚国立大学）
Despite the recent advances in opinion mining for written reviews, few works have tackled the problem on other sources of reviews. In light of this issue, we propose a multi-modal approach for mining fine-grained opinions from video reviews that is able to determine the aspects of the item under review that are being discussed and the sentiment orientation towards them. Our approach works at the sentence level without the need for time annotations and uses features derived from the audio, video and language transcriptions of its contents. We evaluate our approach on two datasets and show that …
Pontifical Catholic University of Minas Gerais（米纳斯吉拉斯天主教大学）；University of Chile（智利大学）
Research in the Vision and Language area encompasses challenging topics that seek to connect visual and textual information. When the visual information is related to videos, this takes us into Video-Text Research, which includes several challenging tasks such as video question answering, video summarization with natural language, and video-to-text and text-to-video conversion. This paper reviews the video-to-text problem, in which the goal is to associate an input video with its textual description. This association can be mainly made by retrieving the most relevant descriptions from a corpus…
This review article surveys the current progresses made toward video-based anomaly detection. We address the most fundamental aspect for video anomaly detection, that is, video feature representation. Much research works have been done in finding the right representation to perform anomaly detection in video streams accurately with an acceptable false alarm rate. However, this is very challenging due to large variations in environment and human movement, and high space-time complexity due to huge dimensionality of video data. The weakly supervised nature of deep learning algorithms can help in…
Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
Bytedance Inc.（字节跳动公司）；AAII, UTS（悉尼科技大学人工智能研究院）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University of Science and Technology of China（中国科学技术大学）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织数据61部门）
Shandong University of Finance and Economics（山东财经大学）；Shandong University（山东大学）；Shandong Normal University（山东师范大学）
Content-based near-duplicate video detection (NDVD) is essential for effective search and retrieval, and robust video fingerprinting is a good solution for NDVD. Most existing video fingerprinting methods use a single feature or concatenating different features to generate video fingerprints, and show a good performance under single-mode modifications such as noise addition and blurring. However, when they suffer combined modifications, the performance is degraded to a certain extent because such features cannot characterize the video content completely. By contrast, the assistance and consens…
State Key Laboratory of Media Convergence Production Technology and Systems（媒体融合生产技术与系统国家重点实验室）；Xinhua Zhiyun Technology Co., Ltd.（新华智云科技有限公司）
Soccer videos can serve as a perfect research object for video understanding because soccer games are played under well-defined rules while complex and intriguing enough for researchers to study. In this paper, we propose a new soccer video database named SoccerDB, comprising 171,191 video segments from 346 high-quality soccer games. The database contains 702,096 bounding boxes, 37,709 essential event labels with time boundary and 17,115 highlight annotations for object detection, action recognition, temporal action localization, and highlight detection tasks. To our knowledge, it is the large…
Xinhua Zhiyun Technology Co., Ltd.（新华智云科技有限公司）；Columbia University in the City of New York（纽约哥伦比亚大学）
Video summarization aims to extract keyframes/shots from a long video. Previous methods mainly take diversity and representativeness of generated summaries as prior knowledge in algorithm design. In this paper, we formulate video summarization as a content-based recommender problem, which should distill the most useful content from a long video for users who suffer from information overload. A scalable deep neural network is proposed on predicting if one video segment is a useful segment for users by explicitly modelling both segment and video. Moreover, we accomplish scene and action recognit…
Indian Institute of Technology Roorkee（印度理工学院鲁尔基分校）；Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）
Video skimming, also known as dynamic video summarization, generates a temporally abridged version of a given video. Skimming can be achieved by identifying significant components either in uni-modal or multi-modal features extracted from the video. Being dynamic in nature, video skimming, through temporal connectivity, allows better understanding of the video from its summary. Having this obvious advantage, recently, video skimming has drawn the focus of many researchers benefiting from the easy availability of the required computing resources. In this paper, we provide a comprehensive survey…
School of Engineering and Applied Science, Ahmedabad University（艾哈迈达巴德大学工程与应用科学学院）
Video Question Answering (VQA) is a recent emerging challenging task in the field of Computer Vision. Several visual information retrieval techniques like Video Captioning/Description and Video-guided Machine Translation have preceded the task of VQA. VQA helps to retrieve temporal and spatial information from the video scenes and interpret it. In this survey, we review a number of methods and datasets for the task of VQA. To the best of our knowledge, no previous survey has been conducted for the VQA task.
VABench: A Comprehensive Benchmark for Audio-Video Generation
Peking University（北京大学）；Zhongguancun Academy（中关村学院）；Beijing Key Laboratory of Data Intelligence (Peking University)（北京市数据智能重点实验室（北京大学））；Ant Group（蚂蚁集团）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Huazhong University of Science and Technology（华中科技大学）
This article describes a comprehensive system for surveillance and monitoring applications. The development of an efficient real time video motion detection system is motivated by their potential for deployment in the areas where security is the main concern. The paper presents a platform for real time video motion detection and subsequent generation of an alarm condition as set by the parameters of the control system. The prototype consists of a mobile platform mounted with RF camera which provides continuous feedback of the environment. The received visual information is then analyzed by use…
OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts
Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；X-LANCE Lab, Shanghai Jiao Tong University（上海交通大学X-LANCE实验室）
Dalian University of Technology（大连理工大学）；National University of Defense Technology（国防科技大学）
Video-Text Retrieval (VTR) aims to search for the most relevant video related to the semantics in a given sentence, and vice versa. In general, this retrieval task is composed of four successive steps: video and textual feature representation extraction, feature embedding and matching, and objective functions. In the last, a list of samples retrieved from the dataset is ranked based on their matching similarities to the query. In recent years, significant and flourishing progress has been achieved by deep learning techniques, however, VTR is still a challenging task due to the problems like ho…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；ByteDance Inc.（字节跳动公司）；University of Oregon（俄勒冈大学）
Video instance segmentation is a complex task in which we need to detect, segment, and track each object for any given video. Previous approaches only utilize single-frame features for the detection, segmentation, and tracking of objects and they suffer in the video scenario due to several distinct challenges such as motion blur and drastic appearance change. To eliminate ambiguities introduced by only using single-frame features, we propose a novel comprehensive feature aggregation approach (CompFeat) to refine features at both frame-level and object-level with temporal and spatial context in…
Text data present in multimedia contain useful information for automatic annotation, indexing. Extracted information used for recognition of the overlay or scene text from a given video or image. The Extracted text can be used for retrieving the videos and images. In this paper, firstly, we are discussed the different techniques for text extraction from images and videos. Secondly, we are reviewed the techniques for indexing and retrieval of image and videos by using extracted text.
VBench: Comprehensive Benchmark Suite for Video Generative Models
Nanyang Technological University（南洋理工大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Nanjing University（南京大学）；The Chinese University of Hong Kong（香港中文大学）
University of New South Wales（新南威尔士大学）；Princess Nourah bint Abdulrahman University（诺拉·本特·阿卜杜勒拉赫曼公主大学）
With the increasing interest in the content creation field in multiple sectors such as media, education, and entertainment, there is an increasing trend in the papers that uses AI algorithms to generate content such as images, videos, audio, and text. Generative Adversarial Networks (GANs) in one of the promising models that synthesizes data samples that are similar to real data samples. While the variations of GANs models, in general, have been covered to some extent in several survey papers, to the best of our knowledge, this is among the first survey papers that reviews the state-of-the-art…
Video capsule endoscopy (VCE) is used widely nowadays for visualizing the gastrointestinal (GI) tract. Capsule endoscopy exams are prescribed usually as an additional monitoring mechanism and can help in identifying polyps, bleeding, etc. To analyze the large scale video data produced by VCE exams automatic image processing, computer vision, and learning algorithms are required. Recently, automatic polyp detection algorithms have been proposed with various degrees of success. Though polyp detection in colonoscopy and other traditional endoscopy procedure based images is becoming a mature field…
Iran University of Science & Technology（伊朗科技大学）；University of Bonn（波恩大学）；Fundamental Sciences (IPM)（基础科学研究所（IPM））
This paper gives an overview on semantic segmentation consists of an explanation of this field, it's status and relation with other vision fundamental tasks, different datasets and common evaluation parameters that have been used by researchers. This survey also includes an overall review on a variety of recent approaches (RDF, MRF, CRF, etc.) and their advantages and challenges and shows the superiority of CNN-based semantic segmentation systems on CamVid and NYUDv2 datasets. In addition, some areas that is ideal for future work have mentioned.
Centro de Investigación en Ciencias de Información Geoespacial AC（地理空间信息科学研究中心（AC））；Consejo Nacional de Ciencia y Tecnología (CONACyT)（国家科学技术委员会（CONACyT））
Video captioning is the process of describing the content of a sequence of images capturing its semantic relationships and meanings. Dealing with this task with a single image is arduous, not to mention how difficult it is for a video (or images sequence). The amount and relevance of the applications of video captioning are vast, mainly to deal with a significant amount of video recordings in video surveillance, or assisting people visually impaired, to mention a few. To analyze where the efforts of our community to solve the video captioning task are, as well as what route could be better to …
