With recent advancements in video backbone architectures, combined with the remarkable achievements of large language models (LLMs), the analysis of long-form videos spanning tens of minutes has become both feasible and increasingly prevalent. However, the inherently redundant nature of video sequences poses significant challenges for contemporary state-of-the-art models. These challenges stem from two primary aspects: 1) efficiently incorporating a larger number of frames within memory constraints, and 2) extracting discriminative information from the vast volume of input data. In this paper, we introduce a novel end-to-end schema for long-form video understanding, which includes an information-density-based adaptive video sampler (AVS) and an autoencoder-based spatiotemporal video compressor (SVC) integrated with a multimodal large language model (MLLM). Our proposed system offers two major advantages: it adaptively and effectively captures essential information from video sequences of varying durations, and it achieves high compression rates while preserving crucial discriminative information. The proposed framework demonstrates promising performance across various benchmarks, excelling in both long-form video understanding tasks and standard video understanding benchmarks. These results underscore the versatility and efficacy of our approach, particularly in managing the complexities of prolonged video sequences.
Edith Cowan University（伊迪丝·考恩大学）；The University of Western Australia（西澳大学）
Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNNs can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation …
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences（中国科学院重庆绿色智能技术研究院）；Chongqing School, University of Chinese Academy of Sciences（中国科学院大学重庆学院）；University College London（伦敦大学学院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人学会）
Video question answering (VideoQA) is challenging given its multimodal combination of visual understanding and natural language understanding. While existing approaches seldom leverage the appearance-motion information in the video at multiple temporal scales, the interaction between the question and the visual information for textual semantics extraction is frequently ignored. Targeting these issues, this paper proposes a novel Temporal Pyramid Transformer (TPT) model with multimodal interaction for VideoQA. The TPT model comprises two modules, namely Question-specific Transformer (QT) and Vi…
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室 OpenGVLab）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Monash University（莫纳什大学）；Nanjing University（南京大学）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）
Recognizing and localizing events in videos is a fundamental task for video understanding. Since events may occur in auditory and visual modalities, multimodal detailed perception is essential for complete scene comprehension. Most previous works attempted to analyze videos from a holistic perspective. However, they do not consider semantic information at multiple scales, which makes the model difficult to localize events in different lengths. In this paper, we present a Multimodal Pyramid Attentional Network (\textbf{MM-Pyramid}) for event localization. Specifically, we first propose the atte…
With the assumption that a video dataset is multimodality annotated in which auditory and visual modalities both are labeled or class-relevant, current multimodal methods apply modality fusion or cross-modality attention. However, effectively leveraging the audio modality in vision-specific annotated videos for action recognition is of particular challenge. To tackle this challenge, we propose a novel audio-visual framework that effectively leverages the audio modality in any solely vision-specific annotated dataset. We adopt the language models (e.g., BERT) to build a semantic audio-video lab…
GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding
China Agricultural University（中国农业大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；AniEye Technology（安尼眼科技）；State Key Laboratory of Animal Nutrition and Feeding（动物营养与饲料国家重点实验室）
School of Computer Science and Engineering, Southeast University, Nanjing, China（计算机科学与工程学院，东南大学，南京，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其跨学科应用关键实验室（东南大学），教育部，中国）
针对多模态视频模型中强模态主导导致的过拟合与模态不平衡问题，提出 MM 及改进的 B-MM 方法，通过动态调整模态混合比例提升泛化能力与鲁棒性。
StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQA
Henan Institute of Advanced Technology, Zhengzhou University（河南高级技术研究所，郑州大学）；Institute of Automation, CAS（自动化研究所，中国科学院）；UCAS（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）
Nowadays, the videos on the Internet are prevailing. The precise and in-depth understanding of the videos is a difficult but valuable problem for both platforms and researchers. The existing video understand models do well in object recognition tasks but currently still cannot understand the abstract and contextual features like highlight humor frames in comedy videos. The current industrial works are also mainly focused on the basic category classification task based on the appearances of objects. The feature detection methods for the abstract category remains blank. A data structure that inc…
HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；ARC Lab, Tencent PCG（腾讯PCG ARC实验室）；The University of Hong Kong（香港大学）；Xi’an JiaoTong University（西安交通大学）
SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding
Hong Kong Institute of Science & Innovation（香港科学与工业创新研究院）；CAMP, Technische Universität München（CAMP，慕尼黑技术大学）；Department of Surgery, Faculty of Medicine, The Chinese University of Hong Kong（香港中文大学医学院外科部）
Ochanomizu University（御茶水女子大学）；The University of Tokyo（东京大学）；Keio University（庆应义塾大学）
This paper introduces a new video-and-language dataset with human actions for multimodal logical inference, which focuses on intentional and aspectual expressions that describe dynamic human actions. The dataset consists of 200 videos, 5,554 action labels, and 1,942 action triplets of the form <subject, predicate, object> that can be translated into logical semantic representations. The dataset is expected to be useful for evaluating multimodal inference systems between videos and semantically complicated sentences including negation and quantification.
Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学信息科学与电子工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）
Livestream videos have become a significant part of online learning, where design, digital marketing, creative painting, and other skills are taught by experienced experts in the sessions, making them valuable materials. However, Livestream tutorial videos are usually hours long, recorded, and uploaded to the Internet directly after the live sessions, making it hard for other people to catch up quickly. An outline will be a beneficial solution, which requires the video to be temporally segmented according to topics. In this work, we introduced a large Livestream video dataset named MultiLive, …
Multimodal Semantic Attention Network for Video Captioning
University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Tencent PCG（腾讯平台与内容事业群）；University of Sydney（悉尼大学）
Multimodal sentiment analysis in videos is a key task in many real-world applications, which usually requires integrating multimodal streams including visual, verbal and acoustic behaviors. To improve the robustness of multimodal fusion, some of the existing methods let different modalities communicate with each other and modal the crossmodal interaction via transformers. However, these methods only use the single-scale representations during the interaction but forget to exploit multi-scale representations that contain different levels of semantic information. As a result, the representations…
GEM-VPC: A dual Graph-Enhanced Multimodal integration for Video Paragraph Captioning
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）
A Computational Model of Message Sensation Value in Short Video Multimodal Features that Predicts Sensory and Behavioral Engagement
Department of Communication, University of Utah（犹他大学通讯系）；Department of Communication, University of California, Davis（加州大学戴维斯分校通讯系）；Department of Media and Communication, City University of Hong Kong（香港城市大学媒体与传播系）；Division of Emerging Interdisciplinary Areas, Hong Kong University of Science and Technology（香港科学与技术大学新兴跨学科领域 division）
In the latest social networks, more and more people prefer to express their emotions in videos through text, speech, and rich facial expressions. Multimodal video emotion analysis techniques can help understand users' inner world automatically based on human expressions and gestures in images, tones in voices, and recognized natural language. However, in the existing research, the acoustic modality has long been in a marginal position as compared to visual and textual modalities. That is, it tends to be more difficult to improve the contribution of the acoustic modality for the whole multimoda…
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Tianjin University（天津大学）；Carnegie Mellon University（卡内基梅隆大学）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）
The extraction of text information in videos serves as a critical step towards semantic understanding of videos. It usually involved in two steps: (1) text recognition and (2) text classification. To localize texts in videos, we can resort to large numbers of text recognition methods based on OCR technology. However, to our knowledge, there is no existing work focused on the second step of video text classification, which will limit the guidance to downstream tasks such as video indexing and browsing. In this paper, we are the first to address this new task of video text classification by fusi…
