We present TUMTraffic-VideoQA, a novel dataset and benchmark designed for spatio-temporal video understanding in complex roadside traffic scenarios. The dataset comprises 1,000 videos, featuring 85,000 multiple-choice QA pairs, 2,300 object captioning, and 5,700 object grounding annotations, encompassing diverse real-world conditions such as adverse weather and traffic anomalies. By incorporating tuple-based spatio-temporal object expressions, TUMTraffic-VideoQA unifies three essential tasks-multiple-choice video question answering, referred object captioning, and spatio-temporal object grounding-within a cohesive evaluation framework. We further introduce the TUMTraffic-Qwen baseline model, enhanced with visual token sampling strategies, providing valuable insights into the challenges of fine-grained spatio-temporal reasoning. Extensive experiments demonstrate the dataset's complexity, highlight the limitations of existing models, and position TUMTraffic-VideoQA as a robust foundation for advancing research in intelligent transportation systems. The dataset and benchmark are publicly available to facilitate further exploration.
It is well known that most of the conventional video question answering (VideoQA) datasets consist of easy questions requiring simple reasoning processes. However, long videos inevitably contain complex and compositional semantic structures along with the spatio-temporal axis, which requires a model to understand the compositional structures inherent in the videos. In this paper, we suggest a new compositional VideoQA method based on transformer architecture with a deformable attention mechanism to address the complex VideoQA tasks. The deformable attentions are introduced to sample a subset o…
Keyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answering
Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局（A*STAR）信息通信研究院（I2R））；School of Computing, National University of Singapore（新加坡国立大学计算学院）；SCSE, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
The task of Video Question Answering (VideoQA) consists in answering natural language questions about a video and serves as a proxy to evaluate the performance of a model in scene sequence understanding. Most methods designed for VideoQA up-to-date are end-to-end deep learning architectures which struggle at complex temporal and causal reasoning and provide limited transparency in reasoning steps. We present the HySTER: a Hybrid Spatio-Temporal Event Reasoner to reason over physical events in videos. Our model leverages the strength of deep learning methods to extract information from video fr…
StreamingCoT: A Dataset for Temporal Dynamics and Multimodal Chain-of-Thought Reasoning in Streaming VideoQA
Henan Institute of Advanced Technology, Zhengzhou University（河南高级技术研究所，郑州大学）；Institute of Automation, CAS（自动化研究所，中国科学院）；UCAS（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）
VideoQA-SC: Adaptive Semantic Communication for Video Question Answering
State Key Laboratory of Advanced Rail Autonomous Operation（先进轨道交通自主运行全国重点实验室）；School of Electronic and Information Engineering, Beijing Jiaotong University（北京交通大学电子信息工程学院）；Frontiers Science Center for Smart High-speed Railway System（智能高速铁路系统前沿科学中心）；School of Information Engineering, Zhengzhou University（郑州大学信息工程学院）
Videos convey rich information. Dynamic spatio-temporal relationships between people/objects, and diverse multimodal events are present in a video clip. Hence, it is important to develop automated models that can accurately extract such information from videos. Answering questions on videos is one of the tasks which can evaluate such AI abilities. In this paper, we propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Specifically, we first employ dense image captions to help identify o…
Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences（中国科学院重庆绿色智能技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）
This paper presents a new method for end-to-end Video Question Answering (VideoQA), aside from the current popularity of using large-scale pre-training with huge feature extractors. We achieve this with a pyramidal multimodal transformer (PMT) model, which simply incorporates a learnable word embedding layer, a few convolutional and transformer layers. We use the anisotropic pyramid to fulfill video-language interactions across different spatio-temporal scales. In addition to the canonical pyramid, which includes both bottom-up and top-down pathways with lateral connections, novel strategies a…
Sea AI Lab（Sea人工智能实验室）；Sea-NExT Joint Lab（Sea-NExT联合实验室）；National University of Singapore（新加坡国立大学）
This paper proposes a Video Graph Transformer (VGT) model for Video Quetion Answering (VideoQA). VGT's uniqueness are two-fold: 1) it designs a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations, and dynamics for complex spatio-temporal reasoning; and 2) it exploits disentangled video and text Transformers for relevance comparison between the video and text to perform QA, instead of entangled cross-modal Transformer for answer classification. Vision-text communication is done by additional cross-modal interaction modules. With more …
REVEAL: Relation-based Video Representation Learning for Video-Question-Answering
Goethe University Frankfurt（法兰克福大学）；Tuebingen AI Center（蒂宾根人工智能中心）；University of Tuebingen（蒂宾根大学）；University of Bonn（波恩大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Video-EM: Event-Centric Episodic Memory for Long-Form Video Understanding
City University of Hong Kong（香港城市大学）；University of Science and Technology of China（中国科学技术大学）；Tianjin University（天津大学）；Nanjing University（南京大学）；Zhejiang University（浙江大学）；The Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究所（TeleAI））
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
A video-grounded dialogue system referred to as the Structured Co-reference Graph Attention (SCGA) is presented for decoding the answer sequence to a question regarding a given video while keeping track of the dialogue context. Although recent efforts have made great strides in improving the quality of the response, performance is still far from satisfactory. The two main challenging issues are as follows: (1) how to deduce co-reference among multiple modalities and (2) how to reason on the rich underlying semantic structure of video with complex spatial and temporal dynamics. To this end, SCG…
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science, Peking University（北京大学计算机科学学院）；Center for Data Science, Peking University（北京大学数据科学中心）
