Explainable Video Action Reasoning via Prior Knowledge and State Transitions
National University of Singapore（新加坡国立大学）；Shandong Computer Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院）山东计算中心（国家超级计算济南中心））；Northwestern Polytechnical University（西北工业大学）
Tao Zhuo, Zhiyong Cheng, Peng Zhang, Yongkang Wong, Mohan Kankanhalli
Human action analysis and understanding in videos is an important and challenging task. Although substantial progress has been made in past years, the explainability of existing methods is still limited. In this work, we propose a novel action reasoning framework that uses prior knowledge to explain semantic-level observations of video state changes. Our method takes advantage of both classical reasoning and modern deep learning approaches. Specifically, prior knowledge is defined as the information of a target video domain, including a set of objects, attributes and relationships in the target video domain, as well as relevant actions defined by the temporal attribute and relationship changes (i.e. state transitions). Given a video sequence, we first generate a scene graph on each frame to represent concerned objects, attributes and relationships. Then those scene graphs are linked by tracking objects across frames to form a spatio-temporal graph (also called video graph), which represents semantic-level video states. Finally, by sequentially examining each state transition in the video graph, our method can detect and explain how those actions are executed with prior knowledge, just like the logical manner of thinking by humans. Compared to previous works, the action reasoning results of our method can be explained by both logical rules and semantic-level observations of video content changes. Besides, the proposed method can be used to detect multiple concurrent actions with detailed information, such as who (particular objects), when (time), where (object locations) and how (what kind of changes). Experiments on a re-annotated dataset CAD-120 show the effectiveness of our method.
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
We develop a novel framework for single-scene video anomaly localization that allows for human-understandable reasons for the decisions the system makes. We first learn general representations of objects and their motions (using deep networks) and then use these representations to build a high-level, location-dependent model of any particular scene. This model can be used to detect anomalies in new videos of the same scene. Importantly, our approach is explainable - our high-level appearance and motion features can provide human-understandable reasons for why any part of a video is classified …
University of Arkansas（阿肯色大学）；FPT Software（FPT软件公司）
Video understanding is a growing field and a subject of intense research, which includes many interesting tasks to understanding both spatial and temporal information, e.g., action detection, action recognition, video captioning, video retrieval. One of the most challenging problems in video understanding is dealing with feature extraction, i.e. extract contextual visual representation from given untrimmed video due to the long and complicated temporal structure of unconstrained videos. Different from existing approaches, which apply a pre-trained backbone network as a black-box to extract vis…
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Frame sampling is a fundamental problem in video action recognition due to the essential redundancy in time and limited computation resources. The existing sampling strategy often employs a fixed frame selection and lacks the flexibility to deal with complex variations in videos. In this paper, we present a simple, sparse, and explainable frame sampler, termed as Motion-Guided Sampler (MGSampler). Our basic motivation is that motion is an important and universal signal that can drive us to adaptively select frames from videos. Accordingly, we propose two important properties in our MGSampler d…
We aim to dismantle the prevalent black-box neural architectures used in complex visual reasoning tasks, into the proposed eXplainable and eXplicit Neural Modules (XNMs), which advance beyond existing neural module networks towards using scene graphs --- objects as nodes and the pairwise relationships as edges --- for explainable and explicit reasoning with structured knowledge. XNMs allow us to pay more attention to teach machines how to "think", regardless of what they "look". As we will show in the paper, by using scene graphs as an inductive bias, 1) we can design XNMs in a concise and fle…
Sun Yat-sen University（中山大学）；DarkMatter AI Research（DarkMatter人工智能研究）
Explanation and high-order reasoning capabilities are crucial for real-world visual question answering with diverse levels of inference complexity (e.g., what is the dog that is near the girl playing with?) and important for users to understand and diagnose the trustworthiness of the system. Current VQA benchmarks on natural images with only an accuracy metric end up pushing the models to exploit the dataset biases and cannot provide any interpretable justification, which severally hinders advances in high-level question answering. In this work, we propose a new HVQR benchmark for evaluating e…
Explainable Forensics of Manipulated Segments in Untrimmed Long Videos
MoE Key Laboratory of Brain-Machine Intelligence Technology, College of Artificial Intelligence, Nanjing University of Aeronautics（脑机智能技术关键实验室、人工智能学院、南京航空航天大学）；Dalian University of Technology（大连理工大学）；Nanjing University（南京大学）；National University of Singapore（新加坡国立大学）
GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Engine
College of Biosystems Engineering and Food Science, Zhejiang University（浙江大学生物系统工程与食品科学学院）；ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University（浙江大学杭州国际科创中心）
Towards Explainable Industrial Anomaly Detection via Knowledge-Guided Latent Reasoning
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区计算机科学与技术学院）；School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）
University of California, Berkeley（加州大学伯克利分校）；Boston University（波士顿大学）
In complex inferential tasks like question answering, machine learning models must confront two challenges: the need to implement a compositional reasoning process, and, in many applications, the need for this reasoning process to be interpretable to assist users in both development and prediction. Existing models designed to produce interpretable traces of their decision-making process typically require these traces to be supervised at training time. In this paper, we present a novel neural modular approach that performs compositional reasoning by automatically inducing a desired sub-task dec…
ManiVid: Unified and Explainable Forensic Analysis of Manipulated Videos
Shanghai Jiao Tong University（上海交通大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）
VADTree: Explainable Training-Free Video Anomaly Detection via Hierarchical Granularity-Aware Tree
School of Software, Xi’an Jiaotong University（西安交通大学软件学院）；China Railway Xi’an Group（中国铁路西安集团）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
EvolveReason: Self-Evolving Reasoning Paradigm for Explainable Deepfake Facial Image Identification
School of Software Technology, Zhejiang University（浙江大学软件技术学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Ant Group（蚂蚁集团）
Explainable Artificial Intelligence for Quantifying Interfering and High-Risk Behaviors in Autism Spectrum Disorder in a Real-World Classroom Environment Using Privacy-Preserving Video Analysis
Emory University School of Medicine（埃默里大学医学院）；The Center for Discovery（发现中心）；Emory University（埃默里大学）；Georgia Institute of Technology（佐治亚理工学院）
Crime and Security Research Institute（犯罪与安全研究所）；Cardiff University（卡迪夫大学）；School of Engineering（工程学院）
The popularity of Deep Learning for real-world applications is ever-growing. With the introduction of high performance hardware, applications are no longer limited to image recognition. With the introduction of more complex problems comes more and more complex solutions, and the increasing need for explainable AI. Deep Neural Networks for Video tasks are amongst the most complex models, with at least twice the parameters of their Image counterparts. However, explanations for these models are often ill-adapted to the video domain. The current work in explainability for video models is still ove…
Agentic Tool-Augmented Reasoning for Explainable Image Forgery Detection
Nanyang Technological University（南洋理工大学）；Ant Digital Technologies, Ant Group（蚂蚁数字科技，蚂蚁集团）；Singapore University of Technology and Design（新加坡科技设计大学）；Singapore Management University（新加坡管理大学）
VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding
University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；University of Toronto（多伦多大学）；University of Washington（华盛顿大学）；University of Michigan（密歇根大学）
Video-CoM: Interactive Video Reasoning via Chain of Manipulations
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；University of California Merced（加州梅尔 Ced 大学）；Google Research（谷歌研究院）；Linköping University（林奈大学）；Australian National University（澳大利亚国立大学）
EDVD-LLaMA: Explainable Deepfake Video Detection via Multimodal Large Language Model Reasoning
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（电子与电气工程系，香港理工大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电子与电气工程学院，南洋理工大学）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（智能工程学院，华南理工大学）
Explainable AI-Powered Framework for Video-Based Skill Assessment in Cataract Surgery
K.N. Toosi University of Technology（霍加托西大学）；Faculties of Electrical and Computer Engineering（电气与计算机工程学院）；Applied Robotics and AI Solutions (ARAS)（应用机器人与人工智能解决方案（ARAS））
XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoning
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；th Research Institute, China Electronics Technology Group Corporation（中国电子科技集团第五十四研究所）
From Visual Attribution to Clinical Reasoning: Explainable Parkinson's Disease Screening from Hand-Drawn Patterns
Indian Institute of Technology Indore（印度理工学院印多尔分校）；Indian Institute of Technology Patna（印度理工学院巴特那分校）；M. R. Bangur Hospital（M. R. 班古尔医院）；University of Western Australia（西澳大利亚大学）
Explainable Cross-Disease Reasoning for Cardiovascular Risk Assessment from Low-Dose Computed Tomography
Department of Computer Science, Emory University（埃默里大学计算机科学系）；Department of Computer Science, Johns Hopkins University（约翰霍普金斯大学计算机科学系）；Department of Radiation Oncology（放射肿瘤学部）；Winship Cancer Institute, Emory University（埃默里大学Winship癌症研究所）
Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks
DeepWisdom；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Renmin University of China（中国人民大学）；University of Oxford（牛津大学）；National University of Singapore（新加坡国立大学）；Xiamen University（厦门大学）；Hong Kong University of Science and Technology (GuangZhou)（香港科技大学（广州））
Video understanding has achieved great success in representation learning, such as video caption, video object grounding, and video descriptive question-answer. However, current methods still struggle on video reasoning, including evidence reasoning and commonsense reasoning. To facilitate deeper video understanding towards video reasoning, we present the task of Causal-VidQA, which includes four types of questions ranging from scene description (description) to evidence reasoning (explanation) and commonsense reasoning (prediction and counterfactual). For commonsense reasoning, we set up a tw…
