Glance and Focus: Memory Prompting for Multi-Event Video Question Answering
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS)（中国科学院智能信息处理重点实验室）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Video Question Answering (VideoQA) has emerged as a vital tool to evaluate agents' ability to understand human daily behaviors. Despite the recent success of large vision language models in many multi-modal tasks, complex situation reasoning over videos involving multiple human-object interaction events still remains challenging. In contrast, humans can easily tackle it by using a series of episode memories as anchors to quickly locate question-related key moments for reasoning. To mimic this effective reasoning strategy, we propose the Glance-Focus model. One simple way is to apply an action detection model to predict a set of actions as key memories. However, these actions within a closed set vocabulary are hard to generalize to various video domains. Instead of that, we train an Encoder-Decoder to generate a set of dynamic event memories at the glancing stage. Apart from using supervised bipartite matching to obtain the event memories, we further design an unsupervised memory generation method to get rid of dependence on event annotations. Next, at the focusing stage, these event memories act as a bridge to establish the correlation between the questions with high-level event concepts and low-level lengthy video content. Given the question, the model first focuses on the generated key event memory, then focuses on the most relevant moment for reasoning through our designed multi-level cross-attention mechanism. We conduct extensive experiments on four Multi-Event VideoQA benchmarks including STAR, EgoTaskQA, AGQA, and NExT-QA. Our proposed model achieves state-of-the-art results, surpassing current large models in various challenging reasoning tasks. The code and models are available at https://github.com/ByZ0e/Glance-Focus.
Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
School of Information Science and Technology, Hangzhou Normal University（杭州师范大学信息科学与技术学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；School of Computer Science and Technology, Hangzhou Dianzi University（杭州电子科技大学计算机科学与技术学院）
Unifying Image Processing as Visual Prompting Question Answering
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Macau（澳门大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
State Key Laboratory of VR Technology and Systems, School of CSE, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学计算机科学与工程学院）；College of AI, Tsinghua University（清华大学人工智能学院）
Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering
Key Laboratory of Complex Systems Modeling and Simulation, the School of Computer Science, Hangzhou Dianzi University（复杂系统建模与仿真重点实验室、计算机科学学院、杭州电子大学）；HDU-ITMO Joint Institute, Hangzhou Dianzi University（杭州电子大学-ITMO联合学院）；School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院、合肥工业大学）；School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen（智能科学与工程学院、哈尔滨工业大学深圳校区）
Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion
School of Artificial Intelligence, Tiangong University（天津工业大学人工智能学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Linying Medical Technology (Shenzhen) Co., Ltd.（临影医疗科技（深圳）有限公司）
