This paper tackles the intricate challenge of video question-answering (VideoQA). Despite notable progress, current methods fall short of effectively integrating questions with video frames and semantic object-level abstractions to create question-aware video representations. We introduce Local-Global Question Aware Video Embedding (LGQAVE), which incorporates three major innovations to integrate multi-modal knowledge better and emphasize semantic visual concepts relevant to specific questions. LGQAVE moves beyond traditional ad-hoc frame sampling by utilizing a cross-attention mechanism that precisely identifies the most relevant frames concerning the questions. It captures the dynamics of objects within these frames using distinct graphs, grounding them in question semantics with the miniGPT model. These graphs are processed by a question-aware dynamic graph transformer (Q-DGT), which refines the outputs to develop nuanced global and local video representations. An additional cross-attention module integrates these local and global embeddings to generate the final video embeddings, which a language model uses to generate answers. Extensive evaluations across multiple benchmarks demonstrate that LGQAVE significantly outperforms existing models in delivering accurate multi-choice and open-ended answers.
Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
School of Information Science and Technology, Hangzhou Normal University(杭州师范大学信息科学与技术学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);School of Computer Science and Technology, Hangzhou Dianzi University(杭州电子科技大学计算机科学与技术学院)
KGFR: A Foundation Retriever for Generalized Knowledge Graph Question Answering
School of Computer Science, Nanjing University of Information Science and Technology(南京信息工程大学计算机科学学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学软件新技术国家重点实验室);National Institute of Healthcare Data Science, Nanjing University(南京大学健康数据科学国家研究院);Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology(南京信息工程大学教育部长江数字取证工程研究中心)
You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMs
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学)
Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine(放射肿瘤科,Winship癌症研究所,埃默里大学医学院);Department of Biomedical Engineering, College of Engineering, Georgia Institute of Technology(生物医学工程系,工程学院,佐治亚理工学院);Department of Computer Science and Mathematics, Laney Graduate School, Emory University(计算机科学与数学系,兰尼研究生院,埃默里大学);School of Electrical and Computer Engineering, College of Engineering, Georgia Institute of Technology(电气与计算机工程系,工程学院,佐治亚理工学院)
MetaScientist: A Human-AI Synergistic Framework for Automated Mechanical Metamaterial Design
University of Pennsylvania(宾夕法尼亚大学);Virginia Tech(弗吉尼亚理工大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Davis(加利福尼亚大学戴维斯分校);Princeton University(普林斯顿大学)
MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning
School of Automation, Guangdong University of Technology(广东工业大学自动化学院);College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);State Key Laboratory of Traditional Chinese Medicine Syndrome, The Second Affiliated Hospital of Guangzhou University of Chinese Medicine, Guangdong Provincial Hospital of Chinese Medicine, Guangdong Provincial Academy of Chinese Medical Sciences(广东省中医药科学院中医证候重点实验室,广州中医药大学第二附属医院,广东省中医院,广东省中医药科学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机与数据科学学院);Department of Electrical and Computer Engineering, Sungkyunkwan University(成均馆大学电子与计算机工程系)
PulseLM: A Foundation Dataset and Benchmark for PPG-Text Learning
Singapore Management University(新加坡管理大学);Queen Mary University of London(伦敦玛丽女王大学);Eindhoven University of Technology(埃因霍温理工大学);University of Cambridge(剑桥大学)
Video Event Reasoning and Prediction by Fusing World Knowledge from LLMs with Vision Foundation Models
INRIA(法国国家信息与自动化研究所);Max Planck Institute for Intelligent Systems(人工智能研究所);San Francisco State University(旧金山州立大学);Seoul AI Institute (SAII)(首尔人工智能研究所);Vision & Robotics Center, Tsinghua University(清华大学视觉与机器人中心);Polytechnic University of Madrid(马德里理工大学)
Understanding Synergistic Interactions among Pathology Foundation Models via Adaptive Fusion
School of Computer Science and Engineering, South China University of Technology(华南理工大学计算机科学与工程学院);School of Computing and Mathematical Sciences, University of Leicester(莱斯特大学计算与数学科学学院);Leicester Cancer Research Centre, University of Leicester(莱斯特大学莱斯特癌症研究中心);Department of Engineering Science, University of Oxford(牛津大学工程科学系);Nuffield Department of Medicine, University of Oxford(牛津大学纳菲尔德医学院);A*STAR, Singapore(新加坡科技研究局)
HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways
University of Calabria(卡拉布里亚大学);University of Catanzaro(卡坦扎罗大学);University of Eastern Piedmont(东皮埃蒙特大学);University of Pisa(比萨大学);University of Cambridge(剑桥大学)