Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding.
Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
University of Electronic Science and Technology of China(电子科技大学);Sun Yat-sen University(中山大学);University of Washington(华盛顿大学);Microsoft(微软公司);The Chinese University of Hong Kong(香港中文大学)
本文提出覆盖数学、物理、化学和编程的 EMMA 多模态推理基准,揭示先进 MLLMs 在复杂跨模态、多步推理上仍存在明显不足。
Reasoning Guided Embeddings: Leveraging MLLM Reasoning for Improved Multimodal Retrieval
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);Sensetime Research(商汤科技研究院);Beijing Institute of Technology(北京理工大学);Shanghai AI Lab(上海人工智能实验室)
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Institute for AI Industry Research (AIR), Tsinghua University(清华大学智能产业研究院(AIR))
V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval
Tsinghua University(清华大学);University of Central Florida(中央佛罗里达大学);Fudan University(复旦大学);The Australian National University(澳大利亚国立大学);The University of Hong Kong(香港大学);Pengcheng Laboratory(鹏城实验室);Central South University(中南大学)
Tsinghua University(清华大学);Meituan Inc(美团公司);Central South University(中南大学);Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);Nanyang Technological University(南洋理工大学)
ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge
College of Computer Science, Sichuan University, Chengdu, China(四川大学计算机学院);National Key Laboratory of Fundamental Algorithms and Models for Engineering Simulation, Sichuan University, China(四川省工程仿真基础算法与模型国家重点实验室)
Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers
Archimedes, Athena Research Center(阿基米德研究中心);National Technical University of Athens(希腊国家技术大学);University of Crete(克里特大学);IACM-Forth(第四研究机构(IACM-Forth))
ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning
National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);University of Washington(华盛顿大学);Stanford University(斯坦福大学);absolute AI;The Chinese University of Hong Kong(香港中文大学)
Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models
Harvard Medical School, Harvard University(哈佛医学院、哈佛大学);Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital(阿提尼乌拉A.马丁努斯生物医学成像中心、麻省总医院);Knovel Engineering Lab, Singapore(Knovel工程实验室、新加坡)
EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Ability
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China(计算机科学与技术系,人工智能研究院,清华大学,北京,中国);Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究院(AIR),清华大学,北京,中国);School of Management, Fudan University, Shanghai, China(管理学院,复旦大学,上海,中国)
SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);OPPO AI Center(OPPO AI中心)
mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Tencent(腾讯);University of California, Los Angeles(加州大学洛杉矶分校);Lenovo Research(联想研究院)
HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);ARC Lab, Tencent PCG(腾讯PCG ARC实验室);The University of Hong Kong(香港大学);Xi’an JiaoTong University(西安交通大学)