We introduce SPUR, a comprehensive benchmark for scientific experimental image perception, understanding, and reasoning, comprising 4,264 question-answering (QA) pairs derived from 1,084 expert-curated images. SPUR features three key innovations: (1) Panel-Level Fine-Grained Perception: evaluating the visual perception of multimodal large language models (MLLMs) across three dimensions (numerical, morphological, and information localization) on six fine-grained panel types; (2) Cross-Panel Relation Understanding: utilizing complex images with an average of 14.3 panels per sample to evaluate MLLMs' ability to decipher intricate cross-panel relations; (3) Expert-Level Reasoning: assessment of qualitative and quantitative reasoning across five experimental paradigms to determine if models can infer conclusions from evidence as human experts do. Comprehensive evaluation of 20 MLLMs and four multimodal Chain-of-Thought (MCoT) methods reveals that current models fall significantly short of the expert-level requirements for scientific image interpretation, underscoring a critical bottleneck in AI for Science (AI4S) research.
State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Models
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Shanghai Innovation Institute(上海创新研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);ByteDance, LarkAI(字节跳动 LarkAI);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室)
Unbiased Visual Reasoning with Controlled Visual Inputs
Arizona State University(亚利桑那州立大学);University of Southern California(南加州大学);University of Pennsylvania(宾夕法尼亚大学);University of California, Davis(加州大学戴维斯分校)
Mitigating Object Hallucination via Robust Local Perception Search
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Center for Future Media & School of Computer Science and Engineering, University of Electronic Science and Technology of China(未来媒体中心及电子科技大学计算机科学与工程学院)
Speculative Decoding Reimagined for Multimodal Large Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(多媒体可信感知与高效计算重点实验室、中华人民共和国教育部、厦门大学)
SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding
Department of Electrical and Computer Engineering, Seoul National University(首尔大学电气与计算机工程系);Interdisciplinary Program in Artificial Intelligence, Seoul National University(首尔大学人工智能跨学科项目)
CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs
Tsinghua University(清华大学);Pengcheng Laboratory(鹏城实验室);Carnegie Mellon University(卡内基梅隆大学);University of California, Berkeley(加州大学伯克利分校);The University of Texas at Dallas(德克萨斯大学达拉斯分校);Chang’an University(长安大学)
MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
School of Future Technology, Shanghai University(上海大学未来技术学院);Institute of Artificial Intelligence, Shanghai University(上海大学人工智能研究院);School of Computer Engineering(计算机工程学院)
Language Models Can See Better: Visual Contrastive Decoding For LLM Multimodal Reasoning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);University of California, Santa Cruz(加州大学圣克鲁兹分校)
Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人工智能与机器人研究所);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd(星辰通用人工智能实验室,中国电信人工智能技术(北京)有限公司);University of Science and Technology Beijing(北京科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);Shanghai Jiao Tong University(上海交通大学)
Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimization
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);Pengcheng Laboratory(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Dual-Latent Collaborative Decoding for Fidelity-Perception Balanced Image Compression
School of Information and Communication Engineering and the State Key Laboratory of Media Convergence and Communication, Communication University of China(信息与通信工程学院和媒体融合与通信国家重点实验室,中国通信大学);School of Information Science and Electronic Engineering, Shanghai Jiao Tong University(信息科学与电子工程学院,上海交通大学);Department of Computer Science, City University of Hong Kong(计算机科学系,香港城市大学);State Key Laboratory of Multimedia information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学)
Real-time Reconstruction of Human Visual Perception from fMRI
Princeton University(普林斯顿大学);Dartmouth College(达特茅斯学院);Icahn School of Medicine at Mount Sinai(西奈山伊坎医学院);Harvard University(哈佛大学);University of Texas at Dallas(德克萨斯大学达拉斯分校);Medical AI Research Center (MedARC)(医学人工智能研究中心)
Neuro-3D: Towards 3D Visual Decoding from EEG Signals
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tsinghua University(清华大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Jiao Tong University(上海交通大学);South China University of Technology(华南理工大学)
Decoding Functional Networks for Visual Categories via GNNs
The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev(内盖夫本-古里安大学电气与计算机工程学院);Psychology Department, Ben-Gurion University of the Negev(内盖夫本-古里安大学心理学系);The School of Brain Sciences and Cognition, Ben-Gurion University of the Negev(内盖夫本-古里安大学脑科学与认知学院)
University of Rome, Tor Vergata Department of Biomedicine and Prevention(罗马大学Tor Vergata医学院与预防学院);A.A. Martinos Center for Biomedical Imaging(A.A. Martinos生物医学成像中心);Harvard Medical School/MGH, Boston (US)(哈佛医学院/麻省总医院,波士顿(美国))
Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding
Dept. of Electrical & Computer Engineering, University of Pittsburgh, USA(宾夕法尼亚大学电气与计算机工程系);Dept. of Biomedical Engineering, Tsinghua University, China(清华大学生物医学工程系);Dept. of Neurology, University of Southern California, USA(美国南加州大学神经病学系);Dept. of Computer Science, University of Texas Rio Grande Valley, USA(德克萨斯理工大学里奥格兰德谷分校计算机科学系)
A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli
Beijing Institute of Basic Medical Sciences(北京基础医学研究所);School of Computer Science and Information Engineering, Hefei University of Technology (HFUT)(合肥工业大学计算机与信息工程学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Zhejiang University(浙江大学);Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney(悉尼科技大学工程与信息技术学院澳大利亚人工智能研究所);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学技术学院)
NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction
Tongji University(同济大学);Ohio State University(俄亥俄州立大学);University of Technology Sydney(悉尼科技大学);Chinese Academy of Sciences(中国科学院);Beijing Anding Hospital(北京安定医院)