在长篇科研论文中寻找问题答案是一个重要研究方向，能够帮助读者快速解答疑问。然而，现有的基于科学论文的问答（QA）数据集不仅规模有限，还仅聚焦于文本内容。我们推出了SPIQA（Scientific Paper Image Question Answering，科学论文图像问答），这是首个专门用于解读计算机科学各领域科研论文上下文内复杂图表的大规模问答数据集。我们借助多模态大语言模型（MLLMs）的广泛专业能力与图像理解能力，采用自动加人工策展的方式构建该数据集。我们设计了一项针对图文交错内容的信息探寻任务，涉及的图像类型包括曲线图、柱状图、表格、示意图以及结果可视化图等多种。SPIQA包含27万个问题，被划分为训练集、验证集以及三个不同的测试子集。我们通过对12个主流基础模型开展大量实验，评估了当前多模态系统理解科研论文细节内容的能力。此外，我们提出了一种结合上下文检索的思维链（CoT）评估策略，可实现细粒度的逐步评估，同时提升了模型性能。我们还进一步探究了引入额外文本信息后性能提升的上限，凸显了其在未来研究中的可观潜力，以及该数据集对革新我们与科学文献交互方式的重要作用。
英文摘要：
Seeking answers to questions within long scientific research articles is a crucial area of study that aids readers in quickly addressing their inquiries. However, existing question-answering (QA) datasets based on scientific papers are limited in scale and focus solely on textual content. We introduce SPIQA (Scientific Paper Image Question Answering), the first large-scale QA dataset specifically designed to interpret complex figures and tables within the context of scientific research articles across various domains of computer science. Leveraging the breadth of expertise and ability of multimodal large language models (MLLMs) to understand figures, we employ automatic and manual curation to create the dataset. We craft an information-seeking task on interleaved images and text that involves multiple images covering plots, charts, tables, schematic diagrams, and result visualizations. SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits. Through extensive experiments with 12 prominent foundational models, we evaluate the ability of current multimodal systems to comprehend the nuanced aspects of research articles. Additionally, we propose a Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that allows fine-grained, step-by-step assessment and improves model performance. We further explore the upper bounds of performance enhancement with additional textual information, highlighting its promising potential for future research and the dataset's impact on revolutionizing how we interact with scientific literature.
SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation
GreenDynamics Pty. Ltd（格林动力私人有限公司）；City University of Hong Kong（香港城市大学）；University of Chicago（芝加哥大学）；University of Sydney（悉尼大学）；University of New South Wales（新南威尔士大学）
MuISQA: Multi-Intent Retrieval-Augmented Generation for Scientific Question Answering
Zhongke Zidong Taichu (Beijing)（中科紫东太初（北京））；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究院）
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
Nanjing University of Science and Technology（南京理工大学）；Baidu Inc（百度公司）；AIML, Adelaide University（阿德莱德大学AIML）；SUTD（新加坡科技设计大学）；Southeast University（东南大学）；East China Normal University（华东师范大学）；University of Oxford（牛津大学）
ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Institute of Tibetan Plateau Research, Chinese Academy of Sciences（中国科学院青藏高原研究所）；Lanzhou University（兰州大学）；College of Resources and Environment, University of Chinese Academy of Sciences（中国科学院大学资源与环境学院）
A Pathway to General-Purpose Scientific AI: Multimodal Comprehension of Scientific Images
TIB Leibniz Information Centre for Science and Technology（莱布尼茨科学与技术信息中心（TIB））；Eindhoven University of Technology（埃因霍温理工大学）；Freie Universität Berlin（柏林自由大学）；International Center for Chemical and Biological Sciences, University of Karachi（卡拉奇大学国际化学与生物科学中心）；National University of Sciences and Technology（国家科技大学）；University of Warwick（华威大学）；PSG College of Technology（PSG理工学院）；Aalto University（阿尔托大学）
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
Allen Institute for AI（艾伦人工智能研究所）；Yale University（耶鲁大学）；University of Washington（华盛顿大学）；Hebrew University（希伯来大学）；MIT（麻省理工学院）；Northwestern University（西北大学）
SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs
Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；University of California, Los Angeles（加州大学洛杉矶分校）；Peking University（北京大学）
EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong（香港中文大学）
