PaperMind：多模态大语言模型中科学论文代理推理与批判的基准测试
PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs
AI总结：
PaperMind通过整合多模态信息和跨源推理，评估科学论文的综合推理能力，揭示多模态大语言模型在科学推理和批判中的性能差距。
摘要：
理解科学论文需要超越孤立回答问题或总结内容，涉及整合文本和视觉信息、解释实验证据、跨源综合信息和批判性评估科学主张的综合推理过程。然而，现有基准通常孤立评估这些能力，难以评估科学论文理解作为统一的交互认知能力。本文引入PaperMind，一个评估多模态大语言模型中科学论文综合推理的基准。PaperMind基于七个领域的真实科学论文构建，包含四个互补的任务家族，涵盖多模态 grounding、实验解释、跨源证据推理和批判评估等不同认知方面。通过分析模型在多个任务中的行为，PaperMind能够诊断评估整合的科学推理行为，这些行为难以通过孤立任务评估来评估。对开源和闭源多模态大语言模型的广泛实验揭示了任务间一致的性能差距，突显了整合科学推理和批判的持续挑战。我们的基准和数据集可在https://github.com/Yanjun-Zhao/PaperMind获取。
英文摘要：
Understanding scientific papers requires more than answering isolated questions or summarizing content. It involves an integrated reasoning process that grounds textual and visual information, interprets experimental evidence, synthesizes information across sources, and critically evaluates scientific claims. However, existing benchmarks typically assess these abilities in isolation, making it difficult to evaluate scientific paper understanding as a unified set of interacting cognitive abilities. In this work, we introduce PaperMind, a benchmark designed to evaluate integrated and agent-oriented scientific reasoning over research papers. PaperMind is constructed from real scientific papers across seven domains, including agriculture, biology, chemistry, computer science, medicine, physics, and economics. It comprises four complementary task families that collectively operationalize distinct cognitive facets of scientific paper reasoning, including multimodal grounding, experimental interpretation, cross-source evidence reasoning, and critical assessment. By analyzing model behavior across multiple tasks, PaperMind enables a diagnostic evaluation of integrated scientific reasoning behaviors that are difficult to assess through isolated task evaluations. Extensive experiments on both opensource and closed-source multimodal LLMs reveal consistent performance gaps across tasks, highlighting persistent challenges in integrated scientific reasoning and critique. Our benchmark and dataset are available at https:// github.com/Yanjun-Zhao/PaperMind.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
