评估大型语言模型在多模态化学奥林匹克考试中的表现
Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams
AI总结：
本文评估了多种多模态LLM在化学奥林匹克考试中的表现，发现其在多模态融合和科学推理方面存在显著局限，提出通过链式思维提示提升模型性能的方法。
摘要：
多模态科学推理仍然是大型语言模型（LLMs）面临的主要挑战，尤其是在化学领域，问题解决依赖于符号图表、分子结构和结构化视觉数据。本文系统评估了40个专有和开源的多模态LLM，包括GPT-5、o3、Gemini-2.5-Pro和Qwen2.5-VL，在一个精心挑选的奥林匹克风格化学问题基准上，这些问题是来自超过二十年的美国全国化学奥林匹克竞赛（USNCO）考试。这些问题需要跨多种模态进行整合的视觉和文本推理。我们发现，许多模型在模态融合方面存在困难，在某些情况下，甚至移除图像也能提高准确性，表明视觉-语言整合存在偏差。链式思维提示法在准确性和视觉定位方面始终表现出色，通过消融研究和遮挡基的可解释性得到了验证。我们的结果揭示了当前多模态LLM在科学推理能力方面的关键限制，为开发更稳健和可解释的多模态系统提供了可行策略。这项工作为衡量领域特定多模态AI的进步提供了及时的基准，并突显了人工智能与科学推理交汇处进一步发展的需求。
英文摘要：
Multimodal scientific reasoning remains a significant challenge for large language models (LLMs), particularly in chemistry, where problem-solving relies on symbolic diagrams, molecular structures, and structured visual data. Here, we systematically evaluate 40 proprietary and open-source multimodal LLMs, including GPT-5, o3, Gemini-2.5-Pro, and Qwen2.5-VL, on a curated benchmark of Olympiad-style chemistry questions drawn from over two decades of U.S. National Chemistry Olympiad (USNCO) exams. These questions require integrated visual and textual reasoning across diverse modalities. We find that many models struggle with modality fusion, where in some cases, removing the image even improves accuracy, indicating misalignment in vision-language integration. Chain-of-Thought prompting consistently enhances both accuracy and visual grounding, as demonstrated through ablation studies and occlusion-based interpretability. Our results reveal critical limitations in the scientific reasoning abilities of current MLLMs, providing actionable strategies for developing more robust and interpretable multimodal systems in chemistry. This work provides a timely benchmark for measuring progress in domain-specific multimodal AI and underscores the need for further advances at the intersection of artificial intelligence and scientific reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
