AstroMMBench：一个用于评估多模态大语言模型天文学能力的基准
AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
AI总结：
该研究推出首个评估多模态大语言模型天文图像理解能力的综合性基准AstroMMBench，含6个天体物理子领域621道专家审核题，对25款模型评估发现Ovis2-34B准确率最高，为AI与天文交叉发展提供支撑。
摘要：
天文图像解译为多模态大语言模型（MLLMs）应用于专业科学任务带来了重大挑战。现有基准聚焦于通用多模态能力，但无法体现天文数据的复杂性。为填补这一空白，我们推出了AstroMMBench，这是首个专为评估MLLMs天文图像理解能力而设计的综合性基准。AstroMMBench包含6个天体物理子领域的621道选择题，由15位领域专家策划并审核，以确保质量和相关性。我们利用AstroMMBench对25种不同的MLLMs开展了全面评估，其中包括22个开源模型和3个闭源模型。结果显示，Ovis2-34B取得了最高的总体准确率（70.5%），即便与强大的闭源模型相比也展现出领先能力。模型在6个天体物理子领域的表现存在差异，在宇宙学和高能天体物理等领域的任务尤其具有挑战性，而在仪器科学和太阳天体物理等其他领域的表现相对更好。这些发现凸显了AstroMMBench这类领域特定基准在严格评估MLLM性能、指导其面向科学应用的定向开发方面的重要作用。AstroMMBench提供了基础资源和动态工具，以推动人工智能与天文学交叉领域的发展。
英文摘要：
Astronomical image interpretation presents a significant challenge for applying multimodal large language models (MLLMs) to specialized scientific tasks. Existing benchmarks focus on general multimodal capabilities but fail to capture the complexity of astronomical data. To bridge this gap, we introduce AstroMMBench, the first comprehensive benchmark designed to evaluate MLLMs in astronomical image understanding. AstroMMBench comprises 621 multiple-choice questions across six astrophysical subfields, curated and reviewed by 15 domain experts for quality and relevance. We conducted an extensive evaluation of 25 diverse MLLMs, including 22 open-source and 3 closed-source models, using AstroMMBench. The results show that Ovis2-34B achieved the highest overall accuracy (70.5%), demonstrating leading capabilities even compared to strong closed-source models. Performance showed variations across the six astrophysical subfields, proving particularly challenging in domains like cosmology and high-energy astrophysics, while models performed relatively better in others, such as instrumentation and solar astrophysics. These findings underscore the vital role of domain-specific benchmarks like AstroMMBench in critically evaluating MLLM performance and guiding their targeted development for scientific applications. AstroMMBench provides a foundational resource and a dynamic tool to catalyze advancements at the intersection of AI and astronomy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
