RxnBench: 一个用于评估大语言模型在科学文献中理解化学反应的多模态基准
RxnBench: A Multimodal Benchmark for Evaluating Large Language Models on Chemical Reaction Understanding from Scientific Literature
AI总结：
RxnBench通过两个任务评估大语言模型在化学反应理解上的能力，揭示模型在深度化学逻辑和结构识别上的不足，强调需改进视觉编码和推理能力以推动AI化学发展。
摘要：
多模态大语言模型（MLLMs）在化学领域的整合有望革新科学发现，但其在真实文献中理解密集的图形语言（反应）的能力仍待探索。本文介绍了RxnBench，一个多层次的基准，旨在严格评估MLLMs在科学PDFs中对化学反应的理解能力。RxnBench包含两个任务：单图问答（SF-QA），通过305个精心挑选的反应方案衍生出的1525个问题，测试细粒度的视觉感知和机理推理；全文档问答（FD-QA），挑战模型从108篇文章中综合信息，要求跨模态整合文本、方案和表格。我们对MLLMs的评估揭示了一个关键能力差距：尽管模型在提取显式文本方面表现出色，但在深入化学逻辑和精确结构识别方面却存在困难。值得注意的是，具有推理时间推理的模型显著优于标准架构，但无一达到FD-QA的50%准确率。这些发现强调了迫切需要领域特定的视觉编码器和更强的推理引擎来推进自主AI化学家。
英文摘要：
The integration of Multimodal Large Language Models (MLLMs) into chemistry promises to revolutionize scientific discovery, yet their ability to comprehend the dense, graphical language of reactions within authentic literature remains underexplored. Here, we introduce RxnBench, a multi-tiered benchmark designed to rigorously evaluate MLLMs on chemical reaction understanding from scientific PDFs. RxnBench comprises two tasks: Single-Figure QA (SF-QA), which tests fine-grained visual perception and mechanistic reasoning using 1,525 questions derived from 305 curated reaction schemes, and Full-Document QA (FD-QA), which challenges models to synthesize information from 108 articles, requiring cross-modal integration of text, schemes, and tables. Our evaluation of MLLMs reveals a critical capability gap: while models excel at extracting explicit text, they struggle with deep chemical logic and precise structural recognition. Notably, models with inference-time reasoning significantly outperform standard architectures, yet none achieve 50\% accuracy on FD-QA. These findings underscore the urgent need for domain-specific visual encoders and stronger reasoning engines to advance autonomous AI chemists.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
