arXiv 2511.023842026-01-06cs.CV

RxnCaption：将反应图解析重新表述为视觉提示引导的描述生成

RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning

Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Beihang University（北航）；Peking University（北京大学）；South China Normal University（华南师范大学）；Northwestern Polytechnical University（西北工业大学）

Jiahe Song, Chuang Wang, Bowen Jiang, Yinfan Wang, Hao Zheng, Xingjian Wei, Chengjin Liu, Rui Nie, Junyuan Gao, Jiaxing Sun, Yubin Wang, Lijun Wu, Zhenhua Huang… 展开作者

Jiahe Song, Chuang Wang, Bowen Jiang, Yinfan Wang, Hao Zheng, Xingjian Wei, Chengjin Liu, Rui Nie, Junyuan Gao, Jiaxing Sun, Yubin Wang, Lijun Wu, Zhenhua Huang, Jiang Wu, Qian Yu, Conghui He

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

RxnCaption通过将反应图解析转化为视觉提示引导的描述生成任务，提升化学反应数据的机器可读性，并构建大规模数据集推动AI在化学领域的应用。

摘要：

大规模的化学反应数据集对化学领域的AI研究至关重要。然而，现有的化学反应数据往往以图像形式出现在论文中，使它们不具机器可读性，无法用于训练机器学习模型。为解决这一挑战，我们提出了RxnCaption框架，用于化学反应图解析（RxnDP）任务。我们的框架将传统的坐标预测驱动的解析过程重新表述为图像描述问题，由大型视觉语言模型（LVLMs）自然处理。我们引入了一种称为BBox和Index作为视觉提示（BIVP）的策略，利用我们最先进的分子检测器MolYOLO，在输入图像上预先绘制分子边界框和索引。这将下游解析转化为自然语言描述问题。广泛的实验表明，BIVP策略在提高结构提取质量的同时简化了模型设计。我们进一步构建了RxnCaption-15k数据集，比先前的现实世界文献基准大一个数量级，具有四个布局范式中的平衡测试子集。实验表明，RxnCaption-VL在多个指标上实现了最先进的性能。我们认为我们的方法、数据集和模型将推动从化学文献中提取结构信息，并催化更广泛的AI在化学中的应用。我们将发布数据、模型和代码到GitHub。

英文摘要：

Large-scale chemical reaction datasets are crucial for AI research in chemistry. However, existing chemical reaction data often exist as images within papers, making them not machine-readable and unusable for training machine learning models. In response to this challenge, we propose the RxnCaption framework for the task of chemical Reaction Diagram Parsing (RxnDP). Our framework reformulates the traditional coordinate prediction driven parsing process into an image captioning problem, which Large Vision Language Models (LVLMs) handle naturally. We introduce a strategy termed BBox and Index as Visual Prompt (BIVP), which uses our state-of-the-art molecular detector, MolYOLO, to pre-draw molecular bounding boxes and indices directly onto the input image. This turns the downstream parsing into a natural-language description problem. Extensive experiments show that the BIVP strategy significantly improves structural extraction quality while simplifying model design. We further construct the RxnCaption-15k dataset, an order of magnitude larger than prior real-world literature benchmarks, with a balanced test subset across four layout archetypes. Experiments demonstrate that RxnCaption-VL achieves state-of-the-art performance on multiple metrics. We believe our method, dataset, and models will advance structured information extraction from chemical literature and catalyze broader AI applications in chemistry. We will release data, models, and code on GitHub.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。