Scientific compound figures combine multiple labeled panels into a single image, and downstream pretraining and retrieval require panel-aligned visual-text pairs. However, in a PubMed Central (PMC)-scale crawl of 346,567 compound figures, 16.3% have no caption and are discarded by existing caption-decomposition pipelines. We propose FigEx2, a visual-conditioned framework that takes only a compound figure as input and jointly produces labeled panel boxes and panel-wise captions. FigEx2 introduces an Entity-Attention Kullback-Leibler (KL) regularizer that aligns the detector's cross-attention with scientific entities annotated for each panel, providing a stable conditioning signal that also improves localization, and applies Group Relative Policy Optimization (GRPO) with a panel-level Entity-F1 reward to optimize scientific faithfulness. We curate BioSci-Fig-Cap for in-domain supervision and contribute physics and chemistry test suites for cross-disciplinary evaluation. FigEx2 achieves 0.751 mAP@0.5:0.95 on BioSci-Fig-Cap, and outperforms Qwen3-VL-8B by 6.80 Entity-F1 on MedICaT for captioning. It also transfers zero-shot to out-of-distribution domains. The source code is available at https://github.com/Huang-AI4Medicine-Lab/FigEx2.
From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France（斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、斯特拉斯堡，法国）；Technical University of Munich（慕尼黑技术大学）；Stanford University（斯坦福大学）；DSAI, The Hong Kong Polytechnic University（香港理工大学DSAI实验室）；University of British Columbia（不列颠哥伦比亚大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；IHU Strasbourg, Strasbourg（斯特拉斯堡IHU医院，斯特拉斯堡）；Vector Institute for AI（人工智能向量研究所）；Yale University（耶鲁大学）
MatMMExtract: An Open-Source Pipeline for Panel-Level Extraction of Grounded Image-Text Pairs from Materials Science Literature
Mehta Family School of Data Science and Artificial Intelligence（梅塔家庭数据科学与人工智能学院）；Indian Institute of Technology Roorkee（印度理工学院罗奥克学院）；Department of Metallurgical and Materials Engineering（冶金与材料工程系）
Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023
Pennsylvania State University（宾夕法尼亚州立大学）；National Tsing Hua University（国立清华大学）；AllSci（奥尔赛公司）；MetaMetrics Inc.（美塔麦特里克公司）；Adobe Research（奥多比研究院）；Institute of Information Science, Academia Sinica（中央研究院资讯科学研究所）
