每个部分都重要：基于多模态大语言模型的科学图表完整性验证
Every Part Matters: Integrity Verification of Scientific Figures Based on Multimodal Large Language Models
AI总结：
本文提出图表完整性验证新任务，构建大规模数据集Figure-seg，并创新EPM框架利用多模态大语言模型通过类比推理提升文本与图表的细粒度对齐与验证，显著改进了复杂科学图表的精确分析。
摘要：
本文解决科学图表解释中的一个关键问题：文本与图表的细粒度对齐。它超越了先前主要处理简单的、数据驱动的可视化（如柱状图和饼状图）且仅通过描述和分类提供对图表基本理解的研究。我们引入了一项新任务——图表完整性验证，旨在评估技术将文本知识与科学图表中视觉元素对齐的精确度。为此，我们开发了一种半自动化方法来构建大规模数据集 Figure-seg，专门用于此任务。此外，我们提出了一种创新框架 Every Part Matters (EPM)，该框架利用多模态大语言模型（MLLMs），不仅能逐步改善文本与图表完整性的对齐和验证，还能通过类比推理增强完整性。我们的全面实验表明，这些创新显著改进了现有方法，能够对复杂科学图表进行更精确、更彻底的分析。这一进展不仅加深了我们对多模态技术的理解，还激发了需要准确解释复杂视觉数据的各个领域的进一步研究与实际应用。
英文摘要：
This paper tackles a key issue in the interpretation of scientific figures: the fine-grained alignment of text and figures. It advances beyond prior research that primarily dealt with straightforward, data-driven visualizations such as bar and pie charts and only offered a basic understanding of diagrams through captioning and classification. We introduce a novel task, Figure Integrity Verification, designed to evaluate the precision of technologies in aligning textual knowledge with visual elements in scientific figures. To support this, we develop a semi-automated method for constructing a large-scale dataset, Figure-seg, specifically designed for this task. Additionally, we propose an innovative framework, Every Part Matters (EPM), which leverages Multimodal Large Language Models (MLLMs) to not only incrementally improve the alignment and verification of text-figure integrity but also enhance integrity through analogical reasoning. Our comprehensive experiments show that these innovations substantially improve upon existing methods, allowing for more precise and thorough analysis of complex scientific figures. This progress not only enhances our understanding of multimodal technologies but also stimulates further research and practical applications across fields requiring the accurate interpretation of complex visual data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
