Image2Struct：面向视觉-语言模型的结构提取基准测试
Image2Struct: Benchmarking Structure Extraction for Vision-Language Models
AI总结：
本研究推出自动化基准Image2Struct，通过往返渲染评估法，覆盖三大领域、结合五种图像指标，可有效区分不同视觉-语言模型的图像结构提取能力，且任务难度梯度明确。
摘要：
我们推出Image2Struct，这是一款用于评估视觉-语言模型（VLM）从图像中提取结构能力的基准测试。该基准具备三大特点：1）覆盖真实世界应用场景；2）完全自动化，无需人工判断；3）基于可更新的新鲜数据流构建。在Image2Struct测试框架中，系统会提示VLM根据输入图像（如网页截图）生成其底层结构（如LaTeX代码或HTML），随后将生成的结构渲染为输出图像（如渲染后的网页），再与输入图像对比得出相似度评分。这种“往返式”评估方法，让我们能够针对存在多种有效结构的任务，对VLM开展定量评估。我们搭建了一套流水线，运行时可从活跃的在线社区下载新鲜数据，全程无需人工干预即可完成VLM评估。基准涵盖三个领域（网页、LaTeX、乐谱），并采用五种图像度量指标（像素相似度、Inception向量间余弦相似度、学习感知图像块相似度、结构相似性指数测度、推土机相似度），可高效自动完成图像对的对比。我们用14款主流VLM对Image2Struct进行了测试，发现各模型得分差异显著，说明该基准能够区分不同VLM的性能表现。此外，不同领域的最优得分差距较大（如乐谱任务最优分为0.402，而LaTeX公式任务最优分为0.830），表明Image2Struct包含难度各异的任务。为保证透明度，我们已在https://crfm.stanford.edu/helm/image2struct/v1.0.1/ 发布全部测试结果。
英文摘要：
We introduce Image2Struct, a benchmark to evaluate vision-language models (VLMs) on extracting structure from images. Our benchmark 1) captures real-world use cases, 2) is fully automatic and does not require human judgment, and 3) is based on a renewable stream of fresh data. In Image2Struct, VLMs are prompted to generate the underlying structure (e.g., LaTeX code or HTML) from an input image (e.g., webpage screenshot). The structure is then rendered to produce an output image (e.g., rendered webpage), which is compared against the input image to produce a similarity score. This round-trip evaluation allows us to quantitatively evaluate VLMs on tasks with multiple valid structures. We create a pipeline that downloads fresh data from active online communities upon execution and evaluates the VLMs without human intervention. We introduce three domains (Webpages, LaTeX, and Musical Scores) and use five image metrics (pixel similarity, cosine similarity between the Inception vectors, learned perceptual image patch similarity, structural similarity index measure, and earth mover similarity) that allow efficient and automatic comparison between pairs of images. We evaluate Image2Struct on 14 prominent VLMs and find that scores vary widely, indicating that Image2Struct can differentiate between the performances of different VLMs. Additionally, the best score varies considerably across domains (e.g., 0.402 on sheet music vs. 0.830 on LaTeX equations), indicating that Image2Struct contains tasks of varying difficulty. For transparency, we release the full results at https://crfm.stanford.edu/helm/image2struct/v1.0.1/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
