World to Code：通过自指示组合式描述与过滤实现多模态数据生成
World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering
AI总结：
本文提出World to Code（W2C）多模态数据构建流程，利用VLM自指示提取跨模态信息并以Python代码形式组织输出，通过一致性过滤提升数据质量，在多个视觉问答与定位基准上验证了有效性。
摘要：
视觉语言模型（VLMs）的最新进展以及高质量多模态对齐数据的稀缺，激发了大量关于合成VLM数据生成的研究。VLM数据构建中的传统惯例是使用描述和OCR方面的专家组合，或使用更强的VLM API以及昂贵的人工标注。在本文中，我们提出了World to Code（W2C），这是一个精心构建的多模态数据构建流程，将最终生成输出组织为Python代码格式。该流程利用VLM自身通过不同的提示提取跨模态信息，并通过一致性过滤策略再次过滤生成输出。实验通过在不同VLM上改进各种现有的视觉问答和视觉定位基准，证明了W2C的高质量。进一步分析还表明，VLM的新代码解析能力比常用的详细描述能力表现出更好的跨模态等价性。我们的代码可在 https://github.com/foundation-multimodal-models/World2Code 获取。
英文摘要：
Recent advances in Vision-Language Models (VLMs) and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation. The conventional norm in VLM data construction uses a mixture of specialists in caption and OCR, or stronger VLM APIs and expensive human annotation. In this paper, we present World to Code (W2C), a meticulously curated multi-modal data construction pipeline that organizes the final generation output into a Python code format. The pipeline leverages the VLM itself to extract cross-modal information via different prompts and filter the generated outputs again via a consistency filtering strategy. Experiments have demonstrated the high quality of W2C by improving various existing visual question answering and visual grounding benchmarks across different VLMs. Further analysis also demonstrates that the new code parsing ability of VLMs presents better cross-modal equivalence than the commonly used detail caption ability. Our code is available at https://github.com/foundation-multimodal-models/World2Code.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
