arXiv 2504.070892025-06-02cs.CVcs.CL

OmniCaptioner：一个统一它们的字幕生成器

OmniCaptioner: One Captioner to Rule Them All

Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）

Yiting Lu, Jiakang Yuan, Zhen Li, Shitian Zhao, Qi Qin, Xinyue Li, Le Zhuo, Licheng Wen, Dongyang Liu, Yuewen Cao, Xiangchao Yan, Xin Li, Tianshuo Peng, Shufei … 展开作者

Yiting Lu, Jiakang Yuan, Zhen Li, Shitian Zhao, Qi Qin, Xinyue Li, Le Zhuo, Licheng Wen, Dongyang Liu, Yuewen Cao, Xiangchao Yan, Xin Li, Tianshuo Peng, Shufei Zhang, Botian Shi, Tao Chen, Zhibo Chen, Lei Bai, Peng Gao, Bo Zhang

备注：: More visualizations on Homepage: https://alpha-innovator.github.io/OmniCaptioner-project-page and Official code: https://github.com/Alpha-Innovator/OmniCaptioner

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出通用视觉字幕生成框架OmniCaptioner，统一处理自然图像、视觉文本和结构化视觉，生成细粒度文本描述，以增强LLM多模态推理、改进图像生成并提升SFT效率。

摘要：

我们提出了OmniCaptioner，一个通用的视觉字幕生成框架，用于在各种视觉域中生成细粒度的文本描述。与以往局限于特定图像类型（例如自然图像或几何视觉）的方法不同，我们的框架为自然图像、视觉文本（例如海报、UI、教科书）和结构化视觉（例如文档、表格、图表）的字幕生成提供了统一解决方案。通过将低级像素信息转换为语义丰富的文本表示，我们的框架弥合了视觉和文本模态之间的差距。我们的结果突出了三个关键优势：（i）借助LLM增强视觉推理，其中视觉模态的长上下文字幕使LLM，特别是DeepSeek-R1系列，能够在多模态场景中有效推理；（ii）改进图像生成，其中详细字幕改进了文本到图像生成和图像转换等任务；以及（iii）高效的监督微调（SFT），这使得用更少的数据实现更快的收敛成为可能。我们相信OmniCaptioner的通用性和适应性可以为弥合语言和视觉模态之间的差距提供新的视角。

英文摘要：

We propose OmniCaptioner, a versatile visual captioning framework for generating fine-grained textual descriptions across a wide variety of visual domains. Unlike prior methods limited to specific image types (e.g., natural images or geometric visuals), our framework provides a unified solution for captioning natural images, visual text (e.g., posters, UIs, textbooks), and structured visuals (e.g., documents, tables, charts). By converting low-level pixel information into semantically rich textual representations, our framework bridges the gap between visual and textual modalities. Our results highlight three key advantages: (i) Enhanced Visual Reasoning with LLMs, where long-context captions of visual modalities empower LLMs, particularly the DeepSeek-R1 series, to reason effectively in multimodal scenarios; (ii) Improved Image Generation, where detailed captions improve tasks like text-to-image generation and image transformation; and (iii) Efficient Supervised Fine-Tuning (SFT), which enables faster convergence with less data. We believe the versatility and adaptability of OmniCaptioner can offer a new perspective for bridging the gap between language and visual modalities.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。