arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.03463cs.CVcs.AIcs.CL
收藏

文本打印图像:为以文本为中心的大型视觉-语言模型训练弥合图像-文本模态差距

Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models

Turing Inc.(图灵公司);Institute of Science Tokyo(东京科学研究院);The University of Tokyo(东京大学);Tohoku University(东北大学)

Shojiro Yamabe, Futa Waseda, Daiki Shiono, Tsubasa Takahashi

AI总结:

本研究提出文本打印图像(TPI)技术,通过生成合成图像弥合图像-文本模态差距,提升以文本为中心的大型视觉-语言模型训练效果。

摘要:

近期大型视觉-语言模型(LVLMs)已被应用于多样化的视觉问答(VQA)任务。然而,实现实用性能通常需要任务特定的微调,使用大量的图像-文本对,这需要大量的数据收集,成本高昂。在本工作中,我们研究了以文本为中心的训练,这是一种设置,其中只有文本描述可用,而没有真实的图像提供,作为一种低成本数据扩展的范式。与图像不同,其收集往往受到隐私约束和在小众领域中的稀缺性限制,而文本则广泛可用。此外,文本易于编辑,能够通过LLMs自动多样化和扩展,以最小的人力成本。尽管这在可扩展性和成本方面明显优于图像收集,但仅在文本上训练而不使用图像仍然在VQA任务上获得有限的收益,因为存在图像-文本模态差距。为了解决这个问题,我们提出了文本打印图像(TPI),它通过直接在纯白画布上渲染给定的文本描述来生成合成图像。这种简单的渲染将文本投射到图像模态中,并且可以以低成本整合到任意现有的LVLM训练流程中。此外,TPI保留了文本的语义,而文本到图像模型往往无法做到。在四个模型和七个基准测试中,我们的系统实验表明,TPI比由扩散模型生成的合成图像更有效地实现了以文本为中心的训练。我们进一步探索TPI作为低成本数据增强策略,并展示了其实际用途。总体而言,我们的发现突显了以文本为中心的训练的巨大潜力,并更广泛地指明了为LVLMs实现完全自动化数据生成的道路。

英文摘要:

Recent large vision-language models (LVLMs) have been applied to diverse VQA tasks. However, achieving practical performance typically requires task-specific fine-tuning with large numbers of image-text pairs, which are costly to collect. In this work, we study text-centric training, a setting where only textual descriptions are available and no real images are provided, as a paradigm for low-cost data scaling. Unlike images, whose collection is often restricted by privacy constraints and scarcity in niche domains, text is widely available. Moreover, text is easily editable, enabling automatic diversification and expansion with LLMs at minimal human effort. While this offers clear advantages over image collection in terms of scalability and cost, training on raw text without images still yields limited gains on VQA tasks because of the image-text modality gap. To address this issue, we propose a Text-Printed Image (TPI), which generates synthetic images by directly rendering the given textual description on a plain white canvas. This simple rendering projects text into the image modality and can be integrated into arbitrary existing LVLM training pipelines at low cost. Moreover, TPI preserves the semantics of the text, whereas text-to-image models often fail to do. Across four models and seven benchmarks, our systematic experiments show that TPI enables more effective text-centric training than synthetic images generated by a diffusion model. We further explore TPI as a low-cost data-augmentation strategy and demonstrate its practical utility. Overall, our findings highlight the significant potential of text-centric training and, more broadly, chart a path toward fully automated data generation for LVLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑