协调视觉文本理解与生成
Harmonizing Visual Text Comprehension and Generation
AI总结：
本文提出TextHarmony模型和Slide-LoRA方法，通过动态聚合模态特定与模态无关的LoRA专家，在单一模型内协调视觉文本理解与生成，以仅增加2%参数实现与模态特定微调相当的性能，并构建了DetailedTextCaps-100K数据集以增强生成能力。
摘要：
在这项工作中，我们提出了TextHarmony，这是一个统一且多功能的的多模态生成模型，擅长理解和生成视觉文本。同时生成图像和文本通常会导致性能下降，这是因为视觉和语言模态之间存在固有的不一致性。为了克服这一挑战，现有方法依赖于模态特定数据进行监督微调，从而需要不同的模型实例。我们提出了Slide-LoRA，它动态聚合模态特定和模态无关的LoRA专家，部分解耦了多模态生成空间。Slide-LoRA在单一模型实例内协调了视觉和语言的生成，从而促进了更统一的生成过程。此外，我们开发了一个高质量的图像描述数据集DetailedTextCaps-100K，该数据集使用一个复杂的闭源MLLM合成，以进一步增强视觉文本生成能力。在各种基准上的全面实验证明了所提出方法的有效性。在Slide-LoRA的支持下，TextHarmony以仅增加2%的参数实现了与模态特定微调结果相当的性能，并在视觉文本理解任务中平均提高了2.5%，在视觉文本生成任务中平均提高了4.0%。我们的工作描绘了在视觉文本领域内采用集成方法进行多模态生成的可行性，为后续研究奠定了基础。代码可在https://github.com/bytedance/TextHarmony获取。
英文摘要：
In this work, we present TextHarmony, a unified and versatile multimodal generative model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities. To overcome this challenge, existing approaches resort to modality-specific data for supervised fine-tuning, necessitating distinct model instances. We propose Slide-LoRA, which dynamically aggregates modality-specific and modality-agnostic LoRA experts, partially decoupling the multimodal generation space. Slide-LoRA harmonizes the generation of vision and language within a singular model instance, thereby facilitating a more unified generative process. Additionally, we develop a high-quality image caption dataset, DetailedTextCaps-100K, synthesized with a sophisticated closed-source MLLM to enhance visual text generation capabilities further. Comprehensive experiments across various benchmarks demonstrate the effectiveness of the proposed approach. Empowered by Slide-LoRA, TextHarmony achieves comparable performance to modality-specific fine-tuning results with only a 2% increase in parameters and shows an average improvement of 2.5% in visual text comprehension tasks and 4.0% in visual text generation tasks. Our work delineates the viability of an integrated approach to multimodal generation within the visual text domain, setting a foundation for subsequent inquiries. Code is available at https://github.com/bytedance/TextHarmony.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
