arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.07086cs.CL
收藏

通过大型多模态模型中的多语言提示提升文本到图像生成效果

Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models

Northeastern University(东北大学);Dalian Jiaotong University(大连交通大学);NiuTrans Research(牛 tran 研究中心)

Yongyu Mu, Hengyu Li, Junxin Wang, Xiaoxuan Zhou, Chenglong Wang, Yingfeng Luo, Qiaozhi He, Tong Xiao, Guocheng Chen, Jingbo Zhu

AI总结:

本研究针对大型多模态模型文本到图像生成中输入文本理解不足的问题,提出构建平行多语言提示的PMT2I方法,实验证明其在多类评估中性能更优,结合重排序效果更佳。

摘要:

以往针对增强大型多模态模型(LMMs)文本到图像(T2I)生成能力的研究,聚焦于丰富上下文学习(ICL)的输入空间,包括提供少量示例、优化图像描述使其更具细节性和逻辑性。然而,随着对更复杂灵活的图像描述需求增长,在ICL范式下提升模型对输入文本的理解能力仍是关键但探索不足的领域。本研究通过构建平行多语言提示拓展该方向研究,旨在利用LMMs的多语言能力:具体而言,将输入文本翻译成多种语言,同时为模型提供原文与译文。在2个LMM上、跨越3个基准的实验表明,我们的方法PMT2I在通用、组合式及细粒度评估中均取得更优性能,尤其在人类偏好对齐方面表现突出。此外,凭借生成更多样化图像的优势,PMT2I与重排序方法结合时,性能显著优于基线提示。我们的代码与平行多语言数据可在https://github.com/takagi97/PMT2I获取。

英文摘要:

Previous work on augmenting large multimodal models (LMMs) for text-to-image (T2I) generation has focused on enriching the input space of in-context learning (ICL). This includes providing a few demonstrations and optimizing image descriptions to be more detailed and logical. However, as demand for more complex and flexible image descriptions grows, enhancing comprehension of input text within the ICL paradigm remains a critical yet underexplored area. In this work, we extend this line of research by constructing parallel multilingual prompts aimed at harnessing the multilingual capabilities of LMMs. More specifically, we translate the input text into several languages and provide the models with both the original text and the translations. Experiments on two LMMs across 3 benchmarks show that our method, PMT2I, achieves superior performance in general, compositional, and fine-grained assessments, especially in human preference alignment. Additionally, with its advantage of generating more diverse images, PMT2I significantly outperforms baseline prompts when incorporated with reranking methods. Our code and parallel multilingual data can be found at https://github.com/takagi97/PMT2I.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑