arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.18302cs.CV
收藏

LDGen:通过大语言模型驱动的语言表示增强文本到图像合成

LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation

Li Auto Inc(理想汽车公司)

Pengzhi Li, Pengfei Yu, Zide Liu, Wei He, Xuhao Pan, Xudong Rao, Tao Wei, Wei Chen

AI总结:

LDGen通过大语言模型驱动的语言表示,结合轻量适配器与跨模态精炼器,实现高效零样本多语言文本到图像生成,提升提示遵循与美学质量。

摘要:

在本文中,我们介绍了LDGen,一种在最小化计算需求的同时将大语言模型(LLMs)集成到现有文本到图像扩散模型中的新方法。传统的文本编码器,如CLIP和T5,在多语言处理方面表现出局限性,阻碍了跨不同语言的图像生成。我们通过利用LLMs的高级能力来解决这些挑战。我们的方法采用了一种语言表示策略,应用层次化标题优化和人类指令技术来获取精确的语义信息。随后,我们引入了一个轻量级适配器和一个跨模态精炼器,以促进LLMs与图像特征之间的高效特征对齐和交互。LDGen减少了训练时间,并实现了零样本多语言图像生成。实验结果表明,我们的方法在提示遵循度和图像美学质量方面均超越了基线模型,同时无缝支持多种语言。项目页面:https://zrealli.github.io/LDGen。

英文摘要:

In this paper, we introduce LDGen, a novel method for integrating large language models (LLMs) into existing text-to-image diffusion models while minimizing computational demands. Traditional text encoders, such as CLIP and T5, exhibit limitations in multilingual processing, hindering image generation across diverse languages. We address these challenges by leveraging the advanced capabilities of LLMs. Our approach employs a language representation strategy that applies hierarchical caption optimization and human instruction techniques to derive precise semantic information,. Subsequently, we incorporate a lightweight adapter and a cross-modal refiner to facilitate efficient feature alignment and interaction between LLMs and image features. LDGen reduces training time and enables zero-shot multilingual image generation. Experimental results indicate that our method surpasses baseline models in both prompt adherence and image aesthetic quality, while seamlessly supporting multiple languages. Project page: https://zrealli.github.io/LDGen.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑