One critical prerequisite for faithful text-to-image generation is the accurate understanding of text inputs. Existing methods leverage the text encoder of the CLIP model to represent input prompts. However, the pre-trained CLIP model can merely encode English with a maximum token length of 77. Moreover, the model capacity of the text encoder from CLIP is relatively limited compared to Large Language Models (LLMs), which offer multilingual input, accommodate longer context, and achieve superior text representation. In this paper, we investigate LLMs as the text encoder to improve the language understanding in text-to-image generation. Unfortunately, training text-to-image generative model with LLMs from scratch demands significant computational resources and data. To this end, we introduce a three-stage training pipeline that effectively and efficiently integrates the existing text-to-image model with LLMs. Specifically, we propose a lightweight adapter that enables fast training of the text-to-image model using the textual representations from LLMs. Extensive experiments demonstrate that our model supports not only multilingual but also longer input context with superior image generation quality.
Advancing AI-Powered Medical Image Synthesis: Insights from MedVQA-GI Challenge Using CLIP, Fine-Tuned Stable Diffusion, and Dream-Booth + LoRA
Department of Computer Science, SCMNS School, Morgan State University, Baltimore, Maryland 21251, USA;Electrical \& Computer Engineering Dept., School of Engineering, Morgan State University, Baltimore, Maryland 21251, USA
LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology(北京智能信息科技重点实验室,计算机科学与技术学院);Beijing Institute of Technology(北京理工大学);Guangdong Laboratory of Machine Perception and Intelligent Computing(广东机器感知与智能计算实验室);Shenzhen MSU-BIT University(深圳MSU-BIT大学)
Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for English to Indian Languages
Indian Institute of Technology Patna India;Indian Institute of Technology Jodhpur \& Indian Institute of Technology Patna India;Indian Institute of Technology Patna;Indian Institute of Technology Jodhpur \& Indian Institute of Technology Patna
RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models
School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);School of Computing, Clemson University(克莱姆森大学计算机学院);Washington University School of Medicine in St. Louis(圣路易斯华盛顿大学医学院);The Keck School of Medicine of the University of Southern California(南加州大学凯克医学院);Mayo Clinic(梅奥诊所);Banner Alzheimer’s Institute(班纳阿尔茨海默病研究所)