arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.10141cs.CVcs.CLcs.MM

ANCHOR:基于大语言模型的文本到图像合成中的主体条件化

ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesis

Optum AI;The Pennsylvania State University(宾夕法尼亚州立大学)

Aashish Anantha Ramakrishnan, Sharon X. Huang, Dongwon Lee

AI总结:

ANCHOR通过大规模抽象式标题数据集研究文本到图像合成中多主体理解与上下文推理的缺陷,提出基于大语言模型的主体感知微调方法,提升图像-标题一致性与人类偏好对齐。

摘要:

ANCHOR通过大规模抽象式标题数据集研究文本到图像合成中多主体理解与上下文推理的缺陷,提出基于大语言模型的主体感知微调方法,提升图像-标题一致性与人类偏好对齐。

英文摘要:

Text-to-image (T2I) models have achieved remarkable progress in high-quality image synthesis, yet most benchmarks rely on simple, self-contained prompts, failing to capture the complexity of real-world captions. Human-written captions often involve multiple interacting subjects, rich contextual references, and abstractive phrasing, conditions under which current image-text encoders like CLIP struggle. To systematically study these deficiencies, we introduce ANCHOR, a large-scale dataset of 70K+ abstractive captions sourced from five major news media organizations. Analysis with ANCHOR reveals persistent failures in multi-subject understanding, context reasoning, and nuanced grounding. Motivated by these challenges, we propose Subject-Aware Fine-tuning (SAFE), which uses Large Language Models (LLMs) to extract key subjects and enhance their representation at the embedding-level. Experiments with contemporary models show that SAFE significantly improves image-caption consistency and human preference alignment, serving as a practical and scalable solution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑