arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.12652cs.CVcs.AI
收藏

PromptEcho:基于视觉语言模型的无标注奖励用于文本到图像强化学习

PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning

Alibaba Group(阿里巴巴集团);Zhejiang University(浙江大学)

Jinlong Liu, Wanggui He, Peng Zhang, Mushui Liu, Hao Jiang, Pipei Huang

AI总结:

PromptEcho通过冻结的视觉语言模型计算token级交叉熵损失,无需标注或训练奖励模型,提升文本到图像模型的提示跟随能力,实验显示在多个基准上取得显著提升。

摘要:

强化学习(RL)可以提升文本到图像(T2I)模型的提示跟随能力,但获取高质量奖励信号仍具挑战:CLIP分数过于粗粒度,而基于视觉语言模型的奖励模型(如RewardDance)需要昂贵的人工标注偏好数据和额外微调。我们提出PromptEcho,一种无需标注和无需奖励模型训练的奖励构建方法。给定生成的图像和引导查询,PromptEcho计算冻结的视觉语言模型与原始提示的token级交叉熵损失,直接提取图像-文本对齐知识,该知识在视觉语言模型预训练期间编码。奖励是确定性的、计算高效且随着更强的开源视觉语言模型的可用性而自动提升。为评估,我们开发了DenseAlignBench,一个概念丰富的密集描述基准,用于严格测试提示跟随能力。实验结果表明,PromptEcho在两个最先进的T2I模型(Z-Image和QwenImage-2512)上在DenseAlignBench上取得显著提升(+26.8pp / +16.2pp净胜率),并在GenEval、DPG-Bench和TIIFBench上取得一致提升,无需任何任务特定训练。消融研究证实,PromptEcho在相同VLM下全面优于基于推理的评分,并且奖励质量随VLM大小而增加。我们将开源训练好的模型和DenseAlignBench。

英文摘要:

Reinforcement learning (RL) can improve the prompt following capability of text-to-image (T2I) models, yet obtaining high-quality reward signals remains challenging: CLIP Score is too coarse-grained, while VLM-based reward models (e.g., RewardDance) require costly human-annotated preference data and additional fine-tuning. We propose PromptEcho, a reward construction method that requires \emph{no} annotation and \emph{no} reward model training. Given a generated image and a guiding query, PromptEcho computes the token-level cross-entropy loss of a frozen VLM with the original prompt as the label, directly extracting the image-text alignment knowledge encoded during VLM pretraining. The reward is deterministic, computationally efficient, and improves automatically as stronger open-source VLMs become available. For evaluation, we develop DenseAlignBench, a benchmark of concept-rich dense captions for rigorously testing prompt following capability. Experimental results on two state-of-the-art T2I models (Z-Image and QwenImage-2512) demonstrate that PromptEcho achieves substantial improvements on DenseAlignBench (+26.8pp / +16.2pp net win rate), along with consistent gains on GenEval, DPG-Bench, and TIIFBench without any task-specific training. Ablation studies confirm that PromptEcho comprehensively outperforms inference-based scoring with the same VLM, and that reward quality scales with VLM size. We will open-source the trained models and the DenseAlignBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑