AutoRubric-T2I: 一种用于文本到图像对齐的鲁棒基于规则的奖励模型
AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignment
AI总结：
本文提出AutoRubric-T2I，一种首个用于文本到图像生成的规则学习框架，通过自动合成和选择显式规则来指导视觉语言模型（VLM）法官。该方法通过合成偏好对的推理轨迹生成候选规则，并利用VLM法官在每种规则下对配对图像进行评分，产生配对规则评分差异用于偏好学习。通过ℓ1正则化逻辑回归精简器去除噪声和冗余规则，从而在少量标注偏好数据下生成高质量、可解释的奖励信号，并在多个图像奖励基准测试中优于现有奖励模型基线。
摘要：
将文本到图像（T2I）生成模型与人类偏好对齐越来越依赖于图像奖励模型，这些模型根据提示对齐和感知质量对生成图像进行评分或排序。现有的奖励模型通常在大规模人类偏好语料上训练为Bradley-Terry（BT）偏好模型，这使得训练成本高、适应困难且评估标准不透明。同时，视觉语言模型（VLM）法官可以通过文本评分规则提供更细致的评估，但其手动设计或启发式生成的评分规则可能无法可靠地反映人类偏好。在本文中，我们提出AutoRubric-T2I，这是首个用于T2I的规则学习框架，能够自动合成和选择显式规则以指导VLM法官。AutoRubric-T2I首先通过合成偏好对的推理轨迹生成候选规则，然后利用VLM法官在每种规则下对配对图像进行评分，产生配对规则评分差异用于偏好学习。为了去除噪声和冗余规则，我们进一步采用ℓ1正则化逻辑回归精简器，选择Top-N最判别性的规则。广泛评估表明，AutoRubric-T2I在使用不到0.01%的标注偏好数据的情况下，能够生成高质量、可解释的奖励信号，大幅减少了大规模奖励模型训练的需求。在图像奖励基准如MMRB2上，AutoRubric-T2I优于强奖励模型基线。我们进一步验证AutoRubric-T2I作为强化学习奖励在下游T2I任务中的效果，包括TIIF和UniGenBench++，其中它通过流-GRPO管道在扩散模型上提升了生成质量，优于标量奖励模型。
英文摘要：
Aligning Text-to-Image (T2I) generation models with human preferences increasingly relies on image reward models that score or rank generated images according to prompt alignment and perceptual quality. Existing reward models are commonly trained as Bradley-Terry (BT) preference models on large-scale human preference corpora, making them costly to train, difficult to adapt, and opaque in their evaluation criteria. Meanwhile, Vision-Language Model (VLM) judges can provide more fine-grained assessments through textual rubrics, but their manually designed or heuristically generated scoring rules may fail to reliably reflect human preferences. In this paper, we propose AutoRubric-T2I, the first rubric learning framework in T2I that automatically synthesizes and selects explicit rubrics for guiding VLM judges. AutoRubric-T2I first synthesizes reasoning traces from preference pairs into candidate rubrics, then uses a VLM judge to score paired images under each rubric, producing pairwise rubric-score differences for preference learning. To remove noisy and redundant rules, we further employ a $\ell_1$-Regularized Logistic Regression Refiner, which selects the Top-$N$ most discriminative rubrics. Extensive evaluations show that AutoRubric-T2I produces high-quality, interpretable reward signals using less than 0.01% of the annotated preference data, substantially reducing the need for large-scale reward-model training. On image reward benchmarks such as MMRB2, AutoRubric-T2I outperforms strong reward model baselines. We further validate AutoRubric-T2I as an RL reward on downstream T2I tasks, including TIIF and UniGenBench++, where it improves generation quality over scalar reward models using the Flow-GRPO pipeline on diffusion models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
