arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16807cs.LGcs.CLcs.CV
收藏

超越点赞/点踩:解开文本到图像生成中细粒度反馈的挑战

Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation

University of Cambridge(剑桥大学);Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院);Field AI;University of Washington(华盛顿大学);University of Tennessee(田纳西大学);Korean Advanced Institute of Science and Technology(韩国高等科学技术院);University of California San Diego(加州大学圣地亚哥分校);The Alan Turing Institute(艾伦·图灵研究所);ServiceNow
Katherine M. Collins, Najoung Kim, Yonatan Bitton, Verena Rieser, Shayegan Omidshafiei, Yushi Hu, Sherol Chen, Senjuti Dutta, Minsuk Chang, Kimin Lee, Youwei Li… 展开作者

Katherine M. Collins, Najoung Kim, Yonatan Bitton, Verena Rieser, Shayegan Omidshafiei, Yushi Hu, Sherol Chen, Senjuti Dutta, Minsuk Chang, Kimin Lee, Youwei Liang, Georgina Evans, Sahil Singla, Gang Li, Adrian Weller, Junfeng He, Deepak Ramachandran, Krishnamurthy Dj Dvijotham

AI总结:

本文研究文本到图像生成中细粒度反馈相比粗粒度反馈的有效性,发现其优越性并非自动实现,并揭示了模型选择与反馈类型交互带来的复杂性,呼吁重新评估细粒度反馈的实用性与建模方法。

摘要:

人类反馈在学习和优化文本到图像生成的奖励模型中起着关键作用,但学习准确奖励函数所需的反馈最佳形式尚未得到确凿确立。本文研究了细粒度反馈的有效性,该反馈捕捉了图像质量和提示对齐中的细微差别,并将其与传统粗粒度反馈(例如,点赞/点差或在一组选项之间进行排名)进行了比较。尽管细粒度反馈具有前景,特别是对于迎合多样化社会偏好的系统,但我们表明,证明其优于粗粒度反馈并非自动实现的。通过对真实和合成偏好数据的实验,我们揭示了由于模型选择、反馈类型以及人类判断与计算解释之间对齐的相互作用,构建有效模型的复杂性。我们识别了在引出和利用细粒度反馈方面的关键挑战,促使人们重新评估其假设的益处和实用性。我们的发现——例如,在某些设置中,对于固定预算,细粒度反馈可能导致更差的模型;然而,在具有已知属性的控制设置中,细粒度奖励确实可以更有帮助——呼吁仔细考虑反馈属性,并可能召唤新的建模方法,以适当地解锁细粒度反馈在野外的潜在价值。

英文摘要:

Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for learning an accurate reward function has not been conclusively established. This paper investigates the effectiveness of fine-grained feedback which captures nuanced distinctions in image quality and prompt-alignment, compared to traditional coarse-grained feedback (for example, thumbs up/down or ranking between a set of options). While fine-grained feedback holds promise, particularly for systems catering to diverse societal preferences, we show that demonstrating its superiority to coarse-grained feedback is not automatic. Through experiments on real and synthetic preference data, we surface the complexities of building effective models due to the interplay of model choice, feedback type, and the alignment between human judgment and computational interpretation. We identify key challenges in eliciting and utilizing fine-grained feedback, prompting a reassessment of its assumed benefits and practicality. Our findings -- e.g., that fine-grained feedback can lead to worse models for a fixed budget, in some settings; however, in controlled settings with known attributes, fine grained rewards can indeed be more helpful -- call for careful consideration of feedback attributes and potentially beckon novel modeling approaches to appropriately unlock the potential value of fine-grained feedback in-the-wild.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑