We study preference alignment for image inpainting. Rather than proposing yet another method, we revisit the problem from first principles and reassess its core challenges. We adopt the widely used direct preference optimization framework and construct preference training data with publicly available reward models. Our empirical study spans nine reward models, two benchmarks, and two baseline inpainting models that differ in architecture and generative mechanism. Our main findings are: (1) Most reward models provide valid signals for preference data construction, although some are unreliable as evaluators. (2) Across models and benchmarks, preference data exhibits consistent trends under both candidate and sample scaling. (3) Reward models display pronounced biases--particularly in brightness, composition, and color scheme--that make them prone to inducing reward hacking. (4) A simple ensemble of reward models mitigates such biases and yields robust, generalizable performance. {\color{rebuttal_blue}(5) Preference alignment is transferable to the object removal task, where the goal shifts from open-ended creative generation to coherent background completion. (6) Further analysis reveals that a calibrated ensemble method further mitigates hacking and improves robustness.} Without modifying model architectures or introducing additional datasets, our models substantially outperform prior state-of-the-art models on standard metrics, large vision-language model evaluations, and human assessments. Our code is available at: https://github.com/shenytzzz/Follow-Your-Preference.
Rethinking Reward Signals in Video GRPO: When Scores Become Targets
University of Science and Technology of China（中国科学技术大学）；Southeast University（东南大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究院（TeleAI），中国电信）
Guidance Matters: Rethinking the Evaluation Pitfall for Text-to-Image Generation
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Cogniser Information Technology（Cogniser信息科技）
