Direct Preference Optimization (DPO) has been successfully used to align large language models (LLMs) according to human preferences, and more recently it has also been applied to improving the quality of text-to-image diffusion models. However, DPO-based methods such as SPO, Diffusion-DPO, and D3PO are highly susceptible to overfitting and reward hacking, especially when the generative model is optimized to fit out-of-distribution during prolonged training. To overcome these challenges and stabilize the training of diffusion models, we introduce a self-entropy regularization mechanism in reinforcement learning from human feedback. This enhancement improves DPO training by encouraging broader exploration and greater robustness. Our regularization technique effectively mitigates reward hacking, leading to improved stability and enhanced image quality across the latent space. Extensive experiments demonstrate that integrating human feedback with self-entropy regularization can significantly boost image diversity and specificity, achieving state-of-the-art results on key image generation metrics.
Curriculum-DPO++: Direct Preference Optimization via Data and Model Curricula for Text-to-Image Generation
Department of Computer Science, University of Bucharest（布加勒斯特大学计算机科学系）；University of Trento（特伦托大学）；Center for Research in Computer Vision (CRCV), University of Central Florida（中央佛罗里达大学计算机视觉研究中心）
Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Xi’an Jiaotong University（西安交通大学）；The Chinese University of Hong Kong（香港中文大学）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）
DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Shenzhen Loop Area Institute（深圳河套学院）；Shanghai Research Institute for Intelligent Autonomous Systems（上海智能自主系统研究院）
AVC-DPO: Aligned Video Captioning via Direct Preference Optimization
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；College of Artificial Intelligence, Nankai University（南开大学人工智能学院）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）
P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology（广东省计算人工智能模型与认知智能重点实验室，计算机科学与工程学院，华南理工大学）；Pazhou Lab, Guangzhou, China（琶洲实验室，广州，中国）；Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China（教育部健康智能感知与并行数字人工程研究中心，广州，中国）
Region-Normalized DPO for Medical Image Segmentation under Noisy Judges
Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Essen, Germany（人工智能医学研究所（IKIM）、埃森大学医院（AöR））；Department of Nuclear Medicine, University Hospital Essen (AöR), Essen, Germany（核医学系、埃森大学医院（AöR））；Department of Physics, TU Dortmund, Dortmund, Germany（物理系、多特蒙德技术大学）；Heidelberg University Hospital, Heidelberg, Germany（海德堡大学医院）
Beyond Binary Preferences: Graded Preference Optimization for Limb-Motion Captioning
Zhejiang University（浙江大学）；Alibaba Group（阿里巴巴集团）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）
Benchmarking Direct Preference Optimization for Medical Large Vision-Language Models
Korea University（韩国大学）；AIGEN Sciences（AIGEN公司）；Hanyang University College of Medicine（翰林大学医学院）；Asan Medical Center, University of Ulsan College of Medicine（釜山大学医学院阿桑医院）；Yale University（耶鲁大学）
