Aligning large language models (LLMs) is a central objective of post-training, often achieved through reward modeling and reinforcement learning methods. Among these, direct preference optimization (DPO) has emerged as a widely adopted technique that fine-tunes LLMs on preferred completions over less favorable ones. While most frontier LLMs do not disclose their curated preference pairs, the broader LLM community has released several open-source DPO datasets, including TuluDPO, ORPO, UltraFeedback, HelpSteer, and Code-Preference-Pairs. However, systematic comparisons remain scarce, largely due to the high computational cost and the lack of rich quality annotations, making it difficult to understand how preferences were selected, which task types they span, and how well they reflect human judgment on a per-sample level. In this work, we present the first comprehensive, data-centric analysis of popular open-source DPO corpora. We leverage the Magpie framework to annotate each sample for task category, input quality, and preference reward, a reward-model-based signal that validates the preference order without relying on human annotations. This enables a scalable, fine-grained inspection of preference quality across datasets, revealing structural and qualitative discrepancies in reward margins. Building on these insights, we systematically curate a new DPO mixture, UltraMix, that draws selectively from all five corpora while removing noisy or redundant samples. UltraMix is 30% smaller than the best-performing individual dataset yet exceeds its performance across key benchmarks. We publicly release all annotations, metadata, and our curated mixture to facilitate future research in data-centric preference optimization.
Less is More: Improving LLM Alignment via Preference Data Selection
University of Science and Technology of China(中国科学技术大学);Peking University(北京大学);Massachusetts Institute of Technology(麻省理工学院);Alibaba Cloud Computing(阿里云计算);MoE Key Lab of BIPC, University of Science and Technology of China(中国科学技术大学MoE关键实验室)
Correcting Large Language Model Behavior via Influence Function
Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Pengcheng Laboratory(鹏城实验室);National University of Defense Technology(国防科技大学);The Chinese University of Hong Kong(香港中文大学);King’s College London(伦敦国王学院)
AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters
Allen Institute for AI(艾伦人工智能研究所);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学);Emory University(埃默里大学);Carnegie Mellon University(卡内基梅隆大学)
AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset
Tsinghua University(清华大学);Harbin Institute of Technology(哈尔滨工业大学);Shanghai AI Lab(上海人工智能实验室);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alibaba Group(阿里巴巴集团)
Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation
University of Zurich, Department of Informatics(苏黎世大学信息学院);New York University, Department of Sociology(纽约大学社会学系);University of Oxford, Department of Sociology(牛津大学社会学系);Duke University, Department of Sociology, Computer Science, Political Science, and Public Policy(杜克大学社会学系、计算机科学、政治学和公共政策系);University of Amsterdam, Institute for Logic, Language and Computation (ILLC)(阿姆斯特丹大学逻辑、语言与计算研究所(ILLC))
Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Carnegie Mellon University(卡内基梅隆大学);Yale University(耶鲁大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
Value Drifts: Tracing Value Alignment During LLM Post-Training
Mila - Quebec AI Institute(魁北克人工智能研究所);McGill University(麦吉尔大学);Université de Montréal(蒙特利尔大学);ETH Zurich(苏黎世联邦理工学院);University of British Columbia(不列颠哥伦比亚大学);Vector Institute(向量研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
Multimodal Human-AI Synergy for Medical Imaging Quality Control: A Hybrid Intelligence Framework with Adaptive Dataset Curation and Closed-Loop Evaluation
Zhongda Hospital(中大医院);School of Medicine, Southeast University(东南大学医学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
Singapore University of Technology and Design(新加坡科技设计大学);National University of Singapore(国立新加坡大学);The University of Tokyo(东京大学);MiroMind(MiroMind公司);Institute for Infocomm Research, A*Star, Singapore(新加坡*星研究院信息与通信研究所)
Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
School of Software, Beihang University(北京航空航天大学软件学院);Shanghai Innovation Institute(上海创新研究院);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学)
From Normative Frameworks to Alignment Data: Constructing and Evaluating SFT and Preference Data
Qatar Computing Research Institute, HBKU(卡塔尔计算研究所,哈马德·本·哈利法大学);College of Islamic Studies, HBKU(伊斯兰研究学院,哈马德·本·哈利法大学);Ibn Haldun University(伊本·哈尔顿大学);College of Science and Engineering, HBKU(科学与工程学院,哈马德·本·哈利法大学)
LitGen: Genetic Literature Recommendation Guided by Human Explanations
Stanford University School of Medicine(斯坦福大学医学院);Stanford University(斯坦福大学);Stanford Healthcare(斯坦福医疗保健中心);Baylor College of Medicine(贝勒医学院);Chan-Zuckerberg Biohub(陈-扎克伯格生物中心)