Direct alignment methods are increasingly used to align large language models (LLMs) with human preferences. However, many real-world alignment problems involve multiple conflicting objectives, where naive aggregation of preferences can lead to unstable training and poor trade-offs. In particular, weighted loss methods may fail to identify update directions that simultaneously improve all objectives, and existing multi-objective approaches often rely on explicit reward models, introducing additional complexity and distorting user-specified preferences. The contributions of this paper are two-fold. First, we propose a Reward-free Alignment framework for Conflicted Objectives (RACO) that directly leverages pairwise preference data and resolves gradient conflicts via a novel clipped variant of conflict-averse gradient descent. We provide convergence guarantees to Pareto-critical points that respect user-specified objective weights, and further show that clipping can strictly improve convergence rate in the two-objective setting. Second, we improve our method using some heuristics and conduct experiments to demonstrate the compatibility of the proposed framework for LLM alignment. Both qualitative and quantitative evaluations on multi-objective summarization and safety alignment tasks across multiple LLM families (Qwen 3, Llama 3, Gemma 3) show that our method consistently achieves better Pareto trade-offs compared to existing multi-objective alignment baselines.
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Microsoft Research（微软研究院）；University of Michigan（密歇根大学）；Shanghai Jiao Tong University（上海交通大学）；CUHKSZ（香港中文大学（深圳））；THU（清华大学）
Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
Chinese University of Hong Kong（香港中文大学）；Microsoft Research Asia（微软亚洲研究院）；Peking University（北京大学）；Hong Kong University of Science and Technology（香港科技大学）
ALaRM: Align Language Models via Hierarchical Rewards Modeling
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）；School of Data Science, Fudan University（复旦大学数据科学学院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Research Institute of Intelligent and Complex Systems, Fudan University（复旦大学智能与复杂系统研究院）
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Qwen LLM Application Team, Alibaba（阿里巴巴Qwen大模型应用团队）；Stanford University（斯坦福大学）
Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China（计算与智能研究所，哈尔滨工业大学，深圳，中国）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室，深圳，中国）
One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
The Hong Kong Polytechnic University（香港理工大学）；Huawei Technologies Ltd.（华为技术有限公司）；National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）
The Reward Model Selection Crisis in Personalized Alignment
School of Informatics, University of Edinburgh（信息学院，爱丁堡大学）；Agency for Science, Technology and Research (A*STAR)（科技研究局）；Samsung AI Research Center（三星人工智能研究中心）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
MPO: Multilingual Safety Alignment via Reward Gap Optimization
Harbin Institute of Technology（哈尔滨工业大学）；Singapore Management University（新加坡管理学院）；Monash University（墨尔本大学）；Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（新加坡国立大学）
