Achieving balanced alignment of large language models (LLMs) in terms of Helpfulness, Honesty, and Harmlessness (3H optimization) constitutes a cornerstone of responsible AI. Existing methods like data mixture strategies face limitations, including heavy reliance on expert knowledge and conflicting optimization signals. While model merging offers parameter-level conflict-resolution strategies through integrating specialized models' parameters, its potential for 3H optimization remains underexplored. This paper systematically compares the effectiveness of model merging and data mixture methods in constructing 3H-aligned LLMs for the first time, revealing previously overlooked collaborative and conflict relationships among the 3H dimensions and discussing the advantages and drawbacks of data mixture (\textit{data-level}) and model merging (\textit{parameter-level}) methods in mitigating the conflict for balanced 3H optimization. Specially, we propose a novel \textbf{R}eweighting \textbf{E}nhanced task \textbf{S}ingular \textbf{M}erging method, \textbf{RESM}, through outlier weighting and sparsity-aware rank selection strategies to address the challenges of preference noise accumulation and layer sparsity adaptation inherent in 3H-aligned LLM merging. Extensive evaluations can verify the effectiveness and robustness of RESM compared to previous data mixture (2\%-5\% gain) and model merging (1\%-3\% gain) methods in achieving balanced LLM alignment. We release our models through \href{https://huggingface.co/Jinluan}{3H\_Merging} for further investigations.
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
National Taiwan University（国立台湾大学）；Texas A&M University（德克萨斯A&M大学）；Appier AI Research（Appier人工智能研究院）；Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）
HonestLLM: Toward an Honest and Helpful Large Language Model
MBZUAI（Mohamed bin Zayed 人工智能大学）；Huazhong University of Science and Technology（华中科技大学）；University of Notre Dame（圣母大学）；University of Washington（华盛顿大学）；Peking University（北京大学）；Lehigh University（理海大学）
BeHonest: Benchmarking Honesty in Large Language Models
Shanghai Jiao Tong University（上海交通大学）；Carnegie Mellon University（卡内基梅隆大学）；Fudan University（复旦大学）；Shanghai AI Laboratory（上海人工智能实验室）；Generative AI Research Lab (GAIR)（生成式人工智能研究实验室（GAIR））
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Tencent Inc.（腾讯公司）
Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院）；State Key Lab of AI Safety（人工智能安全国家重点实验室）；University of Chinese Academy of Sciences, CAS（中国科学院大学）；Zhejiang University（浙江大学）
AI Alignment through a Game-theoretic Lens: A Survey
James Cook University（詹姆斯库克大学）；Western Sydney University（西悉尼大学）；Adelaide University（阿德莱德大学）；CSIRO（联邦科学与工业研究组织）；The University of Osaka（大阪大学）；Nanjing University of Science and Technology（南京理工大学）；Macquarie University（麦考瑞大学）；La Trobe University（拉筹伯大学）
