Hybrid Alignment Training for Large Language Models
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);NiuTrans Research(NiuTrans研究所);Institute of Psychology, CAS(中国科学院心理研究所)
Chenglong Wang, Hang Zhou, Kaiyan Chang, Bei Li, Yongyu Mu, Tong Xiao, Tongran Liu, Jingbo Zhu
Alignment training is crucial for enabling large language models (LLMs) to cater to human intentions and preferences. It is typically performed based on two stages with different objectives: instruction-following alignment and human-preference alignment. However, aligning LLMs with these objectives in sequence suffers from an inherent problem: the objectives may conflict, and the LLMs cannot guarantee to simultaneously align with the instructions and human preferences well. To response to these, in this work, we propose a Hybrid Alignment Training (Hbat) approach, based on alternating alignment and modified elastic weight consolidation methods. The basic idea is to alternate between different objectives during alignment training, so that better collaboration can be achieved between the two alignment tasks.We experiment with Hbat on summarization and dialogue tasks. Experimental results show that the proposed \textsc{Hbat} can significantly outperform all baselines. Notably, Hbat yields consistent performance gains over the traditional two-stage alignment training when using both proximal policy optimization and direct preference optimization.
Domain-oriented Language Pre-training with Adaptive Hybrid Masking and Optimal Transport Alignment
Rutgers, The State University of New Jersey(罗格斯大学);NEC Labs America(NEC 美国实验室);Beihang University(北京航空航天大学);The Hong Kong University of Science and Technology(香港科技大学)
Alignment at Pre-training! Towards Native Alignment for Arabic LLMs
Shenzhen Research Institue of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Shenzhen International Center for Industrial and Applied Mathematics(深圳国际工业与应用数学中心)
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
A Survey on Training-free Alignment of Large Language Models
School of Computer Science, Wuhan University, Wuhan, China(武汉大学计算机学院);Faculty of Computer Science and Technology, Qilu University of Technology, Shandong, China(青岛科技大学计算机科学与技术学院);Zhongguancun Academy, Beijing, China(中关村学院)
Value Drifts: Tracing Value Alignment During LLM Post-Training
Mila - Quebec AI Institute(魁北克人工智能研究所);McGill University(麦吉尔大学);Université de Montréal(蒙特利尔大学);ETH Zurich(苏黎世联邦理工学院);University of British Columbia(不列颠哥伦比亚大学);Vector Institute(向量研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
Université de Montréal(蒙特利尔大学);Mila – Quebec AI Institute(魁北克人工智能研究院);Chandar Research Lab(Chandar研究实验室);IBM Research(IBM研究院);Fujitsu Research(富士通研究院);Polytechnique Montréal(蒙特利尔理工学院)
TACR: A Table-alignment-based Cell-selection and Reasoning Model for Hybrid Question-Answering
Tokyo Institute of Technology(东京工业大学);Nanyang Technological University(南洋理工大学);Microsoft Research Asia(微软亚洲研究院);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training
Department of Computer Science and Engineering, Korea University(韩国大学计算机科学与工程系);Department of English Language and Literature, Kyungpook National University(庆北国立大学英语语言文学系)
Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment
Alibaba Group(阿里巴巴集团);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学)
SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint
National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);Nanjing University of Science and Technology(南京理工大学);Microsoft Research Asia(微软亚洲研究院);Zhejiang University(浙江大学)
Phonetic forced alignment for low-resource language varieties: Model training and evaluation on Chengdu Mandarin
Shanghai Jiao Tong University(上海交通大学);City University of Hong Kong(香港城市大学);The Hong Kong Polytechnic University(香港理工大学);Beijing Foreign Studies University(北京外国语大学)