Reinforcement learning from human feedback (RLHF) is a vital strategy for enhancing model capability in language models. However, annotating preference data for RLHF is a resource-intensive and creativity-demanding process, while existing automatic generation methods face limitations in data diversity and quality. In response, we present Safer-Instruct, a novel pipeline for automatically constructing large-scale preference data. Our approach leverages reversed instruction tuning, instruction induction, and expert model evaluation to efficiently generate high-quality preference data without human annotators. To verify the effectiveness of Safer-Instruct, we apply the pipeline to construct a safety preference dataset as a case study. Finetuning an Alpaca model on this synthetic dataset not only demonstrates improved harmlessness but also outperforms models fine-tuned on human-annotated safety preference data, all the while maintaining a competitive edge in downstream tasks. Importantly, our Safer-Instruct framework is versatile and can be applied to generate preference data across various domains, extending its utility beyond safety preferences. It addresses the challenges in preference data acquisition and advances the development of more capable and responsible AI systems. For dataset and code implementation, see https://github.com/uscnlp-lime/safer-instruct
Self-Instruct: Aligning Language Models with Self-Generated Instructions
University of Washington（华盛顿大学）；Tehran Polytechnic（德黑兰理工大学）；Arizona State University（亚利桑那州立大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Allen Institute for AI（艾伦人工智能研究所）
X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions
State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；Institute of Automation, CAS（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
InstructProtein: Aligning Human and Protein Language via Knowledge Instruction
Zhejiang University（浙江大学）；ZJU-Hangzhou Global Scientific and Technological Innovation Center（浙江大学杭州国际科创中心）；AZFT Joint Lab for Knowledge Engine（AZFT知识引擎联合实验室）
Tool Retrieval Bridge: Aligning Vague Instructions with Retriever Preferences via Bridge Model
Department of Gastroenterology, Renmin Hospital, Wuhan University（武汉大学人民医院消化内科）；School of Computer Science, Wuhan University（武汉大学计算机学院）；State Grid Corporation（国家电网公司）
InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China（香港理工大学计算机系）；Department of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）
Evaluating and Aligning CodeLLMs on Human Preference
Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
OpenAI（开放人工智能研究中心）；Anthropic（Anthropic公司）；Alignment Research Center（对齐研究中心）
Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we u…
Reinforcement learning has seen wide success in finetuning large language models to better align with instructions via human feedback. The so-called algorithm, Reinforcement Learning with Human Feedback (RLHF) demonstrates impressive performance on the GPT series models. However, the underlying Reinforcement Learning (RL) algorithm is complex and requires an additional training pipeline for reward and value networks. In this paper, we consider an alternative approach: converting feedback to instruction by relabeling the original one and training the model for better alignment in a supervised m…
Extrapolating Large Language Models to Non-English by Aligning Languages
Nanjing University（南京大学）；Peking University（北京大学）；Shanghai AI Lab（上海人工智能实验室）；The University of Hong Kong（香港大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Xiaohongshu Inc（小红书公司）；Tsinghua University（清华大学）
From Problem-Solving to Teaching Problem-Solving: Aligning LLMs with Pedagogy using Reinforcement Learning
Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）；ETH AI Center（ETH人工智能中心）；Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany（达姆施塔特技术大学计算机科学系、应用网络安全国家研究中心ATHENE）
AlpaCare:Instruction-tuned Large Language Models for Medical Application
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院）；University of Maryland, College Park（马里兰大学帕克分校）
