Direct alignment from preferences (DAP) methods, such as DPO, have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF), that do not require a separate reward model. However, the preference datasets used in DAP methods are usually collected ahead of training and never updated, thus the feedback is purely offline. Moreover, responses in these datasets are often sampled from a language model distinct from the one being aligned, and since the model evolves over training, the alignment phase is inevitably off-policy. In this study, we posit that online feedback is key and improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as annotator: on each training iteration, we sample two responses from the current model and prompt the LLM annotator to choose which one is preferred, thus providing online feedback. Despite its simplicity, we demonstrate via human evaluation in several tasks that OAIF outperforms both offline DAP and RLHF methods. We further show that the feedback leveraged in OAIF is easily controllable, via instruction prompts to the LLM annotator.
University of Tuebingen（蒂宾根大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）
The field of artificial intelligence (AI) alignment aims to investigate whether AI technologies align with human interests and values and function in a safe and ethical manner. AI alignment is particularly relevant for large language models (LLMs), which have the potential to exhibit unintended behavior due to their ability to learn and adapt in ways that are difficult to predict. In this paper, we discuss methodological challenges for the alignment problem specifically in the context of LLMs trained to summarize texts. In particular, we focus on methods for collecting reliable human feedback …
Constructive Large Language Models Alignment with Diverse Feedback
Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）
HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning
Sony AI（索尼人工智能公司）；Graduate Institute of Communication Engineering, National Taiwan University（台湾大学通信工程研究所）；University of Southern California（南加利福尼亚大学）；Stanford University（斯坦福大学）
The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
University of Oxford（牛津大学）；University of Pennsylvania（宾夕法尼亚大学）；Bocconi University（博科尼大学）；ML Commons；Contextual AI；Factored AI；UCL（伦敦大学学院）；Cohere；MetaAI；New York University（纽约大学）；AWS AI Labs（亚马逊云科技AI实验室）；Meedan
Feedback Indicators: The Alignment between Llama and a Teacher in Language Learning
Humboldt-Universität zu Berlin Department of Computer Science（柏林洪堡大学计算机科学系）；Humboldt-Universität zu Berlin Language Centre（柏林洪堡大学语言中心）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
University of Georgia（佐治亚大学）；The Pennsylvania State University（宾夕法尼亚州立大学）
Standard methods for differentially private training of deep neural networks replace back-propagated mini-batch gradients with biased and noisy approximations to the gradient. These modifications to training often result in a privacy-preserving model that is significantly less accurate than its non-private counterpart. We hypothesize that alternative training algorithms may be more amenable to differential privacy. Specifically, we examine the suitability of direct feedback alignment (DFA). We propose the first differentially private method for training deep neural networks with DFA and show t…
Scalable Valuation of Human Feedback through Provably Robust Model Alignment
The University of Osaka（大阪大学）；Lattice Lab, Toyota Motor Corporation（丰田公司Lattice实验室）；Machine Learning Research Group, University of Oxford（牛津大学机器学习研究组）；RIKEN AIP（理化学研究所AIP）
Policy Optimization (PO) is a widely used approach to address continuous control tasks. In this paper, we introduce the notion of mediator feedback that frames PO as an online learning problem over the policy space. The additional available information, compared to the standard bandit feedback, allows reusing samples generated by one policy to estimate the performance of other policies. Based on this observation, we propose an algorithm, RANDomized-exploration policy Optimization via Multiple Importance Sampling with Truncation (RANDOMIST), for regret minimization in PO, that employs a randomi…
