Direct Preference Optimization (DPO) has emerged as a more computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) with Proximal Policy Optimization (PPO), eliminating the need for reward models and online sampling. Despite these benefits, DPO and its variants remain sensitive to hyper-parameters and prone to instability, particularly on mathematical datasets. We argue that these issues arise from the unidirectional likelihood-derivative negative feedback inherent in the log-likelihood loss function. To address this, we propose a novel LLM alignment loss that establishes a stable Bidirectional Negative Feedback (BNF) during optimization. Our proposed BNF loss eliminates the need for pairwise contrastive losses and does not require any extra tunable hyper-parameters or pairwise preference data, streamlining the alignment pipeline to be as simple as supervised fine-tuning. We conduct extensive experiments across two challenging QA benchmarks and four reasoning benchmarks. The experimental results show that BNF achieves comparable performance to the best methods on QA benchmarks, while its performance decrease on the four reasoning benchmarks is significantly lower compared to the best methods, thus striking a better balance between value alignment and reasoning ability. In addition, we further validate the performance of BNF on non-pairwise datasets, and conduct in-depth analysis of log-likelihood and logit shifts across different preference optimization methods.
Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Princeton University(普林斯顿大学);Nanyang Technological University(南洋理工大学);Salesforce Research(Salesforce 研究);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore(高性能计算研究所,新加坡科技研究局)
The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
University of Oxford(牛津大学);University of Pennsylvania(宾夕法尼亚大学);Bocconi University(博科尼大学);ML Commons;Contextual AI;Factored AI;UCL(伦敦大学学院);Cohere;MetaAI;New York University(纽约大学);AWS AI Labs(亚马逊云科技AI实验室);Meedan
Feedback Indicators: The Alignment between Llama and a Teacher in Language Learning
Humboldt-Universität zu Berlin Department of Computer Science(柏林洪堡大学计算机科学系);Humboldt-Universität zu Berlin Language Centre(柏林洪堡大学语言中心);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)
Constructive Large Language Models Alignment with Diverse Feedback
Alibaba Group(阿里巴巴集团);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学)
Position: Towards Bidirectional Human-AI Alignment
NYU Shanghai, New York University(纽约大学上海研究院,纽约大学);MBZUAI(穆桑大学人工智能研究所);Microsoft(微软公司);University of Michigan(密歇根大学);Apple(苹果公司);Google(谷歌公司);Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学);Google DeepMind(谷歌DeepMind)