Iterative preference optimization methods have recently been shown to perform well for general instruction tuning tasks, but typically make little improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this work we develop an iterative approach that optimizes the preference between competing generated Chain-of-Thought (CoT) candidates by optimizing for winning vs. losing reasoning steps that lead to the correct answer. We train using a modified DPO loss (Rafailov et al., 2023) with an additional negative log-likelihood term, which we find to be crucial. We show reasoning improves across repeated iterations of this scheme. While only relying on examples in the training set, our approach results in increasing accuracy on GSM8K, MATH, and ARC-Challenge for Llama-2-70B-Chat, outperforming other Llama-2-based models not relying on additionally sourced datasets. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.
CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Xiaohongshu Inc(小红书公司)
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
School of Computer Science, Peking University(北京大学计算机科学系);Center on Frontiers of Computing Studies, Peking University(北京大学前沿计算研究中心);National Research and Engineering Center of Software Engineering, Peking University(北京大学软件工程研究中心)
Advancing LLM Reasoning Generalists with Preference Trees
Tsinghua University(清华大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Northeastern University(东北大学);Renmin University of China(中国人民大学);BUPT(北京邮电大学);Tencent(腾讯)
Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs
University of Science and Technology of China(中国科学技术大学);Xiaomi Corporation(小米公司);State Key Laboratory of Communication Content Cognition, People’s Daily Online(人民日报社传播内容认知国家重点实验室)
RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning
Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology(计算机科学与信息工程系,台湾科技大学);Department of Systems Engineering, Stevens Institute of Technology(系统工程系,史蒂文斯理工学院);Department of Management Information Systems, National Chengchi University(管理信息系,National Chengchi University);AI Research Center, Agaruda System(Agaruda系统人工智能研究中心);Department of Computer Science, Stevens Institute of Technology(计算机科学系,史蒂文斯理工学院);Far Eastern Memorial Hospital(东部纪念医院);Smart Healthcare Interdisciplinary College, National Taipei University of Nursing and Health Sciences(智慧医疗跨领域学院,台北大学护理及健康科学学院)
PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking
Center for Computational Science and Engineering(计算科学与工程中心);Schmarzman College of Computing(施马茨曼计算学院);Laboratory for Atomistic and Molecular Mechanics (LAMM)(原子与分子力学实验室);Massachusetts Institute of Technology(麻省理工学院)
ReCUT: Balancing Reasoning Length and Accuracy in LLMs via Stepwise Trails and Preference Optimization
School of Computer Science and Engineering, Northeastern University, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学计算机科学与技术系);Beijing National Research Center for Information Science and Technology, China(北京信息科学与技术国家研究中心);Institute of Big Data Science and Industry/School of Computer and Information Technology, Shanxi University, China(山西大学大数据科学与产业研究院/计算机与信息学院)
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
OpenGVLab(开放智能机器研究实验室);Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);Nanjing University(南京大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);SenseTime Research(商汤科技研究院)
APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Training
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(计算与智能学院,哈尔滨工业大学,深圳,中国);Huawei Cloud Computing Technologies Co., Ltd.(华为云计算技术有限公司);School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen, China(智能科学与工程学院,哈尔滨工业大学,深圳,中国)