In this paper, we introduce \emph{refined Direct Preference Optimization} (rDPO), a method for improving the behavioral alignment of Large Language Models (LLMs) without the need for human-annotated data. The method involves creating synthetic data using self-critique prompting by a teacher LLM and then utilising a generalized DPO loss function to distil to a student LLM. The loss function incorporates an additional external reward model to improve the quality of synthetic data, making rDPO robust to potential noise in the synthetic dataset. rDPO is shown to be effective in a diverse set of behavioural alignment tasks, such as improved safety, robustness against role-playing, and reduced sycophancy. Code to be released at https://github.com/vicgalle/refined-dpo.
Direct Value Optimization: Improving Chain-of-Thought Reasoning in LLMs with Refined Values
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究所）；University College London（伦敦大学学院）
Are LLM Agents Behaviorally Coherent? A Latent Profiling Approach for Social Simulation
Department of Computer Science and Engineering, University of Minnesota（明尼苏达大学计算机科学与工程系）；Department of African American & African Studies, University of Minnesota（明尼苏达大学非裔美国人与非洲研究系）；Department of Sociology, University of Chicago（芝加哥大学社会学系）
