Aligning Large Language Models (LLMs) with human intentions and values is crucial yet challenging. Current methods primarily rely on human preferences, which are costly and insufficient in capturing nuanced feedback expressed in natural language. In this paper, we present Self-Refinement Tuning (SRT), a method that leverages model feedback for alignment, thereby reducing reliance on human annotations. SRT uses a base language model (e.g., Tulu2) to generate initial responses, which are critiqued and refined by a more advanced model (e.g., GPT-4-Turbo). This process enables the base model to self-evaluate and improve its outputs, facilitating continuous learning. SRT further optimizes the model by learning from its self-generated feedback and refinements, creating a feedback loop that promotes model improvement. Our empirical evaluations demonstrate that SRT significantly outperforms strong baselines across diverse tasks and model sizes. When applied to a 70B parameter model, SRT increases the win rate from 9.6\% to 25.8\% on the AlpacaEval 2.0 benchmark, surpassing well-established systems such as GPT-4-0314, Claude 2, and Gemini. Our analysis highlights the crucial role of language feedback in the success of SRT, suggesting potential for further exploration in this direction.
Exposía: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedback
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt(普遍知识处理实验室(UKP实验室)、计算机科学系和海斯曼人工智能中心(hessian.AI)、达姆施塔特技术大学);Department of Computer Science at Queens College, City University of New York (CUNY)(纽约城市大学皇后学院计算机科学系)
From Sentiment Classification to Actionable and Responsible Feedback: A Scoping Review and Evidence Map of NLP in Student Evaluation of Teaching, 2015-2026
S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning
Tencent(腾讯);Tsinghua University(清华大学);The University of Hong Kong(香港大学);Fudan University(复旦大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
TasTe: Teaching Large Language Models to Translate through Self-Reflection
Institute of Computing and Intelligence, Harbin Institute of Technology(哈尔滨工业大学计算与智能学院);Pattern Recognition Center, WeChat AI, Tencent Inc(腾讯微信AI模式识别中心)
Teaching LLMs for Step-Level Automatic Math Correction via Reinforcement Learning
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Lab of AI for Education, East China Normal University(华东师范大学教育人工智能实验室)
ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Amazon AGI(亚马逊人工智慧实验室);NAVER Cloud(NAVER云)
Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Wuhan AI Research(武汉人工智能研究院)