Instruction tuning of open-source large language models (LLMs) like LLaMA, using direct outputs from more powerful LLMs such as Instruct-GPT and GPT-4, has proven to be a cost-effective way to align model behaviors with human preferences. However, the instruction-tuned model has only seen one response per instruction, lacking the knowledge of potentially better responses. In this paper, we propose finetuning an instruction-tuned LLM using our novel \textit{probabilistic ranking} and \textit{contextual ranking} approaches to increase the likelihood of generating better responses. Probabilistic ranking enables the instruction-tuned model to inherit the relative rankings of high-quality and low-quality responses from the teacher LLM. On the other hand, learning with contextual ranking allows the model to refine its own response distribution using the contextual understanding ability of stronger LLMs. Furthermore, we apply probabilistic ranking and contextual ranking sequentially to the instruction-tuned LLM. The resulting model, which we call \textbf{Tuna}, consistently improves the performance on Super Natural Instructions (119 test tasks), LMentry (25 test tasks), Vicuna QA, and can even obtain better results than several strong reinforcement learning baselines. Our code and data are available at \url{ https://github.com/microsoft/LMOps}.
Training language models to follow instructions with human feedback
OpenAI(开放人工智能研究中心);Anthropic(Anthropic公司);Alignment Research Center(对齐研究中心)
使语言模型更大并不能使它们更好地遵循用户的意图。例如,大语言模型可能会生成不真实的、有害的或对用户毫无帮助的输出。换句话说,这些模型并不符合其用户。在本文中,我们展示了一种通过人工反馈进行微调,使语言模型与用户在各种任务上的意图保持一致的方法。从一组标注器编写的提示和通过OpenAI API提交的提示开始,我们收集了所需模型行为的标注器演示数据集,我们使用监督学习来微调GPT-3。然后,我们收集一个模型输出排名的数据集,我们使用它来进一步微调这个监督模型,使用基于人类反馈的强化学习。我们将生成的模型称为InstructGPT。在对我们的即时分布的人类评估中,尽管参数少了100倍,但1.3B参数InstructGPT模型的输出优于175 B GPT-3的输出。此外,InstructGPT模型在真实性方面有所改进,减少了有毒输出的生成,同时在公共NLP数据集上具有最小的性能回归。尽管InstructGPT仍然会犯一些简单的错误,但我们的研究结果表明,利用人类反馈进行微调是将语言模型与人类意图对齐的一个有希望的方向。
ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback
Tencent AI Lab(腾讯人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院)
PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback
Department of Educational Psychology, East China Normal University(华东师范大学教育心理学系);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学上海智能教育研究院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院)
Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback
Institute for AI, Peking University(北京大学人工智能研究院);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院);Huawei Noah’s Ark LAB(华为诺亚方舟实验室);Baichuan Inc.(百川智能);Taobao & Tmall Group of Alibaba(阿里巴巴淘宝天猫集团)
Instruct-of-Reflection: Enhancing Large Language Models Iterative Reflection Capabilities via Dynamic-Meta Instruction
Beijing University of Posts and Telecommunications(北京邮电大学);AI Lab of Lenovo Research(联想研究院AI实验室);Tianjin University(天津大学);The Hong Kong University of Science and Technology(香港科技大学)
LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints
Amazon AGI Foundations(亚马逊AGI基础研究院);Télécom Paris(巴黎电信学院);Institut Polytechnique de Paris(巴黎综合理工学院);UNC Chapel Hill(北卡罗来纳大学教堂山分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
School of Computation, Information and Technology, TUM(慕尼黑工业大学计算、信息与技术学院);Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心);Munich Center for Machine Learning(慕尼黑机器学习中心);Bosch Center for Artificial Intelligence(博世人工智能中心)
FASIONAD++ : Integrating High-Level Instruction and Information Bottleneck in FAt-Slow fusION Systems for Enhanced Safety in Autonomous Driving with Adaptive Feedback
The School of Vehicle and Mobility, Tsinghua University(清华大学车辆与出行学院);McGill University(麦吉尔大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Waseda University(早稻田大学);University of Minnesota(明尼苏达大学);AI2Robotics
CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback
Shanghai AI Laboratory(上海人工智能实验室);The University of Hong Kong(香港大学);New York University(纽约大学);Carnegie Mellon University(卡内基梅隆大学);Shanghai Innovation Institute(上海创新研究院)
Interactively shaping robot behaviour with unlabeled human instructions
Laboratoire de Neurosciences Cognitives Computationnelles (LNC2)(计算认知神经科学实验室(LNC2));INSERM U960(法国国家健康与医学研究院 U960);Institute for Intelligent Systems and Robotics(智能系统与机器人研究所);Sorbonne Université(索邦大学);CNRS UMR 7222(法国国家科学研究中心 UMR 7222)
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);Zhipu AI(智谱AI);University of Electronic Science and Technology of China(电子科技大学)
How Real Is AI Tutoring? Comparing Simulated and Human Dialogues in One-on-One Instruction
Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学上海人工智能教育研究院);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学人工智能教育实验室)