Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits
Dartmouth College(达特茅斯学院);Stanford University(斯坦福大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);University of Michigan, Ann Arbor(密歇根大学安娜堡分校)
Ruibo Liu, Chenyan Jia, Ge Zhang, Ziyu Zhuang, Tony X Liu, Soroush Vosoughi
We present Second Thought, a new learning paradigm that enables language models (LMs) to re-align with human values. By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, Second Thought not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability in few-shot scenarios. The generated editing steps also offer better interpretability and ease for interactive error correction. Extensive human evaluations further confirm its effectiveness.
Learning to Edit Knowledge via Instruction-based Chain-of-Thought Prompting
Beijing University of Posts and Telecommunications(北京邮电大学);Peking University(北京大学);Huawei Technologies Ltd.(华为技术有限公司);Chongqing University of Posts and Telecommunications(重庆邮电大学)
Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
AutoCAP: Towards Automatic Cross-lingual Alignment Planning for Zero-shot Chain-of-Thought
School of Computer Science and Engineering, Central South University(中南大学计算机科学与工程学院);Research Center for SCIR, Harbin Institute of Technology(哈尔滨工业大学SCIR研究中心)
OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models
UC San Diego(加州大学圣迭戈分校);The University of New South Wales(新南威尔士大学);East China Normal University(华东师范大学);Adobe Research(奥多比研究中心);CSIRO’s Data61(联邦科学与工业研究组织数据61部门)