arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.00355cs.CLcs.AIcs.CY
收藏

第二个想法是最好的:从文本编辑中学习重新调整人类价值观

Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits

Dartmouth College(达特茅斯学院);Stanford University(斯坦福大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);University of Michigan, Ann Arbor(密歇根大学安娜堡分校)

Ruibo Liu, Chenyan Jia, Ge Zhang, Ziyu Zhuang, Tony X Liu, Soroush Vosoughi

摘要:

我们提出第二思想,一个新的学习范式,使语言模型(LM)重新调整与人类价值观。通过对值未对齐和值对齐文本之间的编辑链进行建模,并通过强化学习进行LM微调和额外的细化,Second Thought不仅在三个价值对齐基准数据集中实现了卓越的性能,而且在Few-Shot场景中显示了强大的人类价值迁移学习能力。生成的编辑步骤还提供了更好的可解释性和易于交互式纠错。广泛的人类评估进一步证实了其有效性。

英文摘要:

We present Second Thought, a new learning paradigm that enables language models (LMs) to re-align with human values. By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, Second Thought not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability in few-shot scenarios. The generated editing steps also offer better interpretability and ease for interactive error correction. Extensive human evaluations further confirm its effectiveness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑