基于独白式社会场景模拟的大语言模型自对齐
Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulation
AI总结:
本文提出社会场景模拟器MATRIX,让LLM在独白式虚拟场景中自行练习并考虑社会后果,通过微调实现与人类价值观的自对齐,实验表明其13B模型在4个基准上超过10个基线并超越GPT-4。
摘要:
将大语言模型(LLMs)与人类价值观对齐,对于缓解其被滥用可能带来的不良影响至关重要。借鉴社会学洞见——承认各方关切是塑造人类价值观的关键因素,本文提出了一种让LLMs通过自身进行对齐的新方向:社会场景模拟。为实现这一目标,我们提出了MATRIX,一种新颖的社会场景模拟器,它围绕用户输入查询模拟真实场景,使LLM能够在回应之前考虑社会后果。MATRIX充当一个虚拟排练空间,类似于独白式(Monopolylogue),LLM在其中扮演与查询相关的多种角色并自行练习。为了注入这种对齐,我们使用MATRIX模拟的数据对LLM进行微调,确保在不牺牲推理速度的情况下遵循人类价值观。我们从理论上证明,在温和假设下,配备MATRIX的LLM优于Constitutional AI。最后,大量实验验证了我们的方法在4个基准上超过10个基线。875条用户评分表明,我们微调的13B规模LLM在对齐人类价值观方面超过了GPT-4。项目页面见 https://shuotang123.github.io/MATRIX。
英文摘要:
Aligning large language models (LLMs) with human values is imperative to mitigate potential adverse effects resulting from their misuse. Drawing from the sociological insight that acknowledging all parties' concerns is a key factor in shaping human values, this paper proposes a novel direction to align LLMs by themselves: social scene simulation. To achieve this, we present MATRIX, a novel social scene simulator that emulates realistic scenes around a user's input query, enabling the LLM to take social consequences into account before responding. MATRIX serves as a virtual rehearsal space, akin to a Monopolylogue, where the LLM performs diverse roles related to the query and practice by itself. To inject this alignment, we fine-tune the LLM with MATRIX-simulated data, ensuring adherence to human values without compromising inference speed. We theoretically show that the LLM with MATRIX outperforms Constitutional AI under mild assumptions. Finally, extensive experiments validate that our method outperforms over 10 baselines across 4 benchmarks. As evidenced by 875 user ratings, our tuned 13B-size LLM exceeds GPT-4 in aligning with human values. See our project page at https://shuotang123.github.io/MATRIX.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。