arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.14221cs.CL
收藏

通过一致性对齐提升大型语言模型的鲁棒性

Improving the Robustness of Large Language Models via Consistency Alignment

Shandong University(山东大学);Baidu Inc.(百度公司);Leiden University(莱顿大学)

Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Shuaiqiang Wang, Chong Meng, Zhicong Cheng, Zhaochun Ren, Dawei Yin

AI总结:

本文定量定义了大型语言模型因指令微小变化导致响应不一致的问题,提出两阶段训练框架(指令增强监督微调与一致性对齐训练),通过模型自我奖励优化鲁棒性,无需外部人类偏好数据,实验证明其有效性。

摘要:

大型语言模型(LLMs)在遵循用户指令和生成有用响应方面取得了巨大成功。然而,它们的鲁棒性仍远未达到理想状态,因为口头化指令的微小变化可能导致模型生成显著不一致的响应。近期文献探讨了这种不一致性问题,强调了持续改进响应生成鲁棒性的重要性。然而,目前仍缺乏系统性的分析和解决方案。本文中,我们定量定义了不一致性问题,并提出了一种两阶段训练框架,包括指令增强的监督微调(instruction-augmented supervised fine-tuning)和一致性对齐训练(consistency alignment training)。第一阶段通过相似的指令增强帮助模型泛化以遵循指令。在第二阶段,我们通过区分相似响应中的细微差异来提高多样性,并帮助模型理解哪些响应更符合人类期望。训练过程通过第一阶段训练模型推断出的自我奖励(self-rewards)完成,无需依赖外部人类偏好资源。我们在近期公开可用的LLMs上对指令遵循任务进行了广泛实验,证明了我们训练框架的有效性。

英文摘要:

Large language models (LLMs) have shown tremendous success in following user instructions and generating helpful responses. Nevertheless, their robustness is still far from optimal, as they may generate significantly inconsistent responses due to minor changes in the verbalized instructions. Recent literature has explored this inconsistency issue, highlighting the importance of continued improvement in the robustness of response generation. However, systematic analysis and solutions are still lacking. In this paper, we quantitatively define the inconsistency problem and propose a two-stage training framework consisting of instruction-augmented supervised fine-tuning and consistency alignment training. The first stage helps a model generalize on following instructions via similar instruction augmentations. In the second stage, we improve the diversity and help the model understand which responses are more aligned with human expectations by differentiating subtle differences in similar responses. The training process is accomplished by self-rewards inferred from the trained model at the first stage without referring to external human preference resources. We conduct extensive experiments on recent publicly available LLMs on instruction-following tasks and demonstrate the effectiveness of our training framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑