arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.11124cs.CLcs.AI
收藏

在人类对齐中扩展数据多样性以微调语言模型

Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment

School of Computer Science, Peking University(北京大学计算机学院);Alibaba Group(阿里巴巴集团)

Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, Yongbin Li

AI总结:

本文研究人类对齐微调中提示与响应多样性的影响,发现更多响应配合更少提示更有效,并提出新的提示多样性度量,证明其与微调后性能线性相关且可用于数据增强。

摘要:

与人类偏好对齐可以防止大语言模型(LLMs)生成误导性或有毒内容,但这需要高昂的人工反馈成本。在假设人工标注资源有限的情况下,有两种不同的分配方式值得考虑:标注更多样的提示(PROMPTS)或更多样的响应(RESPONSES)。然而,目前缺乏对二者影响的直接比较。在本工作中,我们首先根据微调样本数量控制两侧的多样性,从而直接反映其影响。我们发现,相较于大量提示,更多响应但更少提示更能促使大语言模型实现人类对齐。此外,提示多样性的概念可能比通常用单一数字量化的响应多样性更为复杂。因此,我们提出了一种新的提示多样性表示方法,并进一步表明其与微调后大语言模型的最终性能呈线性相关。我们还将其用于数据增强,并通过实验展示其在不同算法上的效果。

英文摘要:

Alignment with human preference prevents large language models (LLMs) from generating misleading or toxic content while requiring high-cost human feedback. Assuming resources of human annotation are limited, there are two different ways of allocating considered: more diverse PROMPTS or more diverse RESPONSES to be labeled. Nonetheless, a straightforward comparison between their impact is absent. In this work, we first control the diversity of both sides according to the number of samples for fine-tuning, which can directly reflect their influence. We find that instead of numerous prompts, more responses but fewer prompts better trigger LLMs for human alignment. Additionally, the concept of diversity for prompts can be more complex than responses that are typically quantified by single digits. Consequently, a new formulation of prompt diversity is proposed, further implying a linear correlation with the final performance of LLMs after fine-tuning. We also leverage it on data augmentation and conduct experiments to show its effect on different algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑