arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.01490cs.CL
收藏

多个篮子里的合成鸡蛋:合成数据多样性对LLM微调的影响

Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning

Utrecht University(乌特勒支大学)

Max Schaffelder, Albert Gatt

AI总结:

本文研究合成数据来源多样性对LLM微调的影响,发现多样化数据可缓解分布崩溃,提升输出质量,并减少自偏好偏差。

摘要:

随着合成数据在语言模型开发中的广泛应用,理解其对模型行为的影响至关重要。本文研究了合成数据来源多样性对微调大型语言模型的影响,重点关注分布崩溃、对抗鲁棒性和自偏好偏差三个关键维度。我们的发现表明,使用来自不同来源的合成数据微调模型可以缓解分布崩溃,保持输出分布的广度和文本的多样性。此外,尽管人类和合成微调数据都能去除保护措施,但我们观察到后者在输出质量上更高,从而使输出可能更具可使用性和危险性。最后,我们还发现微调可减少自偏好偏差,人类数据效果最显著,其次是多源合成数据。

英文摘要:

As synthetic data becomes widely used in language model development, understanding its impact on model behavior is crucial. This paper investigates the impact of the diversity of sources of synthetic data on fine-tuned large language models. We focus on three key dimensions: distribution collapse, adversarial robustness, and self-preference bias. Our findings reveal that fine-tuning models on synthetic data from diverse sources can mitigate distribution collapse, preserving the breadth of the output distribution and the diversity of the output text. Furthermore, while both human and synthetic fine-tuning data can remove safeguards, we observe a tendency for higher output quality in the latter case, thus making outputs potentially more usable and dangerous. Finally, we also find evidence that fine-tuning reduces self-preference bias, with human data being the most effective, followed by multi-source synthetic data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑