arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.04854cs.CL
收藏

面向语言模型对齐的不确定性感知学习

Uncertainty Aware Learning for Language Model Alignment

School of Computer Science, Fudan University(复旦大学计算机科学学院);The University of Sydney(悉尼大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Nanyang Technological University(南洋理工大学)

Yikun Wang, Rui Zheng, Liang Ding, Qi Zhang, Dahua Lin, Dacheng Tao

AI总结:

本文提出不确定性感知学习(UAL),通过根据样本不确定性自适应调整标签平滑值来改进LLM对齐,实验表明其在多个基准上显著优于标准监督微调。

摘要:

随着指令微调大语言模型(LLMs)的演进,对齐预训练基础模型面临越来越多的挑战。现有的对齐策略通常利用多样且高质量的数据源,但往往忽视了任务的内在不确定性,对所有数据样本一视同仁地学习。这可能导致数据效率和模型性能欠佳。为此,我们提出不确定性感知学习(UAL),通过引入样本不确定性(由能力更强的LLMs引出)来改进不同任务场景下的模型对齐。我们以一种简单的方式实现UAL——根据单个样本的不确定性自适应地设置训练的标签平滑值。分析表明,我们的UAL确实促进了特征空间中更好的token聚类,验证了我们的假设。在广泛使用的基准上的大量实验表明,我们的UAL显著且一致地优于标准监督微调。值得注意的是,在混合场景中对齐的LLMs在高熵任务(即AlpacaEval排行榜)上平均提升了10.62%,在复杂的低熵任务(即MetaMath和GSM8K)上平均提升了1.81%。

英文摘要:

As instruction-tuned large language models (LLMs) evolve, aligning pretrained foundation models presents increasing challenges. Existing alignment strategies, which typically leverage diverse and high-quality data sources, often overlook the intrinsic uncertainty of tasks, learning all data samples equally. This may lead to suboptimal data efficiency and model performance. In response, we propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios, by introducing the sample uncertainty (elicited from more capable LLMs). We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Analysis shows that our UAL indeed facilitates better token clustering in the feature space, validating our hypothesis. Extensive experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning. Notably, LLMs aligned in a mixed scenario have achieved an average improvement of 10.62\% on high-entropy tasks (i.e., AlpacaEval leaderboard), and 1.81\% on complex low-entropy tasks (i.e., MetaMath and GSM8K).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑