arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.09719cs.CLcs.AI
收藏

用于学习模糊性的自知识蒸馏

Self-Knowledge Distillation for Learning Ambiguity

Nota Inc.(Nota公司);School of Computing, KAIST(韩国科学技术院计算机学院)

Hancheol Park, Soyeong Jeong, Sukmin Cho, Jong C. Park

AI总结:

本文提出自知识蒸馏方法,利用模型低层知识学习标签分布并重新校准高度模糊样本的置信度,在多个 NLU 基准上缓解错误预测的过度自信问题,且训练更高效。

摘要:

近期的语言模型在自然语言理解(NLU)任务上表现出显著性能。然而,当面对可以用多种方式解释的模糊样本时,它们往往不是最优的,会过度自信地预测单一标签,而不考虑其正确性。为解决这一问题,我们提出一种新颖的自知识蒸馏方法,使模型能够利用从其较低层蒸馏出的知识,更准确地学习标签分布。该方法还包含一个学习阶段,用于根据蒸馏得到的分布知识,对那些被判定为极度模糊的训练样本所不必要增强的置信度进行重新校准。我们在多个不同的 NLU 基准数据集上验证了该方法,实验结果证明了其在产生更好标签分布方面的有效性。特别是,通过对高度模糊样本的置信度进行重新校准,当对未见样本的预测与其真实标签不匹配时出现的过度自信问题得到显著缓解。结果表明,这有助于生成优于现有最先进方法的分布。此外,与现有方法相比,我们的方法在训练模型方面更高效,因为它不涉及用于细化标签分布的额外训练过程。

英文摘要:

Recent language models have shown remarkable performance on natural language understanding (NLU) tasks. However, they are often sub-optimal when faced with ambiguous samples that can be interpreted in multiple ways, over-confidently predicting a single label without consideration for its correctness. To address this issue, we propose a novel self-knowledge distillation method that enables models to learn label distributions more accurately by leveraging knowledge distilled from their lower layers. This approach also includes a learning phase that re-calibrates the unnecessarily strengthened confidence for training samples judged as extremely ambiguous based on the distilled distribution knowledge. We validate our method on diverse NLU benchmark datasets and the experimental results demonstrate its effectiveness in producing better label distributions. Particularly, through the process of re-calibrating the confidence for highly ambiguous samples, the issue of over-confidence when predictions for unseen samples do not match with their ground-truth labels has been significantly alleviated. This has been shown to contribute to generating better distributions than the existing state-of-the-art method. Moreover, our method is more efficient in training the models compared to the existing method, as it does not involve additional training processes to refine label distributions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑