arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.14124cs.AIcs.LG
收藏

基于信息熵的语言模型数据剪枝中样本重要性度量

Measuring Sample Importance in Data Pruning for Language Models based on Information Entropy

Korea University(高丽大学)

Minsang Kim, Seungjun Baek

AI总结:

针对大语言模型数据高效训练需求,提出基于信息熵的数据剪枝方法,通过负对数似然熵和平均逆词频衡量样本信息量,优先剪枝低信息量冗余样本,提升模型多任务表现与泛化能力。

摘要:

语言模型的计算高效训练已成为重要问题。我们针对大语言模型(LLM)的数据高效训练开展数据剪枝研究。本研究探索一种基于信息熵的数据剪枝方法,提出通过熵函数估计训练语料中样本的信息量并据此排序,核心思路是信息量较低的样本更可能含冗余信息,应优先被剪枝。我们采用基于负对数似然的熵函数和样本的平均逆词频作为衡量样本信息量的替代指标。实验表明,所提出的基于信息的剪枝方法能在多种语言建模及下游任务上取得提升,并增强语言模型的泛化能力。

英文摘要:

Compute-efficient training of language models has become an important issue. We consider data pruning for data-efficient training of LLMs. In this work, we consider a data pruning method based on information entropy. We propose that the samples in the training corpus be ranked in terms of their informativeness which we estimate through entropy functions. The key idea is that, less informative samples are likely to contain redundant information, and thus should be pruned first. We use the entropy functions based on the negative log-likelihood and the average inverse word frequency of a sample as a surrogate to measure its informativeness. Experiments reveal that the proposed information-based pruning can improve upon various language modeling and downstream tasks, and enhance the generalization capability of language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑