arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.17465cs.CL
收藏

基于熵的数据选择用于语言模型

Entropy-Based Data Selection for Language Models

School of Computer and Communication Engineering, University of Science and Technology Beijing(计算机与通信工程学院,科学技术大学);National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence(通用人工智能国家重点实验室,北京通用人工智能研究院);Shunde Innovation School, University of Science and Technology Beijing(顺德创新学院,科学技术大学)

Hongming Li, Yang Liu, Chao Huang

AI总结:

本文提出基于熵的无监督数据选择框架,通过减少计算成本和数据需求,提升语言模型在计算受限场景下的微调效率。

摘要:

现代语言模型(LMs)日益需要两种关键资源:计算资源和数据资源。数据选择技术可以有效减少微调LMs所需的训练数据量。然而,其效果与计算资源密切相关,这总是需要高计算预算。由于实际微调场景中的资源限制,我们系统地揭示了数据选择与所选数据不确定性估计之间的关系。尽管大型语言模型(LLMs)在语言理解和生成方面表现出色,提供了缓解数据稀缺的新方法,但评估数据的实用性仍然是一个具有挑战性的问题。这使得高效的数据选择变得不可或缺。为了缓解这些问题,我们提出了基于熵的无监督数据选择(EUDS)框架。在情感分析(SA)、主题分类(Topic-CLS)和问答(Q&A)任务上的实验证实了其有效性。EUDS建立了一个计算高效的数据显示机制。理论分析和实验结果证实了我们方法的有效性。EUDS显著降低了计算成本,并通过较少的数据需求提高了训练时间效率。这为在计算受限场景下LMs的高效微调提供了创新的解决方案。

英文摘要:

Modern language models (LMs) increasingly require two critical resources: computational resources and data resources. Data selection techniques can effectively reduce the amount of training data required for fine-tuning LMs. However, their effectiveness is closely related to computational resources, which always require a high compute budget. Owing to the resource limitations in practical fine-tuning scenario, we systematically reveal the relationship between data selection and uncertainty estimation of selected data. Although large language models (LLMs) exhibit exceptional capabilities in language understanding and generation, which provide new ways to alleviate data scarcity, evaluating data usability remains a challenging task. This makes efficient data selection indispensable. To mitigate these issues, we propose Entropy-Based Unsupervised Data Selection (EUDS) framework. Empirical experiments on sentiment analysis (SA), topic classification (Topic-CLS), and question answering (Q&A) tasks validate its effectiveness. EUDS establishes a computationally efficient data-filtering mechanism. Theoretical analysis and experimental results confirm the effectiveness of our approach. EUDS significantly reduces computational costs and improves training time efficiency with less data requirement. This provides an innovative solution for the efficient fine-tuning of LMs in the compute-constrained scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑