HBO：用于微调大语言模型的分层平衡优化
HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Models
AI总结：
HBO通过分层平衡优化方法，在LLM微调中解决数据不平衡和异质性问题，提升跨数据集和单数据集的训练效果。
摘要：
对大语言模型（LLMs）在混合多样化数据集上进行微调面临数据不平衡和异质性带来的挑战。现有方法往往在跨数据集层面（全局）解决这些问题，但忽略了单个数据集内部的不平衡和异质性（局部），这限制了其效果。我们引入了分层平衡优化（HBO），一种新的方法，使LLMs能够在微调过程中自主调整数据分配，既在跨数据集层面（全局）又在每个单独数据集内部（局部）。HBO采用双层优化策略，包含两种类型的演员：一个全局演员，用于平衡不同训练混合子集之间的数据采样；以及多个局部演员，根据难度级别优化每个子集的数据使用。这些演员由来自LLM训练状态的奖励函数引导，这些奖励函数衡量学习进度和相对性能提升。我们评估了HBO在三个LLM骨干网络上九种多样任务的多语言和多任务设置中的表现。结果表明，HBO在所有情况下都优于现有基线，实现了显著的准确率提升。深入分析进一步表明，HBO的全局演员和局部演员在微调过程中有效调整了数据使用。HBO为LLM微调中的数据不平衡和异质性挑战提供了全面的解决方案，使在多样化数据集上进行更有效的训练成为可能。
英文摘要：
Fine-tuning large language models (LLMs) on a mixture of diverse datasets poses challenges due to data imbalance and heterogeneity. Existing methods often address these issues across datasets (globally) but overlook the imbalance and heterogeneity within individual datasets (locally), which limits their effectiveness. We introduce Hierarchical Balancing Optimization (HBO), a novel method that enables LLMs to autonomously adjust data allocation during fine-tuning both across datasets (globally) and within each individual dataset (locally). HBO employs a bilevel optimization strategy with two types of actors: a Global Actor, which balances data sampling across different subsets of the training mixture, and several Local Actors, which optimizes data usage within each subset based on difficulty levels. These actors are guided by reward functions derived from the LLM's training state, which measure learning progress and relative performance improvement. We evaluate HBO on three LLM backbones across nine diverse tasks in multilingual and multitask setups. Results show that HBO consistently outperforms existing baselines, achieving significant accuracy gains. Our in-depth analysis further demonstrates that both the global actor and local actors of HBO effectively adjust data usage during fine-tuning. HBO provides a comprehensive solution to the challenges of data imbalance and heterogeneity in LLM fine-tuning, enabling more effective training across diverse datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
