arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.24082cs.CLcs.AI
收藏

偏好打包:大型语言模型高效偏好优化

Preference Packing: Efficient Preference Optimization for Large Language Models

AWS GenAI Innovation Center(AWS生成人工智能创新中心)

Jaekyung Cho

AI总结:

偏好打包通过减少重复输入提示的注意力操作和KV缓存内存使用,提升大型语言模型的训练效率,实验显示训练时间减少37%并实现3.22倍速度提升。

摘要:

随着大型语言模型(LLMs)规模的不断增长,资源高效的训练优化技术变得越来越重要。特别是,在预训练和监督微调中,批量打包常被用来实现资源高效的训练。我们提出了偏好打包,这是一种用于增强训练技术中使用相同输入提示的不同响应数据的资源效率的方法,例如奖励模型或直接偏好优化(DPO)。偏好打包通过减少重复输入提示的注意力操作和降低KV缓存内存使用来提高资源效率。我们在纯文本数据集和包含图像的数据集上进行了实验,实现了至少37%的训练时间减少。值得注意的是,这种方法可以与现有的优化技术如批量排序相结合,从而实现3.22倍的速度提升。

英文摘要:

Resource-efficient training optimization techniques are becoming increasingly important as the size of large language models (LLMs) continues to grow. In particular, batch packing is commonly used in pre-training and supervised fine-tuning to achieve resource-efficient training. We propose preference packing, a method to enhance resource efficiency in training techniques that use data with different responses for the same input prompt, such as reward models or Direct Preference Optimization (DPO). Preference packing improves resource efficiency by reducing the attention operations for duplicate input prompts and decreasing KV cache memory usage. We conducted experiments on text-only datasets and image-included datasets and achieved at least 37% reduction in training time. Notably, this method can be applied alongside existing optimization techniques such as batch sorting, resulting in a 3.22x speedup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑