利用 Flash Attention 的打包技术提升训练效率
Enhancing Training Efficiency Using Packing with Flash Attention
AI总结:
本文分析 Hugging Face Transformers 4.44 新增的打包样本掩码功能,证明其结合 Flash Attention 能减少填充浪费、提升 LLM 微调时的 GPU 利用率和训练效率。
摘要:
在微调 LLM 模型时,通常使用填充(padding),通过向较短的训练样本添加特殊 token 以匹配每个批次中最长序列的长度。虽然这确保了批处理的统一性,但由于计算中包含无关的填充 token,导致效率低下并浪费 GPU 资源。Hugging Face SFT trainer 一直提供使用打包(packing)来组合多个训练样本的选项,从而最大限度地利用 GPU 资源。然而,到目前为止,它尚未提供对每个打包训练样本进行适当掩码(masking)的功能。这一能力已被添加到 Hugging Face Transformers 4.44 中。我们分析了这一新特性,并展示了在不同打包变体下的优势。
英文摘要:
Padding is often used in tuning LLM models by adding special tokens to shorter training examples to match the length of the longest sequence in each batch. While this ensures uniformity for batch processing, it introduces inefficiencies by including irrelevant padding tokens in the computation and wastes GPU resources. Hugging Face SFT trainer has always offered the option to use packing to combine multiple training examples, allowing for maximal utilization of GPU resources. However, up till now, it did not offer proper masking of each packed training example. This capability has been added to Hugging Face Transformers 4.44. We analyse this new feature and show the benefits across different variations of packing.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。