arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.04004cs.LG
收藏

理解LLM训练数据中噪声对算法思维链的影响

Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thought

Georgia Institute of Technology(佐治亚理工学院);IBM Research(IBM研究院)

Alex Havrilla, Maia Iyer

AI总结:

本研究通过TInt框架生成可控噪声的算法思维链数据,发现微调模型对静态噪声鲁棒但对动态噪声敏感,提示模型对静态噪声也敏感,强调应优先过滤含全局错误的动态噪声样本。

摘要:

在预训练和微调阶段,大型语言模型(LLMs)都在数万亿个质量参差不齐的文本令牌上进行训练。这两个训练阶段通常都涉及启发式地过滤掉“低质量”或含噪声的训练样本,然而关于噪声的类型或强度如何影响下游性能,目前知之甚少。在这项工作中,我们在高度可控的算法可解任务设置下,研究了思维链(CoT)中的噪声如何影响任务性能。首先,我们开发了追踪整数(TInt)框架,用于为整数列表上的任意算术函数生成高度可定制的带噪声执行轨迹。然后,我们定义了两种类型的噪声:静态噪声,一种局部形式的噪声,在CoT轨迹计算后应用;以及动态噪声,一种全局形式的噪声,在轨迹计算过程中传播错误。接着,我们评估了预训练模型在带有不同污染程度和强度的噪声数据集上进行提示和微调后的测试性能。我们发现,微调后的模型对高水平的静态噪声极为鲁棒,但在较低水平的动态噪声下则明显更困难。相比之下,少样本提示模型似乎对即使是静态噪声也更为敏感。最后,我们讨论了我们的发现对噪声过滤最佳实践的影响,特别强调了移除包含具有全局错误的破坏性动态噪声样本的重要性。

英文摘要:

During both pretraining and fine-tuning, Large Language Models (\textbf{LLMs}) are trained on trillions of tokens of text of widely varying quality. Both phases of training typically involve heuristically filtering out ``low-quality'' or \textit{noisy} training samples, yet little is known quantitatively about how the type or intensity of noise affects downstream performance. In this work, we study how noise in chain of thought (\textbf{CoT}) impacts task performance in the highly-controlled setting of algorithmically solvable tasks. First, we develop the Traced Integer (\textbf{TInt}) framework to generate highly customizable noised execution traces for any arithmetic function on lists of integers. We then define two types of noise: \textit{static} noise, a local form of noise which is applied after the CoT trace is computed, and \textit{dynamic} noise, a global form of noise which propagates errors in the trace as it is computed. We then evaluate the test performance of pretrained models both prompted and fine-tuned on noised datasets with varying levels of dataset contamination and intensity. We find fine-tuned models are extremely robust to high levels of static noise but struggle significantly more with lower levels of dynamic noise. In contrast, few-shot prompted models appear more sensitive to even static noise. We conclude with a discussion of how our findings impact noise filtering best-practices, in particular emphasizing the importance of removing samples containing destructive dynamic noise with global errors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑