指令和推理数据如何塑造训练后阶段：通过层梯度的视角探讨数据质量
How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients
AI总结：
本文通过层梯度的谱分析，揭示了高质量指令和推理数据对大语言模型训练后阶段的影响，统一了数据评估指标，并展示了数据质量与训练稳定性之间的关系。
摘要：
随着大语言模型（LLM）的训练后阶段从遵循指令发展到复杂推理任务，理解不同数据如何影响微调动态仍鲜有研究。本文通过低/高质量指令和推理数据诱导的层梯度进行谱分析，发现广泛研究的数据评估指标，如IFD、InsTag、难度和奖励，可通过梯度奇异值分解（SVD）计算的谱属性解释和统一。具体而言，高质量数据通常与较低的核范数和较高的有效秩相关。值得注意的是，有效秩在捕捉细微质量差异方面比核范数更稳健和精确。例如，推理数据的有效秩显著高于指令数据，暗示在更复杂的任务上具有更丰富的梯度结构。实验还表明，同一家族内的模型无论大小都具有相似的梯度模式，而不同模型家族则显著不同。本文为指令和推理数据的数据质量影响提供了统一视角，揭示了数据质量与训练稳定性之间的相互作用，为训练后阶段更好的数据探索策略提供了新见解。
英文摘要：
As the post-training of large language models (LLMs) advances from instruction-following to complex reasoning tasks, understanding how different data affect finetuning dynamics remains largely unexplored. In this paper, we present a spectral analysis of layer-wise gradients induced by low/high-quality instruction and reasoning data for LLM post-training. Our analysis reveals that widely-studied metrics for data evaluation, e.g., IFD, InsTag, Difficulty, and Reward, can be explained and unified by spectral properties computed from gradients' singular value decomposition (SVD). Specifically, higher-quality data are usually associated with lower nuclear norms and higher effective ranks. Notably, effective rank exhibits better robustness and resolution than nuclear norm in capturing subtle quality differences. For example, reasoning data achieves substantially higher effective ranks than instruction data, implying richer gradient structures on more complex tasks. Our experiments also highlight that models within the same family share similar gradient patterns regardless of their sizes, whereas different model families diverge significantly. Providing a unified view on the effects of data quality across instruction and reasoning data, this work illuminates the interplay between data quality and training stability, shedding novel insights into developing better data exploration strategies for post-training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
