发表机构
University of South Florida; Brandeis University（南佛罗里达大学; 布兰迪斯大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究大语言模型中参数重要性，提出权重调整梯度（WAG）方法，通过捕捉权重与梯度相互作用识别关键参数，揭示新的相互作用关系，展示其在多应用中的效用，为模型分析、调试和控制提供统一方法。
AI 中文摘要
理解大语言模型（LLMs）中哪些参数有影响力对于提高其效率、可靠性和可解释性至关重要。我们引入权重调整梯度（WAG），这是一种简单而有效的估计参数重要性的方法，它明确捕捉模型权重与一阶梯度信息之间的相互作用，识别对模型行为有不成比例影响的参数。在一系列模型和设置中，我们表明WAG揭示了一小部分但关键的参数子集，其修改会导致性能急剧下降，这是现有重要性度量所忽略的失效模式。这些发现揭示了权重和梯度之间以前未被充分探索的相互作用，表明仅通过信号本身无法完全理解参数重要性。WAG的惊人有效性指出了训练网络的基本结构特性，并引发了关于深度学习中零阶和一阶信息作用的新开放性问题。我们展示了WAG在多个应用中的实际效用，包括专家混合架构中的专家分配、特定参数的遗忘、混合精度量化以及知识编辑的层选择。我们的结果将WAG定位为分析、调试和控制LLMs的统一方法，并为有原则的模型级解释开辟了新方向。
英文摘要
Understanding which parameters are influential in Large Language Models (LLMs) is central to improving their efficiency, reliability, and interpretability. We introduce Weight-Adjusted Gradients (WAG), a simple yet effective approach for estimating parameter importance that explicitly captures the interaction between model weights and first-order gradient information and identifies parameters that disproportionately influence model behavior, such as those responsible for collapse phenomena in LLMs. Across a range of models and settings, we show that WAG surfaces a tiny but critical subset of parameters (< 0.5 parts per million or 0.00005% of model size) whose modification leads to dramatic degradation in performance, indicating a novel failure mode. These findings also reveal a previously underexplored interplay between weights and gradients, suggesting that parameter importance cannot be fully understood through either signal alone. We demonstrate the practical utility of WAG across several diverse applications, such as expert allocation in Mixture-of-Experts (MoE) architectures, targeted unlearning, mixed-precision quantization, and layer selection for knowledge editing. In sum, WAG can serve as a unified approach for analyzing, debugging, and controlling LLMs, and opens new directions for principled parameter-level interpretation.
