发表机构
Duke University; Carnegie Mellon University; University of Florida; Wake Forest University; University of Oxford（杜克大学; 卡内基梅隆大学; 佛罗里达大学; 维克森林大学; 牛津大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究揭示低秩压缩大语言模型在恢复过程中的异质性源于可复现的压缩诱导功能结构（恢复压力），并开发了标准化表征方法，证明该结构演化与恢复响应相关，提供超越标量损失的互补视角。
AI 中文摘要
不同的低秩压缩方法可以产生对相同压缩后恢复过程响应不同的压缩大语言模型，且方法之间在压缩终点观察到的相对优势可能在恢复后缩小、增大甚至逆转。我们探究这种恢复异质性是否反映了超越标量损失演化的功能结构，以及该结构在恢复过程中如何演化。我们的结果表明，这种异质性反映了一种可复现的压缩诱导功能结构，我们将其形式化为恢复压力。为了在恢复过程中一致地表征该结构，我们在每个骨干网络内开发了一种标准化的功能表征方法，适用于异构的低秩方法。主要的反向表征揭示了跨独立探针的可复现模块级结构，而补充的仅前向表征无需损失或反向传播即可恢复相关结构。我们进一步发现，在终点测量的恢复压力与后续恢复响应相关；在恢复过程中，其模块级结构被非均匀地重组，且局部更新会引发超出直接更新模块的分布式响应。进一步证据表明，追踪这种演化结构提供了与标量损失互补的恢复进展功能视角。
英文摘要
Different low-rank compression methods can produce compressed LLMs that respond differently to the same post-compression recovery procedure, and relative advantages observed between methods at the compression endpoint may shrink, grow, or even reverse after recovery. We ask whether this recovery heterogeneity reflects functional structure beyond scalar loss evolution, and how that structure evolves throughout recovery. Our results establish that this heterogeneity reflects a reproducible compression-induced functional structure, which we formalize as recovery pressure. To characterize this structure consistently throughout recovery, we develop a standardized functional characterization within each backbone that is applicable across heterogeneous low-rank methods. The primary backward characterization reveals reproducible module-wise structure across independent probes, while a complementary forward-only characterization recovers related structure without loss or backpropagation. We further find that recovery pressure measured at the endpoint is associated with subsequent recovery response; during recovery, its module-wise structure is reorganized non-uniformly, and localized updates induce distributed responses beyond directly updated modules. Further evidence indicates that tracking this evolving structure provides a complementary functional view of recovery progress alongside scalar loss.
