发表机构
Duke University（杜克大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出轨迹感知低秩近似方法Traj-MC，通过蒙特卡洛采样估计轨迹二阶矩，在压缩扩散语言模型时显著保留数学推理能力，优于传统干净校准方法。
AI 中文摘要
扩散语言模型（dLLM）的压缩面临一个已知挑战，因为校准通常在干净、完全可见的激活上进行，而推理过程则遍历部分掩蔽的中间状态。对于低秩压缩，这引发了两个问题。首先，当近似质量在轨迹分布状态上测量时，低秩最优性是否仍能被刻画？其次，校准状态的选择是否影响压缩下数学推理能力的保持？我们通过制定一个在损坏级别和掩蔽实现上定义的轨迹感知低秩目标来解决这些问题。为了高效估计该目标，我们提出了Traj-MC，它通过蒙特卡洛采样估计轨迹二阶矩，并实现了精确的采样状态最优性和总体一致性。在匹配的压缩预算下，轨迹感知校准改善了生成轨迹上的重建，并在数学推理基准上比干净校准保留了显著更多的数学推理能力。我们的结果将轨迹感知低秩最优性与dLLM压缩后保留的推理能力联系起来。我们的代码可在以下网址获取：此https URL。
英文摘要
Diffusion language model (dLLM) compression faces a known challenge because calibration is typically performed on clean, fully visible activations, whereas inference traverses partially masked intermediate states. For low-rank compression, this raises two questions. First, can low-rank optimality still be characterized when approximation quality is measured over trajectory-distributed states, and second, does the choice of calibration states affect mathematical reasoning preservation under compression? We address these questions by formulating a trajectory-aware low-rank objective over corruption levels and masking realizations. To estimate this objective efficiently, we propose Traj-MC, which estimates the trajectory second moment through Monte Carlo sampling and yields exact sampled-state optimality and population consistency. Under matched compression budgets, trajectory-aware calibration improves reconstruction over the generation trajectory and preserves substantially more mathematical reasoning than clean calibration on mathematical reasoning benchmarks. Our results connect trajectory-aware low-rank optimality to the reasoning capability retained after dLLM compression. Our code is available at: https://github.com/Zishan-Shao/traj-mc.git.
