闭式线性探针数据集蒸馏用于预训练视觉模型
Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models
AI总结：
本文提出CLP-DD方法，通过闭式解法和判别损失提升预训练视觉模型的数据集蒸馏效果，相比传统方法在计算效率和性能上均有显著提升。
摘要：
数据集蒸馏将大规模训练集压缩成小合成集，保留下游训练效用。现有方法多针对从头训练网络，而现代视觉迁移学习常使用冻结的预训练编码器后接轻量线性探针。现有蒸馏方法或通过轨迹基梯度匹配反向传播迭代线性探针更新，或依赖最初为从头训练设计的神经切线核（NTK）近似闭式公式。本文提出闭式线性探针数据集蒸馏（CLP-DD），通过样本空间核岭求解器计算合成集诱导的线性探针。合成图像通过评估此诱导分类器在真实特征上的温度缩放softmax交叉熵进行更新，其中分类器列作为特征空间中的学习类别锚点。进一步显示外目标选择至关重要：将闭式内求解器与标准MSE外损失显著劣于轨迹基方法，而判别外损失能缩小大部分差距。在ImageNet-100上，CLP-DD在无DSA情况下显著优于LGM，在有DSA时接近LGM。在ImageNet-1K上，CLP-DD在三个四预训练主干上匹配或超越LGM with DSA，运行速度快约14倍，GPU内存使用少于八分之一。
英文摘要：
Dataset distillation compresses a large training set into a small synthetic set that preserves downstream training utility. While most existing methods target training networks from scratch, modern visual transfer learning often uses frozen pre-trained encoders followed by lightweight linear probing. Existing distillation methods for this setting either unroll iterative linear-probe updates with trajectory-based gradient matching, or rely on closed-form formulations originally designed for from-scratch training with neural-tangent-kernel (NTK) approximations. Neither route exploits the fact that frozen-feature linear probing admits a closed-form solution determined directly by the pre-trained features themselves, with no infinite-width approximation and no inner-loop trajectory. We propose Closed-Form Linear-Probe Dataset Distillation (CLP-DD), a bilevel formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver. The synthetic images are then updated by evaluating this induced classifier on real features through a temperature-scaled softmax cross-entropy, where the classifier columns act as learned class anchors in feature space. We further show that the choice of outer objective is decisive: pairing the closed-form inner solver with a standard MSE outer loss substantially underperforms trajectory-based methods, while the discriminative outer loss closes most of the gap. On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost. On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly $14\times$ faster and using less than one-eighth of the GPU memory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
