CPFD：面向短视频分类的置信度感知特权特征蒸馏
CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification
AI总结：
针对短视频分类中特权稠密特征在线推理成本高、端到端多模态模型易丢失其信息的问题，提出置信度感知特权特征蒸馏方法CPFD，利用教师模型置信度自适应蒸馏，显著缩小师生模型性能差距，已在工业生产部署。
摘要：
针对不同业务场景定制的稠密特征是短视频分类中的关键要素。然而，这类特征的复杂性、特定适配需求以及高昂的计算成本，使其在在线推理阶段资源消耗大且难以获取，因此被归类为“特权稠密特征”。与此同时，端到端多模态模型已在众多计算机视觉任务中展现出可观的效果。在工业应用中，优先使用端到端多模态特征虽能提升效率，但往往会导致历史特权稠密特征中宝贵信息的丢失。为在兼顾效率与可控资源成本的前提下融合两类特征，我们提出了置信度感知特权特征蒸馏（Confidence-aware Privileged Feature Distillation，CPFD），该方法通过在训练阶段自适应蒸馏特权特征，增强端到端多模态模型的特征表达能力。现有的特权特征蒸馏（Privileged Feature Distillation，PFD）方法在蒸馏过程中对所有样本施加统一权重，可能导致不同业务场景下的性能不稳定，且教师模型（稠密特征增强多模态模型 DF-X-VLM）与学生模型（仅多模态模型 X-VLM）之间存在显著性能差距；与这类方法不同，我们的CPFD利用教师模型得出的置信度分数，自适应缓解与学生模型之间的性能差异。我们在五项不同任务上开展了大量离线实验，结果表明，CPFD的视频分类F1分数相较于端到端多模态模型（X-VLM）平均提升6.76%，相较于 vanilla PFD 平均提升2.31%；同时它将性能差距缩小了84.6%，取得了与教师模型DF-X-VLM相当的结果。在线实验进一步证实了CPFD的有效性，我们的框架已在生产系统中部署于十余个模型。
英文摘要：
Dense features, customized for different business scenarios, are essential in short video classification. However, their complexity, specific adaptation requirements, and high computational costs make them resource-intensive and less accessible during online inference. Consequently, these dense features are categorized as `Privileged Dense Features'.Meanwhile, end-to-end multi-modal models have shown promising results in numerous computer vision tasks. In industrial applications, prioritizing end-to-end multi-modal features, can enhance efficiency but often leads to the loss of valuable information from historical privileged dense features. To integrate both features while maintaining efficiency and manageable resource costs, we present Confidence-aware Privileged Feature Distillation (CPFD), which empowers features of an end-to-end multi-modal model by adaptively distilling privileged features during training. Unlike existing privileged feature distillation (PFD) methods, which apply uniform weights to all instances during distillation, potentially causing unstable performance across different business scenarios and a notable performance gap between teacher model (Dense Feature enhanced multimodal-model DF-X-VLM) and student model (multimodal-model only X-VLM), our CPFD leverages confidence scores derived from the teacher model to adaptively mitigate the performance variance with the student model. We conducted extensive offline experiments on five diverse tasks demonstrating that CPFD improves the video classification F1 score by 6.76% compared with end-to-end multimodal-model (X-VLM) and by 2.31% with vanilla PFD on-average. And it reduces the performance gap by 84.6% and achieves results comparable to teacher model DF-X-VLM. The effectiveness of CPFD is further substantiated by online experiments, and our framework has been deployed in production systems for over a dozen models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
