维度共激活用于冻结视觉基础模型中的表征一致性
Dimensional Coactivation for Representational Consistency in Frozen Vision Foundation Models
AI总结:
本文提出维度共激活(DCA)方法,用于评估冻结视觉基础模型中样本内部的表征一致性。通过比较语义子区域的特征维度共激活,DCA在深度伪造检测任务中表现出色,验证了其在冻结模型中的有效性。
摘要:
冻结视觉基础模型不仅提取特征,还通过学习的坐标系统组织图像。本文探讨该坐标系统在单个输入内部是否保持一致性,提出表征一致性概念。引入维度共激活(DCA),一种用于测量这种一致性的工具。DCA通过比较语义区域是否在同一特征维度上共激活来评估一致性。与经典相似性度量不同,DCA有意避免中心化、L2归一化和完全Gram耦合。深度伪造检测提供了自然的验证任务。合成面孔可能在眼睛、鼻子和嘴巴区域生成合理外观,但破坏真实面孔中这些区域的表征结构。使用冻结的DINOv3特征,DCA揭示了这种破坏:眼-嘴-鼻指纹在CelebDF-v2和DFD上分别达到0.9106和0.9289的AUC。设计还通过消融实验验证:重新引入中心化使CelebDF-v2 AUC降至0.459,L2归一化降至0.862,跨维度耦合降至0.478。最后,用FaRL替代DINOv3使CelebDF-v2 AUC降至0.582。DCA因此依赖于稳定的每维度坐标系统,而非单纯的区域提取。这些结果将DCA定位为测量冻结基础模型中样本内部表征一致性的工具,深度伪造检测是首个验证任务。
英文摘要:
Frozen vision foundation models do not merely extract features; they organize images through a learned coordinate system. We ask whether that coordinate system remains internally coherent within a single input. This leads to Representational Consistency: the study of whether a frozen foundation model represents one sample coherently across its semantic subregions. We introduce Dimensional Coactivation (DCA), a per-dimension instrument for measuring this coherence. DCA compares semantic regions by asking whether the same feature dimensions coactivate across them. Unlike classical similarity measures, it deliberately avoids centering, L2 normalization, and full Gram coupling. These operations are useful when comparing different models or distributions, but they are mismatched to the intra-sample setting, where the coordinate system is fixed and raw magnitude carries signal. Deepfake detection provides a natural validation task. Synthetic faces may reproduce plausible eyes, noses, and mouths while breaking the representational structure that links those regions in real faces. Using frozen DINOv3 features, DCA exposes this break: an eyes-mouth-nose fingerprint achieves 0.9106 AUC on CelebDF-v2 and 0.9289 on DFD under FF++ c23 cross-dataset transfer. The design is also sharply validated by ablation: reintroducing centering collapses CelebDF-v2 AUC to 0.459, L2 normalization reduces it to 0.862, and cross-dimension coupling reduces it to 0.478. Finally, replacing DINOv3 with FaRL collapses CelebDF-v2 AUC to 0.582. DCA therefore depends on a stable per-dimension coordinate system, not on region extraction alone. These results position DCA as an instrument for measuring intra-sample representational coherence in frozen foundation models, with deepfake detection as the first validation task.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。