arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12403cs.CV
收藏

MonoLoss:一种用于可解释单语表示的训练目标

MonoLoss: A Training Objective for Interpretable Monosemantic Representations

Department of Computer Science and Software Engineering, Concordia University, Canada(康科迪亚大学计算机科学与软件工程系);Department of Computer Science, Stony Brook University, United States(石溪大学计算机科学系);Department of Electrical and Computer Engineering, Concordia University, Canada(康科迪亚大学电气与计算机工程系);Mila--Quebec AI Institute, Canada(魁北克人工智能研究所)

Ali Nasiri-Sarvi, Anh Tien Nguyen, Hassan Rivaz, Dimitris Samaras, Mahdi S. Hosseini

AI总结:

MonoLoss是一种用于训练可解释单义表示的训练目标,通过奖励语义一致的激活来提高单义性度量和类别纯度。

摘要:

稀疏自编码器(SAEs)将多义神经表示分解为单义特征,这些特征捕捉单一、可解释的概念。然而,标准训练目标仅弱化鼓励这种分解,而现有的单义性度量需要在所有数据集样本之间进行成对比较,这在训练和评估过程中效率低下。我们研究了一种最近的MonoScore度量,并推导出一个单次通过算法,计算量与数据集图像数量成线性增长,而非二次增长。在OpenImagesV7上,我们实现了评估的1200倍和训练的159倍时间加速,同时仅增加约4%的每轮开销。这使我们能够将MonoScore视为训练信号:我们引入了单义性损失(MonoLoss),一种插件目标,直接奖励语义一致的激活以学习可解释的单义表示。在使用BatchTopK、TopK和JumpReLU SAEs训练CLIP、SigLIP2和预训练ViT特征时,MonoLoss在大多数潜在特征上提高了MonoScore。MonoLoss在所有编码器和SAE组合中一致提高了类别纯度(潜在特征激活图像中属于其主导类的比例),最大增益将基线纯度从0.152提高到0.723。在ResNet-50和CLIP-ViT-B/32微调中作为辅助正则化器使用时,MonoLoss在ImageNet-1K上实现了最高0.6%的准确率提升,并在标准基准数据集上产生了单义激活模式。代码在https://github.com/AtlasAnalyticsLab/MonoLoss上公开。

英文摘要:

Sparse autoencoders (SAEs) decompose polysemantic neural representations, where neurons respond to multiple unrelated concepts, into monosemantic features that capture single, interpretable concepts. However, standard training objectives only weakly encourage this decomposition, and existing monosemanticity metrics require pairwise comparisons across all dataset samples, making them inefficient during training and evaluation. We study a recent MonoScore metric and derive a single-pass algorithm that computes exactly the same quantity, but with a cost that grows linearly, rather than quadratically, with the number of dataset images. On OpenImagesV7, we achieve up to a 1200x speedup wall-clock speedup in evaluation and 159x during training, while adding only ~4% per-epoch overhead. This allows us to treat MonoScore as a training signal: we introduce the Monosemanticity Loss (MonoLoss), a plug-in objective that directly rewards semantically consistent activations for learning interpretable monosemantic representations. Across SAEs trained on CLIP, SigLIP2, and pretrained ViT features, using BatchTopK, TopK, and JumpReLU SAEs, MonoLoss increases MonoScore for most latents. MonoLoss also consistently improves class purity (the fraction of a latent's activating images belonging to its dominant class) across all encoder and SAE combinations, with the largest gain raising baseline purity from 0.152 to 0.723. Used as an auxiliary regularizer during ResNet-50 and CLIP-ViT-B/32 finetuning, MonoLoss yields up to 0.6\% accuracy gains on ImageNet-1K and monosemantic activating patterns on standard benchmark datasets. The code is publicly available at https://github.com/AtlasAnalyticsLab/MonoLoss.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑