Time-Aware Feature Selection: Adaptive Temporal Masking for Stable Sparse Autoencoder Training
Yale University（耶鲁大学）；University of Chicago（芝加哥大学）
T. Ed Li, Junyu Ren
备注：
First submitted on February 10th, 2025 to ICLR 2025 Workshop (XAI4Science: From Understanding Model Behavior to Discovering New Scientific Knowledge). The paper was accepted but the workshop does not generate proceedings. Now uploading to arXiv to make the paper publicly available
Understanding the internal representations of large language models is crucial for ensuring their reliability and safety, with sparse autoencoders (SAEs) emerging as a promising interpretability approach. However, current SAE training methods face feature absorption, where features (or neurons) are absorbed into each other to minimize $L_1$ penalty, making it difficult to consistently identify and analyze model behaviors. We introduce Adaptive Temporal Masking (ATM), a novel training approach that dynamically adjusts feature selection by tracking activation magnitudes, frequencies, and reconstruction contributions to compute importance scores that evolve over time. ATM applies a probabilistic masking mechanism based on statistical thresholding of these importance scores, creating a more natural feature selection process. Through extensive experiments on the Gemma-2-2b model, we demonstrate that ATM achieves substantially lower absorption scores compared to existing methods like TopK and JumpReLU SAEs, while maintaining excellent reconstruction quality. These results establish ATM as a principled solution for learning stable, interpretable features in neural networks, providing a foundation for more reliable model analysis.
Aalborg University（奥尔堡大学）；University Politehnica of Bucharest（布加勒斯特理工大学）；University of Bucharest（布加勒斯特大学）；Milestone Systems（迈尔斯通系统公司）；SecurifAI（SecurifAI公司）
本文提出 CL-MAE，利用可学习掩码模块在 MAE 自监督训练中生成由易到难的重建课程，并在 ImageNet 及五个下游任务上验证其优于 MAE 的表示学习效果。
Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS（智能信息处理重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Agriculture Information Institute, CAAS（农业信息研究所，中国农业科学院）
Zhejiang University（浙江大学）；Hongkong Polytechnic University（香港理工大学）；University College London（伦敦大学学院）；Westlake University（西湖大学）；King's College London（国王学院）；Southern University of Science and Technology（南方科技大学）
Where to Mask: Structure-Guided Masking for Graph Masked Autoencoders
School of Computer Science, Wuhan University（武汉大学计算机学院）；JD Explore Academy, JD.com（京东探索研究院）；School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；School of Computer Science, Yunnan University（云南大学计算机学院）；School of Computing, Macquarie University（麦考瑞大学计算学院）
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Shanghai AI Lab（上海人工智能实验室）；Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）
本文提出双重掩码策略和渐进式训练范式，高效扩展 VideoMAE 至十亿参数视频 ViT 模型，在多个视频基准上取得领先性能。
Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
Harvard School of Engineering and Applied Science（哈佛工程与应用科学学院）；Kempner Institute for the Study of Natural & Artificial Intelligence（自然与人工智能研究学院）；Harvard Business School（哈佛商学院）
Sparse Interventions in Language Models with Differentiable Masking
University of Edinburgh（爱丁堡大学）；Facebook AI Research（Facebook 人工智能研究院）；University of Amsterdam（阿姆斯特丹大学）；Innopolis University（因诺波利斯大学）；University of Osnabrück（奥斯纳布吕克大学）
MedFLIP: Medical Vision-and-Language Self-supervised Fast Pre-Training with Masked Autoencoder
University of Copenhagen（哥本哈根大学）；Tsinghua University（清华大学）；Shanghai Medical Image Insights（上海医学影像洞察）；Harvard University（哈佛大学）；University of Waterloo（滑铁卢大学）
Fully Bayesian Autoencoders with Latent Sparse Gaussian Processes
EURECOM（欧洲通信研究院）；University of California, Irvine（加州大学欧文分校）
自动编码器及其变体是表示学习和生成建模中使用最广泛的模型之一。然而，基于自动编码器的模型通常假设学习的表示是i.i.d.。并且不能捕获数据样本之间的相关性。为了解决这个问题，我们提出了一种新的稀疏高斯过程贝叶斯自动编码器（SGPAE）模型，在该模型中，我们对贝叶斯自动编码器的潜在空间施加完全贝叶斯稀疏高斯过程先验。我们通过随机梯度Hamilton Monte Carlo对该模型进行后验估计。我们在广泛的表示学习和生成建模任务上定性和定量地评估了我们的方法，并表明我们的方法始终优于依赖于变分自编码器的多个替代方案。
