Concept Bottleneck Models (CBMs) aim to deliver interpretable and interventionable predictions by bridging features and labels with human-understandable concepts. While recent CBMs show promising potential, they suffer from information leakage, where unintended information beyond the concepts (either when concepts are represented with probabilities or binary states) are leaked to the subsequent label prediction. Consequently, distinct classes are falsely classified via indistinguishable concepts, undermining the interpretation and intervention of CBMs.
This paper alleviates the information leakage issue by introducing label supervision in concept predication and constructing a hierarchical concept set. Accordingly, we propose a new paradigm of CBMs, namely SupCBM, which achieves label predication via predicted concepts and a deliberately-designed intervention matrix. SupCBM focuses on concepts that are mostly relevant to the predicted label and only distinguishes classes when different concepts are presented. Our evaluations show that SupCBM outperforms SOTA CBMs over diverse datasets. It also manifests better generality across different backbone models. With proper quantification of information leakage in different CBMs, we demonstrate that SupCBM significantly reduces the information leakage.
Eliminating Label Leakage in Tree-Based Vertical Federated Learning
The University of Tokyo（东京大学）；Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
Qwen Large Model Application Team, Alibaba（阿里巴巴大模型应用团队）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Act Now: A Novel Online Forecasting Framework for Large-Scale Streaming Data
School of Information Science and Engineering, Shandong University（山东大学信息科学与工程学院）；School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；Ocean College, Jiangsu University of Science and Technology（江苏科技大学海洋学院）；School of Qilu Transportation, Shandong University（山东大学齐鲁交通学院）；Shandong Key Laboratory of Intelligent Communication and Sensing-Computing Integration（山东省智能通信与感知计算集成重点实验室）
