Concept-based explainability methods use human-understandable intermediaries to produce explanations for machine learning models. These methods assume concept predictions can help understand a model's internal reasoning. In this work, we assess the degree to which such an assumption is true by analyzing whether concept predictors leverage "relevant" features to make predictions, a term we call locality. Concept-based models that fail to respect localities also fail to be explainable because concept predictions are based on spurious features, making the interpretation of the concept predictions vacuous. To assess whether concept-based models respect localities, we construct and use three metrics to characterize when models respect localities, complementing our analysis with theoretical results. Each of our metrics captures a different notion of perturbation and assess whether perturbing "irrelevant" features impacts the predictions made by a concept predictors. We find that many concept-based models used in practice fail to respect localities because concept predictors cannot always clearly distinguish distinct concepts. Based on these findings, we propose suggestions for alleviating this issue.
TU Darmstadt(达姆施塔特工业大学);Hessian Center for Artificial Intelligence (hessian.AI)(黑森人工智能中心);Centre for Cognitive Science, TU Darmstadt(达姆施塔特工业大学认知科学中心);German Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)
Università della Svizzera Italiana(瑞士意大利大学);University of Liechtenstein(利赫泰森大学);Politecnico di Torino(托尼诺理工学院);IBM Research(IBM研究院);Scuola Universitaria Professionale della Svizzera Italiana(瑞士意大利专业大学)
Computer Science Department, TU Darmstadt(图宾根大学计算机科学系);Hessian Center for AI (hessian.AI)(海德堡人工智能中心);German Research Center for AI (DFKI)(德国人工智能研究中心);Centre for Cognitive Science, TU Darmstadt(图宾根大学认知科学中心)
SL-CBM: Enhancing Concept Bottleneck Models with Semantic Locality for Better Interpretability
Saarland University(萨尔大学);University of Chinese Academy of Sciences(中国科学院大学);Institute of Science Tokyo(东京科学大学);CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心);Institute of Software, Chinese Academy of Science(中国科学院软件研究所)
Provable Responsible AI and Data Analytics (PRADA) Lab(可证明的负责任AI与数据分析(PRADA)实验室);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Shanghai Jiao Tong University(上海交通大学);The University of Auckland(奥克兰大学);RIKEN Center for Advanced Intelligence Project (AIP)(理化学研究所先进智能项目中心(AIP));Arizona State University(亚利桑那州立大学);Stony Brook University(石溪大学)
Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off
University of Cambridge(剑桥大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);KU Leuven(鲁汶大学);University of Siena(锡耶纳大学);Babylon Health(巴比伦健康公司);Alan Turing Institute(艾伦·图灵研究所)
University of Virginia(弗吉尼亚大学);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Georgia(佐治亚大学);Shanghai Jiao Tong University(上海交通大学);HKUST(香港科技大学);Ant Group(蚂蚁集团)
Department of Applied Mathematics & Statistics(应用数学与统计学系);State University of New York at Stony Brook(石溪州立大学);Halıcıoğlu Data Science Institute(Halıcıoğlu数据科学研究所);University of California, San Diego(加州大学圣地亚哥分校);Thomas J. Watson Research Center(汤普森·J·沃森研究中心);IBM Research(IBM研究院);Department of Biomedical Informatics(生物医学信息学系)
Interpretable Reward Modeling with Active Concept Bottlenecks
Department of Computer Science, ETH Zurich, Zurich, Switzerland(苏黎世联邦理工学院计算机科学系);Department of Applied Mathematics and Theoretical Physics, University of Cambridge, United Kingdom(剑桥大学应用数学与理论物理系)
Università della Svizzera Italiana(瑞士意大利语大学);University of Cambridge(剑桥大学);Scuola Normale Superiore(高等师范学院);Politecnico di Torino(都灵理工大学);University of Siena(锡耶纳大学);KU Leuven(鲁汶大学)