Understanding the behavior of a trained network and finding explanations for its outputs is important for improving the network's performance and generalization ability, and for ensuring trust in automated systems. Several approaches have previously been proposed to identify and visualize the most important features by analyzing a trained network. However, the relations between different features and classes are lost in most cases. We propose a technique to induce sets of if-then-else rules that capture these relations to globally explain the predictions of a network. We first calculate the importance of the features in the trained network. We then weigh the original inputs with these feature importance scores, simplify the transformed input space, and finally fit a rule induction model to explain the model predictions. We find that the output rule-sets can explain the predictions of a neural network trained for 4-class text classification from the 20 newsgroups dataset to a macro-averaged F-score of 0.80. We make the code available at https://github.com/clips/interpret_with_rules.
Distilling neural networks into skipgram-level decision lists
Computational Linguistics and Psycholinguistics Research Center (CLiPS)（计算语言学与心理语言学研究中心（CLiPS））；University of Antwerp（安特卫普大学）；School of Computing and Information Systems, University of Melbourne（墨尔本大学计算与信息系统学院）
Analysis of Error Sources in LLM-based Hypothesis Search for Few-Shot Rule Induction
Department of Statistics, University of California, Los Angeles（统计学系，加州大学洛杉矶分校）；Microsoft, Redmond, WA（微软公司，西雅图，华盛顿州）；Department of Psychology, University of California, Los Angeles（心理学系，加州大学洛杉矶分校）
Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedents
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China（香港科技大学计算机科学与工程系）；School of Law, Tsinghua University, Beijing, China（清华大学法学院）
Logic Rules as Explanations for Legal Case Retrieval
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information Technology and Management, University of International Business and Economics（对外经济贸易大学信息技术与管理学院）
