We study how well large language models (LLMs) explain their generations through rationales -- a set of tokens extracted from the input text that reflect the decision-making process of LLMs. Specifically, we systematically study rationales derived using two approaches: (1) popular prompting-based methods, where prompts are used to guide LLMs in generating rationales, and (2) technical attribution-based methods, which leverage attention or gradients to identify important tokens. Our analysis spans three classification datasets with annotated rationales, encompassing tasks with varying performance levels. While prompting-based self-explanations are widely used, our study reveals that these explanations are not always as "aligned" with the human rationale as attribution-based explanations. Even more so, fine-tuning LLMs to enhance classification task accuracy does not enhance the alignment of prompting-based rationales. Still, it does considerably improve the alignment of attribution-based methods (e.g., InputXGradient). More importantly, we show that prompting-based self-explanation is also less "faithful" than attribution-based explanations, failing to provide a reliable account of the model's decision-making process. To evaluate faithfulness, unlike prior studies that excluded misclassified examples, we evaluate all instances and also examine the impact of fine-tuning and accuracy on alignment and faithfulness. Our findings suggest that inconclusive faithfulness results reported in earlier studies may stem from low classification accuracy. These findings underscore the importance of more rigorous and comprehensive evaluations of LLM rationales.
When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness
LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Imperial College London（伦敦帝国学院）；Technical University of Munich（慕尼黑工业大学）；MaiNLP lab, CIS, LMU Munich（慕尼黑大学CIS研究所MaiNLP实验室）
From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment
Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Lund University（吕勒奥大学）；University of Tübingen（图宾根大学）；Stanford Graduate School of Education（斯坦福大学教育研究生院）；Harvard Graduate School of Education（哈佛大学教育研究生院）
Legal Experts Disagree With Rationale Extraction Techniques for Explaining ECtHR Case Outcome Classification
Trustworthy Human Language Technologies, Research Center Trustworthy Data Science and Security of the University Alliance Ruhr, Ruhr University Bochum（鲁尔大学联盟可信数据科学与安全研究中心，可信人类语言技术，波鸿鲁尔大学）；Center for Critical Computational Studies, Goethe University Frankfurt（法兰克福歌德大学关键计算研究中心）
Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Northeastern University（东北大学）；MD Anderson Cancer Center（MD安德森癌症中心）；Georgia Institute of Technology（佐治亚理工学院）
Learning from Sufficient Rationales: Analysing the Relationship Between Explanation Faithfulness and Token-level Regularisation Strategies
Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam（范德比尔特大学阿姆斯特丹计算语言学与文本挖掘实验室）；Institute of Computer Science, University of Göttingen（哥廷根大学计算机科学学院）
Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning
Stratified Precision；Harvard Medical School（哈佛医学院）；Imperial College London（帝国理工学院伦敦分校）；National Health Service（国家健康服务系统）；Ipsen France（Ipsen法国）；University College London（伦敦大学学院）
FaithLM: Towards Faithful Explanations for Large Language Models
Rice University（里士大学）；Texas A&M University（德克萨斯农工大学）；Wake Forest University（沃杰森林大学）；New Jersey Institute of Technology（新泽西理工学院）；John Hopkins University（约翰霍普金斯大学）；Google Research（谷歌研究）；Meta Platforms, Inc.（元宇宙平台公司）
