Large Language Models (LLMs) offer natural language explanations as an alternative to feature attribution methods for model interpretability. However, despite their plausibility, they may not reflect the model's true reasoning faithfully. While several faithfulness metrics have been proposed, they are often evaluated in isolation, making principled comparisons between them difficult. We present Causal Diagnosticity, a testbed framework for evaluating faithfulness metrics for natural language explanations. We use the concept of diagnosticity, and employ model-editing methods to generate faithful-unfaithful explanation pairs. Our benchmark includes four tasks: fact-checking, analogy, object counting, and multi-hop reasoning. We evaluate prominent faithfulness metrics, including post-hoc explanation and chain-of-thought methods. Diagnostic performance varies across tasks and models, with Filler Tokens performing best overall. Additionally, continuous metrics are generally more diagnostic than binary ones but can be sensitive to noise and model choice. Our results highlight the need for more robust faithfulness metrics.
From Concept Alignment to Causal Grounding: An Intervention Test of Chain-of-Thought Faithfulness
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；Saarland Informatics Campus（萨尔兰信息学园区）；Centre for European Research in Trusted AI (CERTAIN)（欧洲可信人工智能研究中心（CERTAIN））；Technische Universität Berlin（柏林工业大学）；IBM Research（IBM研究院）；University of Copenhagen（哥本哈根大学）；University of Groningen（格罗宁根大学）
Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning
Stratified Precision；Harvard Medical School（哈佛医学院）；Imperial College London（帝国理工学院伦敦分校）；National Health Service（国家健康服务系统）；Ipsen France（Ipsen法国）；University College London（伦敦大学学院）
Causal Consistency Regularization: Training Verifiably Sensitive Reasoning in Large Language Models
Department of Computer Science, Iowa State University（计算机科学系，爱荷华州立大学）；Department of Civil, Construction & Environmental Engineering, Iowa State University（土木、建筑与环境工程系，爱荷华州立大学）
Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness
Indian Institute of Information Technology Allahabad (IIITA)（印度阿拉哈巴德信息技术学院）；National Institute of Electronics and Information Technology (NIELIT)（国家电子与信息技术研究院）
METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models
College of Computer Science, Sichuan University（四川大学计算机学院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究所）；Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China（中国教育部机器学习与产业智能工程研究中心）
Skill Path: Unveiling Language Skills from Circuit Graphs
Hang Chen School of Computer Science and Technology Xi’an Jiaotong University（陈hang 王校计算机科学与技术学院 西安交通大学）；Jiaying Zhu School of Computer Science and Engineering The Chinese University of Hong Kong（朱继燕 王校计算机科学与工程学院 香港中文大学）；Xinyu Yang School of Computer Science and Technology Xi’an Jiaotong University（杨新宇 王校计算机科学与技术学院 西安交通大学）；Wenya Wang School of Computer Science and Engineering Nanyang Technological University（王文雅 王校计算机科学与工程学院 新加坡国立大学）
