Large language models (LLMs) can explain their predictions through post-hoc or Chain-of-Thought (CoT) explanations. But an LLM could make up reasonably sounding explanations that are unfaithful to its underlying reasoning. Recent work has designed tests that aim to judge the faithfulness of post-hoc or CoT explanations. In this work we argue that these faithfulness tests do not measure faithfulness to the models' inner workings -- but rather their self-consistency at output level. Our contributions are three-fold: i) We clarify the status of faithfulness tests in view of model explainability, characterising them as self-consistency tests instead. This assessment we underline by ii) constructing a Comparative Consistency Bank for self-consistency tests that for the first time compares existing tests on a common suite of 11 open LLMs and 5 tasks -- including iii) our new self-consistency measure CC-SHAP. CC-SHAP is a fine-grained measure (not a test) of LLM self-consistency. It compares how a model's input contributes to the predicted answer and to generating the explanation. Our fine-grained CC-SHAP metric allows us iii) to compare LLM behaviour when making predictions and to analyse the effect of other consistency tests at a deeper level, which takes us one step further towards measuring faithfulness by bringing us closer to the internals of the model than strictly surface output-oriented tests. Our code is available at \url{https://github.com/Heidelberg-NLP/CC-SHAP}
FaithLM: Towards Faithful Explanations for Large Language Models
Rice University（里士大学）；Texas A&M University（德克萨斯农工大学）；Wake Forest University（沃杰森林大学）；New Jersey Institute of Technology（新泽西理工学院）；John Hopkins University（约翰霍普金斯大学）；Google Research（谷歌研究）；Meta Platforms, Inc.（元宇宙平台公司）
When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness
LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Imperial College London（伦敦帝国学院）；Technical University of Munich（慕尼黑工业大学）；MaiNLP lab, CIS, LMU Munich（慕尼黑大学CIS研究所MaiNLP实验室）
Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity
ML Alignment and Theory Scholars（机器学习对齐与理论学者计划）；University of Delaware（特拉华大学）；AI Safety Argentina（阿根廷AI安全组织）；Buenos Aires AI Safety Hub（布宜诺斯艾利斯AI安全中心）；University of Buenos Aires（布宜诺斯艾利斯大学）；Center for AI Safety（AI安全中心）
Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation
University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School（内华达大学拉斯维加斯分校管理、创业与技术系，Lee商学院）；DeepNeuro AI
Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulness
Indian Institute of Information Technology Allahabad (IIITA)（印度阿拉哈巴德信息技术学院）；National Institute of Electronics and Information Technology (NIELIT)（国家电子与信息技术研究院）
