Vision-Language Models (VLMs) often produce fluent Natural Language Explanations (NLEs) that sound convincing but may not reflect the causal factors driving predictions. This mismatch of plausibility and faithfulness poses technical and governance risks. We introduce Explanation-Driven Counterfactual Testing (EDCT), a fully automated verification procedure for a target VLM that treats the model's own explanation as a falsifiable hypothesis. Given an image-question pair, EDCT: (1) obtains the model's answer and NLE, (2) parses the NLE into testable visual concepts, (3) generates targeted counterfactual edits via generative inpainting, and (4) computes a Counterfactual Consistency Score (CCS) using LLM-assisted analysis of changes in both answers and explanations. Across 120 curated OK-VQA examples and multiple VLMs, EDCT uncovers substantial faithfulness gaps and provides regulator-aligned audit artifacts indicating when cited concepts fail causal tests.
When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness
LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Imperial College London(伦敦帝国学院);Technical University of Munich(慕尼黑工业大学);MaiNLP lab, CIS, LMU Munich(慕尼黑大学CIS研究所MaiNLP实验室)
FaithLM: Towards Faithful Explanations for Large Language Models
Rice University(里士大学);Texas A&M University(德克萨斯农工大学);Wake Forest University(沃杰森林大学);New Jersey Institute of Technology(新泽西理工学院);John Hopkins University(约翰霍普金斯大学);Google Research(谷歌研究);Meta Platforms, Inc.(元宇宙平台公司)