Concept-based explanations quantify how high-level concepts (e.g., gender or experience) influence model behavior, which is crucial for decision-makers in high-stakes domains. Recent work evaluates the faithfulness of such explanations by comparing them to reference causal effects estimated from counterfactuals. In practice, existing benchmarks rely on costly human-written counterfactuals that serve as an imperfect proxy. To address this, we introduce a framework for constructing datasets containing structural counterfactual pairs: LIBERTy (LLM-based Interventional Benchmark for Explainability with Reference Targets). LIBERTy is grounded in explicitly defined Structured Causal Models (SCMs) of the text generation, interventions on a concept propagate through the SCM until an LLM generates the counterfactual. We introduce three datasets (disease detection, CV screening, and workplace violence prediction) together with a new evaluation metric, order-faithfulness. Using them, we evaluate a wide range of methods across five models and identify substantial headroom for improving concept-based explanations. LIBERTy also enables systematic analysis of model sensitivity to interventions: we find that proprietary LLMs show markedly reduced sensitivity to demographic concepts, likely due to post-training mitigation. Overall, LIBERTy provides a much-needed benchmark for developing faithful explainability methods.
GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
Beihang University（北京航空航天大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answering
King’s College London（伦敦国王学院）；University of Electronic Science and Technology of China（电子科技大学）；Sun Yat-sen University（中山大学）；York University（约克大学）；Southern University of Science and Technology（南方科技大学）；University College London（伦敦大学学院）；Tsinghua University（清华大学）
Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness
Division of Computational Health Sciences, Department of Surgery University of Minnesota（大学医学计算科学系，外科系）；Institute for Health Informatics University of Minnesota（健康信息学研究所，明尼苏达大学）；Department of Electrical and Computer Engineering University of Minnesota（电气与计算机工程系，明尼苏达大学）
SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generated LLMs
Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；University of California, Los Angeles（加州大学洛杉矶分校）；Peking University（北京大学）
SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibration
Holistic AI（整体人工智能公司）；University College London（伦敦大学学院）；Stanford University（斯坦福大学）；Center for Long-term AI（长期人工智能中心）；University of Maryland, College Park（马里兰大学帕克分校）；University of California, Berkeley（加州大学伯克利分校）
FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning
Jilin University（吉林大学）；University of Central Florida（中佛罗里达大学）；Arizona State University（亚利桑那州立大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；City University of Hong Kong（香港城市大学）
