我的故事有多好？面向评估LLM生成的XAI叙事的量化指标研究
How good is my story? Towards quantitative metrics for evaluating LLM-generated XAI narratives
AI总结：
针对LLM生成的XAI叙事缺乏非人工评估方式的问题，本研究提出评估框架与自动化指标，用于表格分类任务解释叙事的质量评测，对比了多款前沿LLM，并识别出相关的LLM幻觉新挑战。
摘要：
大语言模型（LLM）在可解释人工智能（XAI）领域的一项快速发展的应用，是将SHAP这类量化解释转化为用户易读的叙事内容，用以说明小型预测模型的决策过程。在该领域中，无需依赖人类偏好研究或调研即可评估叙事质量的需求正变得愈发重要。本研究提出了一个框架，并探索了多种自动化指标，用于评估针对表格分类任务解释的LLM生成叙事。我们应用该方法，在不同数据集和提示类型下对比了多款前沿LLM。作为效用验证，这些指标帮助我们识别出XAI叙事场景下与LLM幻觉相关的新挑战。
英文摘要：
A rapidly developing application of LLMs in XAI is to convert quantitative explanations such as SHAP into user-friendly narratives to explain the decisions made by smaller prediction models. Evaluating the narratives without relying on human preference studies or surveys is becoming increasingly important in this field. In this work we propose a framework and explore several automated metrics to evaluate LLM-generated narratives for explanations of tabular classification tasks. We apply our approach to compare several state-of-the-art LLMs across different datasets and prompt types. As a demonstration of their utility, these metrics allow us to identify new challenges related to LLM hallucinations for XAI narratives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
