Explanation faithfulness of model predictions in natural language processing is typically evaluated on held-out data from the same temporal distribution as the training data (i.e. synchronous settings). While model performance often deteriorates due to temporal variation (i.e. temporal concept drift), it is currently unknown how explanation faithfulness is impacted when the time span of the target data is different from the data used to train the model (i.e. asynchronous settings). For this purpose, we examine the impact of temporal variation on model explanations extracted by eight feature attribution methods and three select-then-predict models across six text classification tasks. Our experiments show that (i)faithfulness is not consistent under temporal variations across feature attribution methods (e.g. it decreases or increases depending on the method), with an attention-based method demonstrating the most robust faithfulness scores across datasets; and (ii) select-then-predict models are mostly robust in asynchronous settings with only small degradation in predictive performance. Finally, feature attribution methods show conflicting behavior when used in FRESH (i.e. a select-and-predict model) and for measuring sufficiency/comprehensiveness (i.e. as post-hoc methods), suggesting that we need more robust metrics to evaluate post-hoc explanation faithfulness.
Online detection and infographic explanation of spam reviews with data drift adaptation
University of Vigo(维戈大学);Universidade Portucalense(波尔图大学);ISEP, Polytechnic of Porto(波尔图理工学院ISEP);Institute for Systems and Computer Engineering, Technology and Science(系统与计算机工程、技术与科学研究所)
A Framework for Generating Explanations from Temporal Personal Health Data
Rensselaer Polytechnic Institute(伦斯勒理工学院);Center for Computational Health, IBM Research(IBM研究院计算健康中心);Institute for Clarity in Documentation(文档清晰度研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康分局);Rajiv Gandhi University(拉吉夫甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)
When Robots Say No: Temporal Trust Recovery Through Explanation
School of Engineering Mathematics and Technology, University of Bristol, UK(布里斯托大学工程数学与科技学院);Bristol Digital Futures Institute, University of Bristol, UK(布里斯托数字未来研究所);School of Computer Science, University of Birmingham, UK(伯明翰大学计算机科学学院)