We investigate the degree to which human (and LLM) plausibility judgments of multiple-choice commonsense benchmark answers are subject to influence by (im)plausibility arguments for or against an answer, in particular, using rationales generated by LLMs. We collect 3,000 plausibility judgments from humans and another 13,600 judgments from LLMs. Overall, we observe increases and decreases in mean human plausibility ratings in the presence of LLM-generated PRO and CON rationales, respectively, suggesting that, on the whole, human judges find these rationales convincing. Experiments with LLMs reveal similar patterns of influence. Our findings demonstrate a novel use of LLMs for studying aspects of human cognition, while also raising practical concerns that, even in domains where humans are ``experts'' (i.e., common sense), LLMs have the potential to exert considerable influence on people's beliefs.
Tailoring Self-Rationalizers with Multi-Reward Distillation
University of Southern California（南加州大学）；University of Washington（华盛顿大学）；University of California Los Angeles（加州大学洛杉矶分校）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Legal Experts Disagree With Rationale Extraction Techniques for Explaining ECtHR Case Outcome Classification
Trustworthy Human Language Technologies, Research Center Trustworthy Data Science and Security of the University Alliance Ruhr, Ruhr University Bochum（鲁尔大学联盟可信数据科学与安全研究中心，可信人类语言技术，波鸿鲁尔大学）；Center for Critical Computational Studies, Goethe University Frankfurt（法兰克福歌德大学关键计算研究中心）
While pre-trained language models (LMs) have brought great improvements in many NLP tasks, there is increasing attention to explore capabilities of LMs and interpret their predictions. However, existing works usually focus only on a certain capability with some downstream tasks. There is a lack of datasets for directly evaluating the masked word prediction performance and the interpretability of pre-trained LMs. To fill in the gap, we propose a novel evaluation benchmark providing with both English and Chinese annotated data. It tests LMs abilities in multiple dimensions, i.e., grammar, semant…
Decoding Emotions in Abstract Art: Cognitive Plausibility of CLIP in Recognizing Color-Emotion Associations
Institute for Interdisciplinary Studies, University of Amsterdam（阿姆斯特丹大学跨学科研究所）；Department of Information and Computing Sciences, Utrecht University（乌得勒支大学信息与计算科学系）
Current Explainable AI (ExAI) methods, especially in the NLP field, are conducted on various datasets by employing different metrics to evaluate several aspects. The lack of a common evaluation framework is hindering the progress tracking of such methods and their wider adoption. In this work, inspired by offline information retrieval, we propose different metrics and techniques to evaluate the explainability of SA models from two angles. First, we evaluate the strength of the extracted "rationales" in faithfully explaining the predicted outcome. Second, we measure the agreement between ExAI m…
Quantifying the Plausibility of Context Reliance in Neural Machine Translation
Center for Language and Cognition (CLCG), University of Groningen（格罗宁根大学语言与认知中心（CLCG））；Dept. of Cognitive Science and Artificial Intelligence (CSAI), Tilburg University（蒂尔堡大学认知科学与人工智能系（CSAI））
Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models
Human Centric ART, University of Rome Tor Vergata（人类中心ART，罗马大学托尔维加塔分校）；ILCC, School of Informatics, University of Edinburgh（ILCC，信息学院，爱丁堡大学）；School of Computer Science, University of Sheffield（计算机科学学院，谢菲尔德大学）；Almawave S.p.A.（Almawave公司）
University of Sheffield（谢菲尔德大学）；Imperial College London（伦敦帝国理工学院）
Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results when predicting the overall quality of translated sentences. Predicting translation errors, i.e. detecting specifically which words are incorrect, is a more challenging task, especially with limited amounts of training data. We hypothesize that, not unlike humans, successful QE models rely on translation errors to predict overall sentence quality. By exploring a set of feature attribution methods that assign relevance scores to the inputs to explain model predictions, …
X-MuTeST: A Multilingual Benchmark for Explainable Hate Speech Detection and A Novel LLM-consulted Explanation Framework
Indian Institute of Technology Indore（印度理工学院Indore）；Indian Institute of Information Technology Dharwad（印度信息科技学院Dharwad）；Arizona State University（亚利桑那州立大学）；Indian Institute of Technology Mandi（印度理工学院Mandi）
