AI-text detectors achieve high accuracy on in-domain benchmarks, but often struggle to generalize across different generation conditions such as unseen prompts, model families, or domains. While prior work has reported these generalization gaps, there are limited insights about the underlying causes. In this work, we present a systematic study aimed at explaining generalization behavior through linguistic analysis. We construct a comprehensive benchmark that spans 6 prompting strategies, 7 large language models (LLMs), and 4 domain datasets, resulting in a diverse set of human- and AI-generated texts. Using this dataset, we fine-tune classification-based detectors on various generation settings and evaluate their cross-prompt, cross-model, and cross-dataset generalization. To explain the performance variance, we compute correlations between generalization accuracies and feature shifts of 80 linguistic features between training and test conditions. Our analysis reveals that generalization performance for specific detectors and evaluation conditions is significantly associated with linguistic features such as tense usage and pronoun frequency.
A Methodology for Explainable Large Language Models with Integrated Gradients and Linguistic Analysis in Text Classification
Bioinformatics Multidisciplinary Environment (BioME)(生物信息学多学科环境(BioME));Digital Metropolis Institute (IMD)(数字大都市研究所(IMD));Federal University of Rio Grande do Norte (UFRN)(北里奥格兰德联邦大学(UFRN));Mobile Brain(移动大脑(公司));Institute of Psychiatry (IPUB)(精神病学研究所(IPUB));Federal University of Rio de Janeiro (UFRJ)(里约热内卢联邦大学(UFRJ));The University of Exeter(埃克塞特大学);Institute for Data Science and Artificial Intelligence at the University of Exeter(埃克塞特大学数据科学与人工智能研究所);The University of Sheffield(谢菲尔德大学);Pontifical Catholic University of Rio Grande do Sul (PUCRS)(南里奥格兰德天主教大学(PUCRS))
Explaining Character-Aware Neural Networks for Word-Level Prediction: Do They Discover Linguistic Rules?
IDLab, Ghent University - imec(根特大学-imec IDLab实验室)
当前,在不同的基于神经网络的自然语言处理算法中使用了语义级特征。然而,人们对这些模型学习的字符级模式知之甚少。此外,模型往往只是定量比较,而缺乏定性分析。在本文中,我们研究了神经网络学习哪些字符级模式,以及这些模式是否与手动定义的分词和注释一致。为此,我们将上下文分解技术(Murdoch et al. 2018)扩展到卷积神经网络,这使我们能够比较卷积神经网络和双向长短期记忆网络。我们评估和比较这些模型的任务,形态标记三种不同的语言,并表明这些模型隐含地发现可理解的语言规则。我们的实现可以在https://github.com/FredericGodin/ContextualDecomposition-NLP上找到。
Challenges in Explaining Pretrained Clinical Text Classifiers
Faculty of Computer and Information Science, University of Ljubljana, Slovenia(卢布尔雅那大学计算机与信息科学学院);ICAM - Advanced Environmental Research Institute, West University of Timisoara, Romania(蒂米什瓦德西大学先进环境研究所)
A Systematic Analysis of Linguistic Features in AI-Generated English Text Detection Across Domains and Models
Institute for Natural Language Processing, University of Stuttgart(斯图加特大学自然语言处理研究所);Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart(智能系统反思交流论坛,斯图加特大学);GESIS Leibniz Institute for the Social Sciences(莱比锡社会科学院);Heinrich-Heine University Düsseldorf(杜塞尔多夫海因里希-海涅大学)
DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection
NLP2 CT Lab, Department of Computer and Information Science, University of Macau(澳门大学计算机与信息科学学院NLP2 CT实验室);Alibaba Group(阿里巴巴集团);Xiamen University(厦门大学)
Fake News Detectors are Biased against Texts Generated by Large Language Models
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Monash University(莫纳什大学);CSIRO’s Data61(澳大利亚联邦科学与工业研究组织Data61部门)
MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark
Kempelen Institute of Intelligent Technologies(肯佩伦智能技术研究院);The Pennsylvania State University(宾夕法尼亚州立大学);University of Mississippi(密西西比大学);MIT Lincoln Laboratory(麻省理工学院林肯实验室)