评估AI文本检测器的性能：基于DeepSeek生成文本的少样本与思维链提示研究
Evaluating the Performance of AI Text Detectors, Few-Shot and Chain-of-Thought Prompting Using DeepSeek Generated Text
AI总结：
本文评估六款AI检测器识别DeepSeek-v3文本及抵御改写、拟人化攻击的能力，并测试DeepSeek少样本与CoT分类，发现拟人化攻击最有效，五样本提示仅误判1例。
摘要：
大型语言模型（LLM）迅速改变了书面材料的创作方式。LLM引发了关于写作诚信的质疑，从而推动了人工智能（AI）检测技术的发展。标准改写和拟人化改写等对抗性攻击会削弱检测器识别机器生成文本的能力。以往研究主要关注ChatGPT及其他知名LLM，并显示不同检测器的准确率存在差异。然而，关于近期发布的LLM DeepSeek，文献中存在明显空白。因此，本研究考察六款普遍可用的AI检测工具——AI Text Classifier、Content Detector AI、Copyleaks、QuillBot、GPT-2和GPTZero——能否稳定识别DeepSeek生成的文本。研究让这些检测器面对上述对抗性攻击。我们还将DeepSeek视为检测器，通过少样本提示和思维链推理（CoT）对AI文本与人类撰写文本进行分类。我们收集了LLM时代之前的49组人工撰写问答对，并使用DeepSeek-v3生成匹配回答，得到49个AI生成样本。随后，我们应用改写和拟人化等对抗技术，又增加196个样本。这些样本用于挑战检测器的鲁棒性并评估其准确率影响。QuillBot和Copyleaks在原始及改写后的DeepSeek文本上表现出近乎完美的性能，而其他检测器——尤其是AI Text Classifier和GPT-2——结果不稳定。最有效的攻击是拟人化，它将Copyleaks的准确率降至71%，QuillBot降至58%，GPTZero降至52%。少样本和CoT提示表现出较高准确率，其中最佳的五样本结果在49个样本中仅误分类一个（AI召回率96%，人类文本召回率100%）。
英文摘要：
Large language models (LLMs) have rapidly transformed the creation of written materials. LLMs have led to questions about writing integrity, thereby driving the creation of artificial intelligence (AI) detection technologies. Adversarial attacks, such as standard and humanized paraphrasing, inhibit detectors' ability to detect machine-generated text. Previous studies have mainly focused on ChatGPT and other well-known LLMs and have shown varying accuracy across detectors. However, there is a clear gap in the literature about DeepSeek, a recently published LLM. Therefore, in this work, we investigate whether six generally accessible AI detection tools -- AI Text Classifier, Content Detector AI, Copyleaks, QuillBot, GPT-2, and GPTZero -- can consistently recognize text generated by DeepSeek. The detectors were exposed to the aforementioned adversarial attacks. We also considered DeepSeek as a detector by performing few-shot prompting and chain-of-thought reasoning (CoT) for classifying AI and human-written text. We collected 49 human-authored question-answer pairs from before the LLM era and generated matching responses using DeepSeek-v3, producing 49 AI-generated samples. Then, we applied adversarial techniques such as paraphrasing and humanizing to add 196 more samples. These were used to challenge detector robustness and assess accuracy impact. While QuillBot and Copyleaks showed near-perfect performance on original and paraphrased DeepSeek text, others -- particularly AI Text Classifier and GPT-2 -- showed inconsistent results. The most effective attack was humanization, reducing accuracy to 71% for Copyleaks, 58% for QuillBot, and 52% for GPTZero. Few-shot and CoT prompting showed high accuracy, with the best five-shot result misclassifying only one of 49 samples (AI recall 96%, human recall 100%).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
