arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.12600cs.CL
收藏

LLM时代恶意社交文本检测中证据污染的风险研究

On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMs

School of Computer Science and Technology, Xi’an Jiaotong University, China(西安交通大学计算机科学与技术学院);Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China(教育部智能网络与网络 security 重点实验室);Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, China(陕西省大数据知识工程重点实验室);SGIT AI Lab, State Grid Corporation of China(国网SGIT人工智能实验室);National University of Defense Technology(国防科技大学)

Herun Wan, Minnan Luo, Zhixiong Su, Guang Dai, Xiang Zhao

AI总结:

本文研究了LLM时代恶意社交文本检测中证据污染的风险,提出三种防御策略,实验表明污染显著降低检测性能并损害模型校准。

摘要:

证据增强检测器在识别恶意社交文本方面展现出卓越能力。然而,大语言模型(LLMs)的兴起带来了证据污染的潜在风险,可能干扰检测器。本文探讨了潜在的操纵场景,包括基础污染,以及由LLMs改写或生成证据。为减轻其负面影响,我们从数据和模型层面提出了三种防御策略:机器生成文本检测、专家混合和参数更新。在四个恶意社交文本检测任务和十个数据集上的大量实验表明,证据污染会显著损害检测器,其中生成策略导致性能下降高达14.4%。同时,防御策略能够缓解证据污染,但在实际应用中面临局限。进一步分析表明,受污染证据(i)由指标和人类评估确认具有高质量;(ii)会损害模型校准,使预期校准误差增加高达21.6%;(iii)可被整合以放大负面影响,尤其是对于基于编码器的语言模型,其准确率下降了21.8%。

英文摘要:

Evidence-enhanced detectors present remarkable abilities in identifying malicious social text. However, the rise of large language models (LLMs) brings potential risks of evidence pollution to confuse detectors. This paper explores potential manipulation scenarios including basic pollution, and rephrasing or generating evidence by LLMs. To mitigate the negative impact, we propose three defense strategies from the data and model sides, including machine-generated text detection, a mixture of experts, and parameter updating. Extensive experiments on four malicious social text detection tasks with ten datasets illustrate that evidence pollution significantly compromises detectors, where the generating strategy causes up to a 14.4% performance drop. Meanwhile, the defense strategies could mitigate evidence pollution, but they faced limitations for practical employment. Further analysis illustrates that polluted evidence (i) is of high quality, evaluated by metrics and humans; (ii) would compromise the model calibration, increasing expected calibration error up to 21.6%; and (iii) could be integrated to amplify the negative impact, especially for encoder-based LMs, where the accuracy drops by 21.8%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑