LLM时代恶意社交文本检测中证据污染的风险研究
On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMs
AI总结:
本文研究了LLM时代恶意社交文本检测中证据污染的风险,提出三种防御策略,实验表明污染显著降低检测性能并损害模型校准。
摘要:
证据增强检测器在识别恶意社交文本方面展现出卓越能力。然而,大语言模型(LLMs)的兴起带来了证据污染的潜在风险,可能干扰检测器。本文探讨了潜在的操纵场景,包括基础污染,以及由LLMs改写或生成证据。为减轻其负面影响,我们从数据和模型层面提出了三种防御策略:机器生成文本检测、专家混合和参数更新。在四个恶意社交文本检测任务和十个数据集上的大量实验表明,证据污染会显著损害检测器,其中生成策略导致性能下降高达14.4%。同时,防御策略能够缓解证据污染,但在实际应用中面临局限。进一步分析表明,受污染证据(i)由指标和人类评估确认具有高质量;(ii)会损害模型校准,使预期校准误差增加高达21.6%;(iii)可被整合以放大负面影响,尤其是对于基于编码器的语言模型,其准确率下降了21.8%。
英文摘要:
Evidence-enhanced detectors present remarkable abilities in identifying malicious social text. However, the rise of large language models (LLMs) brings potential risks of evidence pollution to confuse detectors. This paper explores potential manipulation scenarios including basic pollution, and rephrasing or generating evidence by LLMs. To mitigate the negative impact, we propose three defense strategies from the data and model sides, including machine-generated text detection, a mixture of experts, and parameter updating. Extensive experiments on four malicious social text detection tasks with ten datasets illustrate that evidence pollution significantly compromises detectors, where the generating strategy causes up to a 14.4% performance drop. Meanwhile, the defense strategies could mitigate evidence pollution, but they faced limitations for practical employment. Further analysis illustrates that polluted evidence (i) is of high quality, evaluated by metrics and humans; (ii) would compromise the model calibration, increasing expected calibration error up to 21.6%; and (iii) could be integrated to amplify the negative impact, especially for encoder-based LMs, where the accuracy drops by 21.8%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。