通过注入错误信息攻击开放域问答
Attacking Open-domain Question Answering by Injecting Misinformation
AI总结：
本文研究开放域问答模型对错误信息污染的脆弱性，通过注入人工及模型生成的虚假文档进行攻击实验，发现模型性能显著下降，并提出需构建问答与错误信息检测联合的错误信息感知QA系统。
摘要：
随着宣传、新闻和社交媒体中虚假、不准确和误导性信息的增多，现实世界中的问答（QA）系统面临着在被错误信息污染的语境中进行综合与推理以得出正确答案的挑战。这一紧迫性使得让QA系统对错误信息具有鲁棒性成为必要，而该主题此前尚未被探索。我们通过研究开放域QA模型对含有错误信息文档的语料库污染的敏感性，来考察错误信息对QA模型的风险。我们整理了人工编写和模型生成的虚假文档，将其注入QA模型的证据语料库中，并评估对这些系统性能的影响。实验表明，QA模型即使面对少量由错误信息带来的证据污染也十分脆弱，所有模型都出现了大幅的绝对性能下降。当虚假文档由神经模型大规模生成或攻击者针对性地攻击特定感兴趣的问题时，错误信息攻击会带来更大威胁。为了防御这种威胁，我们讨论了构建一个具备错误信息感知能力的QA系统的必要性，该系统以联合方式整合问答与错误信息检测。
英文摘要：
With a rise in false, inaccurate, and misleading information in propaganda, news, and social media, real-world Question Answering (QA) systems face the challenges of synthesizing and reasoning over misinformation-polluted contexts to derive correct answers. This urgency gives rise to the need to make QA systems robust to misinformation, a topic previously unexplored. We study the risk of misinformation to QA models by investigating the sensitivity of open-domain QA models to corpus pollution with misinformation documents. We curate both human-written and model-generated false documents that we inject into the evidence corpus of QA models and assess the impact on the performance of these systems. Experiments show that QA models are vulnerable to even small amounts of evidence contamination brought by misinformation, with large absolute performance drops on all models. Misinformation attack brings more threat when fake documents are produced at scale by neural models or the attacker targets hacking specific questions of interest. To defend against such a threat, we discuss the necessity of building a misinformation-aware QA system that integrates question-answering and misinformation detection in a joint fashion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
