arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.10997cs.CL
收藏

Contextual Breach:评估基于Transformer的问答模型的鲁棒性

Contextual Breach: Assessing the Robustness of Transformer-based QA Models

Rochester Institute of Technology(罗切斯特技术研究所);Islamic University of Technology(伊斯兰技术大学)

Asir Saadat, Nahian Ibn Asad

AI总结:

针对上下文问答模型易受对抗性扰动的问题,本文构建含七种不同类型、五个强度水平对抗性噪声的SQuAD数据集,用标准化鲁棒性指标评估基于Transformer的问答模型,揭示其鲁棒性漏洞及真实场景下的性能见解。

摘要:

上下文问答模型易受输入上下文的对抗性扰动影响,这在现实场景中常见。这些对抗性噪声通过扭曲文本输入来降低模型性能。我们引入一个独特的数据集,在SQuAD数据集的上下文里融入七种不同类型的对抗性噪声,每种噪声在五个不同强度水平上应用。为量化鲁棒性,我们使用鲁棒性指标,为评估不同噪声类型和水平下的模型性能提供标准化度量。在基于Transformer的问答模型上的实验揭示了鲁棒性漏洞,以及关于模型在真实文本输入中性能的重要见解。

英文摘要:

Contextual question-answering models are susceptible to adversarial perturbations to input context, commonly observed in real-world scenarios. These adversarial noises are designed to degrade the performance of the model by distorting the textual input. We introduce a unique dataset that incorporates seven distinct types of adversarial noise into the context, each applied at five different intensity levels on the SQuAD dataset. To quantify the robustness, we utilize robustness metrics providing a standardized measure for assessing model performance across varying noise types and levels. Experiments on transformer-based question-answering models reveal robustness vulnerabilities and important insights into the model's performance in realistic textual input.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑