Contextual question-answering models are susceptible to adversarial perturbations to input context, commonly observed in real-world scenarios. These adversarial noises are designed to degrade the performance of the model by distorting the textual input. We introduce a unique dataset that incorporates seven distinct types of adversarial noise into the context, each applied at five different intensity levels on the SQuAD dataset. To quantify the robustness, we utilize robustness metrics providing a standardized measure for assessing model performance across varying noise types and levels. Experiments on transformer-based question-answering models reveal robustness vulnerabilities and important insights into the model's performance in realistic textual input.
COBIAS: Assessing the Contextual Reliability of Bias Benchmarks for Language Models
International Institute of Information Technology, Hyderabad(国际信息技术学院海得拉巴分校);Amazon GenAI(亚马逊生成式人工智能);University of Maryland Baltimore County(马里兰大学巴尔的摩县分校)
Is Context Helpful for Chat Translation Evaluation?
Instituto de Telecomunicações(电信研究所);Unbabel(Unbabel公司);Instituto Superior Técnico(高等技术学院);Universidade de Lisboa(里斯本大学);Carnegie Mellon University(卡内基梅隆大学);INESC-ID(INESC-ID研究所)
Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes
JBE-QA: Japanese Bar Exam QA Dataset for Assessing Legal Domain Knowledge
Institute of Science Tokyo, Japan(东京科学研究所,日本);ROIS-DS, Center for Juris-informatics, Japan(司法信息中心,日本);National Institute of Informatics, Japan(日本信息机构);University of Tokyo, Japan(东京大学,日本)