TAPE：评估少样本俄语语言理解
TAPE: Assessing Few-shot Russian Language Understanding
AI总结：
本文提出俄语少样本 NLU 基准 TAPE，通过六类复杂任务、语言对抗扰动和子群体分析评估模型鲁棒性，并揭示其与人类表现的显著差距。
摘要：
零样本和少样本学习的最新进展已在一系列研究和实际用途中展现出前景。然而，这一快速发展的领域缺乏面向非英语语言的标准化评测套件，阻碍了以英语为中心范式之外的进展。为解决这一研究方向，我们提出 TAPE（Text Attack and Perturbation Evaluation，文本攻击与扰动评估），这是一个新颖的基准，包含六个更复杂的俄语 NLU 任务，涵盖多跳推理、伦理概念、逻辑和常识知识。TAPE 的设计聚焦于系统化的零样本和少样本 NLU 评估：（i）用于分析鲁棒性的面向语言学的对抗攻击与扰动，以及（ii）用于细致解释的子群体。对自回归基线测试的详细分析表明，简单的基于拼写的扰动对性能影响最大，而改写输入的影响则更可忽略。同时，结果显示在大多数任务上，神经基线与人类基线之间存在显著差距。我们公开发布 TAPE（tape-benchmark.com），以促进对鲁棒 LM 的研究，使其在几乎没有或完全没有监督可用时能够泛化到新任务。
英文摘要：
Recent advances in zero-shot and few-shot learning have shown promise for a scope of research and practical purposes. However, this fast-growing area lacks standardized evaluation suites for non-English languages, hindering progress outside the Anglo-centric paradigm. To address this line of research, we propose TAPE (Text Attack and Perturbation Evaluation), a novel benchmark that includes six more complex NLU tasks for Russian, covering multi-hop reasoning, ethical concepts, logic and commonsense knowledge. The TAPE's design focuses on systematic zero-shot and few-shot NLU evaluation: (i) linguistic-oriented adversarial attacks and perturbations for analyzing robustness, and (ii) subpopulations for nuanced interpretation. The detailed analysis of testing the autoregressive baselines indicates that simple spelling-based perturbations affect the performance the most, while paraphrasing the input has a more negligible effect. At the same time, the results demonstrate a significant gap between the neural and human baselines for most tasks. We publicly release TAPE (tape-benchmark.com) to foster research on robust LMs that can generalize to new tasks when little to no supervision is available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
