TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages

TyDi QA-WANA：西亚与北非语言信息检索型问答基准

TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa

本文构建 TyDi QA-WANA 基准，覆盖西亚和北非 10 种语言变体的 28K 个原生信息检索型问答样本，并提供两个基线模型及开放代码数据。

TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languages

We study politeness phenomena in nine typologically diverse languages. Politeness is an important facet of communication and is sometimes argued to be cultural-specific, yet existing computational linguistic study is limited to English. We create TyDiP, a dataset containing three-way politeness annotations for 500 examples in each language, totaling 4.5K examples. We evaluate how well multilingual models can identify politeness levels -- they show a fairly robust zero-shot transfer ability, yet fall short of estimated human accuracy significantly. We further study mapping the English politenes…

Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval

We present Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. The goal of this resource is to spur research in dense retrieval techniques in non-English languages, motivated by recent observations that existing techniques for representation learning perform poorly when applied to out-of-distribution data. As a starting point, we provide zero-shot baselines for this new dataset based on a multi-lingual adaptation of DPR that we call "mDPR". Experiments show that althou…

面向低资源语言的跨语言知识蒸馏用于答案句选择

Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages

本文提出跨语言知识蒸馏方法CLKD，利用英语AS2教师模型在无目标语言标注数据的情况下训练低资源语言AS2模型，并引入Xtr-WikiQA和TyDi-AS2数据集验证其有效性。

Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval

Recent pretrained language models "solved" many reading comprehension benchmarks, where questions are written with access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA. Our controlled experiments suggest two headrooms -- paragraph selection and answerability prediction, i.e. whether the paired evidence …

用于透明比较多语言 NLP 数据集语言多样性的一种度量

A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets

本文提出基于参考语言样本、Jaccard 指数与文本特征的语言多样性度量，用于解释并比较多语言 NLP 数据集，分析九个常用数据集并发现多数缺少（多式）综合语。