Card-660：剑桥稀有词数据集-稀有词表示模型的可靠基准
Card-660: Cambridge Rare Word Dataset - a Reliable Benchmark for Infrequent Word Representation Models
摘要：
近年来，由于对稀有词的有效处理在准确的语义理解中起着至关重要的作用，稀有词的表示受到了人们的极大关注。然而，缺乏可靠的基准来评估和比较这些技术。我们在本文中表明，唯一现有的基准（斯坦福大学稀有词数据集）存在低置信度注释和有限词汇的问题;因此，它并不构成可靠的比较框架。为了填补这一评估空白，我们提出了CAmbridge稀有词数据集（Card-660），这是一个专家注释的词相似度数据集，为稀有词表示技术提供了一个高度可靠但具有挑战性的基准。通过一组实验，我们表明，即使是最好的主流词嵌入，在其词汇表中有数百万个单词，也无法在数据集上实现高于0.43（Pearson相关性）的性能，而人类水平的上限为0.90。我们在https://pilehvar.github.io/card-660/上发布数据集和注释材料。
英文摘要：
Rare word representation has recently enjoyed a surge of interest, owing to the crucial role that effective handling of infrequent words can play in accurate semantic understanding. However, there is a paucity of reliable benchmarks for evaluation and comparison of these techniques. We show in this paper that the only existing benchmark (the Stanford Rare Word dataset) suffers from low-confidence annotations and limited vocabulary; hence, it does not constitute a solid comparison framework. In order to fill this evaluation gap, we propose CAmbridge Rare word Dataset (Card-660), an expert-annotated word similarity dataset which provides a highly reliable, yet challenging, benchmark for rare word representation techniques. Through a set of experiments we show that even the best mainstream word embeddings, with millions of words in their vocabularies, are unable to achieve performances higher than 0.43 (Pearson correlation) on the dataset, compared to a human-level upperbound of 0.90. We release the dataset and the annotation materials at https://pilehvar.github.io/card-660/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
