学习术语辨析
Learning Term Discrimination
摘要：
文档索引是高效信息检索（IR）的关键组件。经过词干提取和停用词删除等预处理步骤后，文档索引通常会存储词频（tf）。与tf（仅反映文档中术语的重要性）一起，传统的IR模型使用术语区分值（TDVs），如逆文档频率（idf），以在检索过程中支持区分术语。在这项工作中，我们建议学习TDVs的文档索引与浅神经网络，近似传统的IR排名功能，如TF-IDF和BM 25。我们的建议优于，无论是在nDCG和召回，传统的方法，即使有几个积极标记的查询文档对作为学习数据。当我们学习的TDVs用于过滤掉具有零区分值的词汇表中的术语时，允许显著降低倒排索引的内存占用并加快检索过程（BM 25快达3倍），而不会降低检索质量。
英文摘要：
Document indexing is a key component for efficient information retrieval (IR). After preprocessing steps such as stemming and stop-word removal, document indexes usually store term-frequencies (tf). Along with tf (that only reflects the importance of a term in a document), traditional IR models use term discrimination values (TDVs) such as inverse document frequency (idf) to favor discriminative terms during retrieval. In this work, we propose to learn TDVs for document indexing with shallow neural networks that approximate traditional IR ranking functions such as TF-IDF and BM25. Our proposal outperforms, both in terms of nDCG and recall, traditional approaches, even with few positively labelled query-document pairs as learning data. Our learned TDVs, when used to filter out terms of the vocabulary that have zero discrimination value, allow to both significantly lower the memory footprint of the inverted index and speed up the retrieval process (BM25 is up to 3~times faster), without degrading retrieval quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
