软余弦测度的实现说明
Implementation Notes for the Soft Cosine Measure
摘要：
标准的词袋向量空间模型（VSM）是信息检索中普遍存在的有效模型，但它低估了语义相同但术语不同的文档之间的相似性。为了克服这个限制，Sidorov等人提出了软余弦测度（SCM），它包含了术语相似性关系。Charlet和Damnati证明了SCM在问答系统中的高效性。然而，Sidorov等人提出的正交归一化算法具有不切实际的时间复杂度$\mathcal O(n^4)$，其中n是词汇表的大小。本文证明了$\mathcal O(n^3)$的一个更严格的最坏情况时间复杂度下界。我们还提出了一个算法来计算文档之间的相似性，我们表明，它的最坏情况下的时间复杂度是$\mathcal O(1)$给定的现实条件。最后，我们描述了在通用向量数据库（如Annoy和Faiss）以及文本搜索引擎（如Apache Lucene和ElasticSearch）的倒排索引中的实现。我们的研究结果使部署的SCM在现实世界的信息检索系统。
英文摘要：
The standard bag-of-words vector space model (VSM) is efficient, and ubiquitous in information retrieval, but it underestimates the similarity of documents with the same meaning, but different terminology. To overcome this limitation, Sidorov et al. proposed the Soft Cosine Measure (SCM) that incorporates term similarity relations. Charlet and Damnati showed that the SCM is highly effective in question answering (QA) systems. However, the orthonormalization algorithm proposed by Sidorov et al. has an impractical time complexity of $\mathcal O(n^4)$, where n is the size of the vocabulary. In this paper, we prove a tighter lower worst-case time complexity bound of $\mathcal O(n^3)$. We also present an algorithm for computing the similarity between documents and we show that its worst-case time complexity is $\mathcal O(1)$ given realistic conditions. Lastly, we describe implementation in general-purpose vector databases such as Annoy, and Faiss and in the inverted indices of text search engines such as Apache Lucene, and ElasticSearch. Our results enable the deployment of the SCM in real-world information retrieval systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
