下一篇:让我们用k-NN搜索取代基于术语的检索
Off the Beaten Path: Let's Replace Term-Based Retrieval with k-NN Search
摘要:
检索管道通常依赖于基于术语的搜索来获得候选记录,这些候选记录随后被重新排名。这种方法错过了一些候选人,例如,因为词汇不匹配我们解决这个问题,取代基于术语的搜索与通用的k-NN检索算法,其中的相似性函数可以考虑到微妙的长期关联。虽然使用此相似性函数的精确蛮力k-NN搜索是缓慢的,但我们证明了近似算法可以快近两个数量级,代价是精度损失很小。使用近似k-NN搜索的检索流水线可以比基于术语的流水线更有效和高效。这为设计有效的检索管道开辟了新的可能性。我们的软件(包括数据生成代码)和基于Stack Overflow收集的衍生数据可在线获得。
英文摘要:
Retrieval pipelines commonly rely on a term-based search to obtain candidate records, which are subsequently re-ranked. Some candidates are missed by this approach, e.g., due to a vocabulary mismatch. We address this issue by replacing the term-based search with a generic k-NN retrieval algorithm, where a similarity function can take into account subtle term associations. While an exact brute-force k-NN search using this similarity function is slow, we demonstrate that an approximate algorithm can be nearly two orders of magnitude faster at the expense of only a small loss in accuracy. A retrieval pipeline using an approximate k-NN search can be more effective and efficient than the term-based pipeline. This opens up new possibilities for designing effective retrieval pipelines. Our software (including data-generating code) and derivative data based on the Stack Overflow collection is available online.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。