arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.14848cs.IR
收藏

使用NMSLIB和FlexNeuART灵活检索

Flexible retrieval with NMSLIB and FlexNeuART

Leonid Boytsov, Eric Nyberg

摘要:

我们的目标是向NLP社区介绍现有的k-NN搜索库NMSLIB,新的检索工具包FlexNeuART以及它们的集成功能。NMSLIB是最快的k-NN搜索库之一,它非常通用,支持各种距离/相似度函数。由于该库依赖于基于距离的结构不可知算法,因此可以通过添加新的距离来进一步扩展。FlexNeuART是一个模块化的,可扩展的和灵活的工具包,用于IR和QA应用中的候选生成,支持经典和神经排名信号的混合。FlexNeuART可以有效地检索混合的密集和稀疏表示(从训练数据中学习权重),这是通过扩展NMSLIB实现的。在这方面,其他检索系统使用纯稀疏表示(例如,Lucene)、纯稠密表示(例如,FAISS和Annoy),或者只在重新排序阶段进行混合。

英文摘要:

Our objective is to introduce to the NLP community an existing k-NN search library NMSLIB, a new retrieval toolkit FlexNeuART, as well as their integration capabilities. NMSLIB, while being one the fastest k-NN search libraries, is quite generic and supports a variety of distance/similarity functions. Because the library relies on the distance-based structure-agnostic algorithms, it can be further extended by adding new distances. FlexNeuART is a modular, extendible and flexible toolkit for candidate generation in IR and QA applications, which supports mixing of classic and neural ranking signals. FlexNeuART can efficiently retrieve mixed dense and sparse representations (with weights learned from training data), which is achieved by extending NMSLIB. In that, other retrieval systems work with purely sparse representations (e.g., Lucene), purely dense representations (e.g., FAISS and Annoy), or only perform mixing at the re-ranking stage.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑