DeDrift:内容漂移下的鲁棒相似性搜索
DeDrift: Robust Similarity Search under Content Drift
AI总结:
针对大规模相似性搜索中查询与数据库分布随时间漂移导致精度下降的问题,本文提出DeDrift方法,通过在线更新嵌入量化器持续适配索引结构,在几乎消除精度退化的同时实现比全量重建快100倍。
摘要:
媒体分享网站上上传和搜索内容的统计分布会因季节性、社会学和技术因素而随时间变化。我们研究了这种“内容漂移”对基于嵌入空间最近邻搜索的大规模相似性搜索工具的影响。除非频繁进行代价高昂的索引重建,否则内容漂移会降低搜索的准确性和效率。由于查询分布和数据库分布通常都会发生变化,这种退化尤为严重。我们引入并分析了在长时间范围内具有时间信息的真实图像和视频数据集。基于这些认识,我们设计了DeDrift,一种更新嵌入量化器以在运行中持续自适应大规模索引结构的方法。DeDrift几乎消除了因查询和数据库内容漂移而导致的准确性退化,同时比完整索引重建快达100倍。
英文摘要:
The statistical distribution of content uploaded and searched on media sharing sites changes over time due to seasonal, sociological and technical factors. We investigate the impact of this "content drift" for large-scale similarity search tools, based on nearest neighbor search in embedding space. Unless a costly index reconstruction is performed frequently, content drift degrades the search accuracy and efficiency. The degradation is especially severe since, in general, both the query and database distributions change. We introduce and analyze real-world image and video datasets for which temporal information is available over a long time period. Based on the learnings, we devise DeDrift, a method that updates embedding quantizers to continuously adapt large-scale indexing structures on-the-fly. DeDrift almost eliminates the accuracy degradation due to the query and database content drift while being up to 100x faster than a full index reconstruction.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。