AI 中文总结
针对大规模训练数据去重中固定探测IVF忽略应用相似度阈值的问题，提出SieveIVF阈值感知IVF执行器，在Lance中实现后可显著提升搜索速度且召回率损失极小。
AI 中文摘要
基于嵌入的训练数据去重会检索出高于应用相似度阈值的候选重复边，但固定探测倒排文件（IVF）搜索在为每个查询分配相同分区预算时忽略了这一谓词。在四个混元（Hunyuan）工作负载中，符合条件的邻居尽管搜索深度差异很大，仍会较早出现。我们提出SieveIVF，一种阈值感知IVF执行器，它在W次连续搜索未找到符合条件的候选后停止。系统层面的挑战在于，当每个查询的剩余工作依赖于先前结果时，需保持以分区为主的批处理。连续批处理按分区分组就绪查询；前瞻调度器构建于其上，仅暴露停止规则确定的已承诺工作，以在不改变停止决策或返回结果的前提下提升并发度。我们在Lance中实现了SieveIVF。在W=8时，SieveIVF在四个10M混元工作负载上比固定探测IVF快4.1至7.6倍，在两个公开100M工作负载上快6.1至8.4倍，且在相同索引和搜索参数下，混元工作负载的汇总过滤前10召回率损失为0.03至1.13个百分点，公开工作负载为1.43至2.29个百分点。这些结果表明，应用谓词可在不改变索引或有界前k接口的情况下指导IVF工作分配。
英文摘要
Embedding-based training data deduplication retrieves candidate duplicate edges above an application similarity threshold, but fixed-probe inverted-file (IVF) search ignores this predicate when giving every query the same partition budget. Across four Hunyuan workloads, qualifying neighbors appear early despite sharply varying search depths. We present SieveIVF, a threshold-aware IVF executor that stops after $W$ consecutive searches find no qualifying candidate. The systems challenge is to preserve partition-major batching when each query's remaining work depends on prior results. Continuous batching groups ready queries by partition. A lookahead scheduler layers on top, exposing only work committed by the stopping rule to increase concurrency without changing stopping decisions or returned results. We implement SieveIVF in Lance. At $W=8$, SieveIVF is $4.1$--$7.6\times$ faster than fixed-probe IVF on four 10M Hunyuan workloads and $6.1$--$8.4\times$ faster on two public 100M workloads under the same index and search parameters, with pooled filtered top-10 recall losses of $0.03$--$1.13$ percentage points on Hunyuan and $1.43$--$2.29$ percentage points on the public workloads. These results show how an application predicate can guide IVF work allocation without changing the index or bounded top-$k$ interface.
