arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.01601cs.AI
收藏

SemHash-LLM:一种用于文档去重的多粒度语义哈希框架

SemHash-LLM: A Multi-Granularity Semantic Hashing Framework for Document Deduplication

Xinyi Fang, Kejian Tong, Jiabei Liu, Tao Ning, Yuhang He

AI总结:

提出多粒度语义哈希框架SemHash-LLM,融合语义投影哈希、注意力加权MinHash、对比边界学习和选择性LLM裁决,实现高效的大规模文档去重。

摘要:

大规模文档去重必须在保持语义等价性的同时,在庞大的语料库上保持高效。我们提出了SemHash LLM,一个多粒度框架,统一了语义投影哈希、注意力加权MinHash、对比边界学习和基于LLM的选择性裁决。该方法通过门控融合结合字符级、令牌级和文档级信号,然后应用级联过滤管道进行高效的候选缩减。语义投影哈希在蒸馏的LLM嵌入空间中学习紧凑的二进制码,而注意力加权MinHash抑制模板内容并强调信息性内容。自适应决策边界和不确定性估计进一步提高了对模板污染、短文本扰动、包含和病毒片段的鲁棒性。实验表明,SemHash LLM以不到百分之一的神经验证成本实现了强大的重复检测质量。

英文摘要:

Large scale document deduplication must preserve semantic equivalence while remaining efficient over massive corpora. We present SemHash LLM, a multi granularity framework that unifies semantic projection hashing, attention weighted MinHash, contrastive boundary learning, and selective LLM based adjudication. The method combines character, token, and document level signals through gated fusion, then applies a cascaded filtering pipeline for efficient candidate reduction. Semantic projection hashing learns compact binary codes in distilled LLM embedding space, while attention weighted Min- Hash suppresses boilerplate and emphasizes informative content. Adaptive decision boundaries and uncertainty estimation further improve robustness across template pollution, short text perturbation, containment, and viral fragments. Experiments show that SemHash LLM achieves strong duplicate detection quality with less than one percent neural verification cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑