通过分块过滤减少检索增强生成中的冗余
Reducing Redundancy in Retrieval-Augmented Generation through Chunk Filtering
AI总结：
本文通过语义、主题和命名实体过滤策略减少检索增强生成中的冗余，实验表明实体过滤可减少向量索引大小25%-36%并保持高检索质量。
摘要：
标准检索增强生成（RAG）分块方法常产生过多冗余，增加存储成本并减慢检索速度。本研究探索分块过滤策略，如基于语义、主题和命名实体的方法，以减少索引库同时保持检索质量。在多个语料库上进行实验，使用基于精度、召回率和交并比的基于令牌的框架评估检索性能。结果表明，基于实体的过滤可减少向量索引大小约25%至36%，同时保持接近基线的高检索质量。这些发现表明，通过轻量级过滤可有效减少分块过程中引入的冗余，提高RAG流水线中面向检索的组件效率。
英文摘要：
Standard Retrieval-Augmented Generation (RAG) chunking methods often create excessive redundancy, increasing storage costs and slowing retrieval. This study explores chunk filtering strategies, such as semantic, topic-based, and named-entity-based methods in order to reduce the indexed corpus while preserving retrieval quality. Experiments are conducted on multiple corpora. Retrieval performance is evaluated using a token-based framework based on precision, recall, and intersection-over-union metrics. Results indicate that entity-based filtering can reduce vector index size by approximately 25% to 36% while maintaining high retrieval quality close to the baseline. These findings suggest that redundancy introduced during chunking can be effectively reduced through lightweight filtering, improving the efficiency of retrieval-oriented components in RAG pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
