arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.00367cs.IRcs.CL
收藏

拆解难题:面向检索增强生成的领域感知语义分割

Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generation

International Institute of Information Technology, Hyderabad, India(国际信息科技学院,海得拉巴,印度)

Aparajitha Allamraju, Maitreya Prafulla Chitale, Hiranmai Sri Adibhatla, Rahul Mishra, Manish Shrivastava

AI总结:

本文提出两种高效的语义分块方法PSC和MFC,通过PubMed数据训练,提升了检索增强生成的性能和跨领域泛化能力。

摘要:

文档分块是检索增强生成(RAG)的关键组成部分,因为它直接影响相关且精确上下文的检索。传统固定长度和递归分块器往往产生任意、不连贯的块,无法保持语义结构。尽管语义分块已获得 traction,但其对生成质量的影响仍不明确。本文介绍两种高效的语义分块方法,投影相似性分块(PSC)和度量融合分块(MFC),在PubMed数据上使用三种不同的嵌入模型进行训练。我们进一步提出一个评估框架,通过将PubMedQA与完整文本PubMed Central文章结合,衡量分块对检索和生成的影响。我们的结果表明,在MRR上PSC实现了24倍的检索改进,并在PubMedQA上实现了更高的Hits@k。我们提供了全面的分析,包括统计显著性和与常见分块库的响应时间比较。尽管在单一领域上训练,PSC和MFC也表现出良好的泛化能力,在多个数据集中实现了强劲的跨领域生成性能。总体而言,我们的发现证实了我们的语义分块器,尤其是PSC,能够持续提供优越的性能。

英文摘要:

Document chunking is a crucial component of Retrieval-Augmented Generation (RAG), as it directly affects the retrieval of relevant and precise context. Conventional fixed-length and recursive splitters often produce arbitrary, incoherent segments that fail to preserve semantic structure. Although semantic chunking has gained traction, its influence on generation quality remains underexplored. This paper introduces two efficient semantic chunking methods, Projected Similarity Chunking (PSC) and Metric Fusion Chunking (MFC), trained on PubMed data using three different embedding models. We further present an evaluation framework that measures the effect of chunking on both retrieval and generation by augmenting PubMedQA with full-text PubMed Central articles. Our results show substantial retrieval improvements (24x with PSC) in MRR and higher Hits@k on PubMedQA. We provide a comprehensive analysis, including statistical significance and response-time comparisons with common chunking libraries. Despite being trained on a single domain, PSC and MFC also generalize well, achieving strong out-of-domain generation performance across multiple datasets. Overall, our findings confirm that our semantic chunkers, especially PSC, consistently deliver superior performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑