MoC:面向检索增强生成系统的文本分块学习器混合方法
MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System
AI总结:
本文提出双指标评估分块质量,并设计粒度感知的MoC框架,通过三阶段机制引导分块器生成正则表达式,以提升RAG系统性能。
摘要:
检索增强生成(RAG)虽作为大型语言模型(LLM)的有效补充,但其流程中的文本分块关键环节常被忽视。本文首先引入一种双指标评估方法,包括边界清晰度和分块粘性,以直接量化分块质量。借助该评估方法,我们揭示了传统分块与语义分块在处理复杂上下文细微差别方面的固有局限,从而证实了将LLM整合进分块过程的必要性。为解决基于LLM的方法在计算效率与分块精度之间的固有权衡,我们设计了粒度感知的分块器混合(MoC)框架,该框架包含一个三阶段处理机制。值得注意的是,我们的目标是引导分块器生成结构化的分块正则表达式列表,随后利用这些表达式从原始文本中提取分块。大量实验表明,我们提出的指标和MoC框架均有效解决了分块任务的挑战,揭示了分块核心,同时提升了RAG系统的性能。
英文摘要:
Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline. This paper initially introduces a dual-metric evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable the direct quantification of chunking quality. Leveraging this assessment method, we highlight the inherent limitations of traditional and semantic chunking in handling complex contextual nuances, thereby substantiating the necessity of integrating LLMs into chunking process. To address the inherent trade-off between computational efficiency and chunking precision in LLM-based approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC) framework, which consists of a three-stage processing mechanism. Notably, our objective is to guide the chunker towards generating a structured list of chunking regular expressions, which are subsequently employed to extract chunks from the original text. Extensive experiments demonstrate that both our proposed metrics and the MoC framework effectively settle challenges of the chunking task, revealing the chunking kernel while enhancing the performance of the RAG system.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。