自适应分块：为RAG优化分块方法选择
Adaptive Chunking: Optimizing Chunking-Method Selection for RAG
AI总结：
本文提出自适应分块框架，通过五个新指标优化分块策略，提升RAG性能，使答案正确率提升至72%，问题解答数增加30%。
摘要：
检索增强生成（RAG）的效果高度依赖文档分块方式，即如何将文档分割成更小单元进行索引和检索。然而，常用的“一刀切”方法往往无法捕捉多样文本的细微结构和语义。尽管分块在其中起关键作用，但缺乏专门的评估框架，使策略评估和比较变得困难。本文引入自适应分块，通过五个新的内在、文档基础指标：参考完整性（RC）、块内凝聚力（ICC）、文档上下文连贯性（DCC）、块完整性（BI）和大小合规性（SC），直接评估分块质量的关键维度。为了支持该框架，我们还引入了两个新分块器，即LLM-regex分隔器和split-then-merge递归分隔器，以及针对性的后处理技术。在涵盖法律、技术和社会科学领域的多样化语料库上，我们的指标引导自适应方法显著提升了下游RAG性能。不改变模型或提示的情况下，我们的框架提高了RAG结果，将答案正确率提升至72%（从62-64%），并使成功回答的问题数量增加超过30%（65 vs. 49）。这些结果表明，由互补的内在指标引导的自适应、文档感知分块，为更稳健的RAG系统提供了一条实用有效的路径。代码可在https://github.com/ekimetrics/adaptive-chunking获取。
英文摘要：
The effectiveness of Retrieval-Augmented Generation (RAG) is highly dependent on how documents are chunked, that is, segmented into smaller units for indexing and retrieval. Yet, commonly used "one-size-fits-all" approaches often fail to capture the nuanced structure and semantics of diverse texts. Despite its central role, chunking lacks a dedicated evaluation framework, making it difficult to assess and compare strategies independently of downstream performance. We challenge this paradigm by introducing Adaptive Chunking, a framework that selects the most suitable chunking strategy for each document based on a set of five novel intrinsic, document-based metrics: References Completeness (RC), Intrachunk Cohesion (ICC), Document Contextual Coherence (DCC), Block Integrity (BI), and Size Compliance (SC), which directly assess chunking quality across key dimensions. To support this framework, we also introduce two new chunkers, an LLM-regex splitter and a split-then-merge recursive splitter, alongside targeted post-processing techniques. On a diverse corpus spanning legal, technical, and social science domains, our metric-guided adaptive method significantly improves downstream RAG performance. Without changing models or prompts, our framework increases RAG outcomes, raising answers correctness to 72% (from 62-64%) and increasing the number of successfully answered questions by over 30% (65 vs. 49). These results demonstrate that adaptive, document-aware chunking, guided by a complementary suite of intrinsic metrics, offers a practical and effective path to more robust RAG systems. Code available at https://github.com/ekimetrics/adaptive-chunking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
