跨文档主题对齐的分块技术用于检索增强生成
Cross-Document Topic-Aligned Chunking for Retrieval-Augmented Generation
AI总结：
本文提出跨文档主题对齐分块技术，通过语料层面的知识重建提升检索增强生成系统的忠实度和信息密度，实验显示其在多跳推理和法律文本任务中均取得显著性能提升。
摘要：
分块质量决定了RAG系统的性能。当前方法对文档进行单独划分，但复杂的查询需要信息分散在多个来源中：知识碎片化问题。我们引入了跨文档主题对齐（CDTA）分块技术，该技术在语料层面重建知识。它首先在文档间识别主题，将段落映射到每个主题，并将它们合成统一的分块。在HotpotQA多跳推理任务中，我们的方法达到了0.93的忠实度，相较于上下文检索的0.83和语义分块的0.78，比当前行业最佳实践提高了12%（p < 0.05）。在UAE法律文本中，它达到了0.94的忠实度，同时具有0.93的引用准确性。在k = 3时，它保持0.91的忠实度，而语义方法降至0.68，单个CDTA分块包含需要多个传统分块的信息。索引成本更高，但合成产生信息密集的分块，减少了查询时的检索需求。对于高查询量应用和分布式知识，跨文档合成在可衡量上优于文档内优化。
英文摘要：
Chunking quality determines RAG system performance. Current methods partition documents individually, but complex queries need information scattered across multiple sources: the knowledge fragmentation problem. We introduce Cross-Document Topic-Aligned (CDTA) chunking, which reconstructs knowledge at the corpus level. It first identifies topics across documents, maps segments to each topic, and synthesizes them into unified chunks. On HotpotQA multi-hop reasoning, our method reached 0.93 faithfulness versus 0.83 for contextual retrieval and 0.78 for semantic chunking, a 12% improvement over current industry best practice (p < 0.05). On UAE Legal texts, it reached 0.94 faithfulness with 0.93 citation accuracy. At k = 3, it maintains 0.91 faithfulness while semantic methods drop to 0.68, with a single CDTA chunk containing information requiring multiple traditional fragments. Indexing costs are higher, but synthesis produces information-dense chunks that reduce query-time retrieval needs. For high-query-volume applications with distributed knowledge, cross-document synthesis improves measurably over within-document optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
