基于意图的动态分块：使文档分块反映预测的信息需求
Intent-Driven Dynamic Chunking: Segmenting Documents to Reflect Predicted Information Needs
AI总结：
本文提出基于意图的动态分块方法，通过预测用户查询优化文档分块，提升检索性能，尤其在长文档中效果显著。
摘要：
将长文档分成较小的段落是信息检索中的基本挑战。无论是搜索引擎、问答系统还是检索增强生成（RAG），有效的分段决定了系统如何定位并返回相关信息。然而，传统方法，如固定长度或基于连贯性的分段，忽略了用户意图，导致分块将答案分割或包含无关的噪声。我们引入了意图驱动的动态分块（IDC），一种新的方法，利用预测的用户查询来指导文档分段。IDC利用大型语言模型生成文档的可能用户意图，然后使用动态规划算法找到全局最优的分块边界。这代表了动态规划在意图感知分段中的新应用，避免了贪心方法的陷阱。我们对六个多样化的问答数据集进行了IDC评估，包括新闻文章、维基百科、学术论文和技术文档。IDC在五个数据集上优于传统分块策略，将top-1检索准确率提高了5%至67%，并在第六个数据集上与最佳基线相匹配。此外，IDC产生的分块比基线方法少40-60%，同时实现了93-100%的答案覆盖率。这些结果表明，将文档结构与预期的信息需求对齐显著提升了检索性能，特别是对于长且异质的文档。
英文摘要：
Breaking long documents into smaller segments is a fundamental challenge in information retrieval. Whether for search engines, question-answering systems, or retrieval-augmented generation (RAG), effective segmentation determines how well systems can locate and return relevant information. However, traditional methods, such as fixed-length or coherence-based segmentation, ignore user intent, leading to chunks that split answers or contain irrelevant noise. We introduce Intent-Driven Dynamic Chunking (IDC), a novel approach that uses predicted user queries to guide document segmentation. IDC leverages a Large Language Model to generate likely user intents for a document and then employs a dynamic programming algorithm to find the globally optimal chunk boundaries. This represents a novel application of DP to intent-aware segmentation that avoids greedy pitfalls. We evaluated IDC on six diverse question-answering datasets, including news articles, Wikipedia, academic papers, and technical documentation. IDC outperformed traditional chunking strategies on five datasets, improving top-1 retrieval accuracy by 5% to 67%, and matched the best baseline on the sixth. Additionally, IDC produced 40-60% fewer chunks than baseline methods while achieving 93-100% answer coverage. These results demonstrate that aligning document structure with anticipated information needs significantly boosts retrieval performance, particularly for long and heterogeneous documents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
