arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.05485cs.CLcs.IRcs.LG
收藏

S2 Chunking:通过整合空间与语义分析实现文档分割的混合框架

S2 Chunking: A Hybrid Framework for Document Segmentation Through Integrated Spatial and Semantic Analysis

Indian Institute of Technology, Patna(印度理工学院巴特那分校)

Prashant Verma

AI总结:

本文提出 S2 Chunking 混合框架,融合 bbox 空间布局、文本语义与元素关系构建加权图并进行谱聚类,在多样版式文档上提升分块准确性并满足 token 长度限制。

摘要:

文档分块是自然语言处理(NLP)中的一项关键任务,涉及将文档划分为有意义的片段。传统方法通常仅依赖语义分析,而忽略元素的空间布局;在复杂文档中,空间布局对于理解元素间关系至关重要。本文提出一种新颖的混合方法,将布局结构、语义分析和空间关系相结合,以提升文档块的内聚性和准确性。该方法利用边界框信息(bbox)和文本嵌入,构建文档元素的加权图表示,随后使用谱聚类对其进行聚类。实验结果表明,该方法优于传统方法,尤其适用于报告、文章和多栏排版等布局多样的文档。所提方法还确保没有任何分块超过指定的 token 长度,因此适合对 token 限制要求严格的用例(例如具有输入大小限制的语言模型)。

英文摘要:

Document chunking is a critical task in natural language processing (NLP) that involves dividing a document into meaningful segments. Traditional methods often rely solely on semantic analysis, ignoring the spatial layout of elements, which is crucial for understanding relationships in complex documents. This paper introduces a novel hybrid approach that combines layout structure, semantic analysis, and spatial relationships to enhance the cohesion and accuracy of document chunks. By leveraging bounding box information (bbox) and text embeddings, our method constructs a weighted graph representation of document elements, which is then clustered using spectral clustering. Experimental results demonstrate that this approach outperforms traditional methods, particularly in documents with diverse layouts such as reports, articles, and multi-column designs. The proposed method also ensures that no chunk exceeds a specified token length, making it suitable for use cases where token limits are critical (e.g., language models with input size limitations)

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑