Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be over-compressed in the embeddings. Consequently, practitioners often split text documents into smaller chunks and encode them separately. However, chunk embeddings created in this way can lose contextual information from surrounding chunks, resulting in sub-optimal representations. In this paper, we introduce a novel method called late chunking, which leverages long context embedding models to first embed all tokens of the long text, with chunking applied after the transformer model and just before mean pooling - hence the term late in its naming. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks. The method is generic enough to be applied to a wide range of long-context embedding models and works without additional training. To further increase the effectiveness of late chunking, we propose a dedicated fine-tuning approach for embedding models.
Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Alibaba Cloud Computing(阿里云);Hong Kong University of Science and Technology(香港科技大学)
Chunking Methods on Retrieval-Augmented Generation - Effectiveness Evaluation Against Computational Cost and Limitations
Department of Artificial Intelligence, Faculty of Information and Communication Technology, Wrocław University of Science and Technology(人工智能系,信息与通信技术学院,沃斯克大学)
FreeChunker: A Cross-Granularity Chunking Framework
Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China(上海教育人工智能研究院,东华师范大学,中国);Education Technology, East China Normal University, China(教育技术,东华师范大学,中国)
Eric Brill介绍了基于转换的学习,并表明它可以以相当高的准确率进行词性标注。相同的方法可以应用于更高级别的文本解释,以定位标记文本中的块,包括非递归的“baseNP”块。为此,可以方便地将组块视为一个标记问题,方法是将组块结构编码为附加到每个单词的新标记。在使用Treebank衍生数据的自动测试中,该技术对baseNP块的召回率和准确率约为92%,对划分句子的更复杂块的召回率和准确率约为88%。一些有趣的适应转换为基础的学习方法也提出了这个应用程序。
School of Natural Sciences, Institute for Advanced Study, Princeton, NJ, 08540, USA(自然科学学院,普林斯顿高级研究院);Department of Brain Sciences, Weizmann Institute of Science, Rehovot, 76100 Israel(脑科学系,魏兹曼科学研究院)
Context-Aware Dynamic Chunking for Streaming Tibetan Speech Recognition
Qinghai Normal University(青海师范大学);University of Electronic Science and Technology of China(电子科技大学);Xizang University(西藏大学);The State Key Laboratory of Tibetan Intelligence(藏语智能 state key laboratory)
Evaluation of Chunking Strategies for Effective Text Embedding in Low-Resource Language on Agricultural Documents
Department of Big Data, Chungbuk National University, Cheongju-si, South Korea(大数据系, Chungbuk国立大学,韩国Cheongju市);Department of Computer Science, Chungbuk National University, Cheongju-si, South Korea(计算机科学系, Chungbuk国立大学,韩国Cheongju市);BigDataLabs Co., Ltd. Department of Management Information Systems, Chungbuk National University, South Korea(BigDataLabs公司 管理信息系, Chungbuk国立大学,韩国)
Beyond Chunking: Discourse-Aware Hierarchical Retrieval for Long Document Question Answering
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)(计算与智能研究院,哈尔滨工业大学(深圳));Shenzhen Loop Area Institute (SLAI)(深圳环形区研究院(SLAI))
QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debate
School of Information, Renmin University of China(中国人民大学信息学院);School of Smart Governance, Renmin University of China(中国人民大学智能治理学院);School of Information, BRAIN, Renmin University of China(中国人民大学信息学院、BRAIN学院);School of Economics and Management, Tsinghua University(清华大学经济管理学院)
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);School of Computer Engineering, Jiangsu Ocean University(江苏海洋大学计算机工程学院)
Efficient RAG with Intent-Aware Retrieval and Semantics-Preserving Chunking
School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);Massachusetts General Hospital, Harvard University(哈佛大学马萨诸塞州总医院);Nota AI