Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs. Context compression techniques can reduce these costs, but the most effective approaches require fine-tuning the target model or even modifying its architecture. This can degrade its general abilities when not used for this specific purpose. Here we explore an alternative approach: an encoder that compresses the context into continuous representations which replace token embeddings in decoder LLMs. First, we perform a systematic study of training strategies and architecture choices for the encoder. Our findings led to the design of an Adaptable text Representations Compressor, named ARC-Encoder, which outputs $x$-times fewer continuous representations (typically $x\!\in\!\{4,8\}$) than text tokens. We evaluate ARC-Encoder across a variety of LLM usage scenarios, ranging from in-context learning to context window extension, on both instruct and base decoders. Results show that ARC-Encoder achieves state-of-the-art performance on several benchmarks while improving computational efficiency at inference. Finally, we demonstrate that our models can be adapted to multiple decoders simultaneously, allowing a single encoder to generalize across different decoder LLMs. This makes ARC-Encoder a flexible and efficient solution for portable encoders that work seamlessly with multiple LLMs. We release a training code at https://github.com/kyutai-labs/ARC-Encoder , fine-tuning dataset and pretrained models are available at https://huggingface.co/collections/kyutai/arc-encoders-68ee18787301407d60a57047 .
Exploring the Potential of Encoder-free Architectures in 3D LMMs
Northwestern Polytechnical University(西北工业大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);Tele AI
Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Models
SIRE, IIT Delhi(IIT德里SIRE);Robert Bosch GmbH, India(印度罗伯特博世集团);IIIT Hyderabad(海得拉巴IIIT);IISER Kolkata(科钦IISER);Microsoft and the University of York(微软和约克大学)
DiffuSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusion
National Engineering Research Center of Speech and Language Information Processing(国家语音及语言信息处理工程技术研究中心);University of Science and Technology of China(中国科学技术大学)
OCR for TIFF Compressed Document Images Directly in Compressed Domain Using Text segmentation and Hidden Markov Model
Indian Institute of Information Technology Allahabad(印度理工学院阿拉哈巴德分校)
在当今的技术时代,文档图像在我们的日常生活中发挥着重要和不可或缺的作用,特别是随着COVID-19的激增,数字扫描文档已成为沟通的主要来源,从而避免了任何形式的身体接触感染。扫描文档图像的存储和传输是一项非常占用内存的任务,因此压缩技术被用于在存档和传输之前减小图像大小。为了提取信息或对压缩图像进行操作,我们有两种方法:第一种方法是对图像进行压缩和操作,然后再次压缩图像,以提高存储和传输的效率。另一种方法是使用底层压缩算法的特性直接处理压缩形式的图像,而不涉及解压缩和重新压缩。本文提出了一种新的思路,即在压缩域中直接为CCITT(The International Telegraph and Telephone Consultative Committee,国际电报电话咨询委员会)压缩机打印的TIFF文档图像开发OCR。在将文本区域分割成行和词之后,使用CCITT的三种编码模式(水平、垂直和通过模式)应用HMM进行识别。实验结果表明,光学字符识别的通过模式给出了一个有前途的结果。