Processing long contexts is increasingly important for Large Language Models (LLMs) in tasks like multi-turn dialogues, code generation, and document summarization. This paper addresses the challenges of achieving high long-context performance, low computational complexity, and compatibility with pretrained models -- collectively termed the ``impossible triangle''. We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. E2LLM divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. To enhance the LLM's reasoning with these soft prompts, we employ two training objectives: encoder output reconstruction and long-context instruction fine-tuning. Extensive experiments reveal that E2LLM not only outperforms 8 state-of-the-art (SOTA) methods in effectiveness and efficiency for document summarization and question answering, but also achieves the best performance on LongBench v2 among models of comparable size.
Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Models
SIRE, IIT Delhi(IIT德里SIRE);Robert Bosch GmbH, India(印度罗伯特博世集团);IIIT Hyderabad(海得拉巴IIIT);IISER Kolkata(科钦IISER);Microsoft and the University of York(微软和约克大学)
Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);Microsoft Research Asia(微软亚洲研究院);Microsoft(微软)
预先训练的语言模型已经在广泛的自然语言处理(NLP)任务中取得了实质性的进展,但已经被证明对于具有高质量输出要求的自然语言生成任务具有局限性,例如常识生成和广告关键字生成。在这项工作中,我们提出了一种新的知识过滤和对比学习网络(KFCNet),参考外部知识,并取得更好的生成性能。具体来说,我们提出了一个基于BERT的过滤器模型,以消除低质量的候选人,并分别应用对比学习的编码器和解码器中的每一个,在一个通用的编码器-解码器架构。编码器对比模块有助于在编码过程中捕获全局目标语义,解码器对比模块在学习一般特征的同时增强检索到的原型的实用性。在CommonGen基准测试上进行的大量实验表明,我们的模型远远优于先前的最先进水平:BLEU-4为+6.6点(42.5 vs. 35.9),SPICE为+3.7点(33.3 vs. 29.6),CIDER为+1.3点(18.3 vs. 17.0)。我们进一步验证了所提出的对比模块在广告关键词生成上的有效性,并表明我们的模型具有潜在的商业价值。
Causal Prompting for Implicit Sentiment Analysis with Large Language Models
School of Computing Technologies, RMIT University(RMIT大学计算机技术学院);Institute of Innovation, Science and Sustainability, Federation University Australia(联邦大学澳大利亚创新、科学与可持续性研究所);CSIRO’s Data61(CSIRO数据61)
EndoCoT: Scaling Endogenous Chain-of-Thought Reasoning in Diffusion Models
Shanghai AI Laboratory(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiaotong University(上海交通大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
Exploring Task Performance with Interpretable Models via Sparse Auto-Encoders
The University of Sheffield(谢菲尔德大学);Tianjin University(天津大学);University of Manchester(曼彻斯特大学);Kyushu University(九州大学);City University of Hong Kong(香港城市大学)
Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMs
The University of Osaka(大阪大学);Center for Information and Neural Networks(信息与神经网络中心);National Institute of Information and Communications Technology(信息与通信技术国家研究所);Kyoto University(京都大学);NII LLMC(日本信息处理学会LLMC)
ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);School of Electronic and Information Engineering, Anhui University(安徽大学电子与信息学院);Beijing Institute of Technology(北京理工大学);Harbin Institute of Technology(哈尔滨工业大学);Peng Cheng Laboratory(鹏城实验室)
Unifying Specialized Visual Encoders for Video Language Models
Department of Computer Science, Princeton University, Princeton, NJ, United States(普林斯顿大学计算机科学系);Salesforce Research, Palo Alto, CA, United States(Salesforce研究)
MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning
University of Toronto(多伦多大学);Mila-Quebec AI Institute, McGill University(魁北克AI研究所,麦吉尔大学);OntarioTech University(安大略技术大学);Saarland University(萨尔兰州立大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
CVPD at QIAS 2025 Shared Task: An Efficient Encoder-Based Approach for Islamic Inheritance Reasoning
University of the Basque Country UPV/EHU(巴斯克大学UPV/EHU);Institute of Applied Sciences and Intelligent Systems – CNR(应用科学与智能系统研究所 – CNR);Laboratory of LESIA, University of Biskra(LESIA实验室,比斯克拉大学);Sorbonne University Abu Dhabi, UAE(阿布扎赫尔索邦大学)
VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
Om AI Research(Om AI研究机构);Binjiang Institute of Zhejiang University(浙江大学滨江研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Peking University(北京大学);Hong Kong University of Science and Technology(香港科学与技术大学);Chinese University of Hong Kong(香港中文大学);University of California, Santa Barbara(加州大学圣芭芭拉分校)