SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
Department of Convergence IT Engineering(融合IT工程系);Department of Computer Science and Engineering(计算机科学与工程系);Graduate School of Artificial Intelligence(人工智能研究生院)
Changhun Lee, Minsang Seok, Jun-gyu Jin, Younghyun Cho, Eunhyeok Park
While many advanced LLMs are designed to handle long sequence data, we can still observe notable quality degradation even within the sequence limit. In this work, we introduce a novel approach called Scaling to Emphasize Attention for Long-context retrieval (SEAL), which enhances the retrieval performance of large language models (LLMs) over long contexts. We observe that specific attention heads are closely tied to long-context retrieval, showing positive or negative correlation with retrieval scores, and adjusting the strength of these heads boosts the quality of LLMs in long context by a large margin. Built on this insight, we propose a learning-based mechanism that leverages generated data to emphasize these heads. By applying SEAL, we achieve significant improvements in long-context retrieval performance across various tasks and models. Additionally, when combined with existing training-free context extension techniques, SEAL extends the contextual limits of LLMs while maintaining highly reliable outputs.
Towards Spatial Transcriptomics-driven Pathology Foundation Models
Department of Pathology, Mass General Brigham, Harvard Medical School, Boston, MA, USA(病理学系,马萨诸塞州总医院与哈佛医学院,波士顿,马萨诸塞州,美国);Department of Computer Science & Technology, University of Cambridge, Cambridge, UK(计算机科学与技术系,剑桥大学,剑桥,英国);Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA(癌症计划,哈佛与麻省理工联合学院,剑桥,马萨诸塞州,美国);Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA(数据科学计划,达纳-法伯癌症研究所,波士顿,马萨诸塞州,美国);Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology, Cambridge, MA, USA(哈佛-麻省理工健康科学与技术 division,麻省理工学院,剑桥,马萨诸塞州,美国);Télécom Paris, Institut Polytechnique de Paris, Paris, France(巴黎电信学院,巴黎理工学院,巴黎,法国);Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA(哈佛大学数据科学倡议,哈佛大学,剑桥,马萨诸塞州,美国)
BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance
FPT Software AI Center(FPT软件AI中心);Aalborg University(奥尔堡大学);Pioneer Centre for AI(先锋人工智能中心);University of Liverpool(利物浦大学);AICV Lab, University of Arkansas(AICV实验室,阿肯色大学)
An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
"Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models
Arizona State University(亚利桑那州立大学);Michigan State University(密歇根州立大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Virginia(弗吉尼亚大学)
Trustworthiness in Retrieval-Augmented Generation Systems: A Survey
Tsinghua University(清华大学);Renmin University of China(中国人民大学);The Chinese University of Hong Kong(香港中文大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Hong Kong Polytechnic University(香港理工大学);Microsoft Research Asia(微软亚洲研究院);University of Illinois(伊利诺伊大学)
PersonaTree: Structured Lifecycle Memory for Person Understanding in LLM Agents
School of ASEE, Beihang University, Beijing, China(北京航空航天大学航空科学与工程学院);The University of Hong Kong, Hong Kong, China(香港大学);Peking University, Beijing, China(北京大学);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学);School of BME, Beihang University, Beijing, China(北京航空航天大学生物医学工程学院);CAIR and CECS, VinUniversity, Hanoi, Vietnam(越南河内 Vin 大学 CAIR 和 CECS)
ConMem: Contribution-Aware Memory for Long-Horizon Manufacturing Inspection Logs
Shandong University(山东大学);Boston University(波士顿大学);North China Electric Power University(华北电力大学);National University of Singapore(新加坡国立大学);Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University(山东大学-南洋理工大学人工智能联合研究中心(C-FAIR));Rizhao Steel Holding Group Co., Ltd.(日照钢铁控股集团有限公司)
A-MapReduce: Executing Wide Search via Agentic MapReduce
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University(北京未来区块链与隐私计算创新中心,人工智能学院,北京航空航天大学);National University of Singapore(新加坡国立大学);Tsinghua University(清华大学);BNRist, Tsinghua University(BNRist,清华大学)