The ability of Large Language Models (LLMs) to process and generate coherent text is markedly weakened when the number of input tokens exceeds their pretraining length. Given the expensive overhead of finetuning large-scale models with longer sequences, we propose Dual Chunk Attention (DCA), which enables Llama2 70B to support context windows of more than 100k tokens without continual training. By decomposing the attention computation for long sequences into chunk-based modules, DCA manages to effectively capture the relative positional information of tokens within the same chunk (Intra-Chunk) and across distinct chunks (Inter-Chunk), as well as integrates seamlessly with Flash Attention. In addition to its impressive extrapolation capability, DCA achieves performance on practical long-context tasks that is comparable to or even better than that of finetuned models. When compared with proprietary models, our training-free 70B model attains 94% of the performance of gpt-3.5-16k, indicating it is a viable open-source alternative. All code and data used in this work are released at \url{https://github.com/HKUNLP/ChunkLlama}.
DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search
TJUNLP Lab, College of Intelligence and Computing, Tianjin University(天津大学智能计算学院);School of New Media and Communication, Tianjin University(天津大学新媒体与传播学院)
Beyond Many-Shot Translation: Scaling In-Context Demonstrations For Low-Resource Machine Translation
Institute of Information Science, Academia Sinica(台湾“中央研究院”信息科学研究所);National Taiwan University of Science and Technology(台湾科技大学);Ecole Centrale de Marseille(马赛中央理工学院);National University of Singapore(新加坡国立大学)
Intrinsic Entropy of Context Length Scaling in LLMs
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Carnegie Mellon University(卡内基梅隆大学);CPHOS;University of Washington(华盛顿大学)
Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China(计算机科学与技术系,人工智能研究院,清华大学,北京,中国);Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究院(AIR),清华大学,北京,中国);Institute of Intelligent Computing, Alibaba Group(智能计算研究院,阿里巴巴集团)
SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
Department of Convergence IT Engineering(融合IT工程系);Department of Computer Science and Engineering(计算机科学与工程系);Graduate School of Artificial Intelligence(人工智能研究生院)
Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Xiaohongshu Inc(小红书科技有限公司)
L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling
NSF AI Institute for Artificial Intelligence and Fundamental Interactions(NSF人工智能与基础相互作用研究院);Massachusetts Institute of Technology(麻省理工学院);Polytechnic University of Catalonia(加泰罗尼亚理工大学);Harvard University(哈佛大学);University of California, Los Angeles(加州大学洛杉矶分校)
Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system
College of Engineering, Peking University(北京大学工学院);Ningbo Institute of Digital Twin, Eastern Institute of Technology(东方理工高等研究院宁波数字孪生研究院);Shenzhen Tenfong Technology Co., Ltd.(深圳腾方科技有限公司);Mechanical Engineering, Pennsylvania State University(宾夕法尼亚州立大学机械工程系)
Data Engineering for Scaling Language Models to 128K Context
University of Edinburgh(爱丁堡大学);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);University of Melbourne(墨尔本大学);Ohio State University(俄亥俄州立大学);University of Washington(华盛顿大学);MIT(麻省理工学院);UIUC(伊利诺伊大学香槟分校)
One Patient, Many Contexts: Scaling Medical AI with Contextual Intelligence
Harvard Medical School(哈佛医学院);Clalit Research Institute(Clalit研究所);Boston Children’s Hospital(波士顿儿童医院);Harvard Data Science Initiative(哈佛数据科学计划);Beth Israel Deaconess Medical Center(贝斯以色列女执事医疗中心);Harvard T.H. Chan School of Public Health(哈佛大学陈曾熙公共卫生学院);Clalit Health Services(Clalit医疗服务集团);Ben Gurion University(本-古里安大学);Ben Gurion University of the Negev(内盖夫本-古里安大学);Brigham and Women’s Hospital(布莱根妇女医院);Harvard University(哈佛大学);Broad Institute of MIT and Harvard(麻省理工学院与哈佛大学布罗德研究所)
Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoning
University of Cambridge(剑桥大学);Pioneer Center for AI, University of Copenhagen(人工智能先锋中心,哥本哈根大学);Hong Kong University of Science(香港科学大学);University of California San Diego(加州大学圣地亚哥分校);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Peking University(北京大学)