Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Transformer language models, which can process much longer sequences than models of the past. However, the ways in which such models take advantage of the long-range context remain unclear. In this paper, we perform a fine-grained analysis of two long-range Transformer language models (including the \emph{Routing Transformer}, which achieves state-of-the-art perplexity on the PG-19 long-sequence LM benchmark dataset) that accept input sequences of up to 8K tokens. Our results reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks. Finally, we discover that PG-19 contains a variety of different document types and domains, and that long-range context helps most for literary novels (as opposed to textbooks or magazines).
Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信)
LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,北京通用人工智能研究院)
LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy, Beijing, China(中关村学院,北京,中国);Wuhan AI Research(武汉人工智能研究所)
LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Zhipu AI(智谱AI);Tsinghua University(清华大学)
Beyond Isolated Capabilities: Bridging Long CoT Reasoning and Long-Context Understanding
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)
LongEmotion: Measuring Emotional Intelligence of Large Language Models in Long-Context Interaction
Shenzhen MSU-BIT University(深圳MSU-BIT大学);The University of Hong Kong(香港大学);City University of Hong Kong(香港城市大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究所);Beijing Normal University(北京师范大学);University of Michigan, Ann Arbor(密歇根大学安娜堡分校)
LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
IDEA Research, International Digital Economy Academy(IDEA研究院,国际数字经济学院);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));DataArc Tech Ltd.(DataArc科技有限公司);Hithink RoyalFlush Information Network Co., Ltd(Hithink皇家Flush信息网络有限公司)
LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);National Key Laboratory of General Artificial Intelligence, BIGAI(BIGAI通用人工智能全国重点实验室)
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Xiaohongshu Inc(小红书公司);Tsinghua University(清华大学)
LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark
School of Artificial Intelligence, Beihang University(北航人工智能学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)